Deploying an AI Chatbot on Next.js 15: Edge Runtime, Streaming, and Cold Start Reality

Production chatbots on Next.js 15 face a single hard constraint: the user sees a blank cursor until the first token arrives. That latency is the sum of function cold start, runtime init, embedding retrieval, and the LLM’s own time-to-first-token. Picking the wrong runtime or streaming model can turn a 600 ms experience into a 3 s one.

Edge vs Node.js runtime tradeoffs for AI chatbot routes (cold start, memory limits, streaming support)

Next.js 15 lets you mark a route handler with export const runtime = 'edge' or leave it in the default Node.js pool.

• Edge runtime: runs on a V8 isolate rather than a full container. Init time is low—typically tens to low-hundreds of milliseconds in warm-path cases, based on typical production deployments. It streams natively over HTTP and has built-in fetch/ReadableStream support, so it is a natural fit for LLM responses. The tradeoff is a restricted API surface: no fs, no native Node modules, and CPU/memory ceilings enforced by the platform. On Vercel, Edge Functions are limited to the standard isolate constraints; on Cloudflare Workers, CPU time is metered per request and wall-clock limits apply.
• Node.js runtime: gives you the full Node API set, larger memory envelopes, and access to heavier libraries. Use it when you need to run local models, parse PDFs, or use native bindings. The cost is a container cold start that is often higher than an Edge isolate, and you must still stream via ReadableStream or Node Readable to avoid buffering the whole LLM response.

For a standard OpenAI/Anthropic chatbot that only calls upstream APIs, Edge is usually the right default. Move to Node.js only when the work cannot be done in an isolate.

Server-Sent Events vs WebSockets for streaming chat — practical implementation

SSE is the pragmatic choice for serverless chat. It is a long-lived HTTP response with a Content-Type: text/event-stream body. The client receives tokens as they are generated, and reconnection is just a normal HTTP retry with your existing auth cookies.

WebSockets are useful when you need true bidirectional traffic—e.g., voice packets or collaborative cursors—but they add connection state, require sticky routing, and are awkward on serverless platforms. For text chat they are usually overkill.

With the Vercel AI SDK, an Edge route can return a data stream directly:

import { streamText } from 'ai';
import { openai } from '@ai-sdk/openai';

export const runtime = 'edge';

export async function POST(req: Request) {
  const { messages } = await req.json();

  const result = streamText({
    model: openai('gpt-4o-mini'),
    messages,
    system: 'You are a concise support assistant.',
  });

  return result.toDataStreamResponse();
}

On the client, the useChat hook consumes that stream and renders chunks as they arrive. The wire format stays standard HTTP, so it works through CDNs, Vercel, and Cloudflare without extra configuration.

The cold start problem: how LLM calls interact with Vercel/Cloudflare function lifecycle

Cold start is not one event; it is everything that must happen before the first byte reaches the client. In an AI route that means:

1. Platform isolates or starts the function.
2. Your runtime loads the handler and any imports.
3. The route parses the request body.
4. It may fetch cached context or generate an embedding.
5. It opens the upstream LLM connection.
6. The LLM begins emitting tokens.
7. The first token travels back through the function to the client.

Edge functions reduce step 1, but they do not eliminate steps 4–6. If your code waits for the full LLM response before writing to the stream, you lose the benefit of streaming. Start the streamText call as soon as you have the prompt, and pipe its ReadableStream into the response immediately. Also keep imports lean: every package loaded at cold-start adds milliseconds.

Platform keep-alive behavior matters. Vercel and Cloudflare reuse warm isolates for a short window after a request, but idle functions are evicted. High-traffic routes stay warm; low-traffic routes pay the cold start on most invocations.

Worked example: streaming chat route, concrete TTFB numbers, cache strategy for repeated queries

Here is a realistic Vercel Edge route with a Redis cache for exact repeated queries, using Upstash:

import { streamText } from 'ai';
import { openai } from '@ai-sdk/openai';
import { Redis } from '@upstash/redis';
import crypto from 'crypto';

export const runtime = 'edge';

const redis = Redis.fromEnv();

export async function POST(req: Request) {
  const { messages } = await req.json();
  const lastMessage = messages.at(-1)?.content;

  const key = `chat:${crypto.createHash('sha256').update(lastMessage).digest('hex')}`;

  const cached = await redis.get<string>(key);
  if (cached) {
    return new Response(cached, {
      headers: { 'Content-Type': 'text/plain; charset=utf-8' },
    });
  }

  const result = streamText({
    model: openai('gpt-4o-mini'),
    messages,
    system: 'You are a support assistant for an e-commerce site.',
    maxTokens: 512,
  });

  return result.toDataStreamResponse();
}

For a route deployed to iad1 calling OpenAI, numbers from typical production deployments look like this:

• Edge isolate cold start: ~80–250 ms
• Request parsing + Redis round-trip: ~20–60 ms
• OpenAI gpt-4o-mini time-to-first-token: ~300–900 ms
• Total uncached TTFB: approximately 450 ms–1.3 s
• Cached hit TTFB: ~40–120 ms

To reduce repeated-query latency, cache the final answer under a hash of the user’s last message with a 5-minute TTL. For RAG systems, cache the vector-search result too, because embedding + nearest-neighbor lookup is often faster than the LLM but still not free.

Embedding generation at ingestion vs query time — architectural decision and cost impact

You need embeddings in two places: for your document chunks and for the user’s query.

• Ingestion-time embedding: when documents are added, chunk them, generate embeddings with a model such as OpenAI text-embedding-3-small, and store them in a vector database like Pinecone, Weaviate, or pgvector. At query time you only embed the user’s question and run a vector search. This is the standard architecture for static knowledge bases.
• Query-time-only embedding: some frameworks embed chunks on the fly. This is simpler in demos but increases query latency and duplicates work for every request.

Latency impact: embedding a short query with text-embedding-3-small typically adds roughly 50–250 ms, depending on provider and region, based on typical production deployments. That time is on the critical path before the LLM call even starts. Cost impact: embedding is cheap—OpenAI’s text-embedding-3-small is around $0.02 per 1 million input tokens as of current pricing—but repeated calls add up at scale.

Rule of thumb: embed documents once at ingestion, embed queries at request time, and cache common query embeddings if you see repeat traffic.

Checklist for production-ready chatbot deployment

1. Pin the route to runtime = 'edge' unless you need Node-only libraries.
2. Stream LLM output directly; never buffer the full response in memory.
3. Add a Redis or equivalent cache for exact repeated queries with a short TTL.
4. Generate document embeddings at ingestion, not query time.
5. Measure TTFB separately from total response time; optimize the path before the first token.
6. Set provider timeouts and max tokens to prevent runaway costs.
7. Use structured logging for cold-start duration, embedding latency, and LLM TTFT.
8. Test idle-function behavior; expect cold starts on low-traffic routes.
9. Validate input length before sending it to the embedding and LLM APIs.
10. Monitor streaming failure modes: aborted connections, partial chunks, and retry storms.

Simple Agent handles Edge runtime selection, streaming SSE wiring, and query caching automatically.