anoman
Resep · UI Chat Streaming

UI chat streaming dengan Next.js.

Streaming token demi token melalui SSE, pembatalan, penanganan error di tengah stream, dan API key tetap di sisi server. ~150 baris TypeScript.

Arsitektur

Tiga bagian

  1. Route handler Next.js di server — memproksikan permintaan browser ke Anoman agar API key Anda tidak pernah sampai ke klien.
  2. Hook useChat — mengelola state pesan, mem-parsing SSE, mengekspos send() + cancel().
  3. Komponen ChatBox — merender pesan + input. Menonaktifkan input saat streaming. Menampilkan tombol Stop.

1. Proksi sisi server

Alirkan stream Anoman langsung ke browser

Route handler memanggil Anoman, lalu memancarkan ulang setiap chunk sebagai SSE untuk browser. Kami tidak memodifikasi chunk — teruskan apa adanya agar frame metadata _anoman sampai ke klien tanpa perubahan.

// Proxies the browser to Anoman so the API key stays server-side.
// Pipes the SSE response straight through.

import OpenAI from "openai";

export const runtime = "edge";  // optional — faster cold start

const client = new OpenAI({
  baseURL: "https://api.anoman.io/v1",
  apiKey: process.env.ANOMAN_API_KEY!,
});

export async function POST(req: Request) {
  const { messages, model = "claude-sonnet-4-6" } = await req.json();

  const stream = await client.chat.completions.create({
    model,
    messages,
    stream: true,
    // Surface guardrail metadata to the client via the _anoman block
    // already present on the final chunk. No special header needed.
  });

  // The OpenAI SDK returns an async iterator over typed chunks.
  // Re-encode as plain SSE for the browser EventSource client.
  const encoder = new TextEncoder();
  const body = new ReadableStream({
    async start(controller) {
      try {
        for await (const chunk of stream) {
          const line = `data: ${JSON.stringify(chunk)}\n\n`;
          controller.enqueue(encoder.encode(line));
        }
        controller.enqueue(encoder.encode("data: [DONE]\n\n"));
      } catch (err: any) {
        const errFrame = {
          error: {
            type: "provider_error",
            message: err.message ?? "stream interrupted",
          },
        };
        controller.enqueue(
          encoder.encode(`data: ${JSON.stringify(errFrame)}\n\n`),
        );
      } finally {
        controller.close();
      }
    },
  });

  return new Response(body, {
    headers: {
      "Content-Type": "text/event-stream",
      "Cache-Control": "no-cache",
      Connection: "keep-alive",
    },
  });
}

2. Hook parser SSE

Tambahkan delta ke pesan terakhir

Buffer persisten untuk toleransi batas chunk. Frame error di tengah stream dilempar agar blok catch memunculkannya. AbortController untuk pembatalan.

import { useCallback, useState, useRef } from "react";

type Message = { role: "user" | "assistant"; content: string };

export function useChat() {
  const [messages, setMessages] = useState<Message[]>([]);
  const [streaming, setStreaming] = useState(false);
  const [error, setError] = useState<string | null>(null);
  const abortRef = useRef<AbortController | null>(null);

  const send = useCallback(async (content: string) => {
    setError(null);
    const userMsg: Message = { role: "user", content };
    const assistantMsg: Message = { role: "assistant", content: "" };
    setMessages((prev) => [...prev, userMsg, assistantMsg]);
    setStreaming(true);

    const controller = new AbortController();
    abortRef.current = controller;

    try {
      const response = await fetch("/api/chat", {
        method: "POST",
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify({ messages: [...messages, userMsg] }),
        signal: controller.signal,
      });

      if (!response.ok || !response.body) {
        throw new Error(`HTTP ${response.status}`);
      }

      const reader = response.body.getReader();
      const decoder = new TextDecoder();
      let buffer = "";

      while (true) {
        const { done, value } = await reader.read();
        if (done) break;
        buffer += decoder.decode(value, { stream: true });

        const lines = buffer.split("\n");
        buffer = lines.pop() ?? "";

        for (const line of lines) {
          if (!line.startsWith("data: ")) continue;
          const payload = line.slice(6).trim();
          if (payload === "[DONE]") continue;
          let parsed: any;
          try {
            parsed = JSON.parse(payload);
          } catch {
            continue;
          }
          // Mid-stream error frame
          if (parsed.error) {
            throw new Error(parsed.error.message);
          }
          const delta = parsed.choices?.[0]?.delta?.content;
          if (delta) {
            setMessages((prev) => {
              const next = [...prev];
              next[next.length - 1] = {
                ...next[next.length - 1],
                content: next[next.length - 1].content + delta,
              };
              return next;
            });
          }
        }
      }
    } catch (e) {
      if ((e as Error).name !== "AbortError") {
        setError((e as Error).message);
      }
    } finally {
      setStreaming(false);
      abortRef.current = null;
    }
  }, [messages]);

  const cancel = useCallback(() => {
    abortRef.current?.abort();
  }, []);

  return { messages, streaming, error, send, cancel };
}

3. Komponen UI

Tombol Stop saat streaming, Send saat idle

"use client";
import { useState } from "react";
import { useChat } from "@/hooks/useChat";

export function ChatBox() {
  const { messages, streaming, error, send, cancel } = useChat();
  const [input, setInput] = useState("");

  return (
    <div className="flex flex-col h-screen max-w-3xl mx-auto">
      <div className="flex-1 overflow-y-auto p-4 space-y-3">
        {messages.map((m, i) => (
          <div
            key={i}
            className={`p-3 rounded-lg ${
              m.role === "user" ? "bg-blue-50 ml-12" : "bg-gray-50 mr-12"
            }`}
          >
            <div className="text-xs font-semibold text-gray-500 mb-1">
              {m.role === "user" ? "You" : "Assistant"}
            </div>
            <div className="whitespace-pre-wrap">{m.content}</div>
          </div>
        ))}
        {error && (
          <div className="p-3 rounded-lg bg-red-50 text-red-700 text-sm">
            Error: {error}
          </div>
        )}
      </div>

      <form
        onSubmit={(e) => {
          e.preventDefault();
          if (input.trim()) {
            send(input);
            setInput("");
          }
        }}
        className="flex gap-2 p-4 border-t"
      >
        <input
          className="flex-1 border rounded-lg px-3 py-2"
          value={input}
          onChange={(e) => setInput(e.target.value)}
          disabled={streaming}
          placeholder="Ask anything..."
        />
        {streaming ? (
          <button
            type="button"
            onClick={cancel}
            className="px-4 py-2 bg-red-600 text-white rounded-lg"
          >
            Stop
          </button>
        ) : (
          <button
            type="submit"
            className="px-4 py-2 bg-blue-600 text-white rounded-lg"
          >
            Send
          </button>
        )}
      </form>
    </div>
  );
}

Tips produksi

  • Kompatibilitas proksi Cloudflare — edge Anoman meneruskan frame SSE begitu tiba. Setiap chunk mereset timeout proksi, sehingga respons streaming bertahan untuk penyelesaian yang panjang sekalipun.
  • Tampilkan status guardrail inline — frame _anoman terakhir memuat rincian guardrail lengkap. Render lencana kecil "Terverifikasi bersih ✓" di bawah setiap pesan asisten.
  • Penanganan rate limit — saat route handler Anda menerima 429 dari Anoman, tampilkan pesan ramah "Pelan-pelan" ke pengguna. Header retry-after memberi tahu lama tunggunya.
  • Konteks multi-giliran — kirim seluruh array messages pada setiap permintaan. Anoman + upstream menangani sisanya.
  • Sesi persisten — tambahkan header anoman-session-id untuk mengelompokkan semua giliran di bawah satu trace di tampilan Sessions dashboard.

Lihat playground untuk UI referensi lengkap.

Perbandingan streaming 3-panel dengan pemilih model + kontrol parameter.

Buka playgroundResep lainnya