Design a ChatGPT-Style Assistant

"Design a ChatGPT-style assistant" looks like a single model call wrapped in a chat window. The interesting part is everything that surrounds that call: remembering a conversation without letting it grow out of control, streaming an answer safely, and keeping the bill and the latency in check as millions of conversations run at once.

Picture a hotel concierge who keeps a file on each guest and re-reads the whole file before answering every question. On the first day the file is a page. By the tenth day it is a binder, and the concierge is slower and more expensive to employ, although each new question is as short as ever. A chat assistant that resends the whole history on every turn behaves exactly like that concierge.

• -Scale: users, concurrent conversations, messages per day.
• -Behavior: multi-turn chat, streaming answers, attachments, tools, or only text?
• -Memory: only within one conversation, or across conversations?
• -Quality and safety bar: what must never be said, and how is it enforced?
• -Latency: time to first token and the pace of streaming.
• -Budget: cost per conversation the business can afford.

The answer streams back to the client from the safety check, and the new messages are saved to the conversation store. Each part has a job.

• -Gateway: authentication, rate limits per user, and abuse controls.
• -Conversation service and store: messages stored in order, with a conversation id. This is an ordinary database design problem, and it must survive a client disconnect mid-answer.
• -Context assembler: builds the prompt from the system prompt, the history (trimmed or summarized), any retrieved memory, and tool results.
• -Model router: sends simple requests to a small, fast model and hard ones to a larger model, with fallbacks if a provider fails.
• -Streaming: tokens are pushed to the client as they are generated, over server-sent events or a websocket. A stop button must cancel generation and free the GPU.
• -Safety: input checks before the model, and output checks on the stream.

On turn 1 the model reads the system prompt and one message. On turn 20 it reads the system prompt and all 19 earlier exchanges. Each turn's input is bigger than the last, so total input over a conversation grows roughly with the square of its length. The code compares three strategies over a 20-turn conversation: sending the full history, sending only the last six turns, and sending the last four turns plus a 300-token summary of everything older. Press Run and compare the totals.

With full history, the input grows from 580 tokens on turn 1 to 3,550 on turn 10 and 6,850 on turn 20, a total of 74,300 tokens and about $0.298 for the conversation. Sending only the last six turns caps each turn at 2,560 tokens and brings the total to 44,270, for $0.208. The summary-plus-recent strategy caps at 2,200 and totals 39,200, for $0.193. The output cost, $0.075 for 20 answers, is the same in all three, so the history strategy is the lever. Trimming loses old details. A summary keeps the gist, at the cost of one extra summarization call and some risk of dropping a fact the user cares about.

• -Long-term memory: store a few durable facts, such as preferences and project names, in a separate store, retrieved when relevant. Let users see, edit, and delete them.
• -Safety on a stream: you cannot unsay tokens already shown. Check the input first, and buffer enough of the output to screen it before it reaches the user, or cut off the stream when a check fires.
• -Provider outage: route to a fallback model, and tell the user when quality may be lower.
• -Abuse and cost: per-user rate limits and token budgets, since a single user can generate a large bill.
• -Quality: evaluate whole conversations, not only single replies, since failures like forgetting earlier instructions appear over many turns.