8 May 2026 3 min read The chat that costs nothing to keep talking By Old Forge Technologies 58 reads Contents <p>A small badge under the chat assistant on the home page says which engine is answering. The honest version of that badge would also tell you what the answer cost. For most live chat assistants that number creeps upward with every turn - politely, invisibly, expensively. Each new message sends the entire conversation history back to the model, <em>and</em> the system prompt, <em>and</em> the tool definitions. A chat that lingers is a chat that bleeds.</p> <p>So the workshop went looking for cheaper line items.</p> <h2>The trick</h2> <p>The trick is <strong>prompt caching</strong>. Anthropic - and most of the serious model vendors - will quietly retain the prefix of a request in a cache for about five minutes, and bill it at roughly a tenth of the usual rate when you send it again. There is no new technique to it. You mark the parts of your prompt that don't change between turns - the system instruction, the tool catalogue, the long preamble - and the next request that arrives within five minutes pays one-tenth for them.</p> <p>Two things have to be true. The first is that you write the prompt with the cache marker in the right place. The second is that the next message has to land before the cache turns over.</p> <p>We rebuilt the chat to make both true.</p> <h2>The bake-off</h2> <p>The original engine was an agent loop running in n8n on top of LangChain - flexible, generic, tractable. The replacement is a hand-built loop talking directly to Anthropic, with explicit <code>cache_control</code> markers on the system prompt and the tool list. You can pick between the two by adding <code>?engine=direct</code> to the URL on the home page.</p> <p>The numbers are unsubtle. The hand-rolled loop is cheaper <em>and</em> faster - the second matters because cache hits skip the model's re-read of the preamble. Conversations that linger get <em>more</em> efficient, not less. That is the right shape for a small public chat.</p> <h2>Two cautions</h2> <p>Caching only pays off if the prefix is stable. A system prompt that interpolates the time of day defeats it. And the five-minute window means a chat where someone asks a question, walks away for an hour, then comes back, won't see the discount on that next turn. Most real conversations are bursty enough for it to matter.</p> <p>The cheapest tool in the workshop is the one you've already paid for. Prompt caching is plumbing, not magic - but a lot of small operators ship without it because the default examples don't bother. For a longer technical writeup of the rebuild, the bugs we hit, and the undocumented bits of Anthropic's caching behaviour we found along the way, see <a href="/blog/two-engines-under-one-widget">Two engines under one widget</a>.</p>
