AI 1 May 2026 9 min read Two engines under one widget: rebuilding chat for Claude Haiku 4.5 By Old Forge Technologies 88 reads Contents The chat widget in the bottom-right of this page used to run on Azure-hosted GPT-4.1. This week I retired that. The model is now Claude Haiku 4.5, served via Anthropic's API directly. By question three of testing, I'd blown the input token rate limit. By the end of the afternoon I'd built a second backend, exposed both as a click-to-toggle in the chat header, and learned more about Anthropic's prompt caching than I bargained for. This is the writeup. ## The starting position The chat widget routes through a small PHP proxy on this server, which forwards to an n8n workflow on a separate host. Inside n8n, the brain is the canonical LangChain Tools Agent: a single black-box node that takes a system prompt, a list of tools (mounted via MCP), some Redis-backed conversation memory, and a chat model. You hand it a user message; it decides whether to call a tool, calls it, decides whether to call another, and eventually returns a final response shaped by a Structured Output Parser so the frontend can render `output`, `suggestions`, and `collect_fields` cleanly. This setup is fine for "make it work". It has an architectural cost the moment you start counting tokens. ## What changed when we swapped Azure for Claude Two surprises arrived together. **Surprise one: tool call ID format.** Anthropic uses `toolu_xxx`. OpenAI and Azure use `call_xxx`. The Redis chat memory was happily storing whatever LangChain wrote into it, and that included tool_use blocks with the OpenAI-shaped IDs from prior turns. When I pointed the agent at Claude, the next user message replayed the saved history, and Claude refused: *"unexpected `tool_use_id` in `tool_result` blocks"*. Easy fix, ugly cause. The lesson: chat memory is not portable across providers if it preserves intermediate tool blocks. Keep a clean text-only history if you ever want to swap models. **Surprise two: rate limits bite earlier than you think.** Tier 1 on `claude-haiku-4-5` is 50,000 input tokens per minute. The LangChain Tools Agent makes at least two API calls per user turn (one to decide, one to synthesise), and each call ships the entire system prompt plus all tool definitions plus the growing message history. After three quick clicks on the suggestion chips, I was over the limit. Flushing Redis fixed the orphan tool IDs. The bigger problem, the agent burning ten thousand input tokens per question, needed actual engineering. ## The cheap wins (still on LangChain) Before any architectural changes, four straightforward edits on the existing workflow: 1. **Trim the system prompt.** It had drifted to ~2,400 tokens, including the entire product catalogue duplicated inline. Since the agent is mandated to call a knowledge-base tool for any product question (which returns the catalogue from the KB anyway), keeping it in the system prompt is pure duplication. Cut to ~950 tokens. That alone halves the per-call constant cost. 2. **Reduce the context window.** Memory was set to ten turns. With each turn carrying tool_use blocks and KB result chunks, that tail grew fast. Five turns is plenty for a chat widget. 3. **Drop the Structured Output Parser.** When you combine tools and structured output in LangChain's Tools Agent, the parser injects an extra synthetic tool ("emit your answer in this schema") and adds a retry round-trip when the model deviates. Both are token taxes. Cheaper to instruct the model to return JSON in the system prompt, then `JSON.parse` it downstream with a small fallback for the occasional misshape. 4. **Switch the agent type to "Tools Agent" if you are still on the legacy "OpenAI Functions Agent".** Function calling formats are not interchangeable; Claude needs the provider-agnostic Tools Agent to handle its `tool_use` blocks correctly. These four changes brought per-question input down from ~10-12k tokens to ~4-6k. Real money saved, real headroom under the rate limit. But the bigger lever was still untouched. ## Why the bigger lever needed a rebuild Anthropic supports prompt caching: mark a content block with `cache_control: { type: "ephemeral" }` and the prefix up to that block is cached for five minutes. After the first call writes the cache, subsequent calls with the same prefix pay 10% for cache reads instead of full price. For a chat with a stable system prompt and a stable tool list, this is exactly the optimisation you want. n8n's `lmChatAnthropic` node, the one wrapping LangChain's `ChatAnthropic`, does not expose `cache_control`. The integration was written before caching went GA, and the option simply isn't there. To use it, the API call has to leave the LangChain wrapper. So I built a second workflow. ## The Direct workflow `Web Chat - Anthropic Direct` runs alongside the original. It does the same job, but the agent loop is hand-rolled. The shape: - A parent workflow handles the webhook, branches the clear-memory action, loads Redis history, initialises an MCP session, then hands off to a sub-workflow. - The sub-workflow runs one Anthropic API call. It builds the request body (system + tools, with `cache_control` on the last tool so system+tools forms a single cached prefix), hits `api.anthropic.com/v1/messages` via an HTTP Request node holding the API key as an `httpHeaderAuth` credential, accumulates token usage from the response, dispatches any `tool_use` blocks to MCP via inline HTTP calls, and decides whether to recurse. - If the response said `stop_reason: tool_use`, it appends the tool results to the messages array and calls itself with the same workflow ID. If it said `stop_reason: end_turn`, it parses the JSON output and returns. That's it. About a dozen nodes for the whole agent loop. The interesting bit is one Code node, around eighty lines, that you could lift wholesale into a Python service if you ever wanted to leave n8n behind. ## Caching, in practice on Haiku 4.5 The documented minimum cacheable prefix length for Haiku is 2,048 tokens. Our system+tools comes in around 4,300 tokens, comfortably above the bar. Cache should activate on the very first call. It does not. Empirically, across a five-call synthetic test: - Calls one and two: `cache_creation = 0`, `cache_read = 0`. Anthropic is sent valid `cache_control` and ignores it. - Call three: still nothing. - Call four: `cache_creation = ~4,700`. The cache finally gets written. (`cache_read` is still zero on call five.) - The "decide which tool to use" sub-call (the smaller first leg of each turn) never caches across the whole test. - The "synthesise the answer from the tool result" sub-call (the larger second leg) starts caching from call four onwards. There appears to be an undocumented "minimum total request size" near 5,000 tokens before Anthropic considers the prefix worth caching, and a warm-up before cache reads kick in. None of this is in the docs. It is the actual product behaviour. What this means in practice: a brand-new visitor to your chat will not see cache savings on the first turn or two. By turn three or four, the cache writes activate. Within the five-minute TTL, the next visitor with the same system prompt benefits from those writes. Across a busy hour, the average input cost per turn drops by 30 to 40%. Across a quiet afternoon with one visitor every hour, savings are smaller because the cache TTL expires between visits. The lever is real, but it is a steady-state lever, not a single-call lever. Worth knowing before you promise a client a flat percentage. A small adjacent gotcha: the system prompt template originally interpolated `{TIME}` to the current minute. That makes every minute boundary a different cache key, so the cache never warms. Date-only is enough to ground "today" and "tomorrow" for a chat widget. The minute precision was theatre. ## What the toggle does The chat widget on this site now has a small click-to-toggle in the status bar where the engine name used to be static. Two options: **LangChain.** The original n8n AI Agent path. Single high-level node, off-the-shelf agent loop, no prompt caching. Fast to build, fast to maintain, decent at moderate traffic. This is what most n8n chat builds look like, and for a lot of use cases it is the correct answer. **Direct (cached).** The hand-rolled HTTP loop. More nodes on the canvas, but every primitive is visible: request body construction with `cache_control`, tool dispatch, recursion, JSON parsing. Prompt caching is on. Suitable when you care about per-call cost and have multi-turn traffic. The toggle persists per browser via `localStorage`, so you can hop between them mid-conversation and watch the response telemetry change. The response payload includes a `_meta.usage` block from Anthropic when you are on the Direct engine: `input_tokens`, `cache_creation_input_tokens`, `cache_read_input_tokens`. Open your browser devtools and you can see the cache warming in real time. ## Honest takeaways Three things I would say to anyone doing the same exercise. **One:** off-the-shelf agent components hide token costs that matter. The LangChain Tools Agent is a good way to get to "working", and a poor way to stay there if your traffic grows. The structured output retry, the synthetic tool definition, the system prompt that quietly grew to 2,400 tokens because nothing forced you to count, all of it adds up. Counting is the work. **Two:** if you are going to use prompt caching, build the request yourself. The wrappers will catch up eventually, but right now `cache_control` is something you set explicitly on a Python or HTTP request, not something a node abstraction does for you. Anthropic's caching also has undocumented behaviours (the warm-up, the soft 5k threshold) that only show themselves when you can read the raw `usage` block back. **Three:** the right tool depends on what you are selling. The engineering-pure answer to "an Anthropic agent with caching" is about eighty lines of Python and the official SDK. We kept this build in n8n because Old Forge sells automation, and a workflow canvas is a deliverable a prospect understands. The Python service would have shipped sooner. The n8n version opens more sales conversations. Both are correct answers to slightly different questions. The chat at the bottom of this page is the demo. Click the engine name in the header to switch backends. If anything misbehaves, that is part of the point. This is real production code being honest about its trade-offs.
