Llm Cost Optimization: Caching, Batching, And Prompt Trimming That Actually Works

Most teams I meet think their LLM costs are driven by “using GPT-4 too much” or “we should fine-tune.” In practice, the biggest line items are usually way more boring: Llm Cost Optimization: Caching, Batching, And Prompt Trimming That Actually Works

• You’re re-sending the same giant instruction blob every call.

• You’re stuffing RAG context like it’s a Thanksgiving turkey.

• Your agent loops retry tools, retry themselves, and then retry the retries.

• Your “helpful” logging includes full tool payloads, and you also send those payloads straight back into the model next turn (yes, people do this).

• You’re paying for tokens you didn’t mean to generate because you didn’t cap output or you let the model ramble.

The painful part: none of this is visible if you only look at “requests per day” and “average latency.” LLM systems leak tokens in a hundred small ways, and the leaks compound.

In this post I’ll give you the mental models and practical steps I’ve used to make real production systems cheaper without turning them into brittle, low-quality prompt soups. We’ll talk about what drives cost, when caching actually saves money (and when it will quietly ruin your answers), how batching helps, and how to detect token waste and stop it without wrecking quality.

What drives LLM cost the most?

Input tokens usually dominate

Everyone stares at output tokens because they’re visible (“wow it wrote a lot”), but in most production apps the input side dominates.

Why?

• System prompts drift into novels.

• Chat history grows without bound.

• RAG adds chunks “just in case.”

• Tool calls inject huge JSON blobs.

• “Safety” prompts double everything.

A real pattern I’ve seen: average output is ~250–500 tokens, but average input is 3,000–12,000 tokens. If you’re doing multi-step workflows (planner → tool → synthesizer), multiply that by steps. You can be “only generating 500 tokens” and still be paying for 10k tokens per user action.

Rule of thumb

if your system is more than a single-turn Q&A, assume input tokens are the budget killer until proven otherwise.

the invisible instruction tax

We had a customer support assistant where the system prompt had evolved over months. It was ~1,800 tokens of rules, examples, and “DO NOT”s. It was sent on every turn. The average conversation was 6 turns. That’s ~10k tokens of instructions per chat before the user even asked anything interesting.

We cut it to ~450 tokens by pulling stable policy text into a short structured “rules summary” and moving examples into a separate “debug mode” prompt only used during evaluation. Costs dropped noticeably and quality didn’t move because the model wasn’t actually using most of that text; it was just being billed for it.

Model choice and workflow shape

Model choice matters, but workflow shape matters more.

A cheap model called 5 times with 8k input each is often more expensive than one good call to a stronger model with 2k input. Conversely, sometimes the best cost move is using a smaller model for 80% of traffic and only escalating to a larger model when needed.

What you want is an architecture that matches your distribution:

• Most requests are easy

  (summaries, extraction, routing). Use a smaller model with tight outputs.

• Some requests need reasoning

  ambiguous, multi-hop, safety-sensitive. Escalate.

• Rare requests need “gold” answers

  enterprise customers, critical flows. Spend the money.

Practical heuristic

build a router early. Even a simple one:

• If input > X tokens, don’t “just pass it through.” Trim or summarize.

• If confidence is low or the answer must be correct, escalate model.

• If the request matches a known template, skip the model or use extraction.

Retries, tool failures, and agent loops

Retries are the silent killer. The bill doesn’t come from your “happy path.”

It comes from the 5–20% of requests where things go sideways:

• Tool call fails → model retries → tool fails again → model “reflects” and tries a different tool → now you have 4 LLM calls for one user action.

• Streaming responses get cut → client retries → duplicated usage.

• JSON schema mismatch → “Fix JSON” loop → you pay twice for the same content.

Agentic systems amplify this because every “thoughtful” loop is more tokens. A small bug in a tool schema can turn into a runaway bill.

the tool JSON black hole

We shipped an agent that called internal APIs. Tool responses included verbose objects: timestamps, nested metadata, debug arrays, and a full “request echo.” A single tool response was often 6–12k tokens. The agent would then include it in the next turn sometimes multiple tool results at once.

We discovered ~40% of total tokens were tool JSON. Fix was simple: define a strict tool response schema and drop fields aggressively. In a week, we cut cost and latency, and the agent got more reliable because it had less junk to reason over.

Quick cost math: where to look first

You don’t need a fancy finance model. You need a flashlight.

Start with this per-user-action breakdown:

Total cost per action

 over calls (input

_tokens + output_ tokens) × price _per_ token(model)

Then ask:

1. How many calls happen per action?

   If it’s >2 on average, your workflow is doing the spending.

2. What’s the median vs p95 tokens per call?

   The p95 often reveals runaway conversations, RAG stuffing, or tool loops.

3. Where are tokens coming from?

   Split input into buckets:

• system + developer instructions

• chat history

• retrieved context (RAG)

• tool results

• user input

If you do nothing else, ship a dashboard that shows these five numbers per call. The first time you see “tool_results = 9,200 tokens,” you’ll suddenly understand your bill.

Rule of thumb

the fastest wins are almost always:

• cap or compress chat history

• reduce RAG chunking

• trim tool outputs

• stop redundant instructions

• cut retries / loops

When it saves real money (and when it backfires)

Caching is one of those topics where everyone nods, then half the implementations quietly make things worse.

Caching is not a moral good. It’s a trade: you’re swapping compute for complexity and risk. It works when your system has repeatability similar inputs produce similar outputs and correctness doesn’t depend heavily on time or user-specific context.

Types of caches that actually show up in LLM systems

Exact-match caching

You hash the input (and the relevant context) and reuse the response if it matches exactly.

This is the safest form.

It’s also less useful than people expect, because inputs are rarely identical unless:

• you have templated prompts,

• you use stable system prompts and few dynamic fields,

• you’re caching sub-steps (like classification or extraction).

Where exact-match shines

• routing/classification

• structured extraction from a known template

• deterministic transforms (format conversion, short summaries)

• internal evaluations / batch jobs

Where it disappoints

• chatty assistants where user phrasing varies a lot

• anything that includes timestamps, personalization, or fresh data

Semantic caching

You embed the “query” (or some canonical representation) and reuse a previous answer if it’s “close enough.”

This can save serious money on high-volume systems where users ask the same thing in different ways. But it’s riskier because similarity ≠ equivalence.

The trap

you’ll serve a confidently wrong cached answer that sounds right.

Mini case 

semantic cache served stale answers

We cached answers for “How do I reset my billing password?” Great. Then the product UI changed. The embedding similarity stayed high, so we kept serving the old steps for weeks. Support tickets went up. The cache did its job (saved money) while the business bled trust.

The fix wasn’t “don’t cache.” The fix was versioning and gating: tie cache keys to product version / doc revision, and refuse cache when the answer depends on UI state or “latest policy.”

Partial caching (a.k.a. caching sub-results)

This is the most underrated one.

Instead of caching the final assistant output, cache intermediate artifacts:

• retrieved documents

• chunk summaries / compression outputs

• tool responses

• classification results

• canonicalized user query

Partial caching tends to be:

• higher hit rate,

• lower correctness risk,

• easier to invalidate.

In real systems, I get more savings from caching “retrieval + compression” than caching “final answer.”

When to cache responses

Here’s the practical decision filter I use:

Cache the final LLM response when:

• The question is frequently repeated across users (FAQs, docs questions).

• The answer is stable over time (or you can version it).

• The answer doesn’t depend on private user data.

• You can tolerate minor phrasing differences (or you store multiple variants).

Cache sub-results when:

• You have expensive upstream steps (retrieval, tool calls, chunk summarization).

• The downstream output still needs personalization or freshness.

• You’re trying to reduce latency as well as cost.

Don’t cache when:

• Freshness matters (prices, availability, “today’s status,” rapidly changing policies).

• User context matters (account-specific responses, permissions).

• The model is doing creative or open-ended generation where “same-ish” isn’t good enough.

Rule of thumb

if your answer starts with “It depends on your account…” you probably shouldn’t cache the final response.

Cache keys that don’t poison results

Most cache disasters come from bad keys. You cached “close enough,” but you forgot the hidden variables.

A safe cache key should include:

• prompt version

  (you will change prompts; treat this as a breaking change)

• model version

  (if you swap models, cached outputs may not match expectations)

• tool versions

  (schema changes can break response format)

• retrieval corpus version

  (docs revision, index build ID)

• user segmentation

  if it affects content (region, plan tier), but avoid storing private identifiers in the key directly hash or normalize

For semantic cache, you also need a stable “canonical query.” Don’t embed the whole prompt with 8k of context.

Embed the user intent, plus a small set of normalized attributes:

• product area

• feature flag / plan tier

• locale

• time sensitivity flag (fresh vs stable)

“Serve cached?” gating rules

A cache should not be “always serve if hit.” It should be “serve if hit and safe.”

I like simple gating rules you can ship quickly:

1. Freshness gate

• If the user asks about “current,” “today,” “latest,” “price,” “availability,” don’t serve cached unless your cache entry is very fresh (e.g., TTL minutes/hours) and tied to the data source timestamp.

2. Context gate

• If the request includes user-specific terms (“my account,” “my invoice,” “my subscription”), don’t serve cached final responses. You can still cache sub-results (like generic help steps).

3. Similarity gate (semantic)

• Use two thresholds:

  • A high threshold where you serve cached directly (e.g., cosine ≥ 0.92)

  • A medium threshold where you use cached as a draft but still call the model to verify/adjust (e.g., 0.85–0.92)
    Below that, don’t use it.

4. Output-format gate

• If the endpoint needs strict JSON, only serve cached if it was generated under the same schema version and passes validation.

5. Safety / policy gate

• If the request touches compliance, legal, medical, or anything that can harm users, either don’t cache final responses or require a verification pass.

In logs, a good cache system tells you why it didn’t serve.

If you can’t explain non-serves, you can’t debug correctness.

Exact-match vs semantic vs partial caching: what I’d actually do

If you’re early-stage:

• Start with exact-match caching for sub-steps (intent classification, retrieval results, compression).

• Keep TTL short on anything tool-related.

• Add prompt/model versioning on day one. Seriously.

If you’re high volume:

• Add semantic caching for stable FAQ-like queries.

• Invest in gating + versioning.

• Measure “cache correctness” with spot checks and complaint metrics, not vibes.

If you’re enterprise / high correctness:

• Avoid semantic caching for final answers unless you have strong validation.

• Prefer partial caching and deterministic transforms.

Batching: the simplest win for high-volume systems

Batching is the most misunderstood “easy win,” mostly because people expect it to do magic it doesn’t do.

What batching is and isn’t

Batching is sending multiple independent requests together so the system can amortize overhead and use the model server more efficiently.

Batching is not a way to reduce token billing by itself. If you send 10 prompts in one request, you’re still paying for the same tokens (sometimes slightly more due to separators/formatting). The savings usually come from:

• lower per-request overhead on your side,

• better throughput,

• fewer connections,

• sometimes better provider-side scheduling efficiency.

So batching is often a latency/throughput win and an infra cost win, not a direct token-cost win.

When batching pays off

Batching is great when:

• you have a queue of similar tasks (moderation, extraction, embeddings, offline jobs)

• you can tolerate small delays (tens of milliseconds to a few seconds)

• you can group by model + prompt template + output format

Batching is less useful when:

• requests are interactive and user-facing with tight latency SLOs

• each request requires different tools or different long context

• you need streaming per user request

How to batch safely

Micro-batching is the practical approach:

• collect requests for ~20–100ms (or until N items)

• batch them

• send as one call

• return results to each caller

A few “don’t shoot yourself” tips:

• Group by prompt template and schema.

  Mixing different formats in one batch is how you get parsing disasters.

• Use explicit delimiters and IDs.

  Have the model return outputs with stable IDs:

• Keep outputs short and structured.

  Batching plus rambling generation is a great way to blow your latency budget.

• Cap max tokens hard.

  If one item in the batch triggers a long response, it hurts everyone.

Myth-busting callout: three myths I keep seeing

1. “Batching reduces token billing.”

   Not really. It helps throughput and overhead. Token costs mostly don’t change.

2. “More RAG context is always better.”

   No. After a point you’re just paying for noise and confusing the model.

3. “Just use the biggest model for safety.”

   Bigger models can be more robust, but “safety” often comes from guardrails, constraints, and good tool design not maxing your spend.

Prompt trimming that actually works

“Prompt trimming” is not “delete stuff until it breaks and then add it back.” It’s designing the system so you stop paying for tokens that aren’t doing work.

Stop re-sending instruction novels

If your system prompt is >400–600 tokens, there’s a good chance it contains:

• duplicated rules,

• long examples that the model doesn’t need every time,

• historical baggage from incidents.

What actually works

• Write a short policy (bullets) and a separate evaluation pack (examples + edge cases) used in testing, not production.

• Use structured instructions

  • “Output must be JSON matching schema X.”

  • “If you can’t answer, return

• Keep “style” separate from “logic.” Style tokens are expensive and rarely worth it.

Rule of thumb

if it’s stable guidance, it should be short. If it’s long, it should be conditional.

Control chat history growth

Unbounded history is a tax that grows with user engagement. Congrats, your best users are your most expensive users.

Two techniques that work in production:

1. Sliding window

• Keep the last N turns (e.g., last 6–10 messages).

• Drop older turns.

2. Rolling summary

• Summarize older turns into a compact “conversation state”:

  • user goal

  • constraints

  • decisions made

  • open questions

• Update the summary every few turns.

Heuristic

keep the rolling summary under ~200–400 tokens. If it grows, you’re summarizing badly (or you’re building a CRM inside the prompt).

RAG: fewer chunks + compression + budgets

RAG is where token waste goes to party.

Common mistake: retrieve top-k chunks (k=10–20), dump them into the prompt, hope the model “figures it out.” It will figure it out… by charging you.

What actually works:

• Set a retrieval budget in tokens

  Chunks vary wildly in length. Aim for a budget like 800–1,500 tokens of context for most requests, more only when needed.

• Use fewer chunks, but better chunks.

  Often top-k = 3–5 beats top-k = 12 if retrieval quality is decent.

• Compress retrieved text.

  Run a cheap compression step that extracts only relevant lines for the question.

  • Yes, that’s another model call.

  • It still saves money when it prevents stuffing 5k tokens into a bigger model call.

• Rerank aggressively.

  A good reranker (even a lightweight model) often saves more money than it costs by cutting irrelevant context.

Rule of thumb starting point

• top-k retrieval: 5

• rerank to: 3

• context budget: ~1,200 tokens

• only expand budget on failure signals (low confidence, missing citations, user asks “are you sure?”)

Tool output trimming

Tool outputs are the sneakiest source of prompt bloat because they feel “free” they’re “just data.” They are not free. They’re tokens.

Make your tool schemas strict

• return only fields the model needs

• avoid nested arrays of objects unless required

• avoid verbose keys yes, keys count too

If tool output is large, summarize it before feeding it back

• Use a cheap summarizer model

• Use a structured summary

• Keep raw output out of the model context unless needed for a specific step

How do you detect token waste?

You can’t optimize what you can’t see. Token waste is not “high usage.” It’s tokens that don’t improve outcomes.

Define token waste

I define token waste as:

Tokens consumed that do not measurably improve success rate, correctness, user satisfaction, or downstream task completion.

That’s intentionally practical. If you cut 30% of tokens and quality stays the same, those tokens were waste.

Instrumentation checklist

If you’re not logging these, you’re flying blind:

• tokens_in / tokens_out per call

• cost per call, aggregated per user action

• input token breakdown:

  • system instructions

  • history

  • RAG context

  • tool results

  • user message

• number of calls per action

• retry counts (LLM retries and tool retries separately)

• tool call sizes (payload tokens)

• response validation failures (JSON parse errors, schema mismatches)

• cache hit / serve decisions + reasons

• latency per stage (retrieval, model, tools)

This is what “it looks like in dashboards” when you’re serious:

• a time series of cost per successful action

• a histogram of input tokens with p50/p90/p99

• a bar chart of token sources (history vs RAG vs tools)

• a top list of worst offenders by endpoint / prompt version

6 common waste patterns + fixes

1. Instruction duplication

• Symptom: system prompt repeated inside user prompt or tool descriptions.

• Fix: centralize instructions; keep tool descriptions short; version prompts.

2. History bloat

• Symptom

  tokens_in grows linearly with conversation length.

• Fix

  sliding window + rolling summary; drop assistant verbosity from history (store it elsewhere if needed).

3. RAG stuffing

• retrieval context is >50% of tokens_in.

• rerank + budget + compression; reduce top-k; shorten chunk size; remove boilerplate from docs.

4. Tool echo

• tool_results tokens spike; model repeats tool JSON in output.

• strict tool schemas; never send raw tool payloads back unless necessary; summarize.

5. Retry storms

• calls per action >2, retries cluster around tool failures.

• cap retries (e.g., 1 model retry, 2 tool retries max); add circuit breakers; validate schemas; fall back to humans or safe default.

6. Runaway generation

• output tokens unexpectedly high; answers include “let me think…” paragraphs.

• cap output tokens; enforce structured outputs; add “be concise” only where it matters (don’t blanket-apply style constraints everywhere).

A lightweight “waste score” you can ship this week

You don’t need perfection. You need a metric that makes waste visible and ranks what to fix first.

Here’s a simple waste score per user action:

• CallsScore

  max(0, calls_per_action − 2)

• Input

• BloatScore

  (tokens _in − target_ tokens_ in) / target_tokens_in, floored at 0

• Retry Score

  retries _per_ action

• Tool Bloat Score

  tool _tokens_ in / tokens _in

• RAGBloatScore

  rag _tokens _in / tokens _in

Then:

= 1.0×CallsScore + 1.0×RetryScore + 0.5×InputBloatScore + 0.5×ToolBloatScore + 0.5×RAGBloatScore

Pick a reasonable per endpoint (start with p50 + a bit). Track Waste Score p50/p95. Sort endpoints by p95 Waste Score and fix the top 3. You’ll usually find one ridiculous culprit.

Implementation playbook: the 7-day cost reduction plan

This is the “do it now” plan I’ve used when a bill jumps and everyone suddenly cares.

Day 1: Get visibility

• Ship token breakdown logging (system/history/RAG/tools/user).

• Add per-action aggregation (not just per-call).

• Add retry counters and tool payload sizes.

Day 2: Stop the biggest leaks

• Cap output tokens per endpoint.

• Add retry caps and circuit breakers (hard limits).

• Validate tool outputs and fail fast.

Day 3: Tame history

• Implement sliding window.

• Add rolling summary with a strict schema.

• Remove verbose assistant messages from history when not needed.

Day 4: Fix RAG

• Add a context token budget.

• Reduce top-k and rerank.

• Add compression step for long docs or large chunks.

Day 5: Trim tools

• Tighten tool schemas (return less).

• Summarize large tool responses before re-injecting.

• Remove request/response echoes and debug fields.

Day 6: Add caching where safe

• Start with exact-match caching for sub-steps.

• Version cache keys (prompt/model/corpus).

• Add gating rules + logging for serve decisions.

Day 7: Batch the boring stuff

• Micro-batch high-volume background tasks (classification, extraction, embeddings).

• Group by template and schema.

• Monitor parse failures and tail latency.

Checklist: Fastest cost wins this week

• Cap output tokens + enforce structured outputs

• Add sliding window + rolling summary

• RAG token budget + rerank to fewer chunks

• Trim tool schemas; summarize tool output

• Add retry caps and circuit breakers

• Exact-match cache for stable sub-steps

• Micro-batch high-volume non-interactive tasks

Common mistakes

• Caching without versioning.

  You will ship a prompt change or docs update and your cache will happily serve old answers. Add versions on day one.

• Optimizing average tokens while ignoring p95.

  Your bill is often driven by tail behavior: huge histories, giant tool payloads, loop storms. Fix p95 first.

• Trimming prompts by deleting constraints.

  If you remove the parts that enforce structure or safety, you’ll “save money” and then spend it on retries, support tickets, and incident postmortems.

• Assuming batching is a token-cost lever.

  Batching helps throughput and overhead. Token savings come from trimming and avoiding redundant context.

• RAG as a dumping ground.

  If retrieval quality is poor, people compensate by retrieving more. That’s like fixing a leaky pipe by turning up the water pressure.

Token waste red flags

• Input tokens grow with conversation length

• RAG context regularly >40–60% of prompt

• Tool outputs are larger than user input by 10×

• Calls per action >2 on average

• Retry rate spikes after deploys

• Output tokens hit max token cap frequently

• Frequent “JSON fix” loops or schema repair prompts

---

You Might Be Interested In

• Do Autonomous Cars Use Ai?
• Top Google Free Ai Tools Offering Practical Ai Solutions
• What Are The Main Components Of an Expert System Explain?
• How Ai For Zero-day Attack Prevention Works?
• How Is Ai Automation Expected To Evolve In Coming Years?

---

Conclusion

LLM cost optimization isn’t about shaving pennies off output length. It’s about system design: reducing redundant context, controlling loops, tightening tools, and caching the right things with the right safety rails.

If you do nothing else this week, instrument token sources and look at p95. You’ll find a leak. You’ll fix it. And your bill will drop without making your product dumber.

If you want a simple next step: implement the waste score, rank your endpoints, and tackle the top three offenders. That’s usually enough to get meaningful savings fast and it sets you up for smarter caching, batching, and prompt trimming over time.

FAQs

What drives LLM cost the most?

The biggest driver in real systems is almost always input tokens, not output. It’s the stuff you keep feeding the model every time: long system prompts that have grown into “instruction novels,” chat history that expands without control, RAG context that gets stuffed in “just in case,” and tool outputs that are basically entire JSON documents dumped into the prompt.

Even if your assistant only generates a few hundred tokens, you can still be paying for thousands (or tens of thousands) of tokens per user action because the model has to read everything you send.

The other multiplier is workflow shape: how many calls you make per user action, and how often you retry. A system that makes three LLM calls, does a tool call in the middle, then retries once on a schema failure can easily cost 5–10× what you expected when you designed the “happy path.”

That’s why the most useful view isn’t cost per request; it’s cost per successful user action, split by where the tokens came from.

When should you cache LLM responses?

Cache final LLM responses when the answer is stable and the demand is repeatable. Think FAQs, docs questions, onboarding steps, and other “many users ask the same thing” flows. In those cases, caching can cut both cost and latency with surprisingly few downsides, as long as you version your cache keys with prompt/model/corpus changes.

This is where LLM caching actually shines: high-volume, low-variance questions where correctness doesn’t depend on the current minute or a specific user’s account state.

Caching backfires when the answer depends on freshness (“latest,” “today,” pricing, availability), private context (“my account,” “my plan”), or fast-moving policy/UI details. In those cases you can still get value by caching sub-results retrieval results, chunk compression outputs, intent classification because they’re more reusable and easier to invalidate.

The safest caching strategy is usually: cache the boring, stable components first; cache final answers only when you have tight gating rules and versioning.

Exact-match vs semantic cache: which should I use?

Exact-match caching is the boring option, which is exactly why it’s the safest. If the input prompt and relevant context match byte-for-byte (or via a stable normalized hash), you serve the cached answer.

This works extremely well for templated tasks classification, extraction, deterministic rewriting, standard summaries especially when you cache intermediate steps rather than final outputs. It’s predictable, easy to debug, and easy to invalidate by including prompt/model versions in the key.

Semantic caching is where you start saving serious money in high-volume products, but it’s also where you can quietly ship wrong answers at scale. Similarity is not equivalence, and embeddings don’t understand your hidden variables (product version, plan tier, region, policy date) unless you explicitly include them.

If you use semantic caching, treat it like a feature with safety rails: version your keys, add freshness/context gating, and use conservative similarity thresholds. In practice, a hybrid approach often works best: exact-match for structured sub-steps, semantic cache for truly stable FAQs, and “verify pass” behavior for medium-confidence matches.

How do you detect token waste?

You detect token waste by making token usage observable and attributable. Log tokens in/out per call, but more importantly, aggregate per user action and break input tokens into buckets: instructions, history, RAG context, tool outputs, and user message.

Once you can see that “history = 5,200 tokens” or “tool output = 9,000 tokens,” you stop guessing and you can fix the thing that’s actually costing you money. Also look at distributions, not averages p95 and p99 are where runaway conversations and failure loops live.

Token waste is not “high usage.” It’s usage that doesn’t buy you quality, success rate, or reliability. The giveaway is when tokens spike without a corresponding improvement in outcomes: schema-fix loops, tool retries, RAG stuffing that doesn’t improve answer accuracy, or huge instruction blocks that aren’t changing behavior.

Once you have the breakdown, you can track it over time (especially after deploys) and build a simple waste score that flags actions with too many calls, too many retries, or too much non-user context relative to a target.

Does batching reduce token costs?

Batching usually does not reduce token costs directly, because token billing is tied to how many tokens you send and generate, not how you package them. If you take ten prompts and send them as one request, you’re still paying for roughly the same total tokens sometimes slightly more because you need separators, IDs, or formatting to keep outputs aligned. So if someone tells you batching is a “token cost hack,” they’re mixing it up with throughput optimization.

Where batching really helps is operational efficiency: fewer network round trips, less per-request overhead, and better throughput especially for high-volume, non-interactive workloads like classification, extraction, moderation, or offline processing. Micro-batching (collecting requests for a short window like 20–100ms) can keep latency acceptable while improving utilization.

It’s a great tool in the LLM cost optimization toolbox, but it’s mostly a systems win; your biggest token savings still come from prompt trimming, controlling history and RAG budgets, trimming tool outputs, caching smartly, and killing retry storms.