B2B SaaS & AI Tooling
"An AI product's OpenAI bill was growing 30% month-on-month. After a full LLM cost audit, token spend dropped by 61% with no degradation in output quality."
↓ Scroll to reveal the cure
A B2B SaaS product had integrated a large language model to power its core feature set. Initial traction was strong, but the AI infrastructure costs were scaling disproportionately — growing 30% month-on-month while revenue grew at 12%. At the current trajectory, AI costs alone would exceed gross margin within two quarters.
Further investigation revealed the root cause: every API call sent the full conversation history regardless of relevance, the most expensive model was used uniformly across all query types, and there was no caching layer for semantically similar queries. The product team had no visibility into per-feature AI spend.
The most expensive frontier model was used for every query — including simple yes/no classification tasks that a smaller model could handle at 1/20th of the cost.
Every API call included the entire conversation history. For long sessions, this sent 6,000+ tokens of context when only the last 400 were relevant.
Semantically identical queries (rephrased versions of the same question) hit the API independently every time, with no deduplication layer.
The engineering team could not tell which product feature was responsible for which percentage of the monthly AI bill. Cost optimisation had no clear target.
We conducted a full LLM cost audit: categorising every query type by complexity, measuring average token consumption per endpoint, and identifying the top-10 cost drivers. We then designed a three-layer optimisation strategy.
First, a model routing layer was designed to direct simple classification and extraction tasks to lightweight models, reserving premium models for complex generation tasks only. Second, a semantic caching layer was architected to intercept and serve repeated or near-identical queries from cache rather than the API. Third, a context windowing strategy was designed to send only the most relevant conversation turns per request, reducing average input token count by over 50%.
−61%
Token Cost Reduction
Month-on-month
310ms
Average Latency
From 1.4s avg
38%
Cache Hit Rate
Free responses
−55%
Cost per Active User
Sustainable unit economics
Does your system look like the "disease state" above?
Get a Free Diagnosis