Is There a Right Way to Use AI?
Almost everyone I know is using it wrong.
They treat it like Google. They ask it to change one sentence in a 2,000-word article. A friend of mine picks actual fights with his Claude Cowork. Then they wonder why they’re spending hundreds of dollars a month on tools that were supposed to make life cheaper.
Enterprises do the same thing at scale. When the API bill gets scary, they shorten prompts, cap output, and switch to cheaper models.
GitHub tested exactly that. They shortened the shell output their coding agent sees, and every response got smaller. But whenever the agent missed something, it ran another command to recover it, and each extra turn dragged the whole history along. Tasks took longer and cost more.
Less context was never the fix. Here’s what is.
together with Upwork:
Here’s the part of every build I still can’t automate: the moment I need someone who has actually done this before.
Until last week, that meant 2 screens, Claude on one and a freelancer marketplace on the other, copy-pasting between them for an afternoon.
Upwork’s new MCP connector for Claude and ChatGPT collapses that into one conversation.
You describe the outcome, and it comes back with a ranked shortlist of freelance experts and the reasoning behind each one.
One client had Claude read 50 profiles against his job post. It took minutes. He’d budgeted a week.
▫️ Identity-verified freelancers across 180+ countries
▫️ Escrow on every payment, and you pay only when you hire
Table of Contents
1. The Bill Nobody Modelled
2. The Experiment That Breaks the Obvious Fix
3. The Four Places Context Goes to Rot
4. The Changes That Cost Nothing to Make
5. The Changes That Need a Test Behind Them
6. The Problem Underneath the Problem
1. The Bill Nobody Modelled
An agent does not cost more than a chatbot because it is cleverer but mostly because of everything that happens behind the scenes.
A thousand times the tokens for work that looks the same
A chatbot sends a message and receives one back.
An agent plans, calls a tool, reads the result, decides the next move, calls another tool, drafts something, runs a test, revises.
Each of those steps writes text into the context window, and all of it gets resentful on every turn that actually follows.
The attention cost scales with the square of sequence length, so a context that doubles does not cost twice as much.
Agentic workloads land somewhere around a thousand times the token consumption of ordinary prompting.
Peter Steinberger, who built OpenClaw, put a number on the top end. His team spent $1.3 million on tokens in one month, 603 billion of them, across 100 coding agents run by three people.
There is a recent position paper on token reduction from Harvard, MIT and Northeastern singles out OpenClaw and Codex as the systems worth studying, simply because of how many tokens they burn through.
The arithmetic nobody runs before shipping
This version actually applies to everyone else. At roughly $2.50 per million input tokens, a single run carrying 30,000 tokens of context costs seven and a half cents.
Ten thousand runs a day is $750. Call it $22,500 a month, for input alone, before the model generates a single word of output.
And big companies are already feeling it. Uber reportedly exhausted its entire 2026 AI budget in four months. Microsoft is reported to have ended its Claude Code licences after a pilot that began in December 2025.
In that survey of 2,500 decision-makers, the rollback rate reached 81% at companies with mature governance frameworks, which is to say the organisations paying closest attention pulled back hardest.
The thing is, none of the decisions behind those bills was crazy on its own, which is why nobody saw it coming.
2. The Experiment That Breaks the Obvious Fix
GitHub publishes its agent cost engineering, which makes it one of the few places where any of this gets tested rather than asserted.
Shorter calls, longer tasks
The team evaluated a utility called Rust Token Killer, which shortens shell output before a coding agent reads it, and ran it against their own agentic coding benchmarks.
It did its job, the tool responses got shorter.
Then, when the omitted text turned out to matter, the agent reopened the original output or reran the command to recover what was missing. Each recovery added a turn, and every added turn dragged the whole accumulated context along with it.
On average the task consumed more tokens and took longer. Completion rates held steady. So each call got cheaper, but the whole task got more expensive.
The unit is the finished job
By counting tokens per tool call means measuring the wrong thing, once you see that, a lot of the usual advice stops making sense.
A response 200 tokens shorter that triggers two recovery turns is a loss. A response that runs longer but lets the agent finish in one pass is a win.
And it works the other way round, too. That token reduction paper argues attention is a finite budget. Bury a decisive fact in the middle of a 40,000 token window and the model pays less attention to it, a well known problem researchers call “lost in the middle”.
Reasoning models left uncapped have been measured spending over 15,000 tokens on maths problems a few hundred tokens would solve.
One method the paper cites, CoT-Influx, pruned the reasoning examples sitting inside a prompt, which freed room for more of them, and accuracy went up.
So cheaper and better aren’t really opposites. You only lose quality when you cut something the system actually needed, so the first step is figuring out where the wasted tokens are.
from our partners:
GitHub's lesson applies to hiring too. The expensive move is skimping on the scope and paying for it in rounds of interviews.
Describe the outcome once, and Upwork's MCP connector lets Claude rank the freelance experts who fit, with the reasoning attached.
3. The Four Places Context Goes to Rot
Waste concentrates in four reservoirs, and each one wants a different fix.
Prompts and tool output
The system prompts tend to grow over time, because people keep adding instructions and almost never take any out. A prompt that started at 500 tokens arrives at 5,000, and most sit at roughly three times the length they need.
Agent harnesses make it even messier. GitHub found the guidance for a single task tool scattered across tool descriptions, schemas, agent definitions, system instructions and companion tools, and no one was in charge of the full picture.
Tool output is worse, because it arrives raw.
One web search result runs 2,000 to 8,000 tokens of stripped text. A 50 page policy document loaded in as reference material is 25,000 to 40,000 tokens, and any given query needs about three paragraphs of it.
History and thinking
Ten turns into a conversation, history alone is 15,000 tokens and climbing. By turn twenty an agent can be sending 30,000 tokens of past exchanges just to ask what happened in the second one.
Developers often make this worse on purpose, writing recent actions, errors and state into the prompt so the agent remembers what it has done.
And then there’s the thinking part. Extended reasoning modes add 5,000 to 20,000 tokens to calls that often need none of it, and if you don’t put a cap on it, the model will happily use all of it.
Three of those four drain without any risk to output quality at all.
4. The Changes That Cost Nothing to Make
Some changes leave the model receiving precisely what it received before. Those go first, because there is nothing to weigh up.
Caching, and the ordering that makes it work
Marking the stable part of a prompt as cacheable drops its price to around a tenth of the normal input rate. Writing to the cache costs roughly 25% extra, so you break even after about two calls.
Basically, anything you send more than once is worth caching.
The order of your prompt is what makes or breaks this. Static content goes first, meaning the system prompt, examples, reference documents and schemas.
Dynamic content goes after, meaning the query, the tool results, anything that varies. OpenAI caches prefixes automatically above 1,024 tokens,
Anthropic uses explicit breakpoints, and the cache lasts about five minutes by default.
One research agent sending a 6,000 token system prompt on every call saw input costs fall 87% on this change alone.
Agents sharing a large system prompt across many users typically save between 60% and 80% on the cached portion.
The catch is that a single dynamic detail, like a timestamp, a session ID or a user name, can slip into the static block without anyone noticing.
When it does, everything after it gets reprocessed at full price.
Formatting nobody uses, files nobody needs
GitHub’s file reading tool had been adding a number to the start of every line of output. Older editing tools needed those numbers to target changes.
Current ones match the surrounding code instead, so the numbers had been dead weight for a while.
Removing them cut inference cost around 5% in offline benchmarks and about 3% per user per day in production, with success rates inside normal variance and no increase in edit failures.
PDFs have a similar hidden cost. Every page gets rendered as an image and processed as visual plus text tokens, costing 1,500 to 3,000 tokens even when it holds 200 words.
Those same words pasted as plain text run about 250. One 42 page report cost 84,000 tokens per call as a PDF and 9,500 after conversion.
And there’s a fourth one, hidden in how the agent juggles its tasks. When a background task finishes and the completion notice doesn’t include the result, the agent wastes a whole turn asking for something the harness already has.
Two tasks running at the same time meant four model calls before work could resume. Batching those completions cut it down to one, which saved about 2.3% without compressing, summarising or leaving out anything.
From here on, the changes need some judgment, plus a way to test them.
5. The Changes That Need a Test Behind Them
The remaining levers are bigger, and all of them change what the model sees.
Difficulty is the cheapest thing to measure
Sending every request to the strongest available model is the equivalent of paying a surgeon’s hourly rate to apply a plaster.
Most tasks can be sorted with 5 simple questions.
Is the work repetitive, or does it need real judgment?
If a mistake slips through, will someone catch it later, or will it spread?
Does the task need actual reasoning, or just reshaping what’s already there?
Will a customer see the output?
Does the job need the best possible answer, or just one that works?
Most of the time, a simple rule is enough to decide which model gets the task
When it isn’t, a call to the cheapest model costs almost nothing next to what it saves. A simple classifier pushes 60% to 70% of production traffic into the bottom tier, where tokens run 10 to 50 times cheaper.
One team shipped a router and watched the monthly bill fall 58% that same week.
Thinking budgets work the same way. Lookups run fine at zero thinking tokens, moderate reasoning at 1,000 to 2,000, and really hard analysis at 10,000 to 16,000.
Teams that adjust the budget to each query report 50% to 75% off their thinking token cost, because most production queries are routine.
Compression that survives contact with an agent
GitHub’s shipped compressor ended up conservative, and its shape is worth copying.
Anything that looks like source code or unpredictable output is left untouched, meaning file contents, diffs and script results.
Those are exactly the places where one missing line sends the agent straight back. Search results get reorganised without losing anything, and every match is kept.
Only predictable noise gets compressed, meaning install logs, build output, test runners and progress bars, and only when it saves a meaningful amount.
Git diff was in the compression set at first. It was taken out after benchmark tasks showed agents reopening the originals.
The agent can always go back to the full original, and how often it does is a better sign of whether compression works than any offline metric.
When compression is safe, the savings are big. Semantic compression cuts oversized tool output by 70% to 90%.
Rolling summarisation keeps the last four or five turns word for word and folds the rest into a short memory block, which takes a 20 turn history from 8,000 to 15,000 tokens down to under 2,000.
Moving logs and state into SQLite and pulling them only when needed swaps 6,000 tokens of carried history for 300.
There’s one more story from the same GitHub work that’s worth telling. They used an automated loop to rewrite an agent prompt, which cut it to half its size, passed offline evaluation, and shipped.
But in production, subagents that used to run at the same time started running one after another. The rewrite had taken a gentle suggestion about running tasks in parallel and turned it into a strict rule.
They stopped the experiment and wrote a test to catch the behaviour users had spotted. The final fix was a single sentence that was both shorter and less restrictive than the original.
An untested behaviour can be deleted by a compression pass without anyone noticing.
6. The Problem Underneath the Problem
Everything above is about making an existing system cheaper. But a lot of agent costs come from systems that were built the wrong way in the first place.
Before you hand a workflow to an agent, you should be able to answer five questions. What does it receive? What exactly does it do? What does it produce? Who checks the result? And how does it know when it’s finished?
If you leave any of them blank, the agent will just guess. Without a stop condition, it keeps rewriting work that was already fine.
Without a review point, it runs steps that change nothing. Without a clear output, agents keep passing work back and forth forever. From the outside it all looks productive, and you pay for every token of it.
Context works the same way. An agent writing headlines doesn’t need the brand wiki. An agent fixing formatting doesn’t need the company strategy.
Most of the work is giving each agent a small pack of context for its own role, instead of handing everyone the same knowledge base.
And that’s the real problem. Most runaway token bills aren’t an efficiency issue at all, they’re a design issue. Teams resend huge histories, pull in too much data, default to the biggest model and ask for long answers, all because nobody ever decided what mattered.
The system doesn’t know what’s important, so it sends everything.
It’s also why measuring AI adoption by how much people use it tends to backfire.
Leaderboards that celebrate whoever burned the most tokens reward activity over good judgment, and you end up with exactly what you measured.
That means premium models doing trivial work, agents used where one prompt would have done, and ten step workflows built for two step problems.
The teams that got this right didn’t just send fewer tokens, they chose which ones to send.
An agent’s invoice was never a record of how hard it worked. It is a record of everything nobody bothered to decide.
Love this framing, good piece - it has always been about the outcome. I think the novelty of agents has caused some to forget that what we should always have in mind is the finished product. Once prices got real folks started allocating less time to playing and more time to how to get to a price efficient outcome.
I keep seeing clients optimise the visible number too — the cheapest term plan premium, the lowest expense ratio fund — while the real cost sits in the gap between what they hold and what the goal actually needs.
A lower premium that leaves a coverage shortfall, or a smaller SIP that forces a top-up scramble later, is the same error as a shorter tool response that triggers two recovery turns.
Measuring the wrong unit feels like progress right up until the finished job comes due.