The Hidden Cost of a Long AI Conversation

Introducing ContextCoach: a free tool that tells you when to hit reset

[This article is part of my “Mastering your AI Footprint” series on generative AI — its impacts on the world, and how to use it in ways that reduce its financial and environmental costs.]

Imagine you’re in a taxi (remember those?). After a few blocks, you say “turn right up here.” The driver then circles back to where you got picked up, then drives to the turn you wanted and makes the right.

Weird. Then you say, “At the third block, take a left.” Again, the driver goes back to the beginning, but then follows your first two instructions. Every direction you give sends the taxi back to the start. The meter and the engine have been running the whole time.

This is what it’s like talking to an AI chatbot (Claude, ChatGPT, Gemini, etc.). For every question or query, the AI re-reads the whole discussion…from the beginning. The whole chat history is called the “context” and it grows with each interaction.

The Context Problem

Context is measured in “tokens,” or roughly, chunks of words (check out my glossary of other AI lingo). And the count rises faster than you’d think, due to a range of choices you make:

· Conversation history. A long conversation can reach many thousands of tokens.

· Uploaded documents. A text file is counted by words, but a PDF could require roughly 2,000 tokens per page. Ask it to summarize a 100-page annual report or industry study and you’re already at 200,000 tokens. (So, quick solution, convert it to text, or just pull out the pages you really need).

· Images. Often more token-intensive than people expect. If you load an image to refer to, that’s now part of the context.

· Generated files. Things you might create with AI, like a graphic or web page coded in HTML (Often called “artifacts”). Every iteration adds many thousands of tokens.

· Project knowledge files. If you’ve created any custom GPTs for ChatGPT or Projects in Claude (e.g., I built one that searches out sustainability news stories), the project instructions are automatically pre-loaded.

· Web search results. Searching the web itself can use a lot of power. Then the results of the search, which could be long, are added to the context.

The AI companies are working on ways to avoid some of the reprocessing (using “caching” to condense some parts of context that don’t change), but within a live conversation, the basic dynamic holds. And as context increases three things go wrong.

The problems

1. Costs rise or you burn through your limits faster than you think

AI can feel free, but it’s not. Flat-rate plans come with daily token limit; hit the limit, and you stop until it resets. Enterprise teams may pay by the million tokens—most run from ~$3 to $15 depending on the tier of the model. “Millions” sounds like a lot of leeway, but context can climb surprisingly fast. Every question you ask could cost hundreds of thousands more tokens just to re-read what came before. Companies have woken up lately to really big AI bills.

2. Energy use compounds

Data centers are growing extremely fast. Big tech companies are building their own natural gas plants to power some of them, and they use enormous amounts of water for cooling.

How much energy does a chat use? Hard to know precisely; model tiers (e.g., Haiku to Opus in Claude) vary dramatically in processing draw and the context size can swing wildly. The estimates on energy use per query have varied a great deal. But this chart from Epoch AI gets at something important: the energy draw of a typical short query may be negligible, but a longer query uses roughly 130 times more energy.

The gap is the point, and it’s why context management is so critical.

Multiple sources of research on context and energy agree on the physics. As one put it, the Prefill Phase (the step where the model reads your entire chat history) is an energy monster.

We don't yet have great data on energy per million tokens, which is what we'd really need. But the direction is clear: more context, more processing, more energy. Reducing unnecessary context in an active conversation—which caching doesn't fully address—cuts that draw meaningfully.

3. Your answers get worse

As token counts rise, accuracy drops, especially for information in the middle of a long conversation. One study called this phenomenon “context rot,” concluding that “the model should handle the 10,000th token just as reliably as the 100th. However, in practice, this assumption does not hold.”

What does “handling it worse” really mean? Not totally clear, but given the well-known problems with AI and hallucination, a longer context increases the odds that you get something wrong. Worse answers mean more follow-up, which means more tokens, higher cost and energy use, and so on. The inefficiency compounds.

Share

A New Tool: ContextCoach

The simplest fix is to start a fresh chat. You may have drifted into a different topic, or built a solid draft of something you’re writing. Starting a new chat removes the noise, gives the AI a more focused problem, and stops the compounding context.

The hard part is knowing when. The AI companies don’t give you a real-time token count. So I built a free Chrome extension called ContextCoach to help (it works if you’re chatting with Claude or ChatGPT). It warns you when the token count is getting high, then helps you summarize the chat and start a new one using far fewer tokens.

It puts a colored bar across the top of your chat: green (you’re fine), yellow (consider a reset), red (really time to start fresh). At yellow and red, a button appears: “Summarize prompt.” Click it, the AI summarizes your conversation. Another button opens a fresh tab and drops the summary in, so you pick up where you left off, without dragging thousands of tokens you no longer need.

Check out the website or the Chrome store. Just click “Add to Chrome.” (Below are some simple instructions on installing it).

Caveat: The program estimates token count so the real number could be higher. For example, if the program detects a PDF in the chat, it uses a 30,000 token estimate, assuming a 15-page document at roughly 2,000 tokens per page. But if you upload a larger document, the context could be much larger.

Why Bother

Sometimes this feels like extra steps with unknown benefit. But even if we can’t pinpoint the exact energy draw per chat, we know it’s not nothing and growing, and we have both the ability and the responsibility to do something about it. Managing your AI context is like eating more vegetarian meals or driving more efficiently.

This footprint reduction also saves you money and gets you better answers.

I encourage you to try it out and see if it’s helpful, and send me feedback. Happy chatting!

Thanks for reading! This post is public so feel free to share it.

Share

Extra: How to…

Once you’re in Chrome, find the little extensions button — see my address bar here with a few icons to the right. The puzzle piece on the far right is the extensions button

When you click it, a pop up of extensions you’re using should appear like…

You click the little push pin to put it in the icons that stay next to the address bar.

The Mastering Your AI Footprint series:

Overview/Pre-quel

· The Second Chance We Shouldn’t Waste

· The Lingo

The Serious Problems with AI

· Focus on Energy (Coming soon)

· Focus on Social Impacts (Coming soon)

Using AI Efficiently

· You’re Probably Using the Wrong AI Model

· The Hidden Cost of A Long AI Conversation

· Focus on Prompting for efficiency (Coming soon)

Using AI Wisely

· Focus on when not to use AI (Coming soon)

· Focus on enhancing what’s uniquely human, and not anthropomorphizing AI (Coming soon)

---

---

---