Claude can't count your customer feedback
Homegrown Claude setups summarize customer calls well, then hit a wall. Why "how many customers asked?" breaks retrieval, and the layer that fixes it.

If you run product at a B2B software company, there is a good chance someone on your team has already wired your calls and tickets into Claude. Maybe it was you. You exported a folder of transcripts, pointed a project at it, and started asking questions.
It works. For a while it works really well.
Then someone asks the question every roadmap meeting eventually asks: “How many customers actually asked for this?” And the answer you get back is a confident paragraph with no number in it.
This post is about why that happens, why it is not a prompt problem, and what to add to the build you already have.
What a homegrown Claude setup does well
Credit where it is due. A homegrown setup is a fast, cheap way to get closer to customers.
- It reads faster than you. Ten call transcripts summarized in a minute is a real gain.
- It finds the quote you half remember. Search plus a good model beats scrolling through recordings.
- It is yours. You chose the sources, the prompts and the format. Builders are rightly proud of that.
For one person working through a handful of calls, this is often enough. In our Build vs Buy research, about 60% of the product teams we spoke with already had some homegrown equivalent. That number alone tells you the instinct is right. Teams want customer evidence inside the tools where they work.
The trouble starts when the setup moves from one person’s assistant to the team’s source of truth.
The wall at around 10 transcripts
Ask a model about 3 calls and it reads all 3. Ask it about 300 and it cannot. Every chat-style setup is bounded by what fits in the context window at the moment you ask. Past that point, a retrieval step picks a handful of chunks that look relevant to your question, and the model answers from those.
That changes the nature of the answer. With a few transcripts, the model is summarizing everything. With hundreds, it is summarizing whatever the search step happened to pull. The answer still reads as complete. It still sounds sure. But it is anecdotal, and nothing on the page tells you so.
Teams describe the same moment in different words. Answers that felt sharp on the first batch of calls start to feel generic. Two people ask the same question and get different stories. The builder starts to add caveats before sharing results.
Why “how many asked?” is the question that breaks it
Most questions you ask a homegrown setup are lookup questions: “What did the customer say about onboarding?” Retrieval is built for those. Find the relevant passages, summarize them, done.
“How many customers asked for this?” is a different kind of question. To answer it honestly you need to:
- Read every conversation, including the ones that never mention your exact words.
- Decide which mentions are the same underlying need, even when customers phrase it differently.
- Count distinct customers, not distinct quotes.
- Join that count to who those customers are and what they are worth.

Retrieval does none of that. It returns the top matches for your phrasing and stops. So the model does the only thing it can: it guesses, and it guesses fluently.
In the same research, fewer than 1 in 8 teams with a homegrown setup could cluster or count across their feedback. One team put the gap plainly: their in-house setup could surface examples but had no true counts behind them.
”Retrieval finds documents. Counting finds the roadmap.”
The five walls
When we looked across the teams that had outgrown their build, the failures fell into the same five places.
| Wall | What it looks like | What fixes it |
|---|---|---|
| 📈 Scale | Answers turn anecdotal once you pass the context window | Process every conversation ahead of time, not at question time |
| 🔢 Counting | ”How many?” gets a guess | Cluster needs across sources and count distinct customers |
| ✅ Reliability | Nobody can tell which claims are real | A citation on every claim, down to the quote |
| 👥 Use by non-builders | Only the person who built it knows what to ask | A shared workspace anyone can use, plus access from their AI tools |
| 🛠️ Maintenance | It breaks when the builder gets busy or leaves | A pipeline someone else keeps running |
The third and fourth walls tend to show up together. A product leader at an AI company told us the real limit was not the model: “your entire team doesn’t know what questions to ask Claude”. A setup that only its builder can use is a personal tool, however good it is.
”Your entire team doesn’t know what questions to ask Claude.”
A product leader at an AI company
The layer your build is missing
None of this means you built the wrong thing. It means you built a retrieval layer, and a retrieval layer is one part of a customer intelligence stack. The missing part sits underneath it:
- Pre-processing. Every call, ticket and note is read when it arrives, so nothing depends on what a search step picks at question time.
- A structured model of what customers need. Needs, pain points and requests are grouped across sources, so different phrasings of the same ask land in one place.
- Counts and value. Each need carries how many distinct customers raised it and which accounts they are, so “how many” and “what are they worth” have real answers.
- Citations. Every claim drills down to the quote and the source it came from, so you can check it before you put it in a roadmap review.
This is the layer Evermuse adds. It sits next to the build you have rather than replacing it. Your Claude setup keeps doing what it is good at, and when it needs a count, a trend or a citation, it can ask Evermuse through MCP and get an answer grounded in every conversation, not the ten that fit.
A quick test for your setup
Ask your current setup these three questions about a feature your team is debating this quarter:
- How many distinct customers asked for this in the last 90 days?
- Which of them are our largest accounts?
- What changed in how often it comes up this month?
If you get numbers you would put in front of your CEO, with a source behind each one, your build is in great shape. If you get fluent paragraphs, you have found the wall.
Want to see the counts behind your own feedback? Connect Evermuse to Claude at evermuse.com/mcp and ask the question your setup can’t answer.
FAQ
Can I use Claude to analyze customer feedback? Yes, for a small number of calls Claude summarizes feedback well. Past what fits in its context window it answers from a sample, so it can’t reliably count how many customers raised a need.
Why can’t Claude count customer requests? Chat setups retrieve the passages that best match your question and summarize them. Counting needs every conversation read, similar requests grouped, and distinct customers tallied, which retrieval does not do.
Do I need to replace my homegrown Claude setup? No. Add a layer that pre-processes every conversation into counted, cited needs, and let your existing setup query it through MCP.