How we cut customer-support inquiries by 80% — and the pitfalls on the way

Before AI Blacksmiths, I led a six-person AI team at a Malta-based services group serving multiple consumer brands. Our brief sounded simple: customer support was drowning, and leadership wanted to know if "this LLM thing" could help.

Eighteen months later, automated handling had cut incoming support inquiries by roughly 80% across the brands we served. Here's what actually did it — and the three pitfalls that nearly killed the project on the way. This is the stuff I wish someone had told me at the start, and it's now baked into how I run every engagement.

What we built

The core was less glamorous than "an AI support agent" sounds:

• A retrieval layer over each brand's real content — help articles, policies, product data — using LangChain, embeddings, and Pinecone as the vector store, so the model answered from the brand's own truth instead of improvising.
• Multiple LLMs behind one interface — OpenAI models and Llama 3 via AWS Bedrock — chosen per task by cost and quality, not loyalty.
• A template engine for content generation that produced a new brand's full initial support content in minutes. Previously that was weeks of copywriting.
• One-click bot setup for launching a new brand, because the second brand onboarding revealed that every manual step would otherwise be repeated forever.
• Plumbing that gets no applause but decides everything: FastAPI services, PostgreSQL, evaluation with RAGAS, tracing with Langfuse, alerts with Sentry.

Pitfall 1: we trusted the demo

The first prototype answered beautifully in demos. In production, real customers asked questions in ways no demo anticipated — typos, mixed languages, questions about things that didn't exist. Quality fell off a cliff nobody saw, because we weren't measuring.

The fix was boring and decisive: an evaluation suite that scored answer quality on real (anonymized) conversations, run on every change. From then on, "did the new prompt make things better?" had a number instead of an opinion. If you take one thing from this post: no eval suite, no production.

Pitfall 2: hallucinations hide in the edge cases

The system was grounded in retrieved content, which prevents most invention — but "most" isn't a word you want in front of a customer. The dangerous cases were the gaps: questions where retrieval found nothing relevant, and the model filled the silence confidently.

We made "I don't know, here's a human" a first-class answer, triggered by retrieval confidence, not model politeness. Counterintuitively, deflection rose when the bot admitted ignorance more readily — customers trusted the answers it did give, and stopped re-asking the same question five ways.

Pitfall 3: the second brand is the real project

Getting one brand live was an achievement. The economics only appeared when launching the next brand took a click instead of a quarter. We spent real engineering time on templating, configuration, and setup automation — work that looked like overhead and turned out to be the product.

If your AI project only works once, you've built a demo with a production bill.

What this means for your support queue

The 80% number wasn't one clever model. It was retrieval over your own content, honest escalation, measurement on every change, and repeatability. None of it is magic; all of it is engineering. That's the difference between a chatbot your customers resent and a system that quietly removes most of a queue.

If your team is buried in repetitive inquiries, this playbook transfers — and the first step is a readiness check, not a rebuild. Book a 30-minute intro and I'll give you an honest read on whether it fits your setup.