The problem
Questrade's contact center agents were searching for accurate answers inside a knowledge base of over 10,000 pages. Tables, diagrams, internal inconsistencies, the works. Finding the right answer fast was literally the job, and the tooling made it slow.
My first instinct was a structured flow-documentation system. It needed dedicated knowledge-management resources, came with a high upfront cost, and didn't get executive support. Meanwhile the data science team was prototyping GenAI capabilities, and there was strong executive appetite to do something with LLMs. Nobody owned the product direction. So I stepped in and defined it end-to-end.
The constraint that shaped everything
This is financial services. Compliance risk is high, and hallucination tolerance is near zero. A chatbot that's confidently wrong about tax-adjusted cost basis isn't a quirky demo, it's a liability.
So we scoped the MVP to internal users only (employees and agents, never customers) until the system earned trust. I'd previously built a rule-based chatbot as a side project, where I partnered with customer service leads to mine real transcripts for high-friction, high-complexity flows: account journaling, cost-basis lookups. Not the most frequent queries. The most misunderstood ones. Perfect for testing.
The hard part: the accuracy dial
The source documents were poorly structured, and the models struggled with table data and conflicting content. Tighten the system too much and you get "I don't know" far too often. Loosen it, and hallucinations creep in. Managing that dial was the real product work: confidence thresholds, explicit disclaimers when the model was unsure ("this may be a partial answer"), fallback paths to humans, and safe-versus-restricted topic classification with audit trails, designed together with legal.
The metric story I'm proudest of
After launch, average handle time stayed flat at 4–5 minutes. On a dashboard, that looks like failure. I'll be honest: for a moment it felt like one, too.
But underneath, two things were cancelling out. Previously long calls got predictably shorter. And some previously short calls got longer, because agents could now walk customers through a full resolution instead of giving a partial answer that turned into a follow-up call next week. First-contact resolution likely improved. We couldn't prove it yet (the measurement infrastructure didn't exist), so building that infrastructure became part of the roadmap conversation.
What shipped, and what it changed
What we shipped wasn't flashy. It was stable, safe, and built with integrity. It reset internal expectations from "magic AI" to "low-risk automation framework," and it surfaced the content-governance gaps that directly informed the knowledge-management roadmap. Yes, the very same investment that didn't get support earlier. Sometimes you earn the right to fix the root cause by shipping around it first.