Explainability in AI Products: Giving Users a Reason to Trust the Output
We initially treated explainability as a technical problem — expose the retrieved sources, show a confidence score, maybe surface attention weights if the client wanted something that looked sophisticated. User testing on a financial-insights product made clear that none of this was what people actually wanted. A confidence score of '87%' told users nothing actionable; they couldn't tell if that number meant the answer was safe to act on or not.
What worked was much more concrete: for every AI-generated insight, we showed the specific underlying data points it was derived from, in the same units and format the user already understood, positioned right next to the claim. Instead of 'spending is trending up' with a confidence badge, we showed the three months of actual transaction totals the claim was based on, inline, so a user could verify the claim themselves in about two seconds without needing to trust our system's self-reported confidence at all.
For cases where the AI synthesized across multiple sources rather than citing one, we found a short, plain-language explanation of the reasoning path mattered more than any visual confidence indicator — something like 'flagged because this vendor charge is 40% above your typical spend and doesn't match a recurring pattern,' rather than a bare anomaly label. That single sentence gave users a specific, falsifiable thing to check against their own knowledge instead of a verdict to simply accept.
We also learned that explainability needs vary a lot by stakes. For a low-stakes suggestion like a phrasing tweak, no explanation was necessary and adding one just added friction users skipped past anyway. For a claim that would influence a real financial or business decision, users wanted the underlying evidence every time, and skipping it measurably reduced how much they trusted and acted on otherwise-accurate outputs in our testing.
The pattern that generalized: explainability isn't about exposing the model's internals, it's about giving users something specific and checkable that lets them verify a claim against what they already know, calibrated to how much the decision actually matters. A number without context builds less trust than three real data points a user can glance at and confirm for themselves.
