Stop Thinking about Hallucinations
By Kunal Datta, Chief Product Officer at Unit21
The first AI feature Unit21 ever shipped internally was a chatbot that would happily invent links. Ask it for a CSV export and it would generate a fake URL, complete with the air of helpfulness. Push back and it would generate another one. Kunal Datta, the company's Chief Product Officer, tells the story without much embarrassment. It was 2023, GPT-3 had just landed, and the entire industry was figuring out what these things were good for by handing them spreadsheets and seeing what came back. The answer, almost universally, was nonsense delivered with confidence.
That old story now sits at the heart of one of Datta's running frustrations. Three years on, the question he's still getting asked, this time by me, is "but what about hallucinations?"
"This problem has been engineered around,over and over and over again, with all these different solutions, but yet that's the question that comes up again and again."
Datta draws a clean distinction between AI bolted onto an existing compliance workflow and AI sitting on top of dedicated infrastructure built to host it. The first camp, in his view, is most of the market. A bank or fintech mandates AI adoption, the compliance team tries ChatGPT or Claude on their own, hits the wall, and an engineering team is brought in to shadow analysts for two or three months and codify their decisions into a custom agent. The output is one automated process, specific to one customer's data and one customer's process.
"AI Risk Infrastructure" is the company's name for the layer that lets a compliance officer skip this journey. The agents are part of it, but so is everything underneath: the testing rails that grade an agent against years of past human decisions, the data hooks that pull from whatever tools a customer is already running, the auditability needed to defend an automated decision to a regulator, the multi-model architecture that swaps in whichever model is currently strongest for a given task. Bolting an LLM onto the old stack doesn't make it AI-native, and that's why Unit21 has built something different.
To understand why Datta is slightly impatient with the hallucination question, it helps to walk through how Unit21 actually stops the AI from making things up. There are roughly four mechanisms, and they work together as layers, not individually.
The first is the most interesting and the hardest to replicate. Eval sets are the AI industry's version of a report card: thousands of examples where you know the right answer, against which you grade the model's output. You write a prompt, run the model against a list of cases where the answer is already known, and measure how often it gets them right. The first time through, you might score 20 percent accuracy. You adjust the prompt, run it again, and hit 60. Another tweak, another run, you land at 90. You keep iterating until you get to a number you trust, and only then do you ship. The cycle is unglamorous, and to me the process sounded more like bookkeeping than cutting-edge technology. Datta says Unit21 was running it on spreadsheets back in 2023, before the AI industry had even settled on a name for what they were doing.
The catch, for most vendors, is that building those test sets in the first place is brutal and bespoke. A sanctions check requires ten thousand worked examples of where the answer was yes and where the answer was no. Unit21 has been running a platform that human analysts have used to do this work for eight years. They have the data already. They know, for any given task, what the right answer looked like when a human did it. They also know, as Datta says quite plainly, that some humans are bad at the job.
One of the first features he ever built at the company was a random-sampling QA tool, the kind of audit trail any accounting firm would recognise, which means they can now grade their AI not against all the humans but against the top performers, the ones whose past decisions hold up to scrutiny.
The second mechanism is structured output through code generation. The LLM is not allowed to look at data and answer questions about it directly. Instead, the model writes code, SQL in some cases, which then deterministically queries the underlying data and returns the answer. The model is fetching the right data, rather than making decisions on the data. This shifts the failure mode from hallucinated answers to potentially incorrect queries, but that is exactly what the eval sets are designed to catch. If the model writes a bad query, the wrong answer shows up in the test results and the prompt gets refined until it doesn't.
The third is context engineering. The model is given just the right amount of context for a given task, no more, no less. Too much and it gets confused. Too little and it guesses. Knowing what to give it is, again, a function of having watched humans do the same task thousands of times. Unit21's architecture takes this further than most: rather than stuffing everything into one giant prompt, the system routes an investigation across specialized parallel workers, each focused on a narrow objective with a narrow slice of context. A data-gathering worker pulls transaction histories. A behavioral-analysis worker evaluates deviation from normal patterns. A sanctions worker checks watchlists. Each operates independently and feeds results into a shared data store. A separate decision layer then aggregates their outputs to reach a final determination. The effect is that no single model call is asked to do too much, which is where context overload typically causes quality to degrade.
The fourth is temperature, a basic parameter on the underlying model that controls randomness. Set it to zero and the output becomes more deterministic, which helps in a domain where you want consistency rather than creativity. It's the simplest of the four mechanisms and the least sufficient on its own. Temperature zero doesn't prevent a model from being confidently wrong. But combined with the structured output, the eval sets, and the scoped context, it removes one more source of variability from a system designed to minimize all of them.
The point Datta wants to land is that none of this is the LLM doing the work. The LLM is, in his phrase, the engine, not the car. "If I gave you an engine and said go on a road trip, what are you going to do?" The answer, sitting in a workshop holding a block of metal, is presumably nothing. Yet plenty of fintechs, he says, are essentially being handed engines and told to drive.
There is an obvious follow-up question: if the eval sets get you to 90 or 95 percent accuracy, what happens with the rest? This is where the system moves from pre-deployment testing to continuous monitoring. Unit21 runs what the AI industry calls an "LLM-as-judge" system: a separate model that evaluates the primary model's work in real time, scoring every output for consistency and accuracy against the established baseline. If quality degrades, whether from a model update, a shift in the data, or a prompt that works well on one class of alerts but poorly on another, the monitoring system flags it and recommends updates: prompt adjustments, parameter changes, or swapping in a different model for that specific task.
And below all of this is a design choice that Datta considers non-negotiable: the human stays in the loop. The AI agent does the investigation, gathers the evidence, writes the narrative, and makes a recommendation. But in any case where the system's confidence is not high enough, or where the decision carries real regulatory weight, the output goes to a human analyst for review. The system is not trying to reach 100 percent automation immediately. It is trying to automate the cases that are clearly false positives, roughly the 60 to 80 percent of alert volume that experienced analysts close in minutes, and escalate everything else. The residual error rate is managed not by pretending it doesn't exist, but by building the infrastructure to detect it, learn from it, and route around it.
There is a split forming in how companies are responding to AI. At one end are the teams who have started using it, without guardrails, and are slowly getting through things and making incremental improvements. At the other end are the teams who have looked at it, recognised the risks, and are also methodically getting on with the work. The big group in the middle, in Datta's view, is the problem. They are spending more time debating AI than doing anything with it. The discussions matter, but the rate of change in the field is fast enough now that six months of internal debate can produce theoretical conclusions about risks the rest of the industry already learned to practically manage two years ago.
And in the meantime, he points out, there are real crimes happening, such as human trafficking, that could be prevented. "It is happening as we speak," he says. "And in absence of using this stuff, people are getting away with it." Hallucinations are not a reason for caution. They are an old objection that has been engineered around, and continuing to treat them as a live problem is letting the bad actors keep their head start.
Unit21's platform knows how to test itself against the work of its best human predecessors, monitor its own quality in production, and route uncertain cases to human judgment. The question of whether it will hallucinate has been answered, not by eliminating the possibility, but by building the layers of infrastructure that make the residual risk lower than the error rate of the manual process it replaces. The harder question is whether the rest of the industry will realise this before the next governance committee meets.
