In some of the early companies I worked with, the most valuable employees were the ones with the most expertise and experience. We called them SMEs. There were only a handful of them in the whole company, and we looked up to them and aspired to be like them.
Cut to 2026. Everyone has access to AI models trained on real-world data, capable of completing weeks of work in a few hours. AI doesn’t replace those SMEs, but much of what once needed their time is now available to everyone. We are now scaling intelligence like infrastructure. And when intelligence becomes free, trust is what we start paying for. But why? And how do you build it?

Over the last three years, I have seen many pharmacovigilance systems use AI to generate outputs. Yes, they make mistakes, but that is not the point. When we ask how they ensure the output is correct, the answer is always the same: “We have human-in-the-loop (HITL).”
“Human-in-the-loop” is the most misused term in healthcare AI. Human review is important, even mandatory. But it is a process control, not a system control. A reviewer checking hundreds of mostly correct AI cases a day will miss the rare error. That’s automation bias, not negligence. If HITL is your only answer, your safety net is a tired human.
What vendors fail to explain are the controls built into the AI system itself. What prevents it from generating incorrect output? What are the acceptance criteria, and does the AI stop generating output when it falls below the threshold? How do you manage a prompt change, and how do you compare its performance against the previous baseline configuration? Imagine a PV intake source document where the product name is an opioid. Would your AI generate the output as expected, or would it fail because the model’s safety filter refuses or sanitises the input?
I believe they cannot answer these basic questions because validation started only after engineering handed the system over. CSV/CSA remains necessary, but it was designed for deterministic systems. It is not sufficient for AI, where output can shift with a prompt change, a model update or the input itself. GAMP 5 and the ISPE GAMP AI Guide both call for AI-specific controls across the lifecycle, not just testing at release! Trust needs to be built into the product, not bolted on later.
At YEZA.ai, we demo trust during the POC itself: transparency and explainability in every decision the AI makes, and guardrails that actively identify and mitigate risks that could lead to unequal safety outcomes or patient harm. And yes, human oversight sits on top of all this, to review safety decisions, not to cover up for missing controls.
Thanks for reading.
😃
If you have any comments, feedback, or requests, please feel free to connect with me on LinkedIn. And if you liked this post, don’t forget to share it with your network!