Retrieval-Augmented Generation stops a clinical AI from inventing facts, and its guarantee ends there. RAG can retrieve two accurate facts and still assemble them into a recommendation a rule-based clinical system would reject, because grounding a fact is a different operation from reasoning about a rule. Safe clinical AI runs four layers together: an LLM for fluent language, RAG for factual grounding, a neurosymbolic layer that applies deterministic clinical rules to the grounded output, and an agentic layer with a human checkpoint before anything reaches patient care. The architecture that connects the four is what carries the safety.
Healthcare is the sector where getting AI wrong costs the most. A hallucinated product description is an inconvenience. A hallucinated clinical note - a physical exam finding that was never performed, a medication that was never discussed - can change a patient’s treatment path. As health systems move from AI pilots to real AI adoption at scale, the question is no longer “should we adopt AI,” but “which architecture keeps it safe.”
Reconfig has worked with NHS Trusts, together with our partner Tricordant, to simulate and design AI operating models. That work is what got us fascinated with another question underneath it: how do you actually stop AI hallucination in a sector where getting it wrong can cost a life. The conclusion is a direct one: for AI safety in healthcare, a large language model with Retrieval-Augmented Generation (RAG) is not enough on its own. You need Neurosymbolic AI alongside it.
It’s an approach that combines a neural network - the part of the system that handles language and pattern recognition - with a symbolic reasoning layer: a set of explicit, deterministic rules (if X, then Y) that the neural output must pass through before it’s accepted. Where an LLM produces the statistically likely answer, the symbolic layer checks that answer against fixed logic - a guideline, a regulation, a safety constraint - and blocks anything that violates it. It’s the difference between a system that’s probably right and a system that’s provably compliant with a rule.
RAG is the right first fix for hallucination. Instead of generating from general training data, the model is constrained to retrieve from a defined, trusted source - the consultation transcript, the patient’s existing record, an approved clinical guideline set. That grounding is real, and it closes the biggest gap in a general-purpose LLM: it stops the model from inventing facts that were never in the source material.
But RAG only guarantees that an output is grounded in a retrieved document. It does not guarantee that the output is clinically safe. RAG has no concept of a drug interaction, a dosage ceiling, or a NICE guideline - it can retrieve accurate, real information and still assemble it into a recommendation that a rule-based clinical system would immediately reject.
An illustrative example. Picture a patient’s transcript accurately mentioning they take warfarin, while their EHR record separately and accurately shows a new prescription being considered for a course of antibiotics. An LLM with RAG can correctly retrieve both facts - nothing is invented, nothing is hallucinated - and still draft a summary that fails to flag a well-known class of interaction between the two, because grounding a fact is not the same as reasoning about a rule. A Neurosymbolic layer sitting on top of that output applies a deterministic check: a known interaction rule is triggered, and the system halts for clinician review before anything reaches the record. That check has nothing to do with retrieval. It is a symbolic, rule-based gate - the same kind of logic a pharmacist’s decision-support system has used for years - applied on top of the language model’s output. (This scenario is illustrative of the failure mode, not a citation of a specific published case.)
This is the distinction worth holding onto: RAG makes the documentation trustworthy. Neurosymbolic AI makes the decision safe. They solve two different failure modes, and skipping the second because the first is in place is the most common - and most dangerous - shortcut we see organizations take.
Getting this right isn’t a single model choice. Drawing on our work in this space, we think about it as four components, each doing a distinct job, working together - not an official industry standard, but the practical framework we’ve found useful:
Remove any one of the four and the system has a gap: skip the LLM layer and you lose usability; skip RAG and you’re back to hallucination; skip the neurosymbolic layer and grounded facts can still combine into an unsafe recommendation; skip the human-in-the-loop checkpoint on the agentic layer and a safe recommendation can still be acted on without oversight. Safety in a clinical AI system isn’t a property of any one model - it’s a property of the architecture that connects all four.
The architecture decision can’t be separated from the governance structure around it. In the NHS context specifically, that means a Clinical Safety Officer and Caldicott Guardian involved from the design stage, compliance mapped to DCB0129/DCB0160 rather than retrofitted after a pilot, an audit trail that lets a clinician trace any AI-generated statement back to its source in seconds, and a mandatory human-in-the-loop checkpoint before anything AI-generated enters the record.
A neurosymbolic safety layer is only as good as the rules encoded into it - and encoding those rules is a stakeholder exercise involving clinical, legal, and information governance leads, not a technical afterthought.
Before selecting a vendor or a model, map your own use cases against these four pillars: which tasks only need fluent language, which need factual grounding, which need a hard safety check no LLM should be trusted to make alone, and which need a supervised action. That mapping - done before procurement, not after - is what separates a safe AI deployment from an expensive one. It is also the point where AI adoption in healthcare stops being a technology project.
Reconfig is an AI transformation platform that helps enterprise leaders move AI beyond experimentation into scaled, ROI-driven impact.
Powered by an Organizational Digital Twin, Reconfig enables you to define your Target Operating Model based on your strategy, discover hidden operational frictions, and develop a prioritized transformation plan with a quantified business case - simulating financial and structural impact before a single change is made.
If you are currently evaluating what your organization’s AI operating model should look like - or how to capture clear ROI on your AI investments - we’d welcome the conversation.
Sources: World Health Organization (2021). Ethics and Governance of Artificial Intelligence for Health. NHS England / NHS Digital (2024/25). Clinical Risk Management Standards for Health IT: DCB0129, its application in the manufacture of health IT systems, and DCB0160, its application in the deployment and use of health IT systems.
A hallucination is content the model generates that has no basis in the source material: a physical exam finding that was never performed, a medication that was never discussed, a lab value that was never recorded. In a clinical setting the consequence is heavier than in other sectors, because a hallucinated line in a consultation note can enter the patient record and change a treatment path.
RAG reduces them substantially. It constrains the model to generate from a defined trusted source such as the consultation transcript, the patient record or an approved guideline set, which closes the invented-fact gap. RAG guarantees that an output is grounded in a retrieved document. Clinical safety is a separate property, because RAG holds no representation of a drug interaction, a dosage ceiling or a guideline threshold.
Neurosymbolic AI combines a neural network, which handles language and pattern recognition, with a symbolic reasoning layer made of explicit deterministic rules of the form if X then Y. The neural output has to pass through that rule layer before it is accepted. An LLM produces the statistically likely answer; the symbolic layer checks that answer against fixed logic such as a clinical guideline, a regulation or a safety constraint, and blocks anything that violates the rule.
One, an LLM with prompting to turn clinical dialogue into a fluent draft. Two, RAG to ground that draft in the verified transcript and patient record. Three, a neurosymbolic layer that applies deterministic rule checks for drug interactions, guideline compliance and dosage limits to the grounded output. Four, an agentic layer with a human-in-the-loop checkpoint for the administrative actions that follow, such as referrals, billing updates and scheduled follow-ups. Removing any one of the four leaves a specific gap open.
A Clinical Safety Officer and a Caldicott Guardian involved from the design stage, while the architecture can still change. Compliance mapped to DCB0129 and DCB0160 from the start, so the pilot runs inside the standard. An audit trail that lets a clinician trace any AI-generated statement back to its source in seconds. And a mandatory human checkpoint before AI-generated content enters the patient record. The rules encoded in a neurosymbolic safety layer are themselves a governance artefact, agreed with clinical, legal and information governance leads.