For your AI governance committee
Model card and evaluation method
Health-system AI governance policies generally require a vendor to document the algorithm and its training model approach before the platform can be approved, and to show how its evaluation metrics are calculated and compared to a baseline. This page is that document, published rather than sent, so your review can start before anyone signs anything.
The system
- Answer model
- Qwen2.5-72B-Instruct-AWQ
- Model adaptation
- None. The model is used exactly as published — it is not fine-tuned, and your documents are never used to train or improve any AI model.
- Context window
- 32K tokens
- Embedding model
- BAAI/bge-m3, 1024 dimensions
- Reranker
- A local TEI reranker
- Retrieval
- Hybrid — vector search, keyword search, and a per-chunk question bank, followed by reranking.
- Inference hardware
- An NVIDIA RTX PRO 6000, on premises
- Third-party model APIs
- None. No third-party model API is called to produce an answer.
- Sampling
- Temperature 0 with a fixed seed. Answers are deterministic: the same question gives the same answer.
- Answer scope
- Your organization's own documents. When they do not cover the question, the assistant refuses rather than answering.
- Identifiers
- It will not name a button, field, code or number that your documents do not name.
The training model approach
This is the shortest section on the page, because there is no training. Model weights are never updated — not on your documents, not on anyone else's. The system is adapted to your organization entirely by retrieval, which has a property that fine-tuning does not: every change to what the assistant knows is a change to a document you control, it takes effect immediately, and it is undone by changing that document back. There is no artefact carrying your content around after you remove it.
Where it runs
All three models — the language model that writes the answer, the embedding model that indexes your documents, and the reranker that orders the evidence — run on hardware we operate, alongside the documents themselves. No third-party model API is called to produce an answer. That is a stronger statement than a policy commitment: there is nothing configured that could send your content to one by mistake.
How accuracy is measured
If your AI governance policy asks for evaluation metrics calculations and a comparison to baseline, this is the artefact that answers it. Accuracy here is not a vendor adjective — it is a repeatable measurement, and this is the method.
A fixed, versioned question set
The same questions every time, kept under version control. Adding or rewording a question creates a new version of the set, so two runs are only ever compared when they were asked the same thing.
Ground truth from your own documents
The correct answer for each question is taken from the customer's own ingested corpus, not from a public benchmark. A public benchmark measures a model; this measures the system as deployed, against the documents it actually has.
Re-run after every retrieval change
Chunking, embedding, ranking, reranking, the prompt — anything that can move an answer triggers a full re-run before it ships. A retrieval change that was not measured did not happen.
Scored by category, not as one number
Five categories, scored separately, because they fail for different reasons and a single average hides all of them.
Compared against the previous run
Every run is scored against the run before it on the same question set. That is what makes a change that lifts one category while breaking another visible, instead of cancelling out in an average.
Stability measured alongside accuracy
Questions are asked more than once and the answers compared to each other. A system that gets a question right one time in three has not got it right.
The five categories
- Factual
- One fact, stated plainly in one document.
- Procedural
- The steps of a documented process, in the documented order.
- Paraphrase
- The question uses words the document does not — the way staff actually ask.
- Multi-step
- The answer cannot be found in one place; it requires combining more than one document.
- False-premise
- The question assumes something the documents do not support. The correct response is to say so, not to answer.
Why there is no score on this page
A pass rate published without the question set it came from, the corpus version it was measured against and the date it was taken is not a fact, it is a decoration — and the corpus that decides whether this works for you is yours, not ours. So the number is not here. During an evaluation we share the current results under NDA, together with the question set they were measured on, so your team can read the failures as well as the passes and judge the method rather than the headline.
What this document is not
It is not a certification, an audit result or a benchmark placing, and nothing on this page should be read as one. It is a description of what the system is built from and how we measure it, published so that a reviewer can check our work rather than take our word for it. The answers your InfoSec team will ask for — subprocessors, retention, isolation, and the things we do not hold — are on the security questionnaire.