synlayer
cense
Evidence / Evaluation and review

Evaluation and review

Each disease pack is reviewed by specialists before release, tested against a held-out case set on every change, and monitored once it is live.

The lifecycle

The review lifecycle

  1. 01 Draft Medical editors write the pack, grading and citing every statement.
  2. 02 Review Two specialists from different countries review independently.
  3. 03 Adjudicate A third specialist settles disagreements.
  4. 04 Evaluate The held-out case set runs against the new version.
  5. 05 Blind grade Two specialists grade outputs without knowing the version.
  6. 06 Release Ships if no gated metric has dropped.
  7. 07 Monitor Live reports and weekly re-grading.
Any change to a pack, a prompt or the model sends the disease back to stage 4. A guideline update in any covered region reopens stage 1 within 30 days.
The evaluation set

100 cases per disease, per language

Cases are held out and never used to draft packs or tune prompts. Each set contains four kinds of case, since rare diseases are missed for different reasons.

100
Typical The textbook picture Should rank first.
Misdiagnosed Arrives with the wrong label Tests whether Cense looks past an earlier diagnosis in the history.
Look-alike A different disease that mimics it Measures false positives. The rare disease should not rank first here.
Incomplete Key results missing Cense should name the missing result and ask for it.
Release gates

Metrics and gates

Every change re-runs the full set and is compared with the version live today. If any gated metric drops, the change does not ship.

Metric What it measures 2028 target
Top-3 hit rate Share of cases where the correct disease is among the first three candidates. ≥ 90%
Verifiable citations Share of displayed statements that open to a source paragraph saying the same thing. 100%
Look-alike false positives Share of look-alike cases where the mimicked rare disease is ranked first. ≤ 3%
Red-flag recall Share of cases with a specialist-defined red-flag combination where the flag is shown. Gated; no drop allowed
Honest scope Share of out-of-scope cases where Cense says the case is not covered. Gated; no drop allowed
Clarifying questions Share of incomplete cases where Cense asks for the missing result instead of guessing. Gated; no drop allowed

These are targets. Published figures will carry the set name, size and date, for example "eval: <set> · n=<N> · <YYYY-MM>".

Blind specialist review

Blind grading before launch

Before launch, two specialists from different countries grade each disease's outputs against the same rubric, without knowing which version produced them. Disagreements go to a third.

1RankingIs the right condition where a specialist would put it?
2Evidence accuracyDoes every sentence say what its source says, at the right grade?
3Next stepsAre the missing evidence and suggested tests the ones that matter most?
4SafetyAre red flags surfaced, and is anything said that could cause harm?
Pack review

Roles in pack review

Role Responsibility
Medical editorsDraft the pack from guidelines and literature; grade every statement; link every statement to its source paragraph.
Two specialist reviewersReview independently, from different countries. Each declares conflicts of interest before review.
AdjudicatorA third specialist settles any disagreement. The decision and reasoning are recorded.
SponsorsMay fund pack maintenance. Have no role in drafting or review, never see content before release, and are disclosed in the app.

Reviewer names and the pack version are shown in the product next to every disease.

Evidence grades

Each statement carries one of four grades

Guideline-recommendedStated in a current national or international guideline.
Moderate evidenceSupported by consistent cohort or diagnostic-accuracy studies.
Limited evidenceSmall studies or case series; shown with that caveat.
Expert opinionReviewer consensus where published evidence is thin.
Live monitoring

Monitoring after release

  1. 48 hA "this is wrong" report reaches expert review
  2. WeeklyA sample of live output is re-graded by reviewers
  3. > 3%Error rate in the sample triggers an alert
  4. 1 stepRoll back to the last good pack, prompt or model version

Confirmed errors are added to the evaluation set, so the same mistake is tested on every future release.

See what each disease pack covers

Disease packs →