Metadata-Version: 2.4
Name: n1r-rosetta
Version: 0.6.8
Summary: Medical concept resolution — resolves any lab test to its true clinical identity using structured decomposition.
Project-URL: Homepage, https://github.com/n1healthcare/n1r-rosetta
Project-URL: Repository, https://github.com/n1healthcare/n1r-rosetta
Project-URL: Issues, https://github.com/n1healthcare/n1r-rosetta/issues
Author-email: Arun <arun@n1.care>
License: Proprietary
Keywords: LOINC,biomarkers,concept-resolution,healthcare,interoperability,laboratory,medical
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Healthcare Industry
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Medical Science Apps.
Requires-Python: >=3.12
Requires-Dist: openai>=1.40.0
Provides-Extra: dev
Requires-Dist: mypy>=1.20.0; extra == 'dev'
Requires-Dist: pytest>=9.0.3; extra == 'dev'
Requires-Dist: redis>=5.0.0; extra == 'dev'
Requires-Dist: ruff>=0.15.11; extra == 'dev'
Provides-Extra: redis
Requires-Dist: redis>=5.0.0; extra == 'redis'
Description-Content-Type: text/markdown

# N1R Rosetta

Medical concept resolution — resolves any lab test to its true clinical identity.

Rosetta takes a lab test name from any lab, any country, any language, and decomposes it into LOINC-style clinical axes. Two tests that share all axes are the same clinical concept, regardless of what name any lab gave them.

## Design principles

These are non-negotiable. Every change to the codebase is reviewed against them.

1. **The LLM does medicine; code does mechanics.** Every medical-knowledge decision lives in `prompt.py` — component conventions, property-axis assignments, system-axis (specimen) choices, qualifier rules, taxonomy labels, plausibility ranges, diurnal flags, standard-unit choices. Code paths NEVER hand-curate medical equivalences. Lookup tables exist only for mechanical invariants (molar masses, LOINC system → display string, Unicode/glyph folds, SI-derived word folds like `seconds → s`).

2. **Structural validation, never content blocklists.** Any LLM output that becomes persistent data must be gated by JSON-schema or known-shape-dict validation, never by content inspection. Blocklists catch yesterday's failure modes; structural validation rejects anything that doesn't match the declared shape, regardless of contents.

3. **Cache prefix versioning is load-bearing.** Three independent cache namespaces (primary Identity, component canonicalizer, group_key index) carry version segments (`v1`, `v2`, …). Bump on any prompt/normalizer/Identity-shape change that could flip the resolution of a previously-cached input. The bump IS the flush — pre-bump entries are orphaned by namespace.

4. **Group_key stability across reruns is sacred.** The same clinical concept must produce the same `group_key` across runs so a patient's data from 2024 and 2026 lands on the same canonical_id. Property-class coarsening (MCnc/SCnc → `cnc`) and write-once NX cache semantics on the canonicalizer + gkey indexes enforce this.

5. **Identity is maximally distinguished; display grouping is downstream.** Cortisol Morning and Cortisol Evening are different identities (different reference ranges, different clinical meaning) — even though they share the analyte. Don't push display-grouping concerns into the resolver; that's the frontend's job.

6. **Canonical specimen ≠ row specimen.** The biomarker row's `sample_source` is the lab-provided string, preserved unchanged. The canonical aggregating those rows uses the medically-correct specimen from `Identity.system` — because lab labels are unreliable as a specimen signal (many labs say "Blood" for tests run on serum). Rosetta's `system` axis is the medical truth; the canonical follows it, not the row label.

## Installation

```bash
pip install n1r-rosetta
```

Requires Python 3.12+ and an OpenAI-compatible API endpoint.

## Quick Start

```python
from rosetta import Resolver

resolver = Resolver(
    model="gemini-3.1-flash-lite-preview",
    redis_url="redis://localhost:6379",  # optional; falls back to in-memory
)

identity = resolver.resolve("MCH", unit="pg", specimen="Blood")
print(identity.canonical_name)        # "Mean Corpuscular Hemoglobin (MCH)"
print(identity.group_key)             # "hemoglobin|entmass|bld|||"
print(identity.medical_specialties)   # ("Hematologic (Blood Health)",)
print(identity.body_systems)          # ("Hematology / Coagulation",)
```

Works in any language:

```python
resolver.resolve("平均红细胞血红蛋白量", unit="pg", specimen="血液")   # Chinese MCH
resolver.resolve("TCMH", unit="pg", specimen="Sang")                # French MCH
resolver.resolve("평균적혈구혈색소량", unit="pg", specimen="혈액")       # Korean MCH
# All resolve to the same identity: hemoglobin|entmass|bld
```

## What Rosetta Solves

| Problem | How Rosetta Handles It |
|---------|----------------------|
| "LDL cholesterol" and "VLDL cholesterol" look similar but are different tests | Decomposes into component: LDL vs VLDL |
| "MCH" and "Mean Corpuscular Hemoglobin" look different but are the same test | Expands abbreviations, normalises to same component |
| "Haemoglobin" and "Haematocrit" are related but completely different measurements | Different components, different properties (MCnc vs VFr) |
| "CRP" and "hs-CRP" use different assays with different clinical utility | Same component, differentiated by method |
| Same test reported as "Serum" by one lab and "Blood" by another | Assigns LOINC SYSTEM based on test type, not lab's label |
| Test names in French, German, Chinese, Japanese, Korean, Spanish | Understands medical terminology in any language |
| "Calcium" from Blood vs Hair vs Urine are different measurements | Resolved by (test_name, unit, specimen) — separate identities |
| "LDL-3" must stay separate from "LDL-5" (different subfractions) | Qualifier field preserves identity-defining information |

## The Identity Model

A lab test's clinical identity is defined by what it measures, how, where, and when. Rosetta captures this in seven fields following the axis model established by LOINC (Logical Observation Identifiers Names and Codes), plus two taxonomy fields that categorise the test for downstream UIs.

The seven **identity fields** — `component`, `property`, `system`, `method`, `time_aspect`, `qualifier`, `canonical_name` — define the clinical identity. Two tests that share all seven are the same clinical concept. The `group_key` is derived from the first six.

The two **taxonomy fields** — `medical_specialties`, `body_systems` — are editorial overlays for grouping and filtering in the UI. They do NOT participate in `group_key`; changing a taxonomy label must never split or merge a clinical group.

### Component

The core substance or entity being measured. This is the primary axis of identity — two tests measuring different components are always different tests.

Rosetta uses LOINC's dot-notation to express component relationships:

| Raw Test Name | Component | What It Means |
|---|---|---|
| LDL cholesterol | `cholesterol.in ldl` | Cholesterol carried in LDL particles |
| VLDL-C | `cholesterol.in vldl` | Cholesterol carried in VLDL particles |
| HDL Cholesterol | `cholesterol.in hdl` | Cholesterol carried in HDL particles |
| Total Cholesterol | `cholesterol` | All cholesterol fractions combined |
| Non-HDL Cholesterol | `cholesterol.in non hdl` | Total minus HDL |

The dot-notation makes the taxonomy explicit: all are cholesterol, but in different lipoprotein fractions. They share a parent concept but are clinically distinct tests with different reference ranges and treatment targets.

For cell counts, Rosetta distinguishes absolute counts from differential fractions using LOINC's component conventions:

| Raw Test Name | Component | What It Is |
|---|---|---|
| Neutrophil Count | `neutrophils` | Absolute count of neutrophils per volume |
| Neutrophils % | `neutrophils/100 leukocytes` | Fraction of white cells that are neutrophils |

These are different measurements. A patient can have a normal neutrophil percentage but a low absolute count (if total WBC is low). They have different reference ranges and different clinical meaning.

For antibody tests, the component includes both the target organism and the immunoglobulin class:

| Raw Test Name | Component |
|---|---|
| Bartonella henselae IgG | `bartonella henselae ab.igg` |
| Bartonella quintana IgM | `bartonella quintana ab.igm` |
| Borrelia burgd. IgG - Band p41 | `borrelia burgdorferi ab.igg` (with qualifier `p41`) |

For free/bound fractions:

| Raw Test Name | Component |
|---|---|
| PSA | `prostate specific ag` |
| Free PSA | `prostate specific ag.free` |
| Free PSA/Total PSA Ratio | `prostate specific ag.free/prostate specific ag.total` |
| Free T4 | `thyroxine.free` |
| Cortisone (free fraction) | `cortisone.free` |

### Property

What physical or chemical property is being measured about the component. Rosetta uses standard LOINC property codes in the Identity, but groups convertible properties together in the `group_key` (see [Property Classes](#property-classes) below).

| Code | Full Name | Typical Units | Clinical Meaning | Examples |
|------|-----------|--------------|------------------|----------|
| MCnc | Mass Concentration | g/dL, mg/L, ng/mL, µg/L | Mass of substance per volume of specimen | Hemoglobin (g/dL), Ferritin (µg/L), CRP (mg/L) |
| SCnc | Substance Concentration | mmol/L, µmol/L, nmol/L | Moles of substance per volume | Glucose (mmol/L), Creatinine (µmol/L), Calcium (mmol/L) |
| NCnc | Number Concentration | /µL, ×10^9/L, cells/µL | Count of entities per volume | WBC (×10^9/L), Platelets (×10^3/µL), RBC (×10^12/L) |
| CCnc | Catalytic Concentration | U/L, IU/L | Enzyme activity per volume | ALT (U/L), AST (U/L), ALP (IU/L), GGT (U/L) |
| ACnc | Arbitrary Concentration | IU/mL, mIU/L | Immunoassay units (antibodies, hormones) | TSH (mIU/L), Anti-TPO (IU/mL), Insulin (µIU/mL) |
| EntMass | Entitic Mass | pg | Mass per individual entity (cell) | MCH — average hemoglobin mass per red blood cell |
| EntMCnc | Entitic Mass Concentration | g/dL (per-cell) | Concentration within an individual entity | MCHC — hemoglobin concentration within packed red cells |
| VFr | Volume Fraction | % | Volume of one component relative to total | Hematocrit — % of blood volume occupied by red cells |
| NFr | Number Fraction | % | Count of one type relative to total count | Neutrophils % — fraction of WBC that are neutrophils |
| Vel | Velocity | mm/h | Rate of movement | ESR — rate at which red cells settle |
| Ratio | Ratio | dimensionless | Ratio of two measurements | Albumin/Globulin ratio, Free PSA/Total PSA |
| PrThr | Presence/Threshold | (qualitative) | Present or absent, positive or negative | Urine protein dipstick, Hepatitis B surface antigen |
| Pres | Pressure | mmHg | Force per area | Systolic/Diastolic blood pressure |
| NRat | Number Rate | /h, beats/min | Events per unit time | Heart rate, Apnea-Hypopnea Index |
| Time | Time | seconds | Duration | Prothrombin Time, APTT |

The EntMass / EntMCnc distinction is critical. MCH (EntMass, pg) and MCHC (EntMCnc, g/dL) both relate to hemoglobin in red blood cells, but MCH is the average mass of hemoglobin per cell, while MCHC is the average concentration of hemoglobin within the cell volume. They have different reference ranges (MCH: 27-33 pg; MCHC: 32-36 g/dL) and respond differently in disease states.

### Property Classes

LOINC distinguishes MCnc (mass/volume) from SCnc (moles/volume) because they are different observations with different LOINC codes. But for clinical identity — "is this the same test?" — they are the same measurement in different unit systems. LDL cholesterol in mg/dL and LDL cholesterol in mmol/L are the same test. A patient switching labs shouldn't get two separate trend lines.

Rosetta preserves the LOINC property code in `identity.property` but uses a coarser **property class** in `group_key`:

| Property Class | LOINC Codes | What It Means |
|---|---|---|
| `cnc` | MCnc, SCnc, ACnc, CCnc | Concentration — amount per volume, any unit system |
| `fr` | VFr, NFr, Ratio, MRto, SRto | Fraction, proportion, or ratio — dimensionless |
| `ncnc` | NCnc | Number concentration — entities per volume |
| `entmass` | EntMass | Mass per entity |
| `entmcnc` | EntMCnc | Concentration within entity |
| All others | (unchanged) | Vel, PrThr, Pres, NRat, Time, Len, Temp |

This means:
- LDL in mg/dL (`MCnc`) and LDL in mmol/L (`SCnc`) → same `group_key` (both `cnc`)
- AFP in ng/mL (`MCnc`) and AFP in IU/mL (`ACnc`) → same `group_key` (both `cnc`)
- HbA1c in % (`NFr`) and HbA1c in mmol/mol (`SFr`) → same `group_key` (both `fr`)
- But Hemoglobin (`MCnc` → `cnc`) and MCH (`EntMass`) → different `group_key`

This is Rosetta's extension to LOINC: grouping by clinical identity rather than observation identity. The LOINC property codes are preserved for downstream use — including eventual LOINC code lookup, where the full code (MCnc vs SCnc) is needed.

### System

The specimen or body system the sample comes from. This determines what the measurement represents physiologically.

| Code | Meaning | Tests Measured Here |
|------|---------|---------------------|
| Bld | Whole blood | CBC (Hemoglobin, Hematocrit, WBC, Platelets, differentials), HbA1c, ESR, coagulation (PT, INR, APTT, Fibrinogen), blood gases |
| Ser/Plas | Serum or Plasma | Chemistry (Glucose, Creatinine, Electrolytes, Liver enzymes, Bilirubin), Lipids (LDL, HDL, Triglycerides), Hormones (TSH, T4, Testosterone, Cortisol), Tumour markers (PSA, AFP, CEA), CRP, Iron studies, Vitamins, Drug levels, Immunoglobulins |
| Urine | Urine | Urinalysis (Protein, Glucose, Blood, pH), Urine chemistry (Creatinine, Calcium, Cortisol), Organic acids, Drug screening |
| Stool | Stool/Feces | Calprotectin, Occult blood, Microbiome analysis, Parasitology |
| Hair | Hair | Mineral analysis (Calcium, Zinc, Copper, Lead, Mercury, Arsenic) |
| Saliva | Saliva/Buccal | Cortisol diurnal profile, Oral microbiome |
| CSF | Cerebrospinal Fluid | Glucose, Protein, Cell count, Microbiology |

The distinction between Bld and Ser/Plas matters clinically. Hemoglobin is measured in whole blood (Bld) because it's inside the red blood cells. Cholesterol is measured in serum or plasma (Ser/Plas) because it circulates dissolved in the liquid fraction. When a lab labels a specimen "Blood", Rosetta determines the correct LOINC system from the test type: a CBC hemoglobin from "Blood" maps to Bld; an LDL cholesterol from "Blood" maps to Ser/Plas.

The same component in different systems is a different clinical measurement:

| Test | System | What It Measures | Reference Range |
|------|--------|-----------------|-----------------|
| Calcium | Ser/Plas | Ionised + protein-bound calcium in blood | 2.2–2.6 mmol/L |
| Calcium | Urine | Renal calcium excretion | <7.5 mmol/24h |
| Calcium | Hair | Chronic mineral deposition | 200–750 µg/g |

### Method

The assay methodology, captured only when it changes clinical interpretation. Most tests have a single standard method and this field is empty.

| Test | Method | Why It Matters |
|------|--------|---------------|
| hs-CRP | `high sensitivity` | Detects CRP levels below 1 mg/L. Standard CRP only detects above 3–5 mg/L. Used for cardiovascular risk stratification where standard CRP shows "normal". |
| HbA1c (NGSP) | `ngsp` | Reports as % (e.g., 6.5%). The IFCC method reports in mmol/mol (e.g., 48). Different numeric scales, same clinical concept with a known conversion. |
| Ejection Fraction (Biplane) | `biplane` | Simpson's biplane method measures EF from two echocardiographic views. Teichholz uses a single-dimension formula. They give systematically different values. |

### Time Aspect

The timing of specimen collection, when it changes clinical meaning. Most tests don't specify timing and this field is empty.

| Test | Time Aspect | Why It Matters |
|------|------------|---------------|
| Fasting Glucose | `fasting` | Must be measured after 8+ hour fast. Non-fasting glucose has different reference ranges. |
| Free Cortisone (1st Morning) | `1st morning` | First void after waking captures overnight cortisone production. 2nd morning is the next void 2-4 hours later. Together they show the cortisol awakening response. |
| Cortisol (Evening) | `evening` | Evening cortisol should be at its diurnal nadir. An elevated evening cortisol suggests Cushing's. Morning cortisol at the same level is normal. |

### Qualifier

Any remaining identity-defining information that doesn't fit the standard axes. The rule: **if removing it would make two clinically different tests indistinguishable, preserve it.**

| Category | Example Tests | Qualifier Values | Why They Must Be Separate |
|----------|--------------|-----------------|--------------------------|
| Subfractions | LDL-1 through LDL-7 | `1`, `2`, ... `7` | Different particle sizes with different atherogenic risk. LDL-1 (large buoyant) is less dangerous than LDL-7 (small dense). |
| Antigen bands | Borrelia IgG Band p41, Band p58, VlsE-Bb | `p41`, `p58`, `vlse bb` | Western blot interpretation requires knowing which specific bands are positive. CDC Lyme criteria: 5/10 specific IgG bands. |
| Laterality | Spherical Equivalent (Left Eye), (Right Eye) | `left`, `right` | Each eye is an independent organ with its own prescription and pathology. |
| Threshold values | Snores > 70 dB, Snores > 80 dB | `>70 db`, `>80 db` | Different severity thresholds for sleep study metrics. |
| Timed collections | Cortisone U0 (Mid-Sleep), U3 (Dinner) | `u0`, `u3` | Serial urine collections at specific times. Each fraction has different expected values reflecting the diurnal cycle. |
| Platform-specific | SYMPHONY Age, TruMe Age, DunedinPACE | `symphony`, `trume`, `dunedinpace` | Different biological age algorithms that produce different values from different methodologies. |

### Medical Specialties

A list of 1-3 specialty labels describing the clinical domain(s) a clinician would order or interpret this test under. Populated by the same LLM call that fills the LOINC axes — no separate enrichment pass. Plural because real tests often span domains.

| Raw Test Name | Medical Specialties | Reasoning |
|---|---|---|
| LDL Cholesterol | `("Lipid",)` | Owned by lipidology / preventive cardiology risk stratification. |
| Creatinine (serum) | `("Kidney Function",)` | Renal filtration marker. |
| HbA1c | `("Metabolic", "Hormonal")` | The glucose-control marker *and* the endocrinology-owned diabetes test. |
| High-sensitivity CRP | `("Inflammatory", "Cardiovascular")` | Inflammation marker ordered primarily for cardiovascular risk. |
| Neutrophil Count | `("Hematologic (Blood Health)", "Immune/Autoimmune")` | Reported on the CBC but clinically interpreted as part of the immune response. |
| NT-proBNP | `("Cardiovascular",)` | Cardiac-specific. |
| Cortisol | `("Hormonal",)` | Endocrine, adrenal axis. |
| Spherical Equivalent (Left Eye) | `("Other/Not categorisable",)` | No medical specialty applies. |

**Preferred vocabulary, not a closed set.** Rosetta's `constants.MEDICAL_SPECIALTIES` holds 13 preferred labels that match the existing N1 UI groupings. The prompt instructs the LLM to prefer them, but to *propose a new accurate label* when none fit — "Oncology" for tumor markers, "Infectious Disease" for pathogen identification, "Coagulation" for PT/INR/APTT, "Reproductive" for fertility panels, "Genetics" for germline variants. "Other/Not categorisable" should be the exception, not the fallback. The resolver does not validate the label against any whitelist; whatever the LLM emits flows through to consumers. This is intentional — closed vocabularies turn under-coverage into silent "Other" dumps rather than surfacing real domains the product may want to add.

### Body Systems

A list of 1-3 body-system labels describing the physiological systems the test informs. Same LLM call as the specialties.

| Raw Test Name | Body Systems |
|---|---|
| LDL Cholesterol | `("Cardiac / Cardiovascular", "Endocrine / Metabolic")` |
| Hemoglobin | `("Hematology / Coagulation",)` |
| HbA1c | `("Endocrine / Metabolic",)` |
| Creatinine (serum) | `("Renal / Kidney",)` |
| Total Bilirubin | `("Gastrointestinal / Hepatic",)` |

**Preferred vocabulary, same open-set rule.** `constants.BODY_SYSTEMS` holds 9 preferred labels matching the existing N1 DB convention, including slash-compound labels like `"Endocrine / Metabolic"` and `"Hematology / Coagulation"` that are treated as single vocabulary items (splitting them would break DB round-trips). When none fit, the prompt instructs the LLM to propose a new label in the same slash-joined style — e.g., `"Pulmonary / Respiratory"`, `"Neurologic / Central Nervous System"`, `"Reproductive / Gonadal"`.

### Identity vs Display Grouping

Rosetta resolves each test to its **true clinical identity**. This is deliberately more granular than what you might display on a single chart, because the identity layer's job is to ensure no two clinically different values ever end up in the same bucket.

Consider a salivary cortisol diurnal profile — a test that measures cortisol at four timepoints across the day to capture the diurnal curve:

```
Cortisol (Morning)   → cortisol|mcnc|saliva||morning|
Cortisol (Midday)    → cortisol|mcnc|saliva||midday|
Cortisol (Afternoon) → cortisol|mcnc|saliva||afternoon|
Cortisol (Evening)   → cortisol|mcnc|saliva||evening|
```

These are four separate identities. This is correct — a morning cortisol of 15 nmol/L is normal, while an evening cortisol of 15 nmol/L is pathological (suggests Cushing's). They have different reference ranges and different clinical interpretation. If they were in one identity, you'd have a value of 15 with no way to know if it's normal or abnormal.

But for **charting**, you want all four on one graph to visualise the diurnal curve. The display layer achieves this by grouping identities that share `(component, property, system)` — all four have `cortisol|mcnc|saliva` in common. The time_aspect tells the charting layer where each point goes on the X-axis.

The same pattern applies to:

**Timed urine collections** (24-hour cortisone with U0-U4 fractions):
```
Cortisone U0 (Mid-Sleep)  → cortisone|mcnc|urine|||u0
Cortisone U1 (Waking)     → cortisone|mcnc|urine|||u1
Cortisone U3 (Dinner)     → cortisone|mcnc|urine|||u3
Total Cortisone            → cortisone|mcnc|urine|||       (no qualifier — it's the sum)
```
Five separate identities. The display layer groups by `cortisone|mcnc|urine` and plots each point labelled by its qualifier. Total Cortisone is plotted separately as a summary value.

**LDL subfractions** (advanced lipoprotein testing):
```
LDL-1  → cholesterol.in ldl|scnc|ser/plas|||1
LDL-2  → cholesterol.in ldl|scnc|ser/plas|||2
...
LDL-7  → cholesterol.in ldl|scnc|ser/plas|||7
```
Seven separate identities. The display layer groups by `cholesterol.in ldl|scnc|ser/plas` and shows the subfraction distribution — the pattern from LDL-1 (large buoyant, less atherogenic) through LDL-7 (small dense, most atherogenic) is the clinical insight.

**Borrelia Western blot bands**:
```
Borrelia IgG Band p41   → borrelia burgdorferi ab.igg|acnc|ser/plas|||p41
Borrelia IgG Band p58   → borrelia burgdorferi ab.igg|acnc|ser/plas|||p58
Borrelia IgG Band VlsE  → borrelia burgdorferi ab.igg|acnc|ser/plas|||vlse bb
```
Each band is a separate identity. The display layer groups by `borrelia burgdorferi ab.igg|acnc|ser/plas` and shows which bands are positive — the pattern of positive bands determines the Lyme disease diagnosis per CDC criteria.

The rule: **Rosetta resolves identity. The display layer groups related identities for visualisation.** Identity grouping uses the full `group_key` (all seven fields). Display grouping uses a subset — typically `(component, property, system)` — to bring related measurements together while preserving each data point's individual identity.

## Architecture

### Resolution Flow

```
Input: (test_name="TCMH", unit="pg", specimen="Sang")
  │
  ├─ Cache lookup: "TCMH||pg||Sang" → miss
  │   (cache key inputs run through normalize_unit + normalize_specimen)
  │
  ├─ LLM call → JSON (constrained by PRIMARY_SCHEMA):
  │   component: "Hemoglobin", property: "EntMass", system: "Bld",
  │   method: null, time_aspect: null, qualifier: null,
  │   plausible_range: [25, 35], diurnal_sensitive: false,
  │   standard_unit: "pg",
  │   canonical_name: "Mean Corpuscular Hemoglobin (MCH)",
  │   medical_specialties: ["Hematologic (Blood Health)"],
  │   body_systems: ["Hematology / Coagulation"]
  │
  ├─ Normalize strings, run validator stack (dimensional analysis)
  │
  ├─ Stabilize: canonicalize component, inherit cross-run display fields
  │
  ├─ Cache store (only when validators agree)
  │
  └─ Return Identity(group_key="hemoglobin|entmass|bld|||")
```

### Batch Flow

```
8,408 biomarker rows
  │
  ├─ Build cache key per row: "test_name||unit||specimen"
  ├─ Cache hits → immediate Identity
  ├─ Cache misses → deduplicate by key → N unique LLM calls
  ├─ Resolve concurrently (ThreadPoolExecutor, 32 workers)
  ├─ Fan results back to all rows sharing each key
  └─ Return 8,408 Identity objects
```

### The Cache (3-tier)

Rosetta keeps three independent caches, each with its own purpose, semantics, and version segment.

| Prefix | Key | Value | Semantics |
|---|---|---|---|
| `rosetta:cache:vN:` | `(test_name, unit, specimen)` | Identity dict | Primary resolution cache; bumped on prompt/normalizer/Identity-shape changes |
| `rosetta:canon:component:vN:` | raw component string | canonical component | Component-name stabilization; **write-once NX** |
| `rosetta:canon:gkey:vN:` | `group_key` | display-field dict | Cross-run canonical_name + taxonomy stabilization; **write-once NX** |

**Primary cache.** Maps the input tuple to a resolved Identity. Shared across all patients — first patient uploads 30 tests, 30 LLM calls; second patient with 25 of the same tests gets 25 cache hits. Cache-key inputs run through `normalize_unit` + `normalize_specimen` so orthographic variants (`µg/g Cr` / `ug/gcr` / `ug/g creatinine`; `^Patient` / `Patient`) collapse to one entry.

The cache key includes unit and specimen because the same test name means different things in different contexts. "Calcium" with mmol/L from Blood is serum calcium; "Calcium" with µg/g from Hair is a mineral analysis. "Neutrophils" with ×10^9/L is an absolute count; "Neutrophils" with % is a differential fraction. Each gets its own cache entry and identity.

**Component canonicalizer.** Records "the first time this raw component string was canonicalized, the answer was X" with write-once NX semantics. The LLM occasionally drifts in component naming (`vanillylmandelate` vs `vanillylmandelic acid`); the canonicalizer maps later resolutions back to the earliest answer for the same raw string. Cross-run component stability emerges.

**Group_key index.** Records the display fields (`canonical_name`, `medical_specialties`, `body_systems`) for each `group_key` on first sight, write-once NX. Two Identities with the same six core axes can still produce different display names across runs ("Free T4" vs "Free Thyroxine (Free T4)"). The index pins the first answer; later resolutions inherit. Without it, downstream canonical-name aggregation fragments.

**Backend abstraction.** `InMemoryCache` (process-local) or `RedisCache` (shared across pods + survives restarts). `make_cache(redis_url)` tries Redis first, falls back to in-memory if Redis is unreachable — callers don't handle the fallback. Pass `redis_url=` to Resolver, or set `ROSETTA_REDIS_URL`. No file-backed backend exists.

### Cache prefix versioning

Bump the relevant prefix whenever a change could flip the resolution of a previously-cached input. Pre-bump entries are orphaned by namespace — there's no flush operation, the bump IS the flush.

History (primary cache):

| Bump | Reason |
|---|---|
| v1 → v2 (0.5.4) | Prompt principle refactor |
| v2 → v3 (0.5.6) | Identity gained `plausible_range`, `diurnal_sensitive`; method-axis change for NGSP/IFCC |
| v3 → v4 (0.5.7) | Structured output mandatory on all LLM calls — wipes entries with poisoned canonicalizer values |
| v4 → v5 (0.5.8) | LLM no longer infers `time_aspect` from unit conventions |
| v5 → v6 (0.6.0) | Identity gained `standard_unit` |
| v6 → v7 (0.6.1) | Cache-key inputs run through `normalize_specimen` + creatinine-suffix collapse |
| v7 → v8 (0.6.3) | Prompt rule for creatinine-preserving `standard_unit` |

### Structured output

Every LLM call passes `response_format={"type": "json_schema", "json_schema": <SCHEMA>}` (translated to Gemini's `responseJsonSchema` by LiteLLM). The model's output is constrained to the declared shape — no free-form text, no Markdown fences, no reasoning-token leakage, no preamble. `PRIMARY_SCHEMA` and `CANONICALIZER_SCHEMA` are defined in `rosetta/prompt.py`.

This is load-bearing. Pre-0.5.7 the canonicalizer cached whatever the LLM returned and reasoning-preamble truncation under `max_tokens=128 + reasoning_effort=high` wrote permanent entries like `"analyze the input"` / `"bilir"` / `"input: \"m"` — collapsing unrelated analytes into one canonical. Structured-output JSON schema makes this class of failure impossible by construction; the model literally cannot emit free-form text that passes schema validation.

## The Validator Stack

After resolution, Rosetta runs a stack of independent validators against the LLM's output. Each validator emits a `ValidationResult` with status `pass` / `warn` / `fail` / `info` and a structured `annotation` payload. The pattern of validator results IS the confidence signal — `confidence(results)` returns `"low"` if any failed, `"high"` if all passed, otherwise `"medium"`.

```python
@dataclass(frozen=True)
class ValidationResult:
    check: str            # "unit_property" | "plausibility" | "timing"
    status: str           # "pass" | "warn" | "fail" | "info"
    detail: str           # Human-readable verdict
    expected: str         # What the validator expected
    actual: str           # What was actually present
    correction: dict|None # Auto-correctable Identity fields (None when ambiguous)
    annotation: dict|None # Structured machine-parseable payload for downstream rendering
```

### `validate_unit_property` — dimensional analysis

Checks that the resolver's `property` axis matches the unit's dimensionality. Independent of the LLM — pure dimensional analysis driven by a `(num_base, den_base) → expected fine property codes` lookup table.

| Unit signature | Expected fine property | Reasoning |
|---|---|---|
| `g/L` (and prefixes) | `MCnc` or `EntMCnc` | Both are mass concentrations; ambiguity is whole-specimen vs per-entity |
| `mol/L` (and prefixes) | `SCnc` | Substance concentration |
| `U/L` / `IU/L` | `CCnc` or `ACnc` | Catalytic / arbitrary concentration |
| `/L` with `x10^N` | `NCnc` | Number concentration |
| `pg` (bare) | `EntMass` | Entitic mass per cell |
| `%` | `VFr`, `NFr`, `Ratio`, `MRto`, `SRto` | Fraction family |

Three-tier verdict: **pass** if property is in the expected set; **warn** if property is in the right *class* but not the expected fine code; **fail** if property is in a different class.

**Auto-correction safety rule:** when the unit signature has EXACTLY ONE valid property (e.g. `mmol/L → SCnc`), the verdict carries a `correction = {"property": "scnc"}` dict and the resolver applies it silently. Ambiguous signatures (e.g. `g/dL ← {MCnc, EntMCnc}` for plain Hemoglobin vs MCHC) DO NOT carry a correction — they retry the LLM with a hint instead. This prevents silent property mutation when the Identity is genuinely ambiguous from the unit alone.

### `validate_plausibility` — range check + alternate-unit probe

Compares an observed numeric value against `Identity.plausible_range`.

| Branch | Status | Behavior |
|---|---|---|
| In range | `pass` | Annotation: `{status: "in_range", observed_value, plausible_range}` |
| Mild excursion (<2× past nearest bound) | `pass` | Annotation: `{status: "out_of_range", direction, severity: "mild", ratio_out_of_range, observed_value, plausible_range}` |
| Severe excursion (≥2× past nearest bound) | `warn` | Same annotation + alternate-unit probe |

**Mild excursions are deliberately not warnings.** Real patient data sits slightly outside reference ranges all the time — that's the diagnostic signal labs were ordered to capture. The validator's only legitimate alarm job is catching values so extreme they're almost certainly unit mislabels (HbA1c=37 in `%`) or data-entry errors (sodium=13 mmol/L). 2× is the threshold that separates "biological extreme" (calcium of 16, blood pressure of 240, severe hypothyroid TSH of 50) from "numerically impossible".

**The alternate-unit probe.** When out-of-range AND a unit string is supplied, walks small SI-prefix shifts on the unit (1000× series only — `p`/`n`/`u`/`m`/""/`k`) plus the HbA1c `%` ↔ `mmol/mol` affine swap. For each candidate, asks the converter whether scaling brings the value back into `plausible_range`. If EXACTLY ONE candidate fits, emits `suggested_unit` and `suggested_converted_value` in the annotation. Multiple-match → silent (ambiguity is worse than a misleading suggestion). No-match → silent.

Example: HbA1c=37 in `%` column → severe excursion (37/15=2.47×); probe finds `mmol/mol` matches → annotation includes `suggested_unit: "mmol/mol"`, `suggested_converted_value: 5.5455`. Universal — no per-analyte tables.

### `validate_timing` — diurnal-window annotation

For diurnal-sensitive analytes (cortisol, ACTH, growth hormone, melatonin, DHEA-S, testosterone), reads `observed_at` (ISO 8601) and emits a structured timing annotation:
- `window` ∈ `{"morning", "midday", "afternoon", "evening", "night"}` — wallclock-hour bucket
- `time_aspect_source` = `"explicit"` if `Identity.time_aspect` was already populated by the LLM (lab tagged the timing), `"wallclock"` if inferred from the timestamp

Status: **info** when annotation is emitted (timing context is metadata, not a problem); **warn** ONLY when the analyte is diurnal-sensitive AND has neither explicit `time_aspect` NOR `observed_at` — clinicians have no way to interpret the value without timing context.

## Auto-correction loop

`Resolver.resolve_groupable` runs a structured retry loop:

```
1. Call LLM, get Identity.
2. Run validator stack → list[ValidationResult].
3. For each failed result:
     - has correction dict? → apply via dataclasses.replace() and re-validate.
     - no correction dict? → flag for LLM retry with the failure detail as a hint.
4. After max_retries (default 2), return groupable=False if validators still fail.
5. Cache only when groupable=True. Failed resolutions retry the LLM next time
   rather than reading a stale wrong answer.
```

Returns a `ResolveResult`:

```python
@dataclass(frozen=True)
class ResolveResult:
    identity: Identity
    groupable: bool                     # False if validators failed irrecoverably
    attempts: int                       # 0 on cache hit, 1+ on LLM calls
    validators: list[ValidationResult]
    converted_value: float | None       # Observed value converted to standard_unit
    converted_range_min: float | None
    converted_range_max: float | None
    conversion_error: str | None
```

When `groupable=False`, downstream consumers (e.g. the grouper) treat the row as a singleton — assign a synthetic group_key derived from the input so the biomarker has a home but doesn't silently merge with anyone else.

## Observation values: convert at resolve time

When `resolve_groupable` is called with `observed_value` and the resolved `Identity.standard_unit` is set, Rosetta converts the observation from the input unit to the standard unit and returns it on `result.converted_value`. Same for `observed_range_min` / `observed_range_max`. The converter is pure math — no embedding model, no external library — using `Identity.molar_mass` for mass↔molar conversions and `Identity.is_hba1c` for the IFCC/NGSP affine formula.

```python
result = resolver.resolve_groupable(
    "Hemoglobin A1c", unit="%", specimen="Blood",
    observed_value=37.0,                 # Lab tagged as % but value is IFCC mmol/mol
    observed_at="2026-04-25T08:30:00",
)

result.identity.standard_unit            # "mmol/mol"
result.converted_value                   # None — % can't be bridged to mmol/mol affinely
                                         # without knowing direction (37 is way out of % range)

# But the plausibility validator catches it:
plaus = next(v for v in result.validators if v.check == "plausibility")
plaus.status                             # "warn" (severe — 37 vs [3.5, 15] is 2.47× over)
plaus.annotation["suggested_unit"]       # "mmol/mol" — alternate-unit probe found the match
plaus.annotation["suggested_converted_value"]  # 5.5455
```

Downstream consumers consume `validators` and `converted_*` directly; no need to re-implement conversion logic in the grouper.

## Normalizer (`rosetta/normalizer.py`)

`normalize(value)` — lowercase + strip hyphens (preserves dots and slashes) + collapse whitespace. Used on LLM-emitted axis fields.

`normalize_unit(unit)` — cache-key + validator-input unit normalization:
- Unicode char replacements (`µ → u`, `² → ^2`, `³ → ^3`, `× → x`, `− → -`)
- Body-surface-area suffix strip — `/1.73m²`, `/1.73 sq m`, `/1.73 square meter`, `/1.73 m squared` all collapse so eGFR units parse as `mL/min`
- Word-fold table (whole-token match) for SI-derived synonyms:
  - Time: `seconds`/`second`/`secs`/`sec` → `s`; `hours`/`hour`/`hrs`/`hr` → `h`; `minutes`/`minute`/`mins` → `min`; `years`/`year`/`yrs`/`yr` → `y`
  - Volume: `milliliter`/`milliliters`/`mls`/`cc` → `ml`; `liter`/`liters` → `l`
  - Activity: `units`/`unit` → `u`. **`IU` is NOT folded to `U`** — international units ≠ enzyme units
  - Heart-rate: `bpm` → `beats/min`
  - Percent: `percent`/`percentage`/`pct` → `%`. **`percentile` is NOT folded** — different LOINC property (population rank)
  - Slash equivalence: `per` → `/`
- Slash-whitespace collapse: `mg / dl` → `mg/dl`
- Creatinine-suffix collapse: trailing `cr` / `creatinine` / `creatine` (with optional whitespace, requires preceding letter/digit) collapses to no-space `cr`. `ug/g cr` / `ug/g creatinine` / `ug/gcr` → `ug/gcr`. **A lone `cr` token survives** (a unit that is JUST `cr` is not modified).

`normalize_specimen(specimen)` — cache-key specimen normalization:
- Strips leading HL7-style admin prefixes (`^`, `*`)
- Lowercases + whitespace-collapses
- Drops the LOINC `Xxx` placeholder (treats as no specimen)
- **No semantic merging** — `Serum`/`Plasma`/`Whole Blood`/`Blood` stay distinct. The LLM's `system` axis is the medical-truth disambiguator; cache-key normalization stops at orthographic equivalence.

The LLM ALWAYS sees the original (un-normalized) input strings. Normalization is for cache-key construction and validator-input consistency only.

### Project Structure

```
n1r-rosetta/
├── rosetta/
│   ├── __init__.py      # Public exports: Resolver
│   ├── resolver.py      # Resolver, Identity, ResolveResult — the orchestrator
│   ├── prompt.py        # Decomposition prompt + JSON schemas (PRIMARY_SCHEMA, CANONICALIZER_SCHEMA)
│   ├── normalizer.py    # normalize, normalize_unit, normalize_specimen
│   ├── cache.py         # InMemoryCache, RedisCache, make_cache(); 3-prefix architecture
│   ├── constants.py     # MOLAR_MASS, SYSTEM_TO_SAMPLE_SOURCE, MEDICAL_SPECIALTIES, BODY_SYSTEMS
│   ├── converter.py     # Unit conversion (pure math; delegates string normalization to normalize_unit)
│   ├── validators.py    # validate_unit_property, validate_plausibility, validate_timing
│   └── eval.py          # Evaluation harness (label-based partition metrics)
├── tests/
│   ├── test_resolver.py            # Resolver, Identity, cache key
│   ├── test_resolve_groupable.py   # Auto-correction loop, retry, cache discipline
│   ├── test_validators.py          # Validator stack
│   ├── test_normalizer.py          # Word folds, BSA suffix, creatinine, specimen
│   ├── test_cache.py               # Cache backends + prefix-version pin
│   ├── test_converter.py           # Unit conversion edge cases
│   ├── test_clinical.py            # Real-world clinical correctness (integration)
│   ├── test_correct_merges.py      # Synonym/abbreviation merging (integration)
│   ├── test_critical_separations.py# Hb/HCT, LDL/VLDL, MCH/MCHC, etc. (integration)
│   ├── test_known_bugs.py          # Regression tests for fixed bugs (integration)
│   ├── test_multilingual.py        # Multilingual coverage (integration)
│   ├── test_evaluation.py          # Bulk eval against tests/test_inputs.csv (1572 rows)
│   ├── test_regression.py          # Snapshot regression (300-row pinned outputs)
│   └── test_negative.py            # Negative paths and edge cases
├── pyproject.toml
├── CLAUDE.md            # Developer-facing architectural reference
└── README.md            # User-facing documentation
```

All clinical knowledge lives in `prompt.py`. The rest is plumbing.

## Configuration

| Environment Variable | Description | Default |
|---------------------|-------------|---------|
| `OPENAI_API_KEY` | API key for the LLM provider | Required |
| `OPENAI_BASE_URL` | Base URL for OpenAI-compatible API | `https://api.openai.com/v1` |
| `ROSETTA_MODEL` | Model identifier passed to the OpenAI client | `gemini-3.1-flash-lite-preview` |
| `ROSETTA_REDIS_URL` | Redis URL for the shared resolution cache. Falls back to a process-local in-memory cache if unset or unreachable. Keys are namespaced `rosetta:cache:*`. | Unset (→ in-memory) |

Or pass directly:

```python
resolver = Resolver(
    api_key="sk-...",
    base_url="https://litellm.n1-research.com/v1",
    redis_url="redis://localhost:6379",
)
```

### Per-call billing attribution

When the Resolver sits behind a LiteLLM proxy that tracks spend per caller, pass `billing_metadata` so every LLM call carries the attribution tags. Rosetta forwards the dict via `extra_body={"metadata": ...}`, which is what LiteLLM inspects for cost attribution. Non-LiteLLM OpenAI-compatible gateways ignore the field.

```python
resolver = Resolver(
    model="gemini-3.1-pro-preview",
    api_key=per_user_key,
    base_url="http://litellm/v1",
    billing_metadata={
        "user_id": "11111111-2222-3333-4444-555555555555",
        "record_id": "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee",
        "service_name": "rosetta-grouper",
    },
)
```

Keep the dict small — only what your billing/observability layer needs.

## Accuracy

Tested on 8,408 real patient biomarkers and 148 multilingual test cases across 7 countries:

| Metric | Result |
|--------|--------|
| Resolution rate | 8,408/8,408 (100%) |
| Critical separations (Hb/HCT, LDL/VLDL, MCH/MCHC, etc.) | 20/20 |
| Correct merges (synonyms, abbreviations, bilirubin variants) | 35/35 |
| LOINC system correctness (Bld vs Ser/Plas) | 18/20 |
| Multilingual (FR, DE, ZH, JA, KO, ES) | 148/148 (100%) |
| Time (5,138 unique tuples, cold cache) | 5.8 minutes |

### Multilingual Coverage

| Language | Tests | Notable |
|----------|-------|---------|
| German | 21 | BSG, GPT/GOT, Quick-Wert, Tsd./µl |
| Korean | 14 | 당화혈색소, 평균적혈구혈색소량, 적혈구침강속도 |
| French | 24 | TCMH, CCMH, VS, ALAT/ASAT, TCA |
| Spanish (Spain + LATAM) | 42 | VSG, HCM/CHCM, TGP/TGO, Glucemia |
| Mandarin | 25 | 红细胞沉降率, 低密度脂蛋白胆固醇, 丙氨酸氨基转移酶 |
| Japanese | 21 | 赤沈, 中性脂肪, γ-GTP, ×10^4/µL convention |

## Known Limitations

### Thinking model token consumption

Rosetta uses `max_tokens=1000` by default because thinking models (like Gemini Flash Lite) consume 400-600 tokens for internal reasoning before producing the ~100-token JSON output. With non-thinking models, `max_tokens=200` would suffice. The higher token budget increases cost by ~5× per call but is necessary for reliable output from thinking models.

If using a non-thinking model, set `max_tokens=200` to reduce cost. The production grouper deployment runs with `reasoning_effort=none` against `gemini-3.1-pro-preview` — Pro is safety-clean for false-merge detection without extended thinking, and reasoning roughly doubles cost + 4× latency for marginal quality gain at this tier.

### Plausibility threshold tuning

The 2× ratio threshold separating "mild" (pass) from "severe" (warn) excursions is universal but blunt at the edges. Real biological extremes occasionally exceed it (severe primary hypothyroidism with TSH around 50 mIU/L vs reference up to ~20 mIU/L is 2.5×). The validator's job is detecting unit mislabels and data-entry errors, NOT clinical extremes — so a `warn` here is a false positive. Caller code (the grouper) currently logs warnings without triggering any user-facing alert; if that changes, the threshold may need analyte-specific tuning via the `plausible_range` widths the LLM emits.

### Per-analyte conversion gaps

The converter handles SI prefix shifts, mass↔molar (via molar mass), HbA1c IFCC↔NGSP affine, and dimensionless ratios. It does NOT handle analyte-specific conversion factors that aren't captured by molar mass alone — Insulin from `µIU/mL` to `pmol/L` requires a specific-activity factor (~0.144), which Rosetta doesn't know. Biomarkers that require these conversions get `unit_mismatch=True` from the converter and surface as fallback-split canonicals downstream. Adding curated conversion factors per analyte would conflict with principle 1 (LLM does medicine, code does mechanics) — these specific-activity values come from regulatory/standardization bodies and are stable enough to encode, but each addition needs review.

### LLM non-determinism

The same test name may produce slightly different resolutions across runs because LLM outputs aren't perfectly deterministic, even at `temperature=0`. The cache mitigates this — once resolved, a test name always returns the same identity until the relevant cache prefix is bumped. The component canonicalizer + group_key index further pin display fields across runs.

This primarily affects:
- Component normalisation (singular vs plural, hyphenation, Latinate vs anglicized forms)
- Property assignment for ambiguous units (%)
- Qualifier extraction for edge cases
- Canonical-name display when a group_key is first resolved (the gkey index pins it after that)

### Proprietary / esoteric test names

Tests from proprietary platforms (SYMPHONY organ ages, bioresonance panels, functional-medicine recommendation rows like "Number of Bacteria Affected") may not decompose correctly because the LLM has limited training data for these non-standard test types. They typically resolve to generic or incorrect components. The validator stack will often emit warnings for these (unrecognized unit signatures, no plausible_range), and downstream consumers can treat them as singletons.

## License

Proprietary — N1 Research LLC
