Deterministic Computation in AI-Assisted Metrology

A summary of the Metrology Institute white paper Deterministic Computation in AI-Assisted Metrology.

Overview

Large language models explain measurement science well, but they generate numbers rather than compute them, and their output can vary from run to run. The white paper argues that this is incompatible with the foundations of metrology and proposes one architectural rule for AI systems in the field: deterministic code computes every reported number, and the language model only explains the final results. It applies the rule to the core calculations of a calibration laboratory, describes how such calculators are validated, and reports a case study of eight validated calculators on the MetTutor platform.

Why generated numbers are a problem

A language model produces each token from a probability distribution learned from text; it has no arithmetic unit. NIST’s Generative AI Profile (NIST AI 600-1) calls the resulting failure confabulation: confidently stated but erroneous content, sometimes with a plausible derivation attached. In metrology that is the worst case. The paper cites a MetTutor skill that asked the model to compute an uncertainty budget itself and reported a combined standard uncertainty 1.2 % low, with nothing on the screen to show it was wrong.

Repeatability is not guaranteed either: inference services can return different outputs for the same prompt even at temperature zero, because server load changes the order of floating-point additions. And a reproducible model would simply return the same wrong number every time. The case for deterministic calculators rests on correctness that can be validated, not on repeatability alone.

The paper distinguishes deterministic systems (fixed rules, one output per input), probabilistic systems (validated only statistically) and generative systems (testable only by sampling). The result is a division of labour: deterministic code computes, and the generative model explains, teaches and guides.

Why metrology requires determinism

• Each requirement that gives a measurement result its authority assumes one answer per input:
• Traceability (VIM 2.41) is a documented chain of calculations. A link whose arithmetic cannot be reproduced breaks the chain.
• Software validation (ISO/IEC 17025:2017 7.11) requires evidence of correct output for defined inputs, which is meaningful only if that output is fixed.
• Uncertainty evaluation under the GUM is a defined procedure. Once choices such as rounding and the truncation of ν_eff are documented, two competent evaluators of the same budget must agree.
• Decision rules (ISO/IEC 17025:2017 7.1.3 and 7.8.6, ILAC-G8, JCGM 106) must give one conformity statement for one set of values.

These are properties of the calculation, not of whatever presents it. A language model can present a result without weakening any of them, provided the result reaches it already computed.

The architecture

A request passes through five stages. A guided form collects values in fixed fields, so every request arrives in the same structure. A parser converts each field to an exact quantity with its unit and refuses anything ambiguous, such as a comma decimal or a missing unit, rather than guessing. The calculation uses exact rational arithmetic wherever a value can be exact, decides square roots and truncations by integer comparison, and truncates ratios so that no threshold is crossed by rounding error. A single presentation function writes every display string, so the card and each export cannot disagree. Finally, the language model receives the results in a block marked as authoritative and writes the explanation; it may not recompute, re-round or dispute any value.

Three design rules hold the boundary: refuse rather than estimate; be exact where possible and validated where not, confining floating point to irrational quantities checked against high-precision references; and keep the model’s instructions and the user’s data in separate messages. Given the same form, the system returns the same card on every run and device. Only the wording of the explanation varies.

Applications

The paper applies the rule to five core calculations, naming for each the error that approximation introduces. SI conversion factors derived from exact definitions are stored as exact fractions and rounded once at display. GUM uncertainty budgets carry full precision and round once, with expanded uncertainty rounded up as a conservative rule. Welch-Satterthwaite effective degrees of freedom are truncated exactly, where floating-point error could otherwise drop a whole degree of freedom and inflate the expanded uncertainty. TUR and TAR are truncated, so a ratio below 4:1 is never displayed as 4.00, and the tolerance convention (Z540.3 versus ILAC-G8) is stated. Decision rules and guard bands give a binary verdict with a sharp boundary, precisely where a probabilistic answer does the most harm.

Validation and the MetTutor case study

Because each input has one correct output, a deterministic calculator can be tested case by case, and a single mismatch is a finding. The methods include an independent reference implementation sharing no code with production, random differential testing, published known answers, edge and refusal cases, a permanent regression suite, before-and-after comparison on every change, and live tests. Those who build the calculators should not verify them, and any change should trigger a scoped revision of the validation record.

In the case study, an AI coding assistant implemented the calculators, a separate AI session verified them from the standards, and the author performed live testing and approved every change. The paper acknowledges the common-mode risk of using models from the same family. Across 68,000 independent random cases there were no mismatches on any displayed value, bounding the per-case failure rate at roughly 0.004 % with 95 % confidence. The regression suite passed in full, and a later display change was checked by re-running all 69,667 earlier validation inputs. Validation also found real defects, including a rule that treated 1,000 or more degrees of freedom as infinite, each traced, corrected and re-validated. The validation covers every number on the cards and exports, but not the model-written explanation.

Benefits and limitations

The benefits follow from the requirements: reproducibility, validatability, auditability, exactness at boundaries, honest refusal, consistency across outputs, better teaching anchored to correct numbers, and change control. A provider’s model update can silently change its arithmetic, whereas a deterministic calculator changes only when its code changes.

The limitations are stated plainly. The explanation remains probabilistic and can be sampled but not validated. Determinism is not correctness: a wrong formula is wrong every time, though reproducibly and therefore findable. Scope is bounded, and correlated inputs, asymmetric specifications, and Monte Carlo models need their own implementations; outside its scope, a calculator must refuse. Floating point remains for irrational values. Choosing the measurement model and inputs is still the metrologist’s judgment. And a laboratory using a validated training calculator in its own work must still validate that use.

Research and policy context

Research shows that language models struggle with chained arithmetic, and approaches such as Program-aided Language Models and Toolformer delegate computation to code while still allowing the model to write the program. The paper’s architecture goes further, removing the model from the computation entirely. It aligns with the NIST AI Risk Management Framework, which places “valid and reliable” at the base of trustworthy AI; with the OECD AI Principles; and with metrology’s established software-validation guidance.

Recommendations

For developers: compute every reported number in deterministic code; use exact arithmetic wherever possible; refuse ambiguous or out-of-scope input; make display rules conservative and explicit; keep instructions and user data separate; validate against independent references, known answers and edge cases under revision control; and state plainly what is not validated.

For laboratories, educators and assessors: ask any AI tool where its numbers were computed and how that was validated; treat a generated number as unverified until reproduced independently; and apply the existing ISO/IEC 17025 software-validation requirements to AI tools exactly as to any other calculation software.

The paper concludes that determinism does not make an AI system intelligent. It makes its numbers trustworthy, which in measurement science is the precondition for everything else.