Metadata Provenance in Imaging AI Validation Sets

An external validation set is an assertion that the data is independent of training in the ways that matter: different patients, ideally different sites, different acquisition instances. The assertion is about provenance, and provenance lives almost entirely in metadata that curation pipelines are designed to remove.

De-identification removes the audit trail by design

The Basic Application Level Confidentiality Profile in PS3.15 requires removing or replacing a large set of attributes, including dates, institution identifiers, device serial numbers and most UIDs. Applied correctly, it also removes the fields that would let you establish that two series came from different scanners at different sites on different days.

The standard anticipates this. Patient Identity Removed (0012,0062) and De-identification Method (0012,0063) exist to record that de-identification happened and how, and the profile permits retaining attributes when they are needed for the data’s purpose. Retention options exist specifically for device identity and for longitudinal temporal information.

The problem in practice is that de-identification is frequently applied with default settings by whoever exported the data, and the resulting cohort cannot distinguish two acquisitions of the same patient a week apart from two different patients. Once UIDs have been remapped without a retained mapping, the information is not merely hidden — it is gone.

The consequence for validation: a same-patient pair split across training and test is undetectable. The model reports a validation figure that includes memorised anatomy, and nothing in the metadata can be interrogated to show it.

Conversion is a lossy step that reports success

Most imaging AI work happens on NIfTI, not DICOM. Conversion collapses a series into an array plus an affine, and discards essentially every attribute discussed in the header fields that break quantitative analysis — kernel, units, rescale provenance, vendor tags, acquisition times.

Conversion tools are good and this is not a criticism of them. It is an observation that the output format has no place to put the information, so the information’s survival depends entirely on someone extracting it into a sidecar before conversion. If that step is skipped, a later question about scanner distribution has no answer, and the honest response — “we cannot determine that” — is one very few papers give.

Series selection is a hidden inclusion criterion

Cohorts are usually assembled by filtering series on description, as covered in what DICOM metadata guarantees. Because Series Description is free text set from local protocol, a description-based filter selects on site naming convention as much as on sequence type.

The effect on validation is specific and worth stating plainly. If the training cohort and the validation cohort were filtered with the same description patterns, and those patterns match one site’s conventions better than another’s, then the validation set is enriched for sites resembling the training set. The cohort is less independent than its site count suggests, and the direction of the bias is optimistic.

What a defensible cohort record contains

Per series, retained through every processing step and never reconstructed after the fact:

• Identity mapping under controlled access. A retained, access-controlled mapping from pseudonymous to original study and series UIDs. Without it, patient-level leakage cannot be excluded, only hoped against.
• Acquisition fingerprint. Manufacturer, model, field strength, kernel or sequence parameters, extracted before conversion and stored alongside the array.
• De-identification record. Which profile, which options, which tool version. (0012,0063) exists for this.
• Selection provenance. The exact filter that admitted the series, stored as data, so that “how was this cohort assembled” is answerable without archaeology.
• Conversion provenance. Tool and version, and whether rescale was applied during conversion or left to downstream code.

The check that is worth running

Before reporting an external validation figure, attempt to falsify the independence claim. Group the validation set by acquisition fingerprint and compare against the training set: shared scanner serial numbers, implausibly similar acquisition parameter clusters, or same-day study timestamps are all signals that the two cohorts overlap more than the site labels imply.

If the metadata to run that check no longer exists, that is itself the finding. A validation set whose independence cannot be tested should be reported as what it is — a second dataset — rather than as evidence of generalisation.