Reproducibility of laboratory test results

[Daniel Kahn Gillmor]

Hey Reproducible-Builds folks--

Apologies for the slightly off-topic message.  I hope interested folks
will follow up with me off-list.  

I'm looking for folks who might be interested in articulating to a USA
state supervisory body an argument about the value of reproducible
software toolchains in a criminal justice context.  This is not about
reproducible compilation specifically, but rather reproducible data
analysis pipelines. There are clearly similar principles at stake, and
in this context they can have an effect on people's liberty.

Background
----------

The New York City police crime lab recently alerted the public defenders
that there has been a bug in the software that produces printouts of
their chemistry pipeline that tests for controlled substances (e.g. "is
this substance cocaine?").  The bug was introduced in 2016 and only
fixed this year, so about 5 years of wrong printouts.  They claim it is
a bug in only the presentation of calibration data, and that no
substantive analysis was affected.

Their memo indicates that the ability to go back and reprint the
affected reports from their stored data using the fixed code is limited.
Most worrisome in terms of reproducibility is this sentence:

> Reprocessing of the data to update the RtIs value of the sample will
> not guarantee that each reprocessing would produce the same values
> in all other areas of the mass spectral data printout due to
> analysts’ selections and variations in software settings.

A copy of the memo is here if you want to read it.  It has sample images
of the printouts with the bug:

   https://dkg.fifthhorseman.net/nypd-gc-ms-bug-memo-2021-05-06.pdf

I'm hoping that a clear argument in favor of reproducibility for
criminal forensics could influence future guidance on lab forensics
policy, on procurement, and potentially on how courts should think about
dealing with data and data analysis in this sensitive context.

If you decide to follow up with me about this, you're not committing to
anything (and i can't promise that i can actually connect you with the
relevant policy-makers either), but i'd be happy to share more details
and talk over the situation with anyone from r-b who is interested.

If you think you might not be the right person to articulate these
ideas, but want to suggest someone else i should talk to, i'd be happy
for a recommendation/referral too.

    --dkg
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 227 bytes
Desc: not available
URL: <http://lists.reproducible-builds.org/pipermail/rb-general/attachments/20210607/247f7bf8/attachment.sig>