Reproducibility of laboratory test results

[Chris Lamb]

Hi Daniel,

Thanks for your email. The implications of not being able to replicate
results such as these in a legal context are, of course, extremely
disturbing.

(No doubt you can provide more historical background, but this appears
to be another instance where the infallibility of novel scientific
techniques are overstated or misunderstood, leading to highly unsafe
convictions.  DNA immediately spring to mind, but so does psychological
graphology, phrenology, Bayesian statistics, etc. But perhaps I digress..)

As it happens, your link to the PDF hosted on dkg.fifthhorseman.net
403s for me, but I would be very interested in learning more about
what you have in mind in any case. And another org that might be worth
contacting is Software Heritage: whilst this would not fall under
their core mission, they would undoubtedly be allies and/or know of
others too.

// Chris



> Apologies for the slightly off-topic message.  I hope interested folks
> will follow up with me off-list.
>
> I'm looking for folks who might be interested in articulating to a USA
> state supervisory body an argument about the value of reproducible
> software toolchains in a criminal justice context.  This is not about
> reproducible compilation specifically, but rather reproducible data
> analysis pipelines. There are clearly similar principles at stake, and
> in this context they can have an effect on people's liberty.
>
> Background
> ----------
>
> The New York City police crime lab recently alerted the public defenders
> that there has been a bug in the software that produces printouts of
> their chemistry pipeline that tests for controlled substances (e.g. "is
> this substance cocaine?").  The bug was introduced in 2016 and only
> fixed this year, so about 5 years of wrong printouts.  They claim it is
> a bug in only the presentation of calibration data, and that no
> substantive analysis was affected.
>
> Their memo indicates that the ability to go back and reprint the
> affected reports from their stored data using the fixed code is limited.
> Most worrisome in terms of reproducibility is this sentence:
>
> > Reprocessing of the data to update the RtIs value of the sample will
> > not guarantee that each reprocessing would produce the same values
> > in all other areas of the mass spectral data printout due to
> > analysts’ selections and variations in software settings.
>
> A copy of the memo is here if you want to read it.  It has sample images
> of the printouts with the bug:
>
>    https://dkg.fifthhorseman.net/nypd-gc-ms-bug-memo-2021-05-06.pdf
>
> I'm hoping that a clear argument in favor of reproducibility for
> criminal forensics could influence future guidance on lab forensics
> policy, on procurement, and potentially on how courts should think about
> dealing with data and data analysis in this sensitive context.
>
> If you decide to follow up with me about this, you're not committing to
> anything (and i can't promise that i can actually connect you with the
> relevant policy-makers either), but i'd be happy to share more details
> and talk over the situation with anyone from r-b who is interested.
>
> If you think you might not be the right person to articulate these
> ideas, but want to suggest someone else i should talk to, i'd be happy
> for a recommendation/referral too.