Reproducibility of laboratory test results

[David A. Wheeler]

> On Jun 7, 2021, at 7:25 PM, Daniel Kahn Gillmor <dkg at fifthhorseman.net> wrote:
> I'm looking for folks who might be interested in articulating to a USA
> state supervisory body an argument about the value of reproducible
> software toolchains in a criminal justice context.  This is not about
> reproducible compilation specifically, but rather reproducible data
> analysis pipelines. There are clearly similar principles at stake, and
> in this context they can have an effect on people's liberty.


Here are a few snippets I know that may be helpful.

Reproducibility is widely considered essential to the scientific method, e.g., see:
Staddon, John (2017). Scientific Method: How Science Works, Fails to Work or Pretends to Work. Taylor and Francis.

In some fields (notably social science) the pressure to publish has led to the publication
of papers that have, after the fact, turned out to be impossible to replicate or reproduce,
suggesting that these published claims are in fact false. This problem is called the
“Replication crisis”: https://en.wikipedia.org/wiki/Replication_crisis <https://en.wikipedia.org/wiki/Replication_crisis>
In short: if results cannot be replicated, it is generally assumed that the claims are
not scientific.

I’m *not* a lawyer. However, my understanding is that in most of the US,
the main ruling involving scientific evidence is Daubert v. Dow Pharmacuticals, which
"created a four-factor test for the reliable determination of scientific evidence. The factors are whether a scientific technique “can be (and has been) tested;” was subjected to peer review and publication; whether there is a “known or potential rate of error” and “the existence and maintenance of standards controlling the technique’s operation;” and whether the technique has “general acceptance.” “
In practice this has many problems:
https://www.abajournal.com/lawscribbler/article/courts_need_help_when_it_comes_to_science_and_tech

At the very least, if something is *NOT* replicatable, it is generally *NOT* met with “general acceptance”
In the scientific community (as shown by the “replication crisis” materials above).
Thus, if it can’t be reproduced, then it might not meet the Daubert requirement.

Again, NOT NOT NOT a lawyer. If you want a legal opinion, you need a lawyer.
But I suspect you & a lawyer could find grounds to contest any tests that cannot be replicated.

--- David A. Wheeler

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.reproducible-builds.org/pipermail/rb-general/attachments/20210608/9eea0b63/attachment.htm>