<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><blockquote type="cite" class="">On Jun 7, 2021, at 7:25 PM, Daniel Kahn Gillmor <<a href="mailto:dkg@fifthhorseman.net" class="">dkg@fifthhorseman.net</a>> wrote:</blockquote><blockquote type="cite" class="">I'm looking for folks who might be interested in articulating to a USA<br class="">state supervisory body an argument about the value of reproducible<br class="">software toolchains in a criminal justice context. This is not about<br class="">reproducible compilation specifically, but rather reproducible data<br class="">analysis pipelines. There are clearly similar principles at stake, and<br class="">in this context they can have an effect on people's liberty.</blockquote></div><div class=""><br class=""></div><div class="">Here are a few snippets I know that may be helpful.</div><div class=""><br class=""></div><div class="">Reproducibility is widely considered essential to the scientific method, e.g., see:</div><div>Staddon, John (2017). Scientific Method: How Science Works, Fails to Work or Pretends to Work. Taylor and Francis.</div><div><br class=""></div><div>In some fields (notably social science) the pressure to publish has led to the publication</div><div>of papers that have, after the fact, turned out to be impossible to replicate or reproduce,</div><div>suggesting that these published claims are in fact false. This problem is called the</div><div>“Replication crisis”: <a href="https://en.wikipedia.org/wiki/Replication_crisis" class="">https://en.wikipedia.org/wiki/Replication_crisis</a></div><div>In short: if results cannot be replicated, it is generally assumed that the claims are</div><div>not scientific.</div><div><br class=""></div><div><div class="">I’m *not* a lawyer. However, my understanding is that in most of the US,</div><div class="">the main ruling involving scientific evidence is Daubert v. Dow Pharmacuticals, which</div><div class="">"created a four-factor test for the reliable determination of scientific evidence. The factors are whether a scientific technique “can be (and has been) tested;” was subjected to peer review and publication; whether there is a “known or potential rate of error” and “the existence and maintenance of standards controlling the technique’s operation;” and whether the technique has “general acceptance.” “</div><div class="">In practice this has many problems:</div><a href="https://www.abajournal.com/lawscribbler/article/courts_need_help_when_it_comes_to_science_and_tech" class="">https://www.abajournal.com/lawscribbler/article/courts_need_help_when_it_comes_to_science_and_tech</a><br class=""><div><br class=""></div><div>At the very least, if something is *NOT* replicatable, it is generally *NOT* met with “general acceptance”</div><div>In the scientific community (as shown by the “replication crisis” materials above).</div><div>Thus, if it can’t be reproduced, then it might not meet the Daubert requirement.</div><div><br class=""></div><div>Again, NOT NOT NOT a lawyer. If you want a legal opinion, you need a lawyer.</div><div>But I suspect you & a lawyer could find grounds to contest any tests that cannot be replicated.</div><div><br class=""></div><div>--- David A. Wheeler</div><div><br class=""></div></div></body></html>