[rb-general] Debian buster, 54% reproducible in practice (Re: Core Debian reproducibility: 57% and rising!)

[Holger Levsen]

Hi,

it's been some time... ;)

On Tue, Oct 30, 2018 at 07:27:31AM +0000, Daniel Shahaf wrote:
> David A. Wheeler wrote on Mon, 29 Oct 2018 06:22 -0400:
> > >I would skip the numbers or put them last in the news bit.

I'm working on something like this now, and hope to have it ready by tomorrow
evening. We'll see. 

> > I like the idea of including the non real world percentage as well, and 
> > explaining the difference. Any suggestions on text that would do that?
> This is as far as I got, based on your text:
> 
> ---
> 
> For a long time, over 93% of all source packages in the Debian archive
> (25561 out of 27427) have been known to be [reproducible in a laboratory
> environment][1].  Last week, Vagrant Cascadian [probed the package
> archives][2] and found that in current Debian Sid, **57% of the binary
> packages installed in a minimal system** are verifiably reproducible (88
> out of 154).
> 
> While 57% is a lower figure, it is a more substantial statistic: it is
> not a measure of packages that behave well under carefully controlled
> conditions, but of actual "real world" Debian artifacts, that get
> installed on end-user systems, that have successfully been reproduced in
> the field.  Furthermore, this statistic only considers essential core
> packages that are installed on all Debian systems.

I've looked at almost all packages in buster/amd64 now, these are the
results gathered by
https://jenkins.debian.net/job/reproducible_compare_Debian_sha1sums/97/console

ftp.debian.org package reproducibility statistics including packages (currently) in an unknown state
-------------------------------------------------------------
packages in unknown reproducibility state in buster/amd64: 6074: (10.5700%)
reproducible packages in buster/amd64: 27740: (48.2700%)
unreproducible packages in buster/amd64: 23644: (41.1500%)
total number of packages in buster/amd64: 57458

ftp.debian.org package reproducibility statistics of packages in known states only
-------------------------------------------------------------
reproducible packages in buster/amd64: 27740: (53.9800%)
unreproducible packages in buster/amd64: 23644: (46.0100%)

reproducible binNMUs in buster/amd64: 0: (0%)
unreproducible binNMU in buster/amd64: 6403: (12.4600%)


These are the numbers for real .debs downloaded from ftp.debian.org,
for which 2 or more buildinfo files exist, which reproduced a deb with
the very same sha1 hash.

I also tried to download the .buildinfo files for all the packages found
as unreproducible here, but couldn't get any. I think I'll need to try
this again and harder.

The problem with binNMUs is described in length at #894441 and should be
revisited at the beginning of the bullseye development cycle or possibly
earlier.

For now, my headline for this is "Debian Buster: 93% reproducible in
theory, 54% in practice". And while I'm saddened by the 'downgrade' in 
percentage, I'm delighted we've reached the next level: real world 
reproducibility. I'm also hopeful we'll find ways to climb from 54% to
somewhat higher soon. 

Though without solving #894441 we cannot reach much higher than 80% 
(because 93% is the current theoretic maximum, of which we need to 
distract 12% binNMUs...)


tschau,
	Holger

-------------------------------------------------------------------------------
               holger@(debian|reproducible-builds|layer-acht).org
       PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C

Our civilization is being sacrificed for the opportunity of a very small number
of people to continue making enormous amounts of money...  It is the sufferings
of the many  which pay  for the luxuries  of the few...  You say  you love your
children  above all else,  and yet  you are stealing  their future  in front of 
their very eyes...
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.reproducible-builds.org/pipermail/rb-general/attachments/20190301/618bdef4/attachment.sig>