[Git][reproducible-builds/reproducible-website][master] hamburg2023: day 3 notes

Arnout Engelen (@raboof-guest) gitlab at salsa.debian.org
Thu Nov 2 16:54:11 UTC 2023 


Arnout Engelen pushed to branch master at Reproducible Builds / reproducible-website


Commits:
7a931ec7 by Arnout Engelen at 2023-11-02T17:53:50+01:00
hamburg2023: day 3 notes

- - - - -


10 changed files:

- _events/hamburg2023/agenda.md
- + _events/hamburg2023/arch-huddle.md
- + _events/hamburg2023/born-reproducible-2.md
- + _events/hamburg2023/born-reproducible-3.md
- + _events/hamburg2023/debian.md
- + _events/hamburg2023/diffoscope-2.md
- + _events/hamburg2023/fedora-packages.md
- + _events/hamburg2023/filtering-diffoscope.md
- + _events/hamburg2023/rpm-sbom.md
- + _events/hamburg2023/using-verification-data.md


Changes:

=====================================
_events/hamburg2023/agenda.md
=====================================
@@ -68,24 +68,30 @@ Day 2 - Wednesday, November 1st TODO
 * 16.35 Closing Circle
 * 17.00 Adjourn
 
-Day 3 - Thursday, November 2nd TODO
+Day 3 - Thursday, November 2nd
 ------------------------------
 
 * 9.00 Opening Circle
   * The day started with a summary of Day 2 outcomes and a Day 3 Agenda Overview.
-* 9.30 Collaborative Working Sessions, break-out discussions continue.
-  * Verifying packages at installation discussion (in-toto): FIXME https://pad.riseup.net/p/rbsummmit2022-installation-keep
-  * [Taxonomy]({{ "/events/venice2022/taxonomy/" | relative_url }})
-  * Debian FIXME https://pad.riseup.net/p/rbsummmit2022-debian-keep
-  * [Firmware]({{ "/events/venice2022/firmware/" | relative_url }})
-* 10.45 Break
-* 11.00 Collaborative Working Sessions, break-out discussions continue.
-  * in-toto vs sbom (spdx) FIXME https://pad.riseup.net/p/rbsummmit2022-intoto-vs-sbom
+* 9.45 Collaborative Working Sessions, break-out discussions continue.
+  * [SBOM for rpm]({{ "/events/hamburg2023/rpm-sbom/" | relative_url }})
+  * [Filtering diffoscope output]({{ "/events/hamburg2023/filtering-diffoscope/" | relative_url }})
+  * [Images/Filesystems/Containers]({{ "/events/hamburg2023/images-filesystems/" | relative_url }})
+  * [Born Reproducible II]({{ "/events/hamburg2023/born-reproducible-2/" | relative_url }})
+  * [Using verification data]({{ "/events/hamburg2023/using-verification-data/" | relative_url }})
+* 11.15 Break
+* 11.30 Collaborative Working Sessions, break-out discussions continue.
+  * [Born Reproducible III]({{ "/events/hamburg2023/born-reproducible-3/" | relative_url }})
+  * [Verification Service III]({{ "/events/hamburg2023/verification-service-3/" | relative_url }})
+  * [Fedora packages]({{ "/events/hamburg2023/fedora-packages/" | relative_url }})
+  * [Arch huddle]({{ "/events/hamburg2023/arch-huddle/" | relative_url }})
+  * [Diffoscope II]({{ "/events/hamburg2023/diffoscope-2/" | relative_url }})
+  * [Debian]({{ "/events/hamburg2023/debian/" | relative_url }})
 * 12.30 Lunch
-* 13.30 Mapping Where From Here
-  * The group took pause before the final session to take stock of the progress made to this point in the week and to inventory action items, next steps and other bridges to post-event collaboration. FIXME https://pad.riseup.net/p/rbsummmit2022-i-will-we-should-keep
-* 16.15 Closing Circle
+* 14.00 Mapping next conversations and next steps
+  * The group paused before the final session to take stock of the progress made to this point in the week and to inventory action items, next steps and other bridges to post-event collaboration.
+* 15.30 Closing Circle
   * Participants summarized key outcomes from the event, and discussed next steps for continuing collaboration after the meeting.
-* 17.00 Adjourn
+* 16.00 Adjourn / Hack Time
 
 


=====================================
_events/hamburg2023/arch-huddle.md
=====================================
@@ -0,0 +1,9 @@
+---
+layout: event_detail
+title: Collaborative Working Sessions - Arch huddle
+event: hamburg2023
+order: 307
+permalink: /events/hamburg2023/arch-huddle
+---
+
+


=====================================
_events/hamburg2023/born-reproducible-2.md
=====================================
@@ -0,0 +1,30 @@
+---
+layout: event_detail
+title: Collaborative Working Sessions - Born Reproducible II
+event: hamburg2023
+order: 303
+permalink: /events/hamburg2023/born-reproducible-2/
+---
+
+Notes
+
+This session is a follow-up on session from day 2 "Born Reproducible"
+
+* Step #0 - Have a clean and defined build environment
+* Step #1 - Know/Define:
+  * input
+  * output
+* Step #2 - Record build metadata (e.g., buildinfo)
+* Step #3 - Try to rebuild in a similar environment
+* Step #4 - Analyse any differences
+* Step #5 - Try to rebuild in a different enviroment
+* Step #6 - Make it possible for users to rebuild/validate
+* Step #7 - Define policy if build is unreproducible
+* Step #8 - Collect build attestations
+* Step #9 - Diversify Builders (e.g., identities, data centers, possibly users)
+* Step #10 - Implement / Deploy Policy
+* Profit! - Announce reproducibility
+Communicate on every step!
+
+
+


=====================================
_events/hamburg2023/born-reproducible-3.md
=====================================
@@ -0,0 +1,82 @@
+---
+layout: event_detail
+title: Collaborative Working Sessions - Born reproducible III
+event: hamburg2023
+order: 305
+permalink: /events/hamburg2023/born-reproducible-3
+---
+
+Follow up of [Born Reproducible II]({{ "/events/hamburg2023/born-reproducible-2/" | relative_url }})
+
+In this session, we were more concrete. We came up with how each step in the framework in part II could be implemented for Java (especially maven) ecosystem.
+
+## Reference Implementation
+https://github.com/jvm-repo-rebuild/reproducible-central.
+
+### Step 0
+Assume that the clean environment is the GitHub action runner. Although you have more control on your system, the information you care about (like `java` version, `mvn` version) is know in GitHub action runner as well.
+
+### Step 1
+
+- Inputs:
+  1. Java source code
+  2. Build configuration (`pom.xml`): Try to have a minimal working version of
+     that. Remove all the unecessary boilerplate code.
+- Outputs:
+  1. all the `jar`s and the SBOMs produced as the build output.
+    
+    `.buildspec` in the reference implementation documents all of this.
+
+### Step 2
+Build metadata for Java could mean to know all the transitive dependencies and system dependencies (for example, the compression tool for classfiles could be a system dependency)
+
+> maybe not too relevant for Java, but it was for C/android
+
+### Step 3
+
+- Compare the artifacts listed in `output` with the ones on maven central (`rebuild.sh <buildspec>` does that in reference implementation).
+- `mvn package` for the first build and `mvn package` without snapshot lookup for the second build (rebuild).
+
+### Step 4
+
+The differences could be
+
+- order of classfiles in archives
+- absolute and relative paths of resources
+- differences in classfiles
+- difference in manifest files
+
+Reference implementation uses `diffoscope`
+	
+### Step 5
+A different environment could be Jenkins, CircleCI, a separate GitHub action runner with a different architecture.
+
+> In theory, architecture should not matter as jars are supposed to be independent of this.
+
+### Step 6
+
+- Let user's reproduce the CI based on the build step
+_ Provide tooling for reproducing it. Example: [gorebuild](https://pkg.go.dev/golang.org/x/build/cmd/gorebuild) is a tool to reproduce go toolchain.
+
+> From here on, it gets abstract
+
+### Step 7
+Document why the artifacts generated are not reproducible and give reasons why they are not. For example, the signature difference. In Step 6, users could be told to strip signature before comparing.
+
+> Step 8,9,10 are release related so we did not discuss them in depth.
+
+### Step 8
+Project README should tell how to verify build attestation.
+
+### Step 9
+Have your CI sign the artifacts.
+
+### Step 10
+Implementing deploy policy is a responsibilty taken by the artifact hosting service (maven central) in Java ecosystem.  For example, it checks your manifest (`pom.xml`)  for certain requirements before deploying. 
+
+Last takeaway of this session was that the framework was supported by people working in C and android ecosystems. However, commercial code is a hard problem.
+
+
+
+
+


=====================================
_events/hamburg2023/debian.md
=====================================
@@ -0,0 +1,8 @@
+---
+layout: event_detail
+title: Collaborative Working Sessions - Debian
+event: hamburg2023
+order: 308
+permalink: /events/hamburg2023/debian/
+---
+


=====================================
_events/hamburg2023/diffoscope-2.md
=====================================
@@ -0,0 +1,8 @@
+---
+layout: event_detail
+title: Collaborative Working Sessions - Diffoscope II
+event: hamburg2023
+order: 308
+permalink: /events/hamburg2023/diffoscope-2/
+---
+


=====================================
_events/hamburg2023/fedora-packages.md
=====================================
@@ -0,0 +1,26 @@
+---
+layout: event_detail
+title: Collaborative Working Sessions - Fedora packages
+event: hamburg2023
+order: 306
+permalink: /events/hamburg2023/fedora-packages/
+---
+
+Bernhard imported a number of packages into openSUSE's open-build-service(OBS) and used the reproducibleopensuse tools to do double-builds and verification builds and additionally compare to official Fedora binary pkgs
+
+Fedora does not normalize build-time in rpm-headers, so no bit-reproducible replication is possible atm. Would need some rpm %macro mechanism to override the value that is separate from $SOURCE_DATE_EPOCH.
+
+Additionally we found some more roadblocks:
+	* Fedora uses dynamic python provides that ended up missing in OBS and resulted in a failed build
+	* pam did not build because xpdf xpdf-libs both provided the same symbol and OBS does not have Fedoras's automatic resolution of using the first shortest name
+	* python .pyc file headers varied - maybe because of additional rb-related macros used
+	* koji only kept details from the last 2 weeks, so buildroot details for all Fedora39 packages were already expired from that cache (though are accessible in a different way)
+	* OBS has a different logic to create Release strings (N.M vs N.fc39)
+	* The name-epoch-version-release string gets embedded in `.note.package` section in all ELF files, which means it'll directly impact reproducibility.
+
+we tested with
+	* 2ping
+	* perl-Alien-Brotli
+	* python-gemfileparser2
+	* pam
+


=====================================
_events/hamburg2023/filtering-diffoscope.md
=====================================
@@ -0,0 +1,40 @@
+---
+layout: event_detail
+title: Collaborative Working Sessions - Filtering diffoscope output
+event: hamburg2023
+order: 301
+permalink: /events/hamburg2023/filtering-diffoscope/
+---
+
+Goal: add patterns to filter out some parts of output, or filters to only show some parts of output
+
+Requirements:
+- print info that parts output are being ignored
+- indicate in return code that files are not identical
+
+A number of options exist:
+- `--exclude`
+- `--exclude-command=REGEXP`: this skips command matching REGEXP
+  (`--exclude-command '^readelf.*gdb_index'`)
+  but then diffoscope tries the next command, possibly falling back to hexdump comparison
+  
+- output formats: `--json`, `--html`, `--htmldir`.
+  Multiple output formats can be use together.
+  
+- `--load-existing-diff FILE`.
+  Diffoscope will produce all kinds of output from JSON.
+  This can be combined with 'jq' filtering or some other way to filter.
+
+Internally, state is a series of deeply nested dictionaries.
+The comparator is called with a paths of keys.
+
+Issues about --exclude* already exist:
+https://salsa.debian.org/reproducible-builds/diffoscope/-/issues/130
+https://salsa.debian.org/reproducible-builds/diffoscope/-/issues/53
+https://salsa.debian.org/reproducible-builds/diffoscope/-/issues/52
+
+Filtering by "output level" is not enough.
+For example, in an RPM header, some specific fields should be ignored, but only those.
+
+Idea: provide a command to filter the output using a jq-like path.
+


=====================================
_events/hamburg2023/rpm-sbom.md
=====================================
@@ -0,0 +1,56 @@
+---
+layout: event_detail
+title: Collaborative Working Sessions - SBOM for rpm
+event: hamburg2023
+order: 300
+permalink: /events/hamburg2023/rpm-sbom/
+---
+
+SBOM discussion led by Marek
+
+rpmbuild should produce buildinfo file during package-build
+
+currently fragmented: OBS, koji, others reinvent their own formats
+
+There was previous discussion with rpm maintainers.
+Idea: produce separate sub-package with that buildinfo file.
+format was too Debian-ish and therefore disliked by rpm maintainers.
+
+buildinfo-rpm can be signed the normal way
+can be published to separate repo (similar to debuginfo)
+
+
+Prior work:
+* https://github.com/rpm-software-management/rpm/pull/1532 + rpmrebuild
+* https://github.com/rpm-software-management/rpm/issues/2389
+* http://download.opensuse.org/update/leap/15.5/sle/x86_64/ has slsa_provenance.json in-toto format
+* https://github.com/opensbom-generator/spdx-sbom-generator#module-json-example
+* https://cyclonedx.org/
+* some Yocto-based medical device collects plenty data from build
+
+goal:
+ * be able to independently verify rpms / containers
+ * common tool for reproducing rpm packages - no matter from which distribution
+ * also for 3rd-party packages such as google-chrome
+
+Ideas:
+ * discuss more with upstream: what value it would provide
+ * let upstream come up with a PR
+ * have prepared shared zstd dict for efficient SBOM compression
+
+result/output-SBOM vs input/build-SBOM
+  => see also notes on Wed discussion on SBOM
+  SPDX + CycloneDX + in-toto file format
+
+consumers for SBOM files:
+* CVE-scanners
+* License-scanners
+
+
+missing link for publishing required buildrequires rpm + fetching via name|shasum
+* URL for provider service
+* archive.org
+* IPFS
+* other
+
+


=====================================
_events/hamburg2023/using-verification-data.md
=====================================
@@ -0,0 +1,14 @@
+---
+layout: event_detail
+title: Collaborative Working Sessions - Using verification data
+event: hamburg2023
+order: 304
+permalink: /events/hamburg2023/using-verification-data
+---
+
+Using verification data
+
+Notes
+
+
+



View it on GitLab: https://salsa.debian.org/reproducible-builds/reproducible-website/-/commit/7a931ec70928b1bad95387935175e8f52cce0b00

-- 
View it on GitLab: https://salsa.debian.org/reproducible-builds/reproducible-website/-/commit/7a931ec70928b1bad95387935175e8f52cce0b00
You're receiving this email because of your account on salsa.debian.org.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.reproducible-builds.org/pipermail/rb-commits/attachments/20231102/2592c84f/attachment.htm>

More information about the rb-commits mailing list