Diffoscope and Mach-O binaries (supporting reproducible iOS app builds)

[Marc Prud'hommeaux]

TL;DR: Should I expect (or hope) to be able to compare two differently-signed-but-otherwise-identical Mach-O binaries using `diffoscope`?

I have developed a project called the App Fair that enables any user to fork a base GitHub repository to create and release a SwiftUI iOS or macOS app, and then submit their app for publication by way of a pull request back to the base repository. The (untrusted) user's forked repository handles building and publishing the .ipa or .app artifacts using a pre-configured GitHub action, and then the subsequent user-submitted PR initiates a verification action in the (trusted) base repository. Validated apps are then signed and submitted for publication through various channels such as TestFlight and AltStore. This automated process is described more fully at https://appfair.net <https://appfair.net/>, and the AGPL source lives mostly in https://github.com/appfair. The project might best be summarized as an "automated and fully-reproducible F-Droid for Swift apps on iOS and macOS".

Verification of reproducibility is accomplished by attempting to replicate the (untrusted) forked release artifact. This is done by checking out the PR's code and re-building the app from a workflow running the trusted base repository. The resulting artifacts are compared for meaningful differences through a hodge-podge of custom tools and scripts. One challenge is the need to be able to tolerate differences in the signatures embedded in the Mach-O binaries, both in the main executable as well any embedded frameworks and extensions. My current solution is to manually strip out all the code signatures before comparing anything that looks like a Macho-O binary, but this doesn't always work due to factors like different padding in the binary archive.

I would rather use diffoscope for all this, but the tool doesn't seem to grok the Macho-O binaries, despite there seemingly (based on my novice-level Ruby comprehension skills) being some support for the format in diffoscope/comparators/macho.py.

To illustrate with a specific example, the contrived app named "Crazy Glue" has an untrusted released artifact "Crazy-Glue-iOS.ipa" available for download from https://github.com/Crazy-Glue/App/releases/0.0.5/. The trusted verification workflow run for this release's PR can be found at https://github.com/appfair/App/actions/runs/2904093446, whose build artifact can be downloaded and extracted to analyze the reproduced "artifact/Crazy-Glue-iOS.ipa" file. If these two binaries are identical (sans signatures and timestamps), then we accept the app for publication to the various supported distribution channels.

I was hoping to be able to use `diffoscope --exclude-directory-metadata=recursive Crazy-Glue-iOS.ipa artifact/Crazy-Glue-iOS.ipa`, but it fails on each of the Mach-O binaries, the first one being:

--- Crazy-Glue-iOS.ipa
+++ artifact/Crazy-Glue-iOS.ipa
├── Payload/Crazy Glue.app/Crazy Glue
│ ├── arm64
│ │┄ Format-specific differences are supported for this file format but no file-specific differences were detected; falling back to a binary diff. file(1) reports: Mach-O 64-bit arm64 executable, flags:<NOUNDEFS|DYLDLINK|TWOLEVEL|PIE>
│ │ @@ -86,15 +86,15 @@
│ │  00000550: 1900 0000 9800 0000 5f5f 4c4c 564d 0000  ........__LLVM..


So to my question: should I expect or hope to be able to compare archives containing Mach-O binaries like this? If not, would this be a contribution that the community would be interested in seeing? 

Thanks for any insights. The importance of verifiable build reproducibility for iOS apps is about to become more relevant as the platform opens up to additional sources of apps.

	–Marc