Monitoring reproducibility of GitHub Actions

Fri Apr 17 10:50:30 UTC 2026

Hi all,

First of all, thank you for the encouraging reaction :) I'll have a look at the ASF's work and probably add it to the related work section of the docs. Below are some responses and thoughts.

> I found quite a difference between JS actions and Docker actions. [...]

This is indeed true and I find it a bit hard to address since they're so different. Basically, as Aman already elaborated, for Docker we're just deferring the "what is equivalent" to diffoci (if we went with bit-wise equal I don't think we'd find any Docker-based actions that are reproducible). I am not particularly interested in aligning the definitions between the two, but it might be worth being more explicit about this aspect in the docs.

For JavaScript actions some prerequisite knowledge may be necessary. GitHub requires that JavaScript-based actions are runnable Node.js applications *from just the source*. In practice this means one of two things: JavaScript GitHub Action repositories contain either node_modules or the build output. In case of the latter, the goal of my project is to reproduce that build output.

The definition of reproducibility here is roughly as follows (would be happy to hear what is missing from the README and what is still missing here to improve the docs): the build is reproducible if I can

  1.  take the git repository
  2.  check out the version (which includes the build output!)
  3.  (re)install the dependencies using npm/yarn/pnpm/deno/bun or w/e the project wishes to use (ideally matching their version of the CLI and/or runtime, if I can derive it from the source)
  4.  run their build command they have (e.g. `npm run build`)
  5.  remove all files that git ignores
  6.  normalize line endings
  7.  end up with no diff

So basically, I try to match the build environment as closely as possible, except for the OS (under the assumption that the runtime binaries are entirely statically linked and the build doesn't shell out to any other system binaries).
and it could be

> From my understanding, this means that the build process of the underlying package manager is reproducible, not the action itself.

It is not clear to me what you mean by this, I hope the above clarifies. Otherwise, please elaborate.

> I also see many unpinned actions in that file. You may want to pin them :)

As a side, but still related note, I'm also monitoring if the releases of actions are immutable using https://github.com/ericcornelissen/reproducing-actions/blob/da6c6a11838790356e8442eeefb93e9cae0d1b16/.github/workflows/0-ci.yml#L49

If a release is immutable it should be fine to use unpinned actions.

>  For Docker actions, it rebuilds images from the Dockerfile and checks for semantic equivalence

"rebuild" here means we build it twice. This is because there's actually two types of Docker actions: Docker-image based vs Dockerfile based. So far we have some coverage for dockerfile-based action (thanks to Aman).

For Docker-image based actions we could test reproducibility in a manner closer resembling that of the JavaScript-based actions, namely we would build the image on our side and compare it to the hosted image.

There's an open PR to support this (https://github.com/ericcornelissen/reproducing-actions/pull/232) but it's proving very difficult to actually reproduce any (action) image in this way.

> reproducibility for composite actions

The reproducibility of composite actions is a bit of an odd one. As opposed to JavaScript or Docker-based actions, which typically have a build. Composite actions typically don't have a build of their own. Technically speaking they could, in which case the build could be literally anything (there's nothing stopping you from compiling, say, a Go binary, committing it, and invoking it in a composite action), but I haven't seen it.

Another aspect for composite actions is if all actions it is composed of (if any) are reproducible. This is a similar (practically identical, theoretically different) question to asking if a given workflow or repository is "composed" of reproducible actions, for which there's an open question: https://github.com/ericcornelissen/reproducing-actions/issues/73.

Regards,

Eric

________________________________
From: Arnout Engelen <arnout at bzzt.net>
Sent: Thursday, April 16, 2026 9:56:40 PM
To: Aman Sharma; rb-general
Cc: Eric Cornelissen
Subject: Re: Monitoring reproducibility of GitHub Actions

On Thu, Apr 16, 2026, at 19:22, Aman Sharma wrote:

Thanks for sharing the infrastructure! It is quite cool.

> At the ASF, we explicitly allowlist action versions,

I am interested to know more about it. I see in your repository that the final output is approved_patterns.yml. How do you enforce this regularly? Is there are CI job updating "Allow or block specified actions and reusable workflow " under "Org Settings > Action > General"<https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/enabling-features-for-your-repository/managing-github-actions-settings-for-a-repository#allowing-select-actions-and-reusable-workflows-to-run>? We also do it, but we don't have a lot of actions so we are okay with doing this manually.

Basically yes, https://github.com/apache/infrastructure-gha-allowlist-manager. Though we'll likely have to figure out something else, as that configuration has a maximum of 'only' 1000 entries ;)

I also see many unpinned actions<https://github.com/apache/infrastructure-actions/blob/8a059befd17ed98f4942c5cf3a67b7378045b669/approved_patterns.yml#L26-L28> in that file. You may want to pin them :)

Yes, those are a remnant of before we required pinning, we do want to remove those but gradually, without breaking too many workflows.

By the way, does your infrastructure also check reproducibility for composite actions?

I don't think so

Kind regards,

Arnout

> This already led to https://github.com/SonarSource/sonarqube-scan-action/pull/228 .

Nice! We also have some "trophies" which we record here<https://github.com/ericcornelissen/reproducing-actions#trophies>.

> Will definitely keep your projects in mind when we plan to extend that!

We will also look at the actions you have and create monitors for them.

Regards,

Aman Sharma
PhD Student
KTH Royal Institute of Technology
School of Electrical Engineering and Computer Science (EECS)
Department of Theoretical Computer Science (TCS)<http://www.kth.se><https://www.kth.se/profile/amansha><https://www.kth.se/profile/amansha>
<https://www.kth.se/profile/amansha>https://algomaster99.github.io/
________________________________

From: Arnout Engelen <arnout at bzzt.net>
Sent: Thursday, April 16, 2026 6:23:47 PM
To: rb-general
Cc: Aman Sharma; Eric Cornelissen
Subject: Re: Monitoring reproducibility of GitHub Actions

Hi,

Nice!

At the ASF, we explicitly allowlist action versions, and recently started checking actions with 'compiled' javascript for reproducibility before allowlisting. You can read more about this at https://github.com/apache/infrastructure-actions?tab=readme-ov-file#management-of-organization-wide-github-actions-allow-list, in particular https://github.com/apache/infrastructure-actions/tree/main/utils/verify_action_build and https://github.com/apache/infrastructure-actions/blob/main/.github/workflows/verify_dependabot_action.yml.

This already led to https://github.com/SonarSource/sonarqube-scan-action/pull/228 .

Will definitely keep your projects in mind when we plan to extend that!

Kind regards,

Arnout

On Thu, Apr 16, 2026, at 18:02, Aman Sharma via rb-general wrote:

Hi all,

I wanted to briefly share a project from our group at KTH Royal Institute of Technology. Eric Cornelissen<https://www.ericcornelissen.dev/>, a PhD student in our CHAINS<https://chains.proj.kth.se/> group, is maintaining an open-source project that monitors the reproducibility of GitHub Actions:
https://github.com/ericcornelissen/reproducing-actions

The goal of the project is to assess whether GitHub Actions can be reproduced. Currently, it focuses on two types of Actions: JavaScript-based actions and Docker-based actions (composite actions are not considered). For JavaScript actions, the project rebuilds the distributed files and compares them bit-by-bit with the repository contents. For Docker actions, it rebuilds images from the Dockerfile and checks for semantic equivalence, using https://github.com/reproducible-containers/diffoci, across builds.

More details about current actions being monitored are available on README. I am one of its contributors, so would be happy to talk about it.

Regards,

Aman Sharma
PhD Student
KTH Royal Institute of Technology
School of Electrical Engineering and Computer Science (EECS)
Department of Theoretical Computer Science (TCS)<http://www.kth.se><https://www.kth.se/profile/amansha><https://www.kth.se/profile/amansha>
<https://www.kth.se/profile/amansha>https://algomaster99.github.io/

--
Arnout Engelen
Engelen Open Source
https://engelen.eu

--
Arnout Engelen
Engelen Open Source
https://engelen.eu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.reproducible-builds.org/pipermail/rb-general/attachments/20260417/f1c3a929/attachment.htm>