Github Actions and the hashFiles incident
kpcyrd
kpcyrd at archlinux.org
Sun Nov 23 20:15:03 UTC 2025
Dear Reproducible Builds list,
one of my pipelines was failing today with the following error:
> Error: The template is not valid. .github/workflows/rust.yml (Line:
57, Col: 14): hashFiles('**/Cargo.lock') failed. Fail to hash files
under directory '/Users/runner/work/tokio-ar/tokio-ar'
It's caused by the following line for cache-key calculation:
> key: ${{ runner.os }}-cargo-debug-${{ hashFiles('**/Cargo.lock') }}
This I had copied over from a different project so I was surprised this
is now failing. I tried to figure out if maybe the commit wasn't checked
out correctly, but then noticed this seems to be an ongoing regression:
- https://github.com/actions/runner/issues/449 (originally from 2020,
but recent comments from 2025)
- https://github.com/actions/runner-images/issues/13341
- https://github.com/orgs/community/discussions/180160
There's a lot of people affected right now. Well, happens, but in this
discussion thread a comment caught my attention however:
> @andrzejressel:
>
https://github.com/andrzejressel/pulumi-gestalt/actions/runs/19598070772/job/56125529525?pr=1503#step:2:5454
- I think someone edited this file manually - it does not match repo:
https://github.com/actions/runner/blob/main/src/Misc/layoutbin/hashFiles/index.js#L5442
Curious!
The comment links to the current `main` branch, I noticed there's
release tags (which would be a simple solution to this mismatch
mystery), I tried the same file from `v2.330.0` first, which didn't
match, then tried `v2.329.0` which _almost_ matches.
But not entirely:
```
--- index-v2.329.0.js 2025-11-23 19:58:21.222607998 +0100
+++ job-logs.js 2025-11-23 19:57:45.889321750 +0100
@@ -1,4 +1,4 @@
-/******/ (() => { // webpackBootstrap
+/******/ (() => { // webpackBootstrap
/******/ var __webpack_modules__ = ({
/***/ 2627:
@@ -2361,7 +2361,7 @@
if (!options.headers) {
throw Error('The request has no headers');
}
- options.headers['Authorization'] = `Bearer ${this.token}`;
+ options.headers['Authorization'] = `***;
}
// This handler cannot handle 401
canHandleAuthentication() {
@@ -5435,4 +5435,13 @@
/******/ module.exports = __webpack_exports__;
/******/
/******/ })()
-;
\ No newline at end of file
+;*****************/
+/******/
+/******/ // startup
+/******/ // Load entry module and return exports
+/******/ // This entry module is referenced by other modules so it
can't be inlined
+/******/ var __webpack_exports__ = __nccwpck_require__(4711);
+/******/ module.exports = __webpack_exports__;
+/******/
+/******/ })()
+;
```
The first hunk is due to `0xef 0xbb 0xbf` and likely inserted by the CI:
https://en.wikipedia.org/wiki/Byte_order_mark
The second hunk I assume is log redaction along the lines of `Bearer
(.*)` to avoid leaking secrets/credentials.
The third hunk I don't have an explanation for. At first it looks like
this is comments-only, but `/******/` starts and also immediately ends a
block comment, so the code after is indeed executed.
I tried to match this to more versions, but v2.325.0-v2.329.0 was still
the closest match (the file is identical for those releases):
9df25845885800455e561f9b630a5c895ba17e821bfcd0724098ba362e39a0b4
index-v2.320.0.js
9df25845885800455e561f9b630a5c895ba17e821bfcd0724098ba362e39a0b4
index-v2.321.0.js
9df25845885800455e561f9b630a5c895ba17e821bfcd0724098ba362e39a0b4
index-v2.322.0.js
9df25845885800455e561f9b630a5c895ba17e821bfcd0724098ba362e39a0b4
index-v2.323.0.js
9df25845885800455e561f9b630a5c895ba17e821bfcd0724098ba362e39a0b4
index-v2.324.0.js
9df25845885800455e561f9b630a5c895ba17e821bfcd0724098ba362e39a0b4
index-v2.325.0.js
ab159da6a2af4d98859963fbdad93b52929adb5f11e6eea9e440d942624d704b
index-v2.326.0.js
ab159da6a2af4d98859963fbdad93b52929adb5f11e6eea9e440d942624d704b
index-v2.327.0.js
ab159da6a2af4d98859963fbdad93b52929adb5f11e6eea9e440d942624d704b
index-v2.328.0.js
ab159da6a2af4d98859963fbdad93b52929adb5f11e6eea9e440d942624d704b
index-v2.329.0.js
fbcad57b299a699a29f1c750949073d2b9c7ad4b3b742ca3355947f99fcf7e56
index-v2.330.0.js
I've attached the relevant files for posterity. Especially the Github
Actions log output (that documents the file present in the actual Runner
image is) auto-deleted after some time.
---
I was surprised to learn Github Actions is based on minified JavaScript,
but more importantly it seems we have evidence about a disconnect
between Github's source code and what's actually in their Runner images.
I'm not sure if Github makes any claims about the exact versions used
for a given CI run, but this may be an instance of the "SBOM blind-spot"
that I've also pointed out just last month:
https://lists.reproducible-builds.org/pipermail/rb-general/2025-October/003893.html
Both Debian and Arch Linux follow the stricter "The build tools/compiler
are a material of your software executable" philosophy of SBOMs; they
fully document the build environment in their .BUILDINFO files, so had
this happened on their build servers, we would have more data points to
work with.
Exciting times,
kpcyrd
-------------- next part --------------
A non-text attachment was scrubbed...
Name: index-v2.329.0.js
Type: application/x-javascript
Size: 183884 bytes
Desc: not available
URL: <http://lists.reproducible-builds.org/pipermail/rb-general/attachments/20251123/360a3585/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: index-v2.330.0.js
Type: application/x-javascript
Size: 184207 bytes
Desc: not available
URL: <http://lists.reproducible-builds.org/pipermail/rb-general/attachments/20251123/360a3585/attachment-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: job-logs.js
Type: application/x-javascript
Size: 184193 bytes
Desc: not available
URL: <http://lists.reproducible-builds.org/pipermail/rb-general/attachments/20251123/360a3585/attachment-0002.bin>
More information about the rb-general
mailing list