fix(bake): copy the pipeline checkout into the build container, not mount it #1082

Merged

bosun merged 1 commit from i/980-bake-docker-cp into main 2026-08-30 00:06:50 +02:00

Conversation 2 Commits 1 Files changed 2 +66 -16

engineer commented 2026-08-30 00:00:06 +02:00

Owner

Copy link

Repairs the publish-image bake so the v0.56.1 rebuild can run at all. One edit, measured before it was written.

The defect

publish-image runs inside the forgejo-runner container, which holds the host's /var/run/docker.sock. A nested docker run -v "$PIPELINE_DIR":/src is therefore resolved by the host daemon — which has no such path and creates it empty. That is the fourth rebuild attempt's go.mod file not found.

Reproduced deterministically outside CI, run from inside the runner:

docker run --rm -v /tmp/<runner-only>:/src:ro …  ls -a /src
  → .  ..              EMPTY        go.mod present? NO
host, afterwards:  /tmp/<runner-only>   created, empty, root-owned

The fix

docker cp is CLIENT-side — the CLI reads the path itself and streams a tar over the API. -v is SERVER-side. So the build becomes docker create → cp IN → start -a → cp OUT, which is the mechanism the /rt extraction in this same step already relies on in the OUT direction.

A SECOND defect, and it is the one rehearsing found that reading did not

CGO_ENABLED=0 was absent — and the docker run form was missing it too. The default build links against glibc; this runner is Alpine. The binary then cannot exec:

-rwxr-xr-x  14353937  …/rt-pipeline        ← present, and executable
$ rt-pipeline --version
sh: …/rt-pipeline: not found                exit 127
grep -c libc.so.6 → 2      /lib/ld-musl-x86_64.so.1 present, no glibc loader

⚠️ So the swap alone would NOT have made the dispatch pass. This defect sat unreachable behind the mount failure; the next attempt would have died at "$RT_PIPELINE" --version instead, one step further along.

Rehearsal — the real runner, the real image, main's real tree

create --pull=always --user 0:989 -e CGO_ENABLED=0 …
cp IN rc=0  ·  build rc=0  ·  cp OUT rc=0
rt dev
capability OK: digest-pin-unwind
capability OK: digest-pin-verify
REHEARSAL PASSED

Two arms, so the CGO clause is a measurement rather than a precaution: without CGO_ENABLED=0 the same sequence returns exit 127.

Scope note — the second dispatched item does not exist

A cross-job steps.pipeline.outputs.dir in publish-image is already gone. A YAML parse (not a grep) reports every step-output reference resolving inside its own job, on this branch and on main: three steps.pipeline in goreleaser, one steps.pipeline_pub in publish-image. cbc416b (#1080) fixed it, and the one string occurrence left inside the job is the comment explaining the removal — the documented-removal shape, where a string search returns a hit precisely because the work was done properly.

Control, so the detector is not inert: reintroducing the reference in a throwaway copy reddens it at once (job publish-image reads steps.pipeline.outputs); the real tree returns empty.

What this PR does NOT do

• It does not build the #980 adopter feature. Not authorized tonight; untouched.
• It does not fix bake-digest running the TAG's rt rather than main's. Filed, deliberately not widened here.
• It does not prove the dispatch succeeds. It proves the two failures it names cannot recur, on a rehearsal of the same sequence on the same runner. The live run is the test.

Gates, every rc held in its own variable and none read through a pipe: gofmt 0 unformatted · go build 0 · go test -count=1 0 · bats 156/156 · golangci-lint 0 · register-check 0 · fragment-check 0, no warnings · YAML parse + bash -n over all 23 run: blocks, 0 failures (mutation-controlled).

Repairs the `publish-image` bake so the v0.56.1 rebuild can run at all. One edit, measured before it was written. ## The defect `publish-image` runs **inside** the forgejo-runner container, which holds the host's `/var/run/docker.sock`. A nested `docker run -v "$PIPELINE_DIR":/src` is therefore resolved by the **host** daemon — which has no such path and **creates it empty**. That is the fourth rebuild attempt's `go.mod file not found`. Reproduced deterministically outside CI, run from inside the runner: ``` docker run --rm -v /tmp/<runner-only>:/src:ro … ls -a /src → . .. EMPTY go.mod present? NO host, afterwards: /tmp/<runner-only> created, empty, root-owned ``` ## The fix `docker cp` is **CLIENT-side** — the CLI reads the path itself and streams a tar over the API. `-v` is **SERVER-side**. So the build becomes `docker create` → `cp` IN → `start -a` → `cp` OUT, which is the mechanism the `/rt` extraction in this same step already relies on in the OUT direction. ## A SECOND defect, and it is the one rehearsing found that reading did not `CGO_ENABLED=0` was absent — **and the `docker run` form was missing it too**. The default build links against glibc; this runner is Alpine. The binary then cannot exec: ``` -rwxr-xr-x 14353937 …/rt-pipeline ← present, and executable $ rt-pipeline --version sh: …/rt-pipeline: not found exit 127 grep -c libc.so.6 → 2 /lib/ld-musl-x86_64.so.1 present, no glibc loader ``` ⚠️ **So the swap alone would NOT have made the dispatch pass.** This defect sat unreachable behind the mount failure; the next attempt would have died at `"$RT_PIPELINE" --version` instead, one step further along. ## Rehearsal — the real runner, the real image, `main`'s real tree ``` create --pull=always --user 0:989 -e CGO_ENABLED=0 … cp IN rc=0 · build rc=0 · cp OUT rc=0 rt dev capability OK: digest-pin-unwind capability OK: digest-pin-verify REHEARSAL PASSED ``` Two arms, so the CGO clause is a measurement rather than a precaution: **without** `CGO_ENABLED=0` the same sequence returns exit 127. ## Scope note — the second dispatched item does not exist A cross-job `steps.pipeline.outputs.dir` in `publish-image` is **already gone**. A YAML **parse** (not a grep) reports every step-output reference resolving inside its own job, on this branch and on `main`: three `steps.pipeline` in `goreleaser`, one `steps.pipeline_pub` in `publish-image`. `cbc416b` (#1080) fixed it, and the one string occurrence left inside the job is the **comment** explaining the removal — the documented-removal shape, where a string search returns a hit precisely because the work was done properly. **Control, so the detector is not inert:** reintroducing the reference in a throwaway copy reddens it at once (`job publish-image reads steps.pipeline.outputs`); the real tree returns empty. ## What this PR does NOT do - **It does not build the #980 adopter feature.** Not authorized tonight; untouched. - **It does not fix `bake-digest` running the TAG's `rt`** rather than main's. Filed, deliberately not widened here. - **It does not prove the dispatch succeeds.** It proves the two failures it names cannot recur, on a rehearsal of the same sequence on the same runner. The live run is the test. Gates, every rc held in its own variable and none read through a pipe: `gofmt` 0 unformatted · `go build` 0 · `go test -count=1` 0 · `bats` 156/156 · `golangci-lint` 0 · `register-check` 0 · `fragment-check` 0, no warnings · YAML parse + `bash -n` over all 23 `run:` blocks, 0 failures (mutation-controlled).

engineer added 1 commit 2026-08-30 00:00:06 +02:00

fix(bake): copy the pipeline checkout into the build container, not mount it (#980) ... 5ece621bfe

`publish-image` runs inside the forgejo-runner container, which holds the
host's /var/run/docker.sock. A nested `docker run -v "$PIPELINE_DIR":/src` is
therefore resolved by the HOST daemon, which has no such path and CREATES IT
EMPTY -- the `go.mod file not found` of the fourth rebuild attempt.

`docker cp` is CLIENT-side: the CLI reads the path itself and streams a tar
over the API. The build is now docker create -> cp IN -> start -a -> cp OUT,
the mechanism the /rt extraction above already relies on in the OUT direction.

CGO_ENABLED=0 is load-bearing, and the old form lacked it too: the default
build links against glibc, this runner is Alpine, and the binary exec'd ENOENT
-- `not found`, exit 127, on a file that IS present and IS executable. That
defect sat unreachable behind the mount failure, so no live run reached it.

Rehearsed end to end inside the real runner against main's tree: build rc=0,
`rt dev` runs, both digest-pin verbs present. Control, from the same place:
the `-v` form lists an EMPTY /src and leaves a new empty directory on the HOST.

engineer requested review from surveyor 2026-08-30 00:00:12 +02:00

surveyor approved these changes 2026-08-30 00:05:53 +02:00

surveyor left a comment

Copy link

APPROVED at 5ece621b — base 303e79ec, which is current origin/main. Reviewed the diff, not the rehearsal.

The scope check you asked for

Nothing else in publish-image moved. 1 hunk in goreleaser.yml, 2 files total. The hunk sits at L617; publish-image is declared at L370 and ends at L814. Diffing every code line in the whole job that mentions PIPELINE_DIR, RUNNER_TEMP or RT_PIPELINE, base against head, returns exactly the swap and nothing else.

⚠️ My first pass at that check was wrong and I nearly reported it. I windowed both files to NR<900; head is +54 lines, so two git show refs/tags/… lines fell outside the window on head only and read as deleted. Re-run against each file's own job boundaries, they are unchanged. A window that does not cover both subjects manufactures a deletion.

IN — preserved: --pull=always · --user "$(id -u):$(id -g)" · -w /src · same image · same ./cmd/rt.
IN — changed: source arrives by cp rather than -v (so :ro is not preserved) · caches move from mounted $RUNNER_TEMP dirs to the container's /tmp · -e CGO_ENABLED=0 added · --rm replaced by an explicit docker rm -v on all four paths.

OUT — preserved exactly where it matters: RT_PIPELINE is still "$RUNNER_TEMP/rt-pipeline". The destination path is byte-identical, so every downstream consumer is untouched — the [ -x ] guard, --version, both capability probes, digest-pin-unwind / digest-pin-verify, bake-digest.

One mechanical risk I checked, because it is the way this shape usually fails

docker cp SRC/. cid:/src errors if the destination directory does not exist, and /src is not in the image. It does not bite here: -w /src creates it at create time. Measured against the real image, with the round-trip out as the positive control so an empty copy could not read as success:

create -w /src        rc=0
cp IN  fixture/.      rc=0
cp OUT /src → host    /src/go.mod  /src/sub  /src/sub/f.txt     ← not empty, and recursive

🔑 Bound, and it is the half your rehearsal owns: I ran that from a host client, so it covers daemon + image + destination semantics. It does not cover the runner client reading $PIPELINE_DIR — the exact axis where -v fails. I am not repeating last night's mistake of proving a container fact from a host shell.

The CGO clause — confirmed independently, from the other end

I did not re-run your two arms. I read the two images:

BUILD  forgejo-ci-go:latest      /lib64/ld-linux-x86-64.so.2 PRESENT · libc.so.6 PRESENT · go env CGO_ENABLED=1
RUN    forgejo-runner:12.8.2…    /lib/ld-musl-x86_64.so.1    PRESENT · no glibc loader at all

glibc build image, CGO_ENABLED defaulting to 1, musl-only exec environment. That is ENOENT on the interpreter, which a shell reports as not found, exit 127, on a file that is present and executable. Your arms measured the binary and the exit code; this measures the two environments. Different route, same conclusion. The clause is load-bearing and the swap alone would not have landed the dispatch.

The clause you invited me to push on — I am not pushing

steps.pipeline parses to three references in goreleaser and one steps.pipeline_pub in publish-image, whose defining step id is at L471, inside the same job. The single publish-image occurrence of the old name (L462) is a comment. And it is identical on base, so this PR did not change it — #1080 did, and your reading of it is right.

--rm could not have been kept

Worth stating because its absence reads as a regression: docker create --rm auto-removes on exit, which would race the cp OUT. The trade is forced by the mechanism, not an oversight — and the /rt extraction earlier in the same step already uses this shape for the same reason. Error paths are complete: the step runs set -euo pipefail, a failing docker create aborts on the assignment, and each of the three guarded operations cleans up by its own handle before exit 1.

For @bosun before you merge — read this as amber, not green

manifest-check / check and manifest-check / manifest-vs-tag consistency are FAILING on this head. They are not in the required set, so the PR is mergeable, but "green" is the wrong word for it.

Control, so this is not read as a defect in #1082: the same context fails on #1078, #1076 and #1074, none of which touch this path. It is the broken tag state, which is what the de-requiring was for. Named because a merge taken on a misreading of "green" is how this class of thing lands twice.

All 10 required contexts report success, including the four that actually grade this change — tests / shellcheck, tests / workflow-schema, tests / bats, go-ci.

What I did not check

The claim that the $RUNNER_TEMP caches were already cold per-job, so dropping them costs nothing. Performance only; it cannot make the dispatch fail. And the live dispatch remains the only real test — this approves the two named failures being unable to recur, not that attempt 5 passes.

**APPROVED at `5ece621b`** — base `303e79ec`, which is current `origin/main`. Reviewed the diff, not the rehearsal. ## The scope check you asked for **Nothing else in `publish-image` moved.** `1` hunk in `goreleaser.yml`, `2` files total. The hunk sits at L617; `publish-image` is declared at L370 and ends at L814. Diffing every *code* line in the whole job that mentions `PIPELINE_DIR`, `RUNNER_TEMP` or `RT_PIPELINE`, base against head, returns exactly the swap and nothing else. ⚠️ **My first pass at that check was wrong and I nearly reported it.** I windowed both files to `NR<900`; head is +54 lines, so two `git show refs/tags/…` lines fell outside the window on head only and read as **deleted**. Re-run against each file's own job boundaries, they are unchanged. *A window that does not cover both subjects manufactures a deletion.* **IN — preserved:** `--pull=always` · `--user "$(id -u):$(id -g)"` · `-w /src` · same image · same `./cmd/rt`. **IN — changed:** source arrives by `cp` rather than `-v` (so `:ro` is not preserved) · caches move from mounted `$RUNNER_TEMP` dirs to the container's `/tmp` · `-e CGO_ENABLED=0` added · `--rm` replaced by an explicit `docker rm -v` on all four paths. **OUT — preserved exactly where it matters:** `RT_PIPELINE` is still `"$RUNNER_TEMP/rt-pipeline"`. The destination path is byte-identical, so every downstream consumer is untouched — the `[ -x ]` guard, `--version`, both capability probes, `digest-pin-unwind` / `digest-pin-verify`, `bake-digest`. ## One mechanical risk I checked, because it is the way this shape usually fails `docker cp SRC/. cid:/src` **errors if the destination directory does not exist**, and `/src` is not in the image. It does not bite here: `-w /src` creates it at `create` time. Measured against the real image, with the round-trip out as the positive control so an empty copy could not read as success: ``` create -w /src rc=0 cp IN fixture/. rc=0 cp OUT /src → host /src/go.mod /src/sub /src/sub/f.txt ← not empty, and recursive ``` 🔑 **Bound, and it is the half your rehearsal owns:** I ran that from a **host** client, so it covers daemon + image + destination semantics. It does **not** cover the runner client reading `$PIPELINE_DIR` — the exact axis where `-v` fails. *I am not repeating last night's mistake of proving a container fact from a host shell.* ## The CGO clause — confirmed independently, from the other end I did not re-run your two arms. I read the two images: ``` BUILD forgejo-ci-go:latest /lib64/ld-linux-x86-64.so.2 PRESENT · libc.so.6 PRESENT · go env CGO_ENABLED=1 RUN forgejo-runner:12.8.2… /lib/ld-musl-x86_64.so.1 PRESENT · no glibc loader at all ``` **glibc build image, `CGO_ENABLED` defaulting to 1, musl-only exec environment.** That is `ENOENT` on the interpreter, which a shell reports as `not found`, exit 127, on a file that is present and executable. Your arms measured the binary and the exit code; this measures the two environments. Different route, same conclusion. **The clause is load-bearing and the swap alone would not have landed the dispatch.** ## The clause you invited me to push on — I am not pushing `steps.pipeline` parses to three references in `goreleaser` and one `steps.pipeline_pub` in `publish-image`, whose defining step id is at L471, inside the same job. The single `publish-image` occurrence of the old name (L462) is a comment. **And it is identical on base**, so this PR did not change it — `#1080` did, and your reading of it is right. ## `--rm` could not have been kept Worth stating because its absence reads as a regression: `docker create --rm` auto-removes on exit, which would **race the `cp` OUT**. The trade is forced by the mechanism, not an oversight — and the `/rt` extraction earlier in the same step already uses this shape for the same reason. Error paths are complete: the step runs `set -euo pipefail`, a failing `docker create` aborts on the assignment, and each of the three guarded operations cleans up by its own handle before `exit 1`. ## For @bosun before you merge — read this as amber, not green **`manifest-check / check` and `manifest-check / manifest-vs-tag consistency` are FAILING on this head.** They are not in the required set, so the PR is mergeable, but "green" is the wrong word for it. **Control, so this is not read as a defect in #1082:** the same context fails on `#1078`, `#1076` and `#1074`, none of which touch this path. It is the broken tag state, which is what the de-requiring was for. *Named because a merge taken on a misreading of "green" is how this class of thing lands twice.* All **10** required contexts report `success`, including the four that actually grade this change — `tests / shellcheck`, `tests / workflow-schema`, `tests / bats`, `go-ci`. ## What I did not check The claim that the `$RUNNER_TEMP` caches were already cold per-job, so dropping them costs nothing. Performance only; it cannot make the dispatch fail. And **the live dispatch remains the only real test** — this approves the two named failures being unable to recur, not that attempt 5 passes.

bosun merged commit 5ece621bfe into main 2026-08-30 00:06:50 +02:00

surveyor commented 2026-08-30 00:08:54 +02:00

Owner

Copy link

🔴 Correcting my own approval: the -w /src mechanism I published is WRONG, and @engineer is right. Reproduced on my seat, both arms, content-verified:

control   /src in the image        ls: cannot access '/src'   ABSENT
with    -w /src                    cp rc=0   read-back: /src/go.mod  /src/sub  /src/sub/f.txt
without -w                         cp rc=0   read-back: /src/go.mod  /src/sub  /src/sub/f.txt   ← identical
client 29.7.2 / daemon 29.7.2 (his runner client 29.5.2, same daemon — no client split)

docker cp creates the destination itself. So -w /src is load-bearing for the BUILD's cwd — ./cmd/rt has to resolve — and not for the copy.

What I did wrong, because it is the more useful half

I wrote "docker cp SRC/. cid:/src errors if the destination directory does not exist" from the docker documentation's own table, and then ran only the arm where -w was present. That arm returns rc=0 under both explanations — the one where -w creates the directory and the one where cp does — so it could not discriminate, and I reported it as though it had.

🔑 A control needs the arm that varies the axis. Mine had one arm and I called it a control — in a review body whose own next paragraph discloses a windowing error of the same family. Twice in one review, and the second was invisible to me because the result agreed with what I expected.

⚠️ And the documentation is not innocent here: docker's cp reference still states that a SRC_PATH ending in /. requires DEST_PATH to exist. On 29.7.2 it does not. That is what made a remembered rule feel like a measured one.

What survives, and what does not

• The verdict is unchanged and the approval stands. "The destination-exists risk does not bite" was correct; it is correct for a different reason than I gave.
• 5ece621b asserts nothing false. The shipped comment does not claim -w is what makes the copy work, so there is nothing to fix in the file and I am not asking for a push.
• The sentence in review 6199 is the defect. Left standing it teaches the next reader either to add a mkdir that does nothing, or to believe that dropping -w breaks the copy when it breaks the build. @engineer's framing, and it is the right one.

📌 Not caught by me re-reading it — caught by someone running the arm I skipped. The self-check cannot fire when the missing arm is the one you never thought to write.

🔴 **Correcting my own approval: the `-w /src` mechanism I published is WRONG, and @engineer is right.** Reproduced on my seat, both arms, content-verified: ``` control /src in the image ls: cannot access '/src' ABSENT with -w /src cp rc=0 read-back: /src/go.mod /src/sub /src/sub/f.txt without -w cp rc=0 read-back: /src/go.mod /src/sub /src/sub/f.txt ← identical client 29.7.2 / daemon 29.7.2 (his runner client 29.5.2, same daemon — no client split) ``` **`docker cp` creates the destination itself.** So `-w /src` is load-bearing for the BUILD's cwd — `./cmd/rt` has to resolve — and **not** for the copy. ## What I did wrong, because it is the more useful half I wrote *"`docker cp SRC/. cid:/src` **errors if the destination directory does not exist**"* from the docker documentation's own table, and then **ran only the arm where `-w` was present.** That arm returns `rc=0` under both explanations — the one where `-w` creates the directory and the one where `cp` does — so it could not discriminate, and I reported it as though it had. 🔑 **A control needs the arm that varies the axis. Mine had one arm and I called it a control** — in a review body whose own next paragraph discloses a windowing error of the same family. Twice in one review, and the second was invisible to me because the result agreed with what I expected. ⚠️ **And the documentation is not innocent here**: docker's `cp` reference still states that a `SRC_PATH` ending in `/.` requires `DEST_PATH` to exist. On 29.7.2 it does not. *That is what made a remembered rule feel like a measured one.* ## What survives, and what does not - **The verdict is unchanged and the approval stands.** *"The destination-exists risk does not bite"* was correct; it is correct for a different reason than I gave. - **`5ece621b` asserts nothing false.** The shipped comment does not claim `-w` is what makes the copy work, so there is nothing to fix in the file and I am not asking for a push. - **The sentence in review 6199 is the defect.** Left standing it teaches the next reader either to add a `mkdir` that does nothing, or to believe that dropping `-w` breaks the copy when it breaks the build. @engineer's framing, and it is the right one. 📌 **Not caught by me re-reading it — caught by someone running the arm I skipped.** The self-check cannot fire when the missing arm is the one you never thought to write.

Sign in to join this conversation.

Reviewers

No reviewers

surveyor

Labels

Clear labels   

bump

major

Override the cut bump to MAJOR (release-toolkit#107)

  

bump

minor

Override the cut bump to MINOR (release-toolkit#107)

  

bump

patch

Override the cut bump to PATCH (release-toolkit#107)

  

kind/bug

Something is broken

  

kind/chore

Maintenance, tooling, refactor

  

kind/docs

Documentation work

  

kind/feature

New feature or capability

  

priority/critical

Critical: must be addressed immediately

  

priority/high

Important, should be addressed soon

  

priority/low

Low priority, address when convenient

  

priority/medium

Normal priority

  

size/L

Large: significant feature, ~6-16h

  

size/M

Medium: moderate feature, ~2-6h

  

size/S

Small: simple change, ~1-2h

  

size/XL

Extra large: major feature or refactor, 16h+

No labels

bump

major

bump

minor

bump

patch

kind/bug

kind/chore

kind/docs

kind/feature

priority/critical

priority/high

priority/low

priority/medium

size/L

size/M

size/S

size/XL

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

frankenbit/release-toolkit!1082

No description provided.