Telling Technology · Learning in public
All episodes
EP.30 — CI GATE
Green on your machine proves nothing

The gate that blocks my own pushes

Every test passed locally. I pushed. Two minutes later GitHub mailed me seven red tests on a public repo. The tests weren't wrong — my laptop was too rich.

A talking-head frame with a translucent card listing the three tools the CI runner installs: pytest, detect-secrets and bandit
⌁ .githooks/pre-push · 3 checks · 1 throwaway venv · 0 red runs since
The fix isn't running the tests again.
01 The mail you don't want

Seven red tests, on a public repo, with my name on it

The commit was ordinary — a test file covering the money formulas in a budget engine. Locally: all green. I pushed. Two minutes later GitHub mailed a “Run failed” notice. Seven failures, none of which had happened on this machine, on a repo anyone can read.

The instinct is to read the failures as a bug in the code. They weren't. Every one of them was the same failure wearing different hats: a thing that exists here and does not exist there. Which makes them invisible to the one check everybody reaches for first — running the tests again.

github actions — the run that failed
# what the runner installs. the whole list.
$ pip install pytest detect-secrets bandit

# what my laptop has
requirements.txt   not installed on the runner
websockets         not installed on the runner
OPENROUTER_API_KEY not set on the runner

# result
7 failed in 2 test modules
both passed locally, seconds earlier
Nothing here is a logic error. The tests import things the runner was never going to have, and reach for a key it was never going to hold.
A talking-head frame with a pull-quote card reading green on my machine is not evidence of anything
the thesis
the sentence the whole build hangs on
02 Why the obvious fix is useless

Running the tests again locally reproduces the same blind spot

This machine has every library and every API key I have ever installed. The runner installs exactly three things: pytest, detect-secrets, bandit. No requirements.txt, no voice stack, no secrets. So a gate that runs the local interpreter inherits the exact blindness that caused the failure — it will pass, cheerfully, every time, right up until the runner disagrees.

So the hook doesn't use this machine's Python. It builds a cached virtualenv at .git/ci-venv holding exactly what the runner holds, and runs the same three checks in the same order against that. The point isn't to run the tests. It's to run them somewhere deliberately poorer than here.

.githooks/pre-push — the mirror
# we do not run the local interpreter.
VENV="$REPO_ROOT/.git/ci-venv"
DS_PIN="detect-secrets==1.5.0"   # pinned to CI

"$VENV_BIN/python" -m pip install --quiet \
    pytest "$DS_PIN" bandit

# same three checks, same order as ci.yml
[1/3] pytest tests/ -q
[2/3] git ls-files -z | xargs -0 \
      detect-secrets-hook --baseline .secrets.baseline
[3/3] bandit -r scripts/ --severity-level high

PUSH BLOCKED — CI would fail on: pytest
Builds itself on first push in about 30 seconds, then it's cached. The version pin matters: an older or newer scanner reports different findings against the same baseline.
A talking-head frame with a terminal-styled card showing the three checks and a PUSH BLOCKED verdict
3 checks · 1 verdict
the push never leaves the laptop
03 Two ways it fails silently

A guard that fails quietly is worse than no guard at all

Both of these were found before they bit, and both would have been invisible: the hook simply stops gating, and every push after that sails through looking exactly like a push that passed.

Line endings. With core.autocrlf=true, a checkout rewrites the hook's shebang to #!/bin/sh\r. There is no such interpreter, so the hook doesn't error — it doesn't run. Fixed by pinning .githooks/** to LF in .gitattributes. The executable bit. Git skips a non-executable hook on a fresh clone, silently. Fixed by committing it mode 100755.

the two silent modes
# 1 — the carriage return
#!/bin/sh     runs
#!/bin/sh\r   no such interpreter → hook skipped

# fix
.githooks/** text eol=lf     # .gitattributes

# 2 — the mode bit
100644 .githooks/pre-push    git skips it
100755 .githooks/pre-push    git runs it

# neither prints a warning. both look
# identical to a push that passed.
This is the reason the episode exists. The gate is easy; making its absence loud is the hard part.
A talking-head frame with a card comparing a working shebang against the same shebang carrying a carriage return, marked as silently never runs
one invisible character
worse than no gate at all
⚠️

The alternative I rejected was turning off the failure emails

That was genuinely tempting and it is the wrong move: it removes the notice, not the red run, and the notice is the backstop for everything the local gate cannot see — scheduled runs, pushes from another machine, anything that never touches this laptop. Silence the cause, not the alarm. There is also an escape hatch, SKIP_CI_GATE=1 git push, and it is honest about what it's for: pushing something you already know will fail, on purpose. A guard with no documented bypass gets bypassed in ways you don't find out about.

04 The rule that came out of it

If a test needs something CI doesn't have, it isn't a unit test

The gate stops bad pushes. It doesn't stop you writing a test that was always going to fail there. That needed a rule, and the rule is now in the repo's own instructions. Tap any image to enlarge it.

SHIPPED 2026-07-30 · 0 RED RUNS SINCE

The AI wrote the check that blocks the AI

Every push now runs an environment deliberately poorer than this one, and if any of the three checks fail, nothing leaves the laptop. The corollary rule went into the repo's instructions the same day: anything needing a package outside CI's three tools, a live key, or a billed API call is an integration test — guard it with pytest.importorskip or skipif so it skips on the runner instead of failing. Never move a test into tests/ after verifying it only on this machine. Red on main isn't caught after the fact anymore. It never gets out the door.

A talking-head frame with a card showing the file mode 100755 that a committed git hook needs
commits 53eacd1 (gate) · d21294a (LF pin) · 328c392 (mode bit) · e6307e6 (the test fix) · decision logged 2026-07-30
06 Steal this

Four rules for a pre-push gate that actually gates

The hook is welded to this repo's three checks, so the copyable thing is the shape. Everything here is free and open — clone it, run it, make it yours.

rule mirror the runner, not the laptop rule pin the tool versions rule make the bypass explicit rule integration tests skip, never fail
the pattern # 1. NEVER RUN THE LOCAL INTERPRETER. # Build a venv holding EXACTLY what CI installs # and run the checks in there. A green local run # proves nothing if your machine is richer than # the runner — which it always is. # 2. PIN THE TOOL VERSIONS TO CI's. # A different scanner version reports different # findings against the same baseline, so an # unpinned gate disagrees with CI at random. # 3. GIVE IT A DOCUMENTED BYPASS. # SKIP_CI_GATE=1 git push — for when you already # know CI fails and mean it. A guard with no # escape hatch gets bypassed invisibly instead. # 4. MAKE ITS ABSENCE LOUD. # Pin hooks to LF (a CRLF shebang silently skips # the hook) and commit them 100755 (git skips a # non-executable hook). Both fail SILENTLY — # which is worse than not having a gate at all.

Rule 1 is the whole episode: run your tests somewhere poorer than your laptop.

Next episode

The gate catches my mistakes. Who catches the gate's?

A pre-push hook is a piece of automation that decides whether other automation gets to ship. It runs on my machine, with my config, and the only thing verifying it is me remembering to check. Every safety net in this system has the same shape — and at some point one of them is going to fail quietly and I'm going to find out from a stranger.

A card comparing a working shebang line against one carrying an invisible carriage return
drops soon · follow so you don't miss it