Sixty containers on one server
One bare-metal box runs dozens to hundreds of Hoody containers. KSM and BTRFS dedup make the marginal cost near zero.
Containers·Snapshots
TypeUnlockedStageFleetDifficultyModerateJobShare streamForDev teamsForDevOps & infraServicesPipeWhy HoodyHTTP-nativeWhy HoodyContainer economics
TypeUnlockedStageFleetDifficultyModerateJobShare streamForDev teamsForDevOps & infraServicesPipeWhy HoodyHTTP-nativeWhy HoodyContainer economics
use-cases / push-one-build-to-thirty-ci-workers / hero
PIPE · CI FAN-OUT
Push one build to thirty CI workers at once
Your matrix CI fans out across thirty test runners. Each one needs the same 800 MB image. Stream the tarball into one pipe path with ?n=30. All thirty workers curl the same URL. The bytes go through once, the server holds nothing, and no registry credentials get rotated.
builder.ci
1 sender
# stream the image oncetar c ./image.tar | curl -T - https://pipe.hoody.com/build-12af?n=30PUT /pipe1× egress
?n=30one stream
ci-fleet
30 receivers01
02
03
04
05
06
07
08
09
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
all connected
GET /pipeONE SENDER ON THE LEFT. THIRTY GET RECEIVERS ON THE RIGHT. THE PIPE WAITS UNTIL EVERYONE IS LISTENING, THEN THE BYTES MOVE THROUGH ONCE.
use-cases / push-one-build-to-thirty-ci-workers / mechanism
How thirty curls become one stream
The pipe is a fan-out router with no disk. The sender's POST to /api/v1/pipe/[path]?n=30 blocks until thirty receivers connect to the same URL with the same n. Then the bytes flow from the build container straight through to every runner, simultaneously, at the speed of the slowest receiver.
Three steps. No registry, no S3, no cache action.PIPE · ?n=30
01
Build streams once
tar c | curl -T - https://pipe/.../build?n=30The build container pipes the tarball straight into curl. No file written, no registry pushed.
02
Pipe waits for the fleet
POST /api/v1/pipe/[path]?n=30The server holds the sender until all thirty receivers connect. Mismatched n returns 400. Pre-connected receivers are fine.
03
All thirty pull the same URL
curl https://pipe/.../build?n=30 | tar xEach runner gets identical bytes. Backpressure flows from the slowest receiver, not from the sender's bandwidth.
Nothing persists. Nothing is cached. The pipe brokers the connection, then steps out of the way. When the slowest runner finishes, the transfer finishes — and the URL is gone.
use-cases / push-one-build-to-thirty-ci-workers / numbers
What it costs the matrix
Naive: thirty registry pulls of the same 800 MB tarball, thirty cold caches, thirty network round-trips. Pipe: one egress, one transfer, the slowest receiver sets the pace.
WALL TIME
12s
One egress at line rate. Slowest receiver sets the pace, but no one re-downloads.
EGRESS
1× / build
Bytes leave the builder once, fan out at the pipe. No S3 GET fees, no Docker Hub pulls.
STORAGE
0 bytes
The pipe holds nothing on disk. No registry to clean up, no cache key to invalidate.
Wall-time figure assumes a 30-way matrix on the same regional network as the build container; cross-region transfers gate on inter-region bandwidth, not the pipe.
use-cases / push-one-build-to-thirty-ci-workers / powers
What the fan-out unlocks
Once the build is one URL and thirty curls, a stack of CI scaffolding goes away. No artifact storage to age out. No registry credentials to rotate. No cache action to debug.
The slowest receiver sets the pace
Backpressure is built into the pipe. The fast workers don't waste a registry round-trip waiting for the slow one — they wait at the pipe, then drink at the same rate. No one re-downloads.
No registry credentials to rotate
Nothing is pushed to a registry, so nothing has to authenticate to one. The URL itself is the credential — short-lived, scoped to one transfer, evicted when the build finishes.
No S3 bill, no egress surprise
Bytes leave the builder once. The pipe broadcasts. You pay one egress per build instead of thirty registry pulls per matrix run.
No cache key to invalidate
The pipe is per-build, not per-key. There is no GitHub Actions cache to mis-hit, no buildx layer mystery, no stale tarball from last week's main.
Works for any tarball, not just images
Same pattern handles node_modules, .pnpm-store, target/, the wheel cache, the dataset shard. If it streams, it fans out.
use-cases / push-one-build-to-thirty-ci-workers / punchline
One sender. Thirty receivers. Zero S3 bills.
A 30-way push that took ninety seconds and an S3 hit takes twelve seconds and a single egress. No one re-downloads. No registry credentials get rotated. The URL evicts itself when the matrix finishes.
- 1 EGRESS
- 30 RECEIVERS
- 0 STORAGE
- 0 CREDENTIALS
use-cases / push-one-build-to-thirty-ci-workers / replaces
What this replaces
The pieces a matrix-CI flow normally has to assemble — registry, cache action, mirror, custom upload step. The pipe folds them into one URL.
- AWS S3 (registry storage + egress)30× GET fees per build, eviction policy, IAM role
- GitHub Actions cache10 GB cap, key collisions, scope-per-branch
- Docker Hub pullsRate-limited, paid mirror to avoid throttling
- npm / pnpm registry mirrorSelf-hosted Verdaccio just to skip the public registry
- Custom CI cache actionBash glue, S3 SDK, expiry logic, on-call when it breaks
- Buildx layer cache exportLayer-format quirks, cross-runner cache misses
use-cases / push-one-build-to-thirty-ci-workers / cta
Stop pushing the same tarball thirty times. Push it once. Let thirty curls share the stream.
use-cases / push-one-build-to-thirty-ci-workers / related
Read the others
