Skip to content
use-cases / a-ci-cache-thats-just-two-curl-commands / hero
PIPE · CI · COST OPTIMIZE

A CI cache that's just two curl commands

The first job in your pipeline tars node_modules and pipes it to a Hoody URL. Twenty downstream jobs curl that same URL and untar. With ?n=20 the producer waits for all twenty workers to connect, then streams once — fanned out to all of them. No S3 bucket, no cache action, no egress bill.

Read the pipe docs
use-cases / a-ci-cache-thats-just-two-curl-commands / flow

The whole mechanic in two shell lines

There is no client library, no daemon, no SDK. The producer streams its tarball into PUT /pipe/cache?n=20. Each worker streams it back out of GET /pipe/cache?n=20. The pipe holds the bytes only while they're in flight — never on disk.

PUT · PRODUCERRUN ONCE PER BUILD

Produce the cache

$tar c node_modules|zstd|curl -T -
https://hoody.com/pipe/cache?n=20
# blocks until 20 receivers connect, then streams once

tar packs node_modules; zstd compresses on the fly; curl PUTs the bytes straight into the pipe path. No tempfile, no upload-artifact step, no bucket credentials.

GET · CONSUMERRUN ON EVERY WORKER

Consume the cache

$curlhttps://hoody.com/pipe/cache?n=20
| zstd -d | tar x
# fan-out: every worker gets an identical copy

Each test worker GETs the same URL, decompresses, and untars into its working directory. Slow workers apply backpressure to the producer but don't block faster ones.

THE WHOLE COORDINATION SURFACE?n=20

One query parameter. Producer and consumer agree on the same n. The pipe holds the upload until exactly that many receivers are connected, then opens the floodgate.

use-cases / a-ci-cache-thats-just-two-curl-commands / relief

What you stop paying for

CI caching used to be a tax: storage you'll overwrite tomorrow, egress every time a worker pulls, engineering time on the cache action's quirks. The pipe deletes all three line items at once.

No bucket. No egress bill.

There is no S3 bucket because there is no storage. The pipe forgets the bytes the second the transfer ends, so there is nothing to charge per-GB-month or per-GB-out for. The cache stops being a line item.

No cache action quirks

No yaml-encoded keys, no save-cache / restore-cache split, no debugging why a cache hit didn't happen on the right runner. Just curl. The same two lines run on GitHub Actions, BuildKite, Jenkins, your laptop, or a cron container.

No key collisions across branches

Branches just use different paths. /pipe/cache/main, /pipe/cache/feat-x, /pipe/cache/PR-742. Nothing to invalidate. Nothing to evict. When the branch dies, its path stops being asked for and that is the entire lifecycle.

use-cases / a-ci-cache-thats-just-two-curl-commands / compare

The cost shape, before and after

On a real workload — node_modules around 800 MB, twenty parallel test workers, a hundred CI runs a day — most of the bill is egress, not storage.

S3 + GH ACTIONS CACHE

20× egress

Each of the twenty workers pulls the cache out of S3. Twenty downloads of an 800 MB tarball is 16 GB of egress per CI run. The bucket itself is the easy part — the egress is what compounds.

VS
HOODY PIPE · ?n=20

1× transfer

The producer streams the 800 MB once. The pipe fans the bytes out to all twenty receivers in flight. One transfer through the wire, no per-receiver multiplier, no storage bill.

Numbers are illustrative for a typical Node monorepo cache. Actual savings depend on tarball size, worker fan-out, and the egress price your provider charges out of the cache region. The shape — linear vs. constant in worker count — is invariant.

use-cases / a-ci-cache-thats-just-two-curl-commands / punchline

The cache layer is HTTP. It always was. We just hadn't noticed.

Caches were never about storage. They were about getting the same bytes to N workers without rebuilding. HTTP already does that — once you let one URL fan out to a known number of receivers. The bucket was a workaround for the fan-out we didn't have.

BYTES

Stream once, fan out

One PUT, n GETs, identical bytes. Backpressure is per-receiver so the slow worker doesn't slow the fast ones.

PUT /pipe/cache?n=20
TIME

Live only in flight

The pipe holds the bytes for the transfer and forgets them when it ends. There is nothing to evict, nothing to lifecycle, nothing to back up.

TTL ≤ 5 min · then evicted
PATH

Branches are paths

Each branch picks its own path. No shared keyspace, no collisions. The path is the cache key and the URL at once.

/pipe/cache/[branch]
Read the pipe API
use-cases / a-ci-cache-thats-just-two-curl-commands / replaces

What this replaces

Most CI caches solve the same problem: get the same tarball to N workers. They do it through storage and egress. The pipe does it through the wire.

  • AWS S3 (cache bucket + egress)Storage you'll overwrite tomorrow + per-pull egress
  • GitHub Actions cacheYaml keys, runner-specific quirks, 10 GB ceiling
  • BuildJet / Garnix CI cachesFaster, still per-vendor, still per-byte
  • Bazel remote cacheExcellent if you're all-in on Bazel; heavy if you're not
  • Turborepo remote cacheVercel-hosted, monorepo-shaped, opinionated
  • Earthly satellite cacheAnother daemon, another bucket, another bill
  • Custom rsync cachesAn NFS box and an SSH key everyone forgets to rotate
use-cases / a-ci-cache-thats-just-two-curl-commands / cta

Two curl commands. One URL. Twenty workers fed.

Read the pipe docs
use-cases / a-ci-cache-thats-just-two-curl-commands / related

Read the others