Skip to content
#0983
FilesContainers

The disk is in front, the bucket is behind, and the path never changes

The pipeline writes 40 GB of parquet a week into /data. The NVMe is fast and finite; the bucket is slow and endless. Every quarter someone moves old months to the bucket and fixes the paths that broke.

One POST to /api/v1/backends/union ends the ritual. Upstreams: the local disk and the S3 remote. create_policy stays at epmfs, so new files land on the upstream with the most free space; min_free_space reserves 50 GB on the NVMe; search_policy: ff reads from the first upstream that has the file. Mount it at /data.

The pipeline keeps writing to /data/2026/09/. Hot files sit on the disk; when it nears its floor, new files go to the bucket. Reads find them either way. Nothing in the code knows there are two tiers.

Tiered storage used to be a migration. Now it's a policy on a folder.