Hourly competitor price scrape, indexed three ways

You sell a B2B product. Two competitors update pricing on their public pages and you want the history. Not a one-off: every hour, forever.

You schedule one hoody-curl request per competitor with an hourly cron, save: true, and a save_path that includes the competitor name. Each run pulls the page, retries on transient failure, and writes the response into the curl service's downloads/ storage. The service symlinks every file into three index trees: by-job/, by-domain/, by-date/: so the same file appears under the run's job id, the source domain, and the date.

Six months later you want to chart "competitor A's pricing on page X over Q3." You walk by-domain/competitor-a.com/ filtered by date. The data is on disk, in your container, in a tree you didn't have to build.

The history isn't a job log. It's the data set.