Two thousand PDFs, async, results land in storage
Two thousand URLs of regulatory filings. You need them on disk, organized, recoverable if anything dies mid-run. The wrong way is a Python script with requests and a tqdm bar that loses everything on a network blip.
You loop through the URLs and POST each to /api/v1/curl/request with mode: "async" and save: true. Each call returns a job id immediately. Downloads run in the background through libcurl, with retries built in. Files land in the curl service's downloads/ storage, symlinked into by-job/, by-domain/, and by-date/: three views of the same files.
You come back two hours later. List /api/v1/curl/storage and see two thousand entries with size, content-type, source domain. The half-dozen that failed show up in /api/v1/curl/jobs with the error message. Re-fire those.
The download queue is the API. The output directory is the URL.