Floor 4 of 5 of the tower
Conveyor
Big files fail in the middle. The pipeline should expect that.
The problem
Media processing breaks halfway: a huge upload, a timeout, a worker that dies. A script that just runs top to bottom loses the whole run and nobody knows where it stopped.
The rules I set
- Orchestration and processing are separate jobs.
- A job’s state is saved before the work starts, not after.
- Every failure has somewhere to go: a retry with a pause, then a queue a person can look at.
What I built
An intake that accepts files, a queue, worker lanes that run FFmpeg, object storage for sources and results, and a delivery step. A state board shows each job. Workers can fail and be restarted without anyone editing anything by hand.
How it’s layered
- Intake Accepts a file, creates a job, puts it in the queue.
- Workers Take one job each and run the transcoding.
- Storage Sources and results kept apart from processing and delivery.
Decisions
- Jobs are idempotent and keyed by their input, so running one twice is harmless.
- Retries wait a little longer each time, and the last attempt goes to a queue for a person.
- A retry resumes from the last finished step instead of starting again.
Where it stands
A demo system. The tower shows one job failing on a worker, going round the recovery lane and arriving on the second pass.
Node.js · TypeScript · Queues + workers · Supabase · AWS object storage · FFmpeg