Queue workers per lane
An API that puts jobs on a Redis list and two worker processes that take them, with a page that shows which worker finished each job. The workers have no port and finish the job in hand before they stop, and each lane keeps its own queue in a shared Redis.
Requirements
Section titled “Requirements”- devlanes, with
devlanes setuprun once - Node 22 or later
- Docker with Compose
Run it
Section titled “Run it”cd workflows-queue-workersdevlanes startdevlanes start --no-ui prints the start as text:
example-queue-workers@0 (main): 3/3 readyready api http://127.0.0.1:20570 https://example-queue-workers.localhostready worker.1 http://127.0.0.1:20571ready worker.2 http://127.0.0.1:20572Each replica gets a port it does not use. Open https://example-queue-workers.localhost. The page shows this lane’s queue, the two workers and a table of finished jobs that refreshes every second. “Enqueue jobs” adds jobs of 2 seconds each. The same queue answers curl:
curl -fsS -X POST "https://example-queue-workers.localhost/api/jobs?count=6" # add --ssl-revoke-best-effort on Windowscurl -fsS https://example-queue-workers.localhost/api/jobsDuring a start
Section titled “During a start”- Redis starts once in the shared Compose project
example-queue-workers(infra.project_name) on an ephemeral loopback port, whichREDIS_URLreads as${{ infra.redis.port }}. - The
installtask runsnpm installat the example root on the lane’s first start. apirunsnpm run devinapps/apiand is ready when/healthzanswers.workerruns as two units,worker.1andworker.2, each withWORKER_IDfrom${{ replica.index }}.ready.logisworker ${{ replica.index }} ready, so each replica waits for its own line. The worker’sdevscript runstsxwithoutwatch, becausetsx watchkills its child on a stop before the job in hand finishes.- Every process gets
QUEUE_PREFIXfromqueue_prefix: jobs:${{ lane.name }}, so lane 0 usesjobs:main.
After enqueuing 6 jobs, devlanes logs worker.1 shows the jobs this replica took. worker.2 took 2, 3 and 5:
worker.1#5 worker 1 ready, waiting on jobs:main:waitingworker.1#6 worker.1 started job 1worker.1#7 worker.1 finished job 1 in 2001 msworker.1#8 worker.1 started job 4worker.1#9 worker.1 finished job 4 in 2001 msworker.1#10 worker.1 started job 6worker.1#11 worker.1 finished job 6 in 2000 msTry next
Section titled “Try next”Start a second lane:
git worktree add ../examples-checkout -b feat/checkout, thencd ../examples-checkout/workflows-queue-workers && devlanes start. The page there shows lane 1 atexample-queue-workers1.localhost, with its ownjobs:<lane name>prefix.In the same folder,
devlanes lane new --name example-queue-workers-review --start --no-uigives a second lane without a worktree:created example-queue-workers@1 named example-queue-workers-review (main) at C:\Users\acme\examplesexample-queue-workers@1 (main): 3/3 readyready api http://127.0.0.1:20570 https://example-queue-workers1.localhostready worker.1 http://127.0.0.1:20571ready worker.2 http://127.0.0.1:20572Names given with
--nameare unique per folder across projects, so pick another one if it is taken.Enqueue jobs at https://example-queue-workers1.localhost. The lane’s
lane.nameislane1, so its queue isjobs:lane1and the jobs go to lane 1’s workers only. Redis holds both lanes’ keys:Terminal window docker compose -p example-queue-workers exec -T redis redis-cli --scan --pattern 'jobs:*'jobs:lane1:donejobs:lane1:next_idjobs:lane1:workersjobs:main:donejobs:main:next_idjobs:main:workersTry to freeze the lane with
devlanes debug freeze. The running workers keep it awake, so it refuses and exits 1:error: lane example-queue-workers@0 runs keep_awake services (worker.1, worker.2), so it stays awakehint: set services.worker.1.keep_awake and services.worker.2.keep_awake to false in the project config to let it freezeThe key to change is
services.worker.keep_awake; the hint names each replica. Withoutkeep_awake, a lane whose only traffic is jobs would freeze afterlanes.freeze_afterand its workers would stop taking jobs.Enqueue 10 jobs, then stop the workers while they work.
devlanes stop workeranswersworker not found, because each replica is its own unit. Select them by tag withdevlanes stop +worker:stopped example-queue-workers@0:worker.1 exit=1stopped example-queue-workers@0:worker.2 exit=1devlanes logs worker.1 worker.2 --tail 8ends with each worker finishing its job before it exits:worker.2#15 worker.2 got SIGINT, finishing job 9 firstworker.1#15 worker.1 got SIGINT, finishing job 10 firstworker.2#16 Terminate batch job (Y/N)?worker.1#16 Terminate batch job (Y/N)?worker.1#17 worker.1 finished job 10 in 2000 msworker.2#17 worker.2 finished job 9 in 2001 msworker.1#18 worker.1 stopped cleanly, jobs:main keeps the waiting jobs for the next startworker.2#18 worker.2 stopped cleanly, jobs:main keeps the waiting jobs for the next startThat log is from Windows, which has no POSIX signals. devlanes sends
SIGINTas Ctrl-C, and as Ctrl-Break for arun:string, whose programs ignore Ctrl-C. This example usesrun_script, so Node gets Ctrl-C and raises itsSIGINTevent as on macOS and Linux. npm’snpm.cmdruns undercmd.exe, which printsTerminate batch job (Y/N)?and makes the stop reportexit=1; the job still finishes.devlanes start +workerbrings both workers back, and they take the jobs that were left waiting. A worker that was killed mid-job puts that job back on the queue when it starts again.Drop the finished-job records with
devlanes run cleanup. Cron-like jobs are tasks, since a service that exits shows as failed:cleanup removed 16 finished jobs from jobs:main:doneClean lane 1 with
devlanes lane clean example-queue-workers-review --yes.infra.teardown.queuedeletes thejobs:lane1:*keys and leaves lane 0’s queue:cleaned example-queue-workers@1: stopped 3 units, teardown queueClean up everything:
devlanes lane clean 0 --yesdeletes lane 0’s keys. Redis outlives every lane, so remove it withdocker compose -p example-queue-workers down -v.
devlanes.yaml: the queue prefix inlanes.vars, the teardown entry,REDIS_URLandQUEUE_PREFIXinglobal_env, theinstallandcleanuptasks,apiand theworkerreplicas.docker-compose.yml: Redis with a pinned tag, an ephemeral loopback port and a healthcheck.packages/queue/: the typed queue both apps use, with every Redis key under the lane’s prefix.apps/api/src/: the Hono server with the page, the enqueue form and/api/jobs.apps/worker/src/index.ts: connects, takes jobs withBLMOVE, prints the ready line and finishes its job onSIGINT.apps/worker/src/scripts/cleanup.ts: thecleanuptask.