Workers in production

The batch article ended on a debt. The jobs table gets reserved with FOR UPDATE SKIP LOCKED , the batch ledger with a bare FOR UPDATE , and the question of why one gets to skip while the other has to wait was left standing. Time to pay it. This is the article where we stop running one worker at a time and start running a fleet: two workers on one table, a kill -9 mid-job, clocks racing one another, and a process manager whose whole job is to notice dead workers. Everything below runs on Postgres, and for the first scenes the camera is not tinker. It is the database's own log.

The two queries#

Postgres can log every statement it executes. Turn that on and the queue has nowhere to hide:

ALTER SYSTEM SET log_statement = 'all'; SELECT pg_reload_conf();

Now dispatch one plain job and let a worker take it. Then dispatch a batch of one and let a worker finish it. Two runs, and the Postgres log catches both locks red-handed:

execute pdo_stmt_00000003: select * from "jobs" where "queue" = $1 and (("reserved_at" is null and "available_at" <= $2) or ("reserved_at" <= $3)) order by "id" asc limit 1 FOR UPDATE SKIP LOCKED

execute pdo_stmt_00000006: select * from "job_batches" where "id" = $1 limit 1 for update

The first line is the reserve query, the one every queue:work tick runs. It is worth reading slowly, because this single statement carries half the article. There is the obvious branch, reserved_at is null and available_at <= $2 , a job nobody holds. There is a second branch, reserved_at <= $3 , which takes jobs that somebody DOES hold. Park that one. There is order by "id" asc , so the queue is scanned first row first. And at the end, FOR UPDATE SKIP LOCKED .

The second line is the ledger decrement from the batch article, byte for byte, and its lock has no SKIP.

Who decides which lock you get? The driver does, by asking the database its own version:

if (($databaseEngine === 'mysql' && version_compare($databaseVersion, '8.0.1', '>=')) ||
    ($databaseEngine === 'mariadb' && version_compare($databaseVersion, '10.6.0', '>=')) ||
    ($databaseEngine === 'pgsql' && version_compare($databaseVersion, '9.5', '>=')) ||
    ($databaseEngine === 'vitess' && version_compare($databaseVersion, '19.0', '>='))
) {
    return $this->lockForPopping = 'FOR UPDATE SKIP LOCKED';
}

Postgres 9.5 and up, MySQL 8 and up, MariaDB 10.6 and up. Anything older falls through to a plain lock, and on SQLite even that compiles away to nothing, the same disappearing act the batch ledger did there. The skipping you are about to watch is a property of your database engine, not of Laravel.

Six orders, two workers#

Here is the actor for this act. One order in, one log line out, a second of packing:

<?php

namespace App\Jobs;

use Illuminate\Bus\Batchable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Queue\Queueable;
use Illuminate\Support\Facades\Log;

class ShipOrder implements ShouldQueue
{
    use Batchable, Queueable;

    public function __construct(public int $orderId) {}

    public function handle(): void
    {
        Log::info("shipped order #{$this->orderId} by pid ".getmypid());
        sleep(1);   // packing takes a second
    }
}

Queue six of them:

> foreach (range(1, 6) as $n) App\Jobs\ShipOrder::dispatch($n);

> DB::table('jobs')->count();

= 6

And start two workers, one in each terminal, at the same moment:

php artisan queue:work --stop-when-empty --sleep=0

2026-07-11 15:46:16 App\Jobs\ShipOrder .. RUNNING
2026-07-11 15:46:17 App\Jobs\ShipOrder .. 1s DONE
2026-07-11 15:46:17 App\Jobs\ShipOrder .. RUNNING
2026-07-11 15:46:18 App\Jobs\ShipOrder .. 1s DONE
2026-07-11 15:46:18 App\Jobs\ShipOrder .. RUNNING
2026-07-11 15:46:19 App\Jobs\ShipOrder .. 1s DONE

2026-07-11 15:46:16 App\Jobs\ShipOrder .. RUNNING
2026-07-11 15:46:17 App\Jobs\ShipOrder .. 1s DONE
2026-07-11 15:46:17 App\Jobs\ShipOrder .. RUNNING
2026-07-11 15:46:18 App\Jobs\ShipOrder .. 1s DONE
2026-07-11 15:46:18 App\Jobs\ShipOrder .. RUNNING
2026-07-11 15:46:19 App\Jobs\ShipOrder .. 1s DONE

Three jobs each, both finished by :19. Six seconds of packing took three seconds of wall clock. The log has the full registry, and every order appears exactly once:

shipped order #1 by pid 3032432
shipped order #2 by pid 3032433
shipped order #3 by pid 3032432
shipped order #4 by pid 3032433
shipped order #5 by pid 3032432
shipped order #6 by pid 3032433

Two pids, perfectly interleaved, no order shipped twice and no order skipped. Both workers ran the same reserve query against the same six rows at the same time, and never collided. That is SKIP LOCKED doing its job. But a clean result like this proves little on its own; the interleaving could be luck and timing. To show the skip itself, we have to stage it.

The skip, staged#

Two orders in the queue, and a tinker session that grabs the first row's lock and does not let go. Watch the ids:

> App\Jobs\ShipOrder::dispatch(201); App\Jobs\ShipOrder::dispatch(202); DB::table('jobs')->count();

= 2


> DB::table('jobs')->orderBy('id')->get(['id','available_at','reserved_at']);

= Illuminate\Support\Collection {#7475
    all: [
      {#7476
        +"id": 100,
        +"available_at": 1783786206,
        +"reserved_at": null,
      },
      {#7477
        +"id": 101,
        +"available_at": 1783786206,
        +"reserved_at": null,
      },
    ],
  }


> DB::beginTransaction();

= null


> DB::table('jobs')->orderBy('id')->limit(1)->lockForUpdate()->get(['id']);

= Illuminate\Support\Collection {#7407
    all: [
      {#7472
        +"id": 100,
      },
    ],
  }

Both rows available at the same second, neither reserved, and row 100 is first in the exact order the reserve query scans. Our open transaction now holds row 100 locked. Leave the session sitting exactly there and send a worker in from a second terminal. It has to arrive from outside: a worker started inside this session would run on this session's connection, inside this very transaction, and a transaction does not block itself. A lock only means something to somebody else.

php artisan queue:work --once

2026-07-11 16:10:18 App\Jobs\ShipOrder .. RUNNING
2026-07-11 16:10:19 App\Jobs\ShipOrder .. 1s DONE

One second, one job, and no waiting. Back in the session, which still has not let go:

> DB::table('jobs')->orderBy('id')->get(['id']);

= Illuminate\Support\Collection {#7438
    all: [
      {#7439
        +"id": 100,
      },
    ],
  }

One job is gone, and the survivor is row 100. The one we hold. The worker met the first row, found it locked, stepped over it and took row 101 instead. Release the lock and send another:

> DB::rollBack();

= null

php artisan queue:work --once

2026-07-11 16:10:25 App\Jobs\ShipOrder .. RUNNING
2026-07-11 16:10:26 App\Jobs\ShipOrder .. 1s DONE

> DB::table('jobs')->count();

= 0

shipped order #202 by pid 168685
shipped order #201 by pid 168706

Order #202 shipped before order #201, against the scan order, and the only difference between the two runs is a held lock. That is the skip, isolated. The worker does not queue up behind a busy row, because for a worker any row is as good as any other. Work is work.

The row that cannot be skipped#

Now the same trick against the batch ledger. A batch of one job, and our session grabs the ledger row instead of a jobs row:

> Illuminate\Support\Facades\Bus::batch([new App\Jobs\ShipOrder(301)])->dispatch(); DB::table('job_batches')->count();

= 1


> DB::beginTransaction();

= null


> DB::table('job_batches')->lockForUpdate()->get(['id','pending_jobs']);

= Illuminate\Support\Collection {#7424
    all: [
      {#7427
        +"id": "a23c027e-5f60-44c8-8232-8d03fc169f8f",
        +"pending_jobs": 1,
      },
    ],
  }

Same move, same second terminal. The worker takes the job from jobs without any trouble, runs the one second of packing, and then has to write its result into the ledger row we are holding. It prints its first line and stops dead there. Ask Postgres who is stuck:

php artisan queue:work --once

> sleep(4); DB::select("select pid, wait_event_type, left(query, 62) as query from pg_stat_activity where wait_event_type = 'Lock'");

= [
    {#7447
      +"pid": 263,
      +"wait_event_type": "Lock",
      +"query": "select * from \"job_batches\" where \"id\" = \$1 limit 1 for update",
    },
  ]

There it is. A worker, frozen, and the query it is frozen on is the ledger lock from the top of this article. Nothing to skip to; the group has one ledger row, and this is it. The worker waits until we let go:

> DB::commit();

= null


> sleep(2); DB::table('job_batches')->get(['pending_jobs']);

= Illuminate\Support\Collection {#7459
    all: [
      {#7455
        +"pending_jobs": 0,
      },
    ],
  }

And the second terminal, which had been sitting on that one line the whole time, finally finishes:

2026-07-11 16:10:53 App\Jobs\ShipOrder .. RUNNING
2026-07-11 16:11:08 App\Jobs\ShipOrder .. 14s DONE

Fourteen seconds DONE, on a job whose work is one second. All but one of them were spent standing in line for a single row. So here is the answer the batch article owed you. A worker skips busy jobs rows because the rows are interchangeable; it waits at the ledger because the ledger is the one row that means the group. Skipping is what you can do when identity does not matter. The moment a specific row is the point, you wait your turn.

The lease#

Back to the branch we parked, reserved_at <= $3 , the one that takes jobs somebody already holds. To watch it work we need a longer job and a bigger cast. Meet the slow one:

<?php

namespace App\Jobs;

use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Queue\Queueable;
use Illuminate\Queue\Attributes\Tries;
use Illuminate\Support\Facades\Log;

#[Tries(2)]
class SyncCatalog implements ShouldQueue
{
    use Queueable;

    public function __construct(public int $feedId) {}

    public function handle(): void
    {
        Log::info("catalog #{$this->feedId} sync started by pid ".getmypid());
        foreach (range(1, 12) as $page) usleep(1_000_000);   // twelve pages, a second each
        Log::info("catalog #{$this->feedId} sync finished by pid ".getmypid());
    }
}

Twelve pages, a second each, and two tries so the story has a second act. Dispatch it and let a worker take it:

> App\Jobs\SyncCatalog::dispatch(7); DB::table('jobs')->count();

= 1

Now kill that worker the way production kills workers. No warning, no cleanup. The job names its own executioner's target, because getmypid() is right there in the log:

catalog #7 sync started by pid 168946

kill -9 168946

The worker is gone mid-page. Now the important question. What did the queue see?

> DB::table('jobs')->get(['id','attempts','reserved_at']);

= Illuminate\Support\Collection {#7456
    all: [
      {#7458
        +"id": 103,
        +"attempts": 1,
        +"reserved_at": 1783786289,
      },
    ],
  }


> [now()->timestamp, DB::table('jobs')->value('reserved_at')];

= [
    1783786304,
    1783786289,
  ]

Nothing. The queue saw nothing. The row is still there, still stamped reserved_at = 1783786289 , attempts = 1 , fifteen seconds stale and counting. No error fired, no cleanup ran, nobody was notified, because the process that would have done any of that is the process that died. The two numbers in that stamp were written at reserve time, by the worker, on its way in:

protected function markJobAsReserved($job)
{
    $this->database->table($this->table)->where('id', $job->id)->update([
        'reserved_at' => $job->touch(),
        'attempts' => $job->increment(),
    ]);

    return $job;
}

So who brings the job back? Send another worker and find out:

> config('queue.connections.database.retry_after');

= 90


> DB::table('jobs')->get(['id','attempts','reserved_at']);

= Illuminate\Support\Collection {#7453
    all: [
      {#7451
        +"id": 103,
        +"attempts": 1,
        +"reserved_at": 1783786289,
      },
    ],
  }

Ninety seconds is the term. Now send a fresh worker in from the second terminal and watch what it does with a job somebody else already holds:

$ php artisan queue:work --once --stop-when-empty
$

It printed nothing at all. It looked, found no job it was allowed to touch, and left. The row has not moved:

> DB::table('jobs')->get(['id','attempts','reserved_at']);

= Illuminate\Support\Collection {#7504
    all: [
      {#7500
        +"id": 103,
        +"attempts": 1,
        +"reserved_at": 1783786289,
      },
    ],
  }


> [now()->timestamp, DB::table('jobs')->value('reserved_at') + 90];

= [
    1783786319,
    1783786379,
  ]

A worker came, looked, and left empty-handed. The row did not move. Not one field changed, because now is 1783786319 and the row only becomes fair game at reserved_at + 90 , which is 1783786379, sixty seconds away. The 90 is retry_after , and this is the second branch of the reserve query in action:

protected function isReservedButExpired($query)
{
    $expiration = Carbon::now()->subSeconds($this->retryAfter)->getTimestamp();

    $query->orWhere(function ($query) use ($expiration) {
        $query->where('reserved_at', '<=', $expiration);
    });
}

Read where $this->retryAfter lives. Not in the row. In the worker doing the asking. A reservation is a lease, and the lease term is set by the reader. The row has not moved. Now send a worker whose config says five instead of ninety:

DB_QUEUE_RETRY_AFTER=5 php artisan queue:work --once

> DB::table('jobs')->get(['id','attempts','reserved_at']);

= Illuminate\Support\Collection {#7404
    all: [
      {#7413
        +"id": 103,
        +"attempts": 2,
        +"reserved_at": 1783786322,
      },
    ],
  }

catalog #7 sync started by pid 168946
catalog #7 sync started by pid 169024

Taken. Fresh reserved_at , attempts bumped to 2, a second start in the log under a new pid. To the ninety-second worker this row was off limits; to the five-second worker it was expired and free. Both were right, by their own config. Keep that in mind the next time a deploy leaves two worker generations running with different retry_after values against one queue. The dead worker never came back and never will. What came back was the row, when a reader's clock said its lease had run out.

A lease shorter than the work#

The lease saved us when the worker died. Now the setup nobody plans on purpose, a lease shorter than the job is long. Set retry_after to five seconds and remember the sync takes twelve. First, with tries back at its default of one:

<?php

namespace App\Jobs;

use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Queue\Queueable;
use Illuminate\Support\Facades\Log;

class SyncCatalog implements ShouldQueue
{
    use Queueable;

    public function __construct(public int $feedId) {}

    public function handle(): void
    {
        Log::info("catalog #{$this->feedId} sync started by pid ".getmypid());
        foreach (range(1, 12) as $page) usleep(1_000_000);   // twelve pages, a second each
        Log::info("catalog #{$this->feedId} sync finished by pid ".getmypid());
    }
}

Worker A takes the job. Six and a half seconds later, while A is on page seven, worker B shows up:

DB_QUEUE_RETRY_AFTER=5 php artisan queue:work --once

2026-07-11 16:20:08 App\Jobs\SyncCatalog .. RUNNING
2026-07-11 16:20:20 App\Jobs\SyncCatalog .. 12s DONE

2026-07-11 16:20:15 App\Jobs\SyncCatalog .. RUNNING
2026-07-11 16:20:15 App\Jobs\SyncCatalog .. 7.66ms FAIL

Check B's clock against A's. B failed the job at 16:20:15, and A finished it, successfully, at 16:20:20, five seconds later. What did B even do in under eight milliseconds? It reserved the row, which bumped attempts to 2, and then the worker did the arithmetic it always does before running anything. Two attempts, one allowed. Off to the morgue, before handle() ever ran:

> Str::before(DB::table('failed_jobs')->value('exception'), ' in ');

= "Illuminate\\Queue\\MaxAttemptsExceededException: App\\Jobs\\SyncCatalog has been attempted too many times."

catalog #7 sync started by pid 3343707
catalog #7 sync finished by pid 3343707

Hold these two outputs next to each other. The morgue says the job died, attempted too many times. The log says the job ran once, started and finished by the same pid, no errors anywhere. Both are true. The sync completed and its corpse is in failed_jobs , because B charged an attempt for a run that never happened and the count crossed the limit while the real run was still going. One misconfigured clock, and your monitoring now believes a successful job failed.

Now give it the second try it had in the lease scene, #[Tries(2)] back on the class, and run the exact same race. This time B's arithmetic passes, two attempts, two allowed. So B runs it:

> DB::table('jobs')->get(['id','attempts','reserved_at']);

= Illuminate\Support\Collection {#7486
    all: [
      {#7485
        +"id": 124,
        +"attempts": 2,
        +"reserved_at": 1783786842,
      },
    ],
  }

$ ps -o pid=,cmd= -C php | grep 'queue:work'
3344023 php artisan queue:work --once
3344067 php artisan queue:work --once

One job row, attempts = 2 , and two live worker processes. Both of them are inside handle() for catalog #7 right now. Let them finish:

2026-07-11 16:20:35 App\Jobs\SyncCatalog .. RUNNING
2026-07-11 16:20:47 App\Jobs\SyncCatalog .. 12s DONE

2026-07-11 16:20:42 App\Jobs\SyncCatalog .. RUNNING
2026-07-11 16:20:54 App\Jobs\SyncCatalog .. 12s DONE

catalog #7 sync started by pid 3344023
catalog #7 sync started by pid 3344067
catalog #7 sync finished by pid 3344023
catalog #7 sync finished by pid 3344067

B went RUNNING at 16:20:42 while A did not finish until 16:20:47. Five seconds of overlap, two full executions of one dispatched job, and at the end jobs is empty, failed_jobs is empty, everything looks perfectly healthy. If this job charged a card instead of syncing a catalog, you just charged it twice and no table anywhere knows. The queue never promised you exactly-once; it promised at-least-once, and retry_after is where the "more than once" leaks in. The lease is a dead-worker detector with one fatal blind spot. It cannot tell a dead worker from a slow one.

The second clock#

What you want to say is "no job may run longer than its lease." There is an attribute for exactly that, and the limits article already showed its enforcer. It is the daemon, plain queue:work with no --once , the long-lived form that loops taking job after job until something stops it. Only the daemon arms the alarm, and it arms a real POSIX one before every job:

pcntl_signal(SIGALRM, function () use ($job, $options) {
    if ($job) {
        $this->markJobAsFailedIfWillExceedMaxAttempts(
            $job->getConnectionName(), $job, (int) $options->maxTries, $e = $this->timeoutExceededException($job)
        );
        // ...
    }

    $this->kill(static::$timedOutExitCode ?? static::EXIT_ERROR, $options, WorkerStopReason::TimedOut);
}, true);

Put #[Timeout(4)] on the twelve-second sync, tries at default, and let a daemon at it:

php artisan queue:work --stop-when-empty

2026-07-11 16:21:01 App\Jobs\SyncCatalog .. RUNNING
2026-07-11 16:21:05 App\Jobs\SyncCatalog .. 4s FAIL

> Str::before(DB::table('failed_jobs')->value('exception'), ' in ');

= "Illuminate\\Queue\\TimeoutExceededException: App\\Jobs\\SyncCatalog has timed out."

Four seconds in, the alarm fired, the worker filed the corpse itself and died. Compare that to the lease. When retry_after reclaimed a job, nothing was written anywhere; when the alarm killed this one, the worker signed the death certificate. Two clocks, and they leave different paperwork. Which means the fix for the double execution is not a bigger lease or fewer tries. It is an ordering. The alarm must fire before the lease expires, timeout strictly less than retry_after . Here is the same race as before, twelve-second job, five-second lease, #[Tries(2)] , and one change to the class, #[Timeout(4)] . Both workers run as daemons this time, because the alarm arms nowhere else:

<?php

namespace App\Jobs;

use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Queue\Queueable;
use Illuminate\Queue\Attributes\Timeout;
use Illuminate\Queue\Attributes\Tries;
use Illuminate\Support\Facades\Log;

#[Tries(2)]
#[Timeout(4)]
class SyncCatalog implements ShouldQueue
{
    use Queueable;

    public function __construct(public int $feedId) {}

    public function handle(): void
    {
        Log::info("catalog #{$this->feedId} sync started by pid ".getmypid());
        foreach (range(1, 12) as $page) usleep(1_000_000);   // twelve pages, a second each
        Log::info("catalog #{$this->feedId} sync finished by pid ".getmypid());
    }
}

Worker A takes the job at t=0. Its alarm is due at four, the lease runs out at five, and B arrives at six and a half. One tinker frame, caught between the alarm and B:

> [now()->timestamp, DB::table('jobs')->get(['id','attempts','reserved_at']), DB::table('failed_jobs')->count()];

= [
    1783786905,
    Illuminate\Support\Collection {#7558
      all: [
        {#7557
          +"id": 127,
          +"attempts": 1,
          +"reserved_at": 1783786899,
        },
      ],
    },
    0,
  ]

Six seconds after the reserve stamp, the row is still leased with attempts = 1 , and the morgue count is zero. A is already dead; its alarm fired at four. But look at what the alarm handler above actually does: it files the corpse only if this attempt would exceed the limit. One attempt of two would not, so the alarm killed the process and wrote nothing, the same silence the lease scene taught us to expect. The row simply sits there, waiting out its lease. Then B arrives:

2026-07-11 16:21:39 App\Jobs\SyncCatalog .. RUNNING

2026-07-11 16:21:45 App\Jobs\SyncCatalog .. RUNNING
2026-07-11 16:21:49 App\Jobs\SyncCatalog .. 4s FAIL

> Str::before(DB::table('failed_jobs')->value('exception'), ' in ');

= "Illuminate\\Queue\\TimeoutExceededException: App\\Jobs\\SyncCatalog has timed out."

catalog #7 sync started by pid 3344515
catalog #7 sync started by pid 3344571

A's output ends at RUNNING. No DONE, no FAIL, nothing; the alarm killed it mid-sentence. B took the expired lease at 16:21:45, six seconds after A started and two after A had already died, hit its own alarm at 16:21:49, and this time the arithmetic said final attempt, so B filed the corpse. Two starts in the log, zero finishes, one grave. The job still failed, twelve seconds of work does not fit in a four-second box no matter who runs it. But compare the shape of the failure to the last scene. There, two processes ran the same job at the same time. Here, B started two seconds after A was already gone. The timeout did not save the job. It saved you from the overlap, because a worker that kills itself before its lease expires can never share a job with its successor.

A worker is a process#

Every scene so far had the same villain, a worker dying at the wrong moment. Time to look at the deaths themselves. Strip the attributes off the sync and kill the worker politely this time, SIGTERM instead of SIGKILL, two seconds into the twelve:

kill -TERM 3344713

2026-07-11 16:22:05 App\Jobs\SyncCatalog .. RUNNING
2026-07-11 16:22:16 App\Jobs\SyncCatalog .. 11s DONE

catalog #7 sync started by pid 3344713
catalog #7 sync finished by pid 3344713

The TERM landed at second two. The worker kept going for nine more seconds, finished the sync, logged DONE, and only then exited. Eleven seconds instead of twelve, because the signal cut short the one-second page it interrupted; the loop carried on with the rest. That is the difference between a deploy that loses jobs and one that does not. SIGKILL takes the process mid-page and leaves a stale lease; SIGTERM sets a flag the daemon checks between jobs, so the current job lands safely and no lease is left behind.

There is a signal-free version of the same courtesy, and you have been told a hundred times to run it after every deploy. Here is what it actually is:

php artisan queue:restart

INFO Broadcasting queue restart signal.

> DB::table('cache')->where('key', 'like', '%queue:restart%')->get(['key','value']);

= Illuminate\Support\Collection {#7516
    all: [
      {#7515
        +"key": "laravel-cache-illuminate:queue:restart",
        +"value": "i:1783786952;",
      },
    ],
  }

A cache row. The broadcast is one timestamp written to your cache, and every daemon compares it to the timestamp it booted with, between jobs. That comparison is a third clock, and unlike the first two it never kills anything; it only tells a worker its generation is over. Run it while a daemon is two seconds into a sync with another job queued behind it:

2026-07-11 16:22:31 App\Jobs\SyncCatalog .. RUNNING
2026-07-11 16:22:43 App\Jobs\SyncCatalog .. 12s DONE

> DB::table('jobs')->count();

= 1

The daemon finished the job it was on, saw the flag, and left, with the second job still sitting in the queue. And that second job will now wait forever, because Laravel can stop workers gracefully all day long, but nothing in the framework brings a dead one back. queue:work is a process. Somebody has to run it, and somebody has to run it again after it exits, and that somebody is not Laravel.

In production that somebody is a process manager. Supervisor is the classic. Its config for a two-worker fleet fits on a screen:

[program:queue-worker]
command=php artisan queue:work
directory=/var/www/app
numprocs=2
process_name=%(program_name)s_%(process_num)02d
autorestart=true

$ supervisorctl status
queue-worker:queue-worker_00     RUNNING   pid 3042958, uptime 0:00:02
queue-worker:queue-worker_01     RUNNING   pid 3042959, uptime 0:00:02

Two workers, two pids, the fleet from the first scene as a managed service. Now murder one of them and ask again:

$ kill -9 3042958
$ supervisorctl status
queue-worker:queue-worker_00     RUNNING   pid 3043000, uptime 0:00:02
queue-worker:queue-worker_01     RUNNING   pid 3042959, uptime 0:00:05

Slot 00 is RUNNING again with a brand-new pid and its uptime reset to two seconds, while slot 01 kept its pid and its uptime just kept counting. The nanny noticed the death and refilled the slot, within a second, no Laravel involved. Note what Supervisor did not do. It did not release the dead worker's job; the lease still has to expire for that. It did not finish anything. It guarantees one thing: that the processes exist. Everything else in this article, the lease, the alarm, the graceful stop, assumes the workers are there to begin with, and this config file is the only reason they are.

One warning before you copy that file, because Supervisor has a clock of its own and it is aimed straight at your jobs. supervisorctl stop sends SIGTERM, the polite signal, and then waits stopwaitsecs for the process to leave. Ten seconds, by default. Stop a worker that has just started a twelve-second sync and count the waits:

$ supervisorctl stop 'queue-worker:*'
queue-worker:queue-worker_00: stopped

2026-07-11 16:28:52,282 INFO waiting for queue-worker_00 to stop
2026-07-11 16:28:54,283 INFO waiting for queue-worker_00 to stop
2026-07-11 16:28:56,285 INFO waiting for queue-worker_00 to stop
2026-07-11 16:28:58,287 INFO waiting for queue-worker_00 to stop
2026-07-11 16:29:00,289 INFO waiting for queue-worker_00 to stop
2026-07-11 16:29:02,292 WARN killing 'queue-worker_00' (3347767) with SIGKILL
2026-07-11 16:29:02,292 INFO waiting for queue-worker_00 to stop
2026-07-11 16:29:02,295 WARN stopped: queue-worker_00 (terminated by SIGKILL)

> DB::table('jobs')->get(['id','attempts','reserved_at']);

= Illuminate\Support\Collection {#7486
    all: [
      {#7485
        +"id": 134,
        +"attempts": 1,
        +"reserved_at": 1783787332,
      },
    ],
  }

catalog #9 sync started by pid 3347767

Ten seconds of patience, then the same SIGKILL the whole SIGTERM scene existed to avoid. A started sync with no finish, and a reserved row nobody owns, the stale lease from the second act, planted this time by the nanny itself. The cure is one line in the program block, stopwaitsecs=3600 , sized above your longest job. Stop the same sync under it:

2026-07-11 16:29:24,208 INFO waiting for queue-worker_00 to stop
2026-07-11 16:29:26,210 INFO waiting for queue-worker_00 to stop
2026-07-11 16:29:28,213 INFO waiting for queue-worker_00 to stop
2026-07-11 16:29:30,216 INFO waiting for queue-worker_00 to stop
2026-07-11 16:29:32,218 INFO waiting for queue-worker_00 to stop
2026-07-11 16:29:33,225 INFO stopped: queue-worker_00 (exit status 0)

catalog #9 sync started by pid 3347965
catalog #9 sync finished by pid 3347965

Same waits, no SIGKILL, exit status 0, and the sync finished before the worker left. A fourth clock, then, and it obeys the same rule as the other three: every outer clock must outlive what runs inside it. The timeout below the lease, the lease below nothing that matters, and Supervisor's patience above your longest job.

Eight articles ago a queue was one table you mail letters to yourself through. Now it is a fleet, and the fleet runs on four clocks and two queries: the lease on the row, the alarm on the process, the restart stamp on the generation, and the nanny's patience on the stop. Six lines worth keeping:

• Workers fan out because rows are interchangeable. The reserve query ends in FOR UPDATE SKIP LOCKED (Postgres 9.5+, MySQL 8+), so a busy row is stepped over. The batch ledger is one specific row, so there workers wait.
• A reservation is a lease, and the term lives in the reader. retry_after belongs to the worker's config, not the row. Different workers can disagree about whether the same row is free.
• The lease cannot tell dead from slow. If the job outlives retry_after , a second worker runs it while the first still is. With default tries you get a false corpse; with more tries you get a true double.
• The timeout is the other clock, and order is everything. Keep timeout below retry_after and a worker kills itself before its lease frees the job. The failure stays; the overlap dies.
• The two clocks leave different paperwork. An expired lease writes nothing, a non-final alarm writes nothing, and only a final failure reaches failed_jobs . Silence in the morgue does not mean nothing died.
• Laravel stops workers; it does not bring them back. SIGTERM and queue:restart (one cache row) end a daemon after its current job. Keeping daemons alive is Supervisor's autorestart=true , a file with no framework in it, and its stopwaitsecs must outlive your longest job or its stop turns into the SIGKILL you just escaped.