Your Queue Workers Are Running Last Week's Code

A developer fixes a bug in a job, reviews it, merges it, watches the deploy go green, and the same failure appears in the log ten minutes later. They redeploy. It happens again. Somewhere around the third attempt the suspicion arrives that production is not running the code in the repository, and the suspicion is correct.

Why a deploy does not reach a worker

The request lifecycle in PHP is what makes this surprising. Every HTTP request boots the framework, does its work and exits, so new code is picked up by definition - the next request loads the new files because there was no old process to keep the old ones.

A queue worker is the opposite. php artisan queue:work boots the framework once and then loops, pulling jobs off the queue for as long as it lives. The classes it loaded at boot stay loaded. Replace every file underneath it and the running process neither knows nor cares: it holds its own compiled copy of your application, from whenever it started.

So after a deploy you have a web tier running the new release and a queue tier running whatever was current when those processes last started - which might be the previous release, or one from three weeks ago.

What that looks like when it goes wrong

The failure modes are recognisable once you know to look for them.

A bug you fixed keeps occurring, exclusively in queued work. A job calls a method that does not exist in the running code, or fails to call one that was just added. A migration adds a column, the controller writes to it happily, and the job that reads the same model throws because its copy of the schema predates the column.

Worst is the version split. Restart workers gradually - or let them cycle on their own memory limits - and for a while some processes run the new code and some run the old. Jobs are distributed between them arbitrarily. You now have failures that appear on roughly half the attempts and reproduce on none of them, which is the most expensive shape a bug can take.

The fix is one line in the deploy script

php artisan queue:restart

It does not kill anything. It writes a timestamp to the cache, and every worker checks that timestamp between jobs; a worker that booted before it finishes its current job and exits. Your process supervisor - systemd, Supervisor, whatever the platform provides - sees the exit and starts a fresh worker, which boots the new code.

Two things have to be true for it to work at all, and both are missed often enough to be worth stating.

The cache store has to be shared. The timestamp is written to the cache. Use the array driver and it goes into the memory of the process that ran artisan, which is not the worker. Use file with workers on another machine and the same thing happens. Redis, Memcached or the database - anything both sides can read.

Something has to restart the exited worker. queue:restart stops workers. It does not start them. Without a supervisor watching, a deploy quietly leaves you with no workers at all, and jobs pile up until somebody notices the queue depth.

Under Horizon the call is php artisan horizon:terminate, and Horizon brings its own supervision - but it still has to be told, because it cannot see your deploy.

Ordering matters more than the command

Where the restart sits in the script decides whether the window between old and new code is dangerous:

# new code in place first
php artisan migrate --force
php artisan config:cache
php artisan queue:restart

Restart before the new code is on disk and the fresh workers boot the old release, which is the failure you were trying to fix. Run migrations after the restart and new workers meet a schema that has not changed yet.

The harder case is a migration that removes something. A dropped column breaks old workers instantly, and there will be old workers for as long as their current jobs take to finish. Any schema change that takes something away wants splitting across two deploys: stop using it, ship, then drop it.

Confirming it worked

Do not trust the script. Ask the workers:

ps -eo lstart,cmd | grep "[q]ueue:work"

Start times older than the deploy mean the restart did not reach them, and that is worth knowing now rather than during the next incident. In Horizon, the same answer is on the dashboard under the supervisor's process list.

One line in a deploy script, and a class of bug that wastes entire afternoons stops existing.

This is one of four steps that decide whether a deploy is invisible; the other three are here. If the queue itself is what you cannot trust, with jobs vanishing, duplicating, or failing where nobody looks, we take that on as its own engagement.