A Laravel job that runs perfectly locally with queue:work on the sync driver tells you almost nothing about how it behaves in production. In production, workers crash, jobs get retried automatically after a failure (sometimes several times), and the same job can run twice for reasons that have nothing to do with the business logic itself. Without an explicit strategy for retries, idempotency and permanent-failure handling, the result is duplicated side effects - an email sent three times, a card charged twice - that surface as support tickets rather than technical alerts.
Why a job that "works in dev" fails in production
In development, a job runs once, synchronously or through a single manually started worker, never interrupted. In production, several mechanisms change the picture: a deployment that restarts workers mid-processing, a PHP timeout that kills a job running too long, a transient network exception to a third-party API, or simply several workers processing the same queue in parallel. Laravel automatically retries a failed job according to its retry configuration - exactly the desired behavior for a transient error, and exactly the problem if the job wasn't designed to be safely replayed.
Retries: the actual mechanism
Each job can define its number of attempts, its backoff strategy between attempts, and a time limit after which it's considered stuck:
class ChargeCustomerInvoice implements ShouldQueue
{
use Queueable;
public int $tries = 5;
public int $maxExceptions = 3;
public function backoff(): array
{
// Increasing delay between attempts: 10s, 30s, 60s, 120s, 300s
return [10, 30, 60, 120, 300];
}
public function retryUntil(): \DateTime
{
// After this point, the job is abandoned even with attempts left
return now()->addHours(2);
}
public function handle(PaymentGateway $gateway): void
{
$gateway->charge($this->invoice);
}
public function failed(\Throwable $exception): void
{
// Called exactly once, after the job has permanently failed
Log::error('Invoice charge permanently failed', [
'invoice_id' => $this->invoice->id,
'exception' => $exception->getMessage(),
]);
}
}
The classic trap is leaving the defaults ($tries = 1, no backoff) on a job that calls an external API prone to transient errors - it fails for good on the first network blip, never getting a second chance.
Idempotency: why a retry without it is dangerous
A retry implicitly assumes that replaying the job produces the same outcome as if it had never failed in the first place. That's true for a read operation, false for anything with an external side effect. If ChargeCustomerInvoice succeeded on the payment gateway's side but the job failed after that point (network timeout waiting for the response, worker crash), the next retry will charge the customer a second time.
public function handle(PaymentGateway $gateway): void
{
// The idempotency key must stay stable across every attempt
// of the same job, not regenerated on each run
$idempotencyKey = "invoice-charge-{$this->invoice->id}";
$lock = Cache::lock("charge-lock-{$this->invoice->id}", 30);
if (! $lock->get()) {
// Another worker is already handling this invoice, retry later
$this->release(15);
return;
}
try {
if ($this->invoice->refresh()->status === 'paid') {
// Already processed by a previous attempt despite the apparent failure
return;
}
$gateway->charge($this->invoice, idempotencyKey: $idempotencyKey);
$this->invoice->update(['status' => 'paid']);
} finally {
$lock->release();
}
}
Most payment gateways (Stripe, for instance) accept an idempotency key on their API, guaranteeing that a repeated call with the same key never charges twice, even if the network dropped the original response. On the application side, the Cache::lock and the check against already-persisted state are the necessary complement, because the external gateway offers no protection against two Laravel workers processing the same invoice in parallel.
Dead letters: where permanently failed jobs go
When a job exhausts its attempts, Laravel writes it to the failed_jobs table (or its Horizon/Redis equivalent) instead of silently dropping it. That table is Laravel's dead letter queue - jobs land there as permanent failures but stay inspectable and replayable:
# List permanently failed jobs
php artisan queue:failed
# Replay a specific job after fixing the underlying bug
php artisan queue:retry 15
# Replay all failed jobs within a given range
php artisan queue:retry --range=10,25
# Discard a job you no longer want to replay
php artisan queue:forget 15
The weak point of this mechanism, often discovered too late, is that a job in failed_jobs alerts nobody by default. Without an explicit integration - a listener on the JobFailed event, a Horizon alert, or a webhook to Slack/PagerDuty - critical jobs can pile up silently for days.
Diagnosing in production
Three signals worth watching before a customer incident reveals them instead:
- Queue size (
horizon:statusorphp artisan queue:monitor emails,payments --max=100), to catch a backlog growing faster than it drains. - Workers restarting in a loop - often a sign of a memory leak caused by missing
--max-jobsor--max-time, which prevents the PHP worker from recycling periodically. - Jobs stuck longer than their configured
timeout, which points to a slow external dependency rather than a code bug.
; Typical Supervisor configuration for resilient workers
[program:laravel-worker]
command=php /var/www/artisan queue:work redis --queue=payments --tries=5 --max-jobs=1000 --max-time=3600
autostart=true
autorestart=true
numprocs=4
stopwaitsecs=30
The fix, and its tradeoffs
The Redis driver combined with Horizon gives real-time visibility (throughput, latency, failure rate) that a database driver doesn't provide as well, at the cost of an extra infrastructure dependency. The WithoutOverlapping middleware prevents the same logical job (same invoice, same user) from running twice across two workers, but introduces processing latency if the lock stays held longer than expected. Finally, explicitly distinguishing "retryable" exceptions (network timeout, 5xx error) from "permanent" ones (business validation, declined card) through dedicated exception classes avoids burning attempts on an error that will never resolve itself by replaying the job.
public function handle(PaymentGateway $gateway): void
{
try {
$gateway->charge($this->invoice);
} catch (CardDeclinedException $e) {
// Permanent error: never retry, fail immediately
$this->fail($e);
} catch (GatewayTimeoutException $e) {
// Transient error: let the standard retry mechanism take over
throw $e;
}
}
When retries and dead letters aren't enough
For a multi-step business process (reserve stock, then charge the customer, then trigger shipping), naively retrying the entire job after a failure at step three can replay the first two steps, which already succeeded. The right answer is an explicit compensation pattern - each step knows how to undo its own effect (release_stock, refund) rather than relying on the global retry to "start from zero" without a double effect.
Another edge case is the "poison" job: a corrupted payload, or a job class renamed after a deployment, that fails systematically during deserialization, before even reaching handle(). Such a job burns through its attempts instantly and pollutes failure metrics without ever being fixable through a simple queue:retry. Prevention comes from careful versioning of job classes and from using Bus::batch() with then/catch callbacks to isolate partial failures within a batch, rather than letting a single corrupted job take down monitoring for the whole batch.
Takeaway
A reliable queue system in production rests on three inseparable pillars: retries configured per error type (not a single default value), idempotency guaranteed at both the job and the external resource level, and active monitoring of the dead letter queue rather than a failed_jobs table checked only after the incident.
