Skip to content

Fix recurring task logging as if it enqueued the same job twice#731

Open
ghiculescu wants to merge 1 commit intorails:mainfrom
ghiculescu:recurring-task-timing
Open

Fix recurring task logging as if it enqueued the same job twice#731
ghiculescu wants to merge 1 commit intorails:mainfrom
ghiculescu:recurring-task-timing

Conversation

@ghiculescu
Copy link
Copy Markdown
Member

A bug exists with recurring task enqueuing, where you will sometimes see doubled up [ActiveJob] Enqueued ... log lines. On Postgres these do not result in doubled up enqueueing, but they do result in confusing log lines. For example, given a QueueHeartbeatJob that's meant to run every minute:

2026-04-09T06:50:00Z app[e823931b293218] syd [info][ActiveJob] Enqueued QueueHeartbeatJob (Job ID: f3f21750-2d6b-416e-a1f3-5a1235744825) to SolidQueue(default)
2026-04-09T06:50:00Z app[e823931b293218] syd [info][ActiveJob] [QueueHeartbeatJob] [f3f21750-2d6b-416e-a1f3-5a1235744825] Performing QueueHeartbeatJob (Job ID: f3f21750-2d6b-416e-a1f3-5a1235744825) from SolidQueue(default) enqueued at 2026-04-09T06:50:00.002768396Z
2026-04-09T06:50:00Z app[e823931b293218] syd [info][ActiveJob] [QueueHeartbeatJob] [f3f21750-2d6b-416e-a1f3-5a1235744825] Performed QueueHeartbeatJob (Job ID: f3f21750-2d6b-416e-a1f3-5a1235744825) from SolidQueue(default) in 11.79ms
2026-04-09T06:51:00Z app[e823931b293218] syd [info][ActiveJob] Enqueued QueueHeartbeatJob (Job ID: e32694c9-eeb8-4079-bc7c-6c77a405fd6f) to SolidQueue(default)
2026-04-09T06:51:00Z app[e823931b293218] syd [info][ActiveJob] [QueueHeartbeatJob] [e32694c9-eeb8-4079-bc7c-6c77a405fd6f] Performing QueueHeartbeatJob (Job ID: e32694c9-eeb8-4079-bc7c-6c77a405fd6f) from SolidQueue(default) enqueued at 2026-04-09T06:51:00.001187865Z
2026-04-09T06:51:00Z app[e823931b293218] syd [info][ActiveJob] [QueueHeartbeatJob] [e32694c9-eeb8-4079-bc7c-6c77a405fd6f] Performed QueueHeartbeatJob (Job ID: e32694c9-eeb8-4079-bc7c-6c77a405fd6f) from SolidQueue(default) in 9.0ms
2026-04-09T06:52:00Z app[e823931b293218] syd [info][ActiveJob] Enqueued QueueHeartbeatJob (Job ID: e60a1384-4b46-4c97-94bc-7dd695d1cfdb) to SolidQueue(default)
2026-04-09T06:53:00Z app[e823931b293218] syd [info][ActiveJob] Enqueued QueueHeartbeatJob (Job ID: d5f795de-080c-47bd-88e2-0e33520aa470) to SolidQueue(default)
2026-04-09T06:53:00Z app[e823931b293218] syd [info][ActiveJob] [QueueHeartbeatJob] [d5f795de-080c-47bd-88e2-0e33520aa470] Performing QueueHeartbeatJob (Job ID: d5f795de-080c-47bd-88e2-0e33520aa470) from SolidQueue(default) enqueued at 2026-04-09T06:53:00.009200319Z
2026-04-09T06:53:00Z app[e823931b293218] syd [info][ActiveJob] [QueueHeartbeatJob] [d5f795de-080c-47bd-88e2-0e33520aa470] Performed QueueHeartbeatJob (Job ID: d5f795de-080c-47bd-88e2-0e33520aa470) from SolidQueue(default) in 15.97ms

To find the root cause, I deployed SQ with extra logging. If we include those logs and remove the job being performed it makes it easier to see what's going on:

2026-04-09T06:50:00Z app[e823931b293218] syd [info]schedule: QueueHeartbeatJob.perform_later() [ * * * * * ], delay = 59.998386254, next time = 2026-04-09 06:51:00 UTC (now = 2026-04-09 06:50:00 UTC, diff = 59.999722945)
2026-04-09T06:50:00Z app[e823931b293218] syd [info][ActiveJob] Enqueued QueueHeartbeatJob (Job ID: f3f21750-2d6b-416e-a1f3-5a1235744825) to SolidQueue(default)
2026-04-09T06:50:00Z app[e823931b293218] syd [info]SolidQueue-1.4.0 Enqueued recurring task (17.0ms)  task: "queue_heartbeat", active_job_id: "f3f21750-2d6b-416e-a1f3-5a1235744825", at: "2026-04-09T06:50:00Z"
2026-04-09T06:51:00Z app[e823931b293218] syd [info]schedule: QueueHeartbeatJob.perform_later() [ * * * * * ], delay = 59.999577626, next time = 2026-04-09 06:51:00 UTC (now = 2026-04-09 06:50:59 UTC, diff = 8.0566e-05)
2026-04-09T06:51:00Z app[e823931b293218] syd [info][ActiveJob] Enqueued QueueHeartbeatJob (Job ID: e32694c9-eeb8-4079-bc7c-6c77a405fd6f) to SolidQueue(default)
2026-04-09T06:51:00Z app[e823931b293218] syd [info]SolidQueue-1.4.0 Enqueued recurring task (19.1ms)  task: "queue_heartbeat", active_job_id: "e32694c9-eeb8-4079-bc7c-6c77a405fd6f", at: "2026-04-09T06:51:00Z"
2026-04-09T06:52:00Z app[e823931b293218] syd [info]schedule: QueueHeartbeatJob.perform_later() [ * * * * * ], delay = 59.99585098, next time = 2026-04-09 06:53:00 UTC (now = 2026-04-09 06:52:00 UTC, diff = 59.999882842)
2026-04-09T06:52:00Z app[e823931b293218] syd [info][ActiveJob] Enqueued QueueHeartbeatJob (Job ID: e60a1384-4b46-4c97-94bc-7dd695d1cfdb) to SolidQueue(default)
2026-04-09T06:52:00Z app[e823931b293218] syd [info]SolidQueue-1.4.0 Skipped recurring task – already dispatched (20.4ms)  task: "queue_heartbeat", at: "2026-04-09T06:51:00Z"
2026-04-09T06:53:00Z app[e823931b293218] syd [info]schedule: QueueHeartbeatJob.perform_later() [ * * * * * ], delay = 59.993155042, next time = 2026-04-09 06:54:00 UTC (now = 2026-04-09 06:52:59 UTC, diff = 60.000016247)
2026-04-09T06:53:00Z app[e823931b293218] syd [info][ActiveJob] Enqueued QueueHeartbeatJob (Job ID: d5f795de-080c-47bd-88e2-0e33520aa470) to SolidQueue(default)
2026-04-09T06:53:00Z app[e823931b293218] syd [info]SolidQueue-1.4.0 Enqueued recurring task (16.4ms)  task: "queue_heartbeat", active_job_id: "d5f795de-080c-47bd-88e2-0e33520aa470", at: "2026-04-09T06:53:00Z"

On line 1, next time = 2026-04-09 06:51:00 UTC - that's when we expect the next Concurrent::ScheduledTask to fire. On line 4 we see it's actually fired at 2026-04-09 06:50:59 UTC. As a result, the new next_time is incorrectly calculated as the current time. A job has already been enqueued for 06:51 so the second attempt will result in a uniqueness validation - but not before the [ActiveJob] Enqueued QueueHeartbeatJob line is logged.

This PR fixes this by calculating next_time in terms of when we thought the Concurrent::ScheduledTask would fire, not the actual time. This handles the case where it executes milliseconds earlier than we expected to.

@ghiculescu ghiculescu changed the title Fix recurring task double-enqueue Fix recurring task logging as if it enqueued the same job twice Apr 9, 2026
@ghiculescu ghiculescu force-pushed the recurring-task-timing branch from d549e95 to 0527244 Compare April 9, 2026 07:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant