[4.x]: 'Generating pending image transforms' jobs are all repeating the same work

[FlorisDerks]

What happened?

Description

When the generateTransformsBeforePageLoad setting isn't set to TRUE, jobs are queued to generate pending transforms. However, these jobs seem to all do the same work when running concurrently, even though there is a 'inProgess' column in the imagetransformindex table.

Since all jobs do the same work, and don't reserve an image prior to starting the transform, all jobs come to a halt when a single image transform fails, as can be seen on this screenshot:

Steps to reproduce

1. Disable all queue-workers
2. Gather some pending image transforms by clicking around the site or CMS
3. Start multiple queue-workers
4. See that they all try to resize the same images, even when they are already completed by another worker, and they all hang on the same longer-lasting images which they all try to complete at the same time.

Expected behavior

Image transforms are only handled by a single job at the same time, and are reserved to that worker when starting. Since a job is added for every single pending image transform, maybe that job should only handle that specific image instead of the whole queue in every job?

Actual behavior

Image transforms are only handled as completed when all image transforms in the job are completed.

Craft CMS version

v4.3.1

PHP version

8.0

Operating system and version

No response

Database type and version

No response

Image driver and version

No response

Installed plugins and versions

[i-just]

At first glance, it seems somewhat related to: #11970 and most importantly #12047.
As a first port of call, could you please check if you're still getting those issues when running only one queue worker?

As per @angrybrad comment here: #12047 (comment) we recommend only running one queue worker.

[FlorisDerks]

The issue I still get when only running a single queue-worker is when the job runs out of it's timeout (which can easily happen when there are thousands of images to be transformed in a single job), it will not save any of it's progress (or mark the timed-out transform as error) and starts again at the start of all pending image transforms, encounters the same timeout-causing transform, times out again and therefor never reaches the rest of the pending transforms.

Basically, the main problem is that this single job sometimes takes way too long due to those retries, which causes other jobs to be needlessly delayed.

[i-just]

Right, thanks for clarifying.

One thing you can do from your end is increase the default ttr in config/app.php. Following code would change the ttr to 1 hour.

'components' => [
    'queue' => [
        'ttr' => 3600,     // 1 hour
    ],
]

Another thing to note is that assettransformindex table already has a fileExists which is taken into consideration when getting pending transform indexes, so if you re-run the job, the query to generate transforms will only grab the ones that haven't yet been generated (fileExists is false). It will also check assettransformindex for inProgress and error and won't actually process them again.

I'd say, at the moment it's working as expected (with one queue worker), but we're discussing internally how we can make it perform better.

[brandonkelly]

This has been addressed in two ways:

• The “Generating pending image transforms” job has been updated for the next Craft 3 and 4 releases, to completely ignore any transforms that have already been generated or are currently in progress. (Previously the job would fetch all pending transforms up front, and then attempt to process each of them, including any transforms that were already processed by a separate request between the time the job started and whenever it got around to that transform.)
• That job is deprecated for Craft 4.4, in favor of a new “Generating image transform” (singular) job, which will be queued up for each ungenerated transform that is requested. This will result in significantly more jobs getting queued, but each of them will be significantly faster, reducing the likelihood that Craft will run out of resources mid-job. (The jobs will also bail immediately if their transform already exists by the time they’re executed.)

[brandonkelly]

Craft 3.7.64 and 4.3.7 have been released with that first fix.

[brandonkelly]

Craft 4.4 is out with the second fix now.

[Bronskiy]

Looks like the problem is back in 4.5.10

[marina-moca]

Confirm we are seeing this too on 4.5.10

Is there any way we can disable the firing of the queue processing on login (given we have a cron job periodically running the queue process (ie * * * * * flock -n /container/application/queue.lock /usr/local/bin/php /container/application/craft queue/run >/dev/null 2>&1)

Thanks

[realjoshharrison]

Is there any way we can disable the firing of the queue processing on login (given we have a cron job periodically running the queue process (ie * * * * * flock -n /container/application/queue.lock /usr/local/bin/php /container/application/craft queue/run >/dev/null 2>&1)

GeneralConfig::$runQueueAutomatically sounds like what you need for that:

Add to config/general.php:
->runQueueAutomatically(false)

Or, add to .env:
CRAFT_RUN_QUEUE_AUTOMATICALLY=false

[Bronskiy]

Is there any way we can disable the firing of the queue processing on login (given we have a cron job periodically running the queue process (ie * * * * * flock -n /container/application/queue.lock /usr/local/bin/php /container/application/craft queue/run >/dev/null 2>&1)

GeneralConfig::$runQueueAutomatically sounds like what you need for that:

Add to config/general.php: ->runQueueAutomatically(false)

Or, add to .env: CRAFT_RUN_QUEUE_AUTOMATICALLY=false

in such case it just adds new line with Pending status every second

[marina-moca]

Thanks for the responses. Should we be logging this as a new issue - I notice this one is closed....

[Bronskiy]

@brandonkelly could you reopen the issue?

[brandonkelly]

@Bronskiy It looks like you are describing the expected behavior, if you read my previous comment (#12274 (comment)).

[marina-moca]

Hi, ok it's Sunday here and I'm on our dev system so no other activity. I have added a video of what happens each time I log in and navigate to an item in the sidebar. Note it only starts queuing transformations when I click on something. The queue eventually clears (a couple of minutes). If I log out and back in, and choose the same entry in the sidebar, the transformations start with the same number of transformations to do. I've repeated this multiple times and get the same result every time. Note that all the queue entries have a Description of "null".

https://share.cleanshot.com/jZ1vCbH7

I've also noted the load on the container (it's all transformation load) so this is not an insignificant process especially when we will have up to 10 people logged in to the backend at any one time once the system is live.

I also note we have maybe 30 other CraftCMS sites where we don't see this behaviour, although this is the most complicated with 8 sites within the multisite setup.

So it could be we are an edge case, but this behaviour is not expected or desired. Any help on this would be much appreciated - maybe I should be logging it somewhere else?

Thanks.

[Bronskiy]

@Bronskiy It looks like you are describing the expected behavior, if you read my previous comment (#12274 (comment)).

yes, but it looks like it generates it unlimited

[marina-moca]

Hi, further to this, we cleared out some data yesterday and re-imported - the problem no longer exists, so we are wondering if one of our previous imports had some issues with images. Not sure, but we are not longer seeing any transforming of images as per my video above. Fingers crossed it stays this way.

Cheers!

[danethomas]

We're seeing this on 4.5.10 and I've just upgraded to 4.7.1 and it seems to be continuing. Continual Generating image transform jobs getting created with the job data

{
    "description": null,
    "transformId": 1
}

I suspect it's something that's gone wrong with a imagetransformindex record. I'll try to re-index assets from the CP to see if that fixes anything (it didn't on 4.5.10) - anything that you can suggest either via the CP or CLI?

[danethomas]

@marina-moca when you say you cleared our some data and re-imported - what did you do? We're still trying to get to the bottom of this issue for our instance and we're only running a single worker.

[danethomas]

This ended up being related to SEOMatic and a site default meta image (gif) not having transforms enabled on it. Turned that setting on and the jobs stopped queueing.

[AmarHuda1]

Confirm we are seeing this too on 4.5.10

if this kind of behaviour occurs in Production Environments, will it affect the server performance? Like, it will cause server error or PHP-timeout?

[DavidKabelitz]

We have the same issue, had today 1.6M jobs in queue. Also seomatic in use but with the setting on. Any other ideas here? It's just on our stage enviroment. We are on 4.8.0