Laravel, Queue, Horizon and >10 Servers (Workers) - Creating several million jobs takes extremely long

[0x7357]

I need to create several 10 million jobs.

I have tried it with for-loops and Bus::batch([]) and unfortunately the creation of the jobs takes longer than the processing of the jobs by the 10 servers/workers. That means the workers have to wait until the job shows up in the database (redis etc). With redis-benchmark I could learn that Redis is not the problem.

Anyway... is there a way to create jobs in BULK (not batch)? I'm just thinking of something like:

INSERT INTO ... () VALUES (), (), (), (), ...

Anyway, creating several million jobs in a for-loop or in batch seems to be much too slow for some reason. Probably because it's always just 1 query at a time and not an "upsert".

For any help I would be very grateful!

[llioor]

Foe me, the reason is not the insert, the problem is with lock time when many workers select the same batch "for update".
Check your slow log in mysql (if you batches table in mysql)

select * from `job_batches` where `id` = '994d96ad-5f3c-4c11-ad8e-313a65c2c69c' limit 1 for update

you will have something like

# Query_time: 0.414555  Lock_time: 0.414371 Rows_sent: 1  Rows_examined: 1

It means 414555 seconds, 6909.25 minutes, 115.15 hous.

Mysql "for update" can not handle it in right way.

The solution I'm searching for is to use faster solution than management the batch in DB table

[llioor]

No one find a solution yet?