[8.x] Add upsert to Eloquent and Base Query Builders

[paras-malhotra]

Motivation
Laravel already supports bulk inserts, bulk insert/ignores. It's only logical to also support bulk upserts, which are already supported on frameworks such as Rails.

Bulk Inserts Already Supported

• Bulk inserts: errors out on any row with invalid data

DB::table('users')->insert([
    ['email' => 'taylor@example.com', 'votes' => 0],
    ['email' => 'dayle@example.com', 'votes' => 0],
]);

• Bulk insert or ignore: rows with invalid data are ignored

DB::table('users')->insertOrIgnore([
    ['id' => 1, 'email' => 'taylor@example.com'],
    ['id' => 2, 'email' => 'dayle@example.com'],
]);

Bulk Upsert (This PR)

This PR supports bulk upserts (insert and on duplicate key, update) like so:

DB::table('users')->upsert([
    ['id' => 1, 'email' => 'taylor@example.com'],
    ['id' => 2, 'email' => 'dayle@example.com'],
], 'email');

The first argument is the values to insert/update and the second argument is the column(s) that uniquely identify records. All databases except SQL Server require these columns to have a PRIMARY or UNIQUE index. This is similar to Rails upsert_all without the returning support.

Eloquent upsets are also supported like so:

User::upsert([
    ['id' => 1, 'email' => 'taylor@example.com'],
    ['id' => 2, 'email' => 'dayle@example.com'],
], 'email');

If the model uses the updated_at timestamp, upsert() will also add the updated_at timestamp columns. Inspired from https://github.com/staudenmeir/laravel-upsert with some slight modifications. Ping @staudenmeir

[rodrigopedra]

Usage of VALUES() on MySQL is deprecated starting on MySQL 8.0.20.

Reference:

https://dev.mysql.com/doc/refman/8.0/en/insert-on-duplicate.html

The use of VALUES() to refer to the new row and columns is deprecated beginning with MySQL 8.0.20, and is subject to removal in a future version of MySQL. Instead, use row and column aliases, as described in the next few paragraphs of this section.

The sample below is from the link above showcasing the new syntax to use:

INSERT INTO t1 (a,b,c) VALUES (1,2,3),(4,5,6) AS new
  ON DUPLICATE KEY UPDATE c = new.a+new.b;

[taylorotwell]

@rodrigopedra does that new syntax work in MySQL 5.7? Though I guess it doesn't really matter as this feature could require MySQL 8.0.

[rodrigopedra]

Old syntax works on both MySQL versions (5 and 8).

New syntax was introduced in MySQL 8.0.19, from the link on my previous message:

Beginning with MySQL 8.0.19, it is possible to use an alias for the row...

To support both versions I think we can keep the old syntax around as the docs says: "...and is subject to removal in a future version of MySQL". No removal timeline was set yet.

In general it takes a while to MySQL really remove deprecated features (for example the GROUP BY ... DESC was deprecated in 5.7 (https://dev.mysql.com/doc/refman/5.7/en/select.html) but was removed only in 8.0 and noted in the upgrade guide.

So we could use the old syntax for now and handle this case when it actually gets removed.

EDIT: corrected the deprecation example about GROUP BY...DESC as it was actually deprecated in MySQL 5.7

[paras-malhotra]

Yeah, it's still supported but deprecated beginning 8.0.20. It isn't removed yet and will be around for a while. So to support both mySQL 5 and 8, it's best to continue using values.

Besides, we're already using VALUES() in all insert operations including insert and insertOrIgnore.

[rodrigopedra]

Actually @paras-malhotra, the usage of VALUES as it is used now in insert and insertOrIgnore is not deprecated and is part of the SQL standard, it is a clause of the INSERT SQL statement.

What is deprecated is the usage of the VALUES() special function on INSERT...ON DUPLICATE KEY UPDATE statement.

https://dev.mysql.com/doc/refman/8.0/en/miscellaneous-functions.html#function_values

[paras-malhotra]

Ohh wow, I didn't know that 😮. Okay, I'll probably look out for the removal notice since row/column aliases don't work in 5.7. @taylorotwell if you'd like I can implement the new way (row/column aliases), but I'd recommend we go with values for now as it's still supported in both versions.

[rodrigopedra]

No problem, maybe one of the reasons to deprecate that function is the confusion with the SQL clause usage.

MariaDB took a similar step, but they renamed the VALUES() function to VALUE() (https://mariadb.com/kb/en/values-value/).

I think this feature is a great addition to the framework and that for new it is best to keep the VALUES() usage until it actually gets removed from MySQL. Maybe by then we can remove MySQL 5.x support.

[mpskovvang]

Great addition! This is the first I add to new projects myself.

However, this PR misses the option to do partial updates. You don't want to keep overwriting e.g. created_at.

@paras-malhotra do you know if partial updates is supported by other than MySQL and PostgreSQL? Thanks for contributing this to the core.

Perhaps a third argument with either an indexed array of column names or an associate array with key-value. The default is to update all columns.

[paras-malhotra]

@mpskovvang absolutely, that was the follow up PR: #34712 😄

[mpskovvang]

@mpskovvang absolutely, that was the follow up PR: #34712 😄

Did't see that. Spot on! Thanks! Much appreciated. 🙏

[webbby]

Does the feature support columns combination that uniquely identifies records?

For example

App\Models\Flight::upsert([
    ['departure' => 'Oakland', 'destination' => 'San Diego', 'price' => 99],
    ['departure' => 'Chicago', 'destination' => 'New York', 'price' => 150]
], ['departure', 'destination'], ['price']);

departure - not unique
destination - not unique
(departure,destination) - unique pair

If not can it be targeted by unique index instead of columns?

[shankhadevpadam]

I have post meta table with fields [id, post_id, meta_key, meta_value]. i want to bulk update a meta values. But It repeatedly create a new row when using upsert method while updating form every time.

foreach ($metas as $key => $value) {
    $arr[] = [
        'post_id' => $post->id,
        'meta_key' => $key,
        'meta_value' => $value ?? '',
    ];
}

Postmeta::upsert($arr, ['post_id', 'meta_key'], ['meta_value']);

but while using updateOrCreate method it work fine.

foreach ($metas as $key => $value) {
    Postmeta::updateOrCreate(['post_id' => $post->id, 'meta_key' => $key], ['meta_value' => $value ?? '']);
}

[paras-malhotra]

@shankhadevpadam all databases except SQL Server require the columns in the second argument of the upsert method to have a "primary" or "unique" index. If that doesn't solve your problem, please create a new issue with steps to replicate.

[eusonlito]

Only as reference, is important understand than INSERT INTO ... ON CONFLICT will update the internal reference to increment primary key on related table because the increment has to happen before the insert is attempted. Then, if you use the upsert method as a regular insert or update action, your ID will be something like:

-[ RECORD 284 ]
id | 19039
-[ RECORD 285 ]
id | 19221
-[ RECORD 286 ]
id | 19306
-[ RECORD 287 ]
id | 19397
-[ RECORD 288 ]
id | 19784
-[ RECORD 289 ]
id | 19828
-[ RECORD 290 ]
id | 20161
-[ RECORD 291 ]
id | 20768
-[ RECORD 292 ]
id | 20799
-[ RECORD 293 ]
id | 21287
-[ RECORD 294 ]
id | 21770
-[ RECORD 295 ]
id | 22205
-[ RECORD 296 ]
id | 22401
-[ RECORD 297 ]
id | 23099
-[ RECORD 298 ]
id | 23727
-[ RECORD 299 ]
id | 23820
-[ RECORD 300 ]
id | 24190
-[ RECORD 301 ]
id | 24653
-[ RECORD 302 ]
id | 24744
-[ RECORD 303 ]
id | 25054
-[ RECORD 304 ]
id | 25295

[fractalf]

I noticed that upsert will add values for created_at and updated_at, but insert does not do this.

Is that intended?

My preference would be that insert also has this behaviour

[ejntaylor]

Does the feature support columns combination that uniquely identifies records?

For example

App\Models\Flight::upsert([
    ['departure' => 'Oakland', 'destination' => 'San Diego', 'price' => 99],
    ['departure' => 'Chicago', 'destination' => 'New York', 'price' => 150]
], ['departure', 'destination'], ['price']);

departure - not unique
destination - not unique
(departure,destination) - unique pair

If not can it be targeted by unique index instead of columns?

Doesn't look like this is the case - anyone had luck with this?

[rognales]

When issuing a mass update or delete query via Eloquent, the saved, updated, deleting, and deleted model events will not be dispatched for the affected models. This is because the models are never actually retrieved when performing mass updates or deletes

This statements applies to UPSERT as well?

Because the created_at and updated_at did get filled by Eloquent

[mdobydullah]

Does the feature support columns combination that uniquely identifies records?

For example

App\Models\Flight::upsert([
    ['departure' => 'Oakland', 'destination' => 'San Diego', 'price' => 99],
    ['departure' => 'Chicago', 'destination' => 'New York', 'price' => 150]
], ['departure', 'destination'], ['price']);

departure - not unique
destination - not unique
(departure,destination) - unique pair

If not can it be targeted by unique index instead of columns?

This feature is badly needed. Do you have any solutions regarding this issue?