Mongoose bulk insert misses critical information

[samislam]

Prerequisites

• I have written a descriptive issue title
• I have searched existing issues to ensure the feature has not already been requested

🚀 Feature Proposal

The picture above shows an example of a request getting sent to the following route:

and the following picture shows what have been inserted to the database in compass (notice how there are three entries):

As we know, Model.create() accepts an array of objects, or an object.

In this example, I am sending an array of objects, to insert them.

Model.create([]) will insert the documents one by one to the database, it doesn't skip the validation part, which is why I chose it.
and when it finds a document with a validation error, it skips it, and moves to the next one.
until it finishes, then it reports the errors it encounters.

That's what it should be, However it's not exactly working like that.

Note that I have two documents which holds validation errors:

However, mongoose is only reporting the first one, it's not reporting the second one, even though it passes by it, and it sees it.

Motivation

Why this information is critical?

Because on the client side, I have to know which documents got inserted, and which did not.

In this case, (when I will know which are the ones got inserted and the ones that did not), I can show the client for example that the documents x, y, z has been inserted, while the documents f, g, h did not. So the user can correct his mistake and send the request again.

The current error report is useless, because it only tells "there was a validation error", but it doesn't tell you the "where"

Example

The error report should include all the documents which refused to be written to the database in an array.

[samislam]

I realized that

const data = await User.insertMany(req.body)

Has exactly the same behavior.
It doesn't only apply to Model.create().
Model.insertMany() has the same problem as well.

[samislam]

This question is on stackoverflow as well:
https://stackoverflow.com/questions/74760543/mongoose-bulk-insert-misses-critical-information

[vkarpov15]

With insertMany(), you can try setting the ordered: false and rawResult: true options. That will give you more info. Specifically, res.insertedIds will contain the _id property of every document that was successfully inserted. And res.mongoose.validationErrors will contain any validation errors that occurred.

      const schema = new Schema({
        name: { type: String, required: true }
      });
      const Movie = db.model('Movie', schema);

      const id1 = new mongoose.Types.ObjectId();
      const id2 = new mongoose.Types.ObjectId();
      const id3 = new mongoose.Types.ObjectId();
      const arr = [
        { _id: id1, foo: 'The Phantom Menace' },
        { _id: id2, foobar: 'The Force Awakens' },
        { _id: id3, name: 'The Empire Strikes Back' }
      ];
      const opts = { ordered: false, rawResult: true };
      const res = await Movie.insertMany(arr, opts);
      // {
      //   acknowledged: true,
      //   insertedCount: 1,
      //   insertedIds: { '0': new ObjectId("63b34b062cfe38622738e510") },
      //   mongoose: { validationErrors: [ [Error], [Error] ] } 
      // }
      console.log(res);
      assert.equal(res.insertedCount, 1);
      assert.equal(res.insertedIds[0].toHexString(), id3.toHexString());
      assert.equal(res.mongoose.validationErrors.length, 2);

That's probably the best way to get info on all the documents that were inserted, and which ones had validation errors. Does that work for you?

One unfortunate issue we noticed: order of res.mongoose.validationErrors isn't strictly guaranteed, so there's no way to map validation errors to original documents being inserted. We'll do some work to make sure that ordering is consistent.

[samislam]

One unfortunate issue we noticed: order of res.mongoose.validationErrors isn't strictly guaranteed, so there's no way to map validation errors to original documents being inserted. We'll do some work to make sure that ordering is consistent.

If it gives the ids of the invalid ones, then that's enough, We can sort them on the client side, it's not a big deal.

You mean we need to generate the id on the client side or otherwise we have no way to map the invalid ones with the valid ones, right?

let me check that out

[vkarpov15]

@samislam we added a res.mongoose.results property, which will contain an array of what happened to each individual document. If res.mongoose.results[i] is a document, then that document was inserted successfully, otherwise the insert failed.