Inconsistent virtual reference population

[MartinDrost]

Do you want to request a feature or report a bug?
Could be interpreted both ways but I'd label it as a bug.

What is the current behavior?
When creating a virtual field referencing an array of other documents, the population of the field works inconsistently. The resulting values will vary between an array filled with users, an empty array and undefined. This is inconsistent and will cause issues and/or unnecessary type checks.

If the current behavior is a bug, please provide the steps to reproduce.
As an example we will attempt to populate all children of a user document which are user documents themselves. The schema will be complemented with an array called childIds containing all children ids and a virtual called children which references the user schema.

const mongoose = require("mongoose");

mongoose.connect("mongodb://localhost:32768/population");

/**
 * Define the user schema
 */
const userSchema = new mongoose.Schema(
  {
    name: { type: String },
    childIds: [
      {
        type: mongoose.Schema.Types.ObjectId,
        ref: "User",
        required: true
      }
    ]
  },
  {
    toObject: { virtuals: true },
    toJSON: { virtuals: true }
  }
);

userSchema.virtual("children", {
  ref: "User",
  localField: "childIds",
  foreignField: "_id",
  justOne: false
});

const userModel = mongoose.model("User", userSchema);

Afterwards we will define a scenario where Joe has two children called John and Jane. When we find and populate all users we will receive three user objects with two of them containing an empty array at the children field and one containing an array with the two previous users. When we find and populate the two users without children however, we will receive those two users without the children field since it is left empty.

/**
 * Play out the scenario
 */
async function scenario() {
  // create users
  const children = [
    await userModel.create({ name: "John", childIds: [] }),
    await userModel.create({ name: "Jane", childIds: [] })
  ];

  await userModel.create({ name: "Joe", childIds: children.map(c => c._id) });

  // output all users (populated)
  const all = await userModel
    .find({})
    .populate("children")
    .exec();

  console.log(all);

  /*
  [
    {
        childIds: [],
        _id: 5d9db29c4066eb4baac97f58,
        name: 'John',
        __v: 0,
        children: [],
        id: '5d9db29c4066eb4baac97f58'
    },
    {
        childIds: [],
        _id: 5d9db29c4066eb4baac97f59,
        name: 'Jane',
        __v: 0,
        children: [],
        id: '5d9db29c4066eb4baac97f59'
    },
    {
        childIds: [ 5d9db29c4066eb4baac97f58, 5d9db29c4066eb4baac97f59 ],
        _id: 5d9db29c4066eb4baac97f5a,
        name: 'Joe',
        __v: 0,
        children: [ [Object], [Object] ],
        id: '5d9db29c4066eb4baac97f5a'
    }
  ]
  */

  // output all users without children (populated)
  const withoutChildren = await userModel
    .find({ childIds: [] })
    .populate("children")
    .exec();

  console.log(withoutChildren);

  /*
  [
    {
      childIds: [],
      _id: 5d9db29c4066eb4baac97f58,
      name: 'John',
      __v: 0,
      id: '5d9db29c4066eb4baac97f58'
    },
    {
      childIds: [],
      _id: 5d9db29c4066eb4baac97f59,
      name: 'Jane',
      __v: 0,
      id: '5d9db29c4066eb4baac97f59'
    }
  ]
  */
}

scenario();

What is the expected behavior?
I expect that, since I define it to be an array, the virtual field is always filled as an array when populated even when empty.

What are the versions of Node.js, Mongoose and MongoDB you are using? Note that "latest" is not a version.

Node.js: v12.10.0
MongoDB: v4.2.0
Mongoose: v5.7.3