Feature: Infer schema type automatically.

[mohammad0-0ahmad]

Tasks to be implemented :

• Infer schema type automatically from document definition.
• Provide TS utility to be able to extract schema type from schema instance to be able to reuse the type as separated type "InferSchemaType".
• Improve Model.create function type to show schema properties in suggestions dialog.
• Adding "statics" property in schema options to support auto type inference for model statics methods.
• Adding "methods" property in schema options to support auto type inference for document instance methods.
• Support custom type key "typeKey" to inference schema type.
• Improve Model new function "document constructor" type to show schema properties in suggestions dialog.
• Improve Model.insertMany function type to show schema properties in suggestions dialog.
• Adding "query" property in schema options to support auto type inference for query helpers methods.

Status:

• Ready to review.
• Docs is updated.
• Ready to merge.

Do you have any suggestions or ideas to be implemented in this PR, please leave a comment 🙂

Get it early:

You can install mongoose including this feature by running:

 npm i Automattic/mongoose#pull/11563/head

Screenshots :

• Auto schema type inference & create function:

• InferSchemaType utility & supporting custom type key:

[Uzlopak]

@mohammad0-0ahmad

When do you think this PR will be ready for merge?

[mohammad0-0ahmad]

@mohammad0-0ahmad

When do you think this PR will be ready for merge?

When it is reviewed 🙂 @Uzlopak

[Uzlopak]

Not easy to digest :D

Made some comments.

[mohammad0-0ahmad]

@Uzlopak
Thanks for reviewing and following the entire process.
I've resolved the latest comments, and I am looking forward for @vkarpov15 review.
Best regards! 🙂

[Uzlopak]

@vkarpov15
I think this should be part of v6.3

[Uzlopak]

Maybe we should talk about documenting https://github.com/Automattic/mongoose/blob/master/docs/typescript.md

[mohammad0-0ahmad]

Hello @vkarpov15!
Would you like to review this?

[uncle-ara]

when will typing become available for general use?

[mohammad0-0ahmad]

when will typing become available for general use?

I am waiting for @vkarpov15 review.

[Uzlopak]

@mohammad0-0ahmad

I hope you are still motivated. Issue is that complex typescript PR are always leaving an unease feeling in the stomach. So dont take it personally. :)

Can you check please if we need some documentation changes?

https://github.com/Automattic/mongoose/tree/master/docs/typescript

[mohammad0-0ahmad]

@mohammad0-0ahmad

I hope you are still motivated. Issue is that complex typescript PR are always leaving an unease feeling in the stomach. So dont take it personally. :)

Can you check please if we need some documentation changes?

https://github.com/Automattic/mongoose/tree/master/docs/typescript

I am still motivated, and I understand that, but I've provided some JS docs for the types, and I ready to explain the unclear things in this PR.
We need absolutely to make some documentation changes and maybe update docs on the website, but I would like to wait until get review response in case something will be not acceptable.

[vkarpov15]

I love the type key support and auto typed schemas. But my big concerns are 1) performance, 2) backwards btreaking changes. Our TS benchmark shows before this PR:

Run node ./benchmarks/typescript
simple instantiations: 35805
simple memory used: 152972.7
simple check time: 3.089
simple total time: 4.831999999999999

After this PR:

Run node ./benchmarks/typescript
simple instantiations: 227139
simple memory used: 191137.1
simple check time: 3.8959999999999995
simple total time: 5.3

After spending over 6 months fixing the performance of our TypeScript types, I do not want to merge something that undoes all the progress we made in #10349.

[vkarpov15]

Nice, thanks 👍 We'll merge this into the 6.4 branch. Thanks for all your hard work on this.

[mohammad0-0ahmad]

Thank you @Uzlopak @vkarpov15 as well.

[magyarb]

Hi, and thanks for all the work done here.

I see that mongoose-tsgen is no longer updated due to this PR.

Will it be possible to export these inferred types as separate files? It would be useful to use these types or interfaces elsewhere, for example on the frontend, where mongoose is not installed, or converting them to json schemas, to perform input validation on both the frontend and backend.

[francescov1]

@magyarb I'm open to continuing support for mongoose-tsgen if it is still useful to the community. I assumed that it became redundant with the type inference, but the use case for exporting types still exists so might be worth maintaining for now.

We could also look at integrating something like it into Mongoose natively if that's something the team is interested in. Happy to help in whatever method is preferred!

[mohammad0-0ahmad]

It might be great if you can make it availabe internally @francescov1 or at least mention that mongoose-tsgen can generate types files in mongoose docs.
What do you think @vkarpov15 @Uzlopak @AbdelrahmanHafez ?

[vkarpov15]

@francescov1 I'd say continue to support mongoose-tsgen for now 👍

[francescov1]

@vkarpov15 Will do 👍

[Starrah]

Sorry that I'm not excellent in Typescript, but what does U extends any here mean?

As far as I know, everything extends any? so the contiditional clause (T & { _id: Types.ObjectId }) is always used, and the clause T & Required<{ _id: U }> is never used.

Thus, _id will be always infered with type ObjectId, even if I have defined another type such as String in the Schema definition.

[Starrah]

For example:

import {model, Schema} from "mongoose";
const schema_with_string_id = new Schema({_id: String, nickname: String})
const theModel = model('test', schema_with_string_id)
const obj = new theModel()
const theId = obj._id

The type of variable theId will be string | Types.ObjectId, rather than string.
Is this designed behaviour for some reason I don't know, or just a bug?

[Starrah]

@mohammad0-0ahmad @Uzlopak Sorry for bothering you!

[mohammad0-0ahmad]

Hello @Starrah !
Firstly, you are not bothering, and you are totally right.
I think this meant to be something like:

  export type Require_id<T> = T extends { _id?: infer U }
    ? IfEquals<U, any> extends true
      ? (T & { _id: Types.ObjectId })
      : T & Required<{ _id: U }>
    : T & { _id: Types.ObjectId };

Thanks for your review, I will make sure to refactor this.
What are your thoughts @Uzlopak about that?

[Uzlopak]

I think I did this to check if T is not never.

[Starrah]

I think I did this to check if T is not never.

First, does you mean you did that to check if {_id: } in T is not never, rather than check T?
Because according to my experiment,

import {Require_id, Types} from "mongoose";

type Required_id_old<T> = T extends { _id?: any } ? (T & { _id: T['_id'] }) : (T & { _id: Types.ObjectId });
type Required_id_new<T> = T extends { _id?: infer U }
    ? U extends any
        ? (T & { _id: Types.ObjectId })
        : T & Required<{ _id: U }>
    : T & { _id: Types.ObjectId };

type t1_old = Required_id_old<{_id: never}>
type t1_new = Required_id_new<{_id: never}>

type t2_old = Required_id_old<never>
type t2_new = Required_id_new<never>

Both t2_old and t2_new is never(which I think is expected behaviour), and for t1, t1_old is {_id: never} but t1_new is never(I have no idea which is your expected behaviour).

Then, if my assumption is true and you does mean check if {_id: } in T is not never, then maybe the two conditional clause should be swapped, as the following:

type Required_id_new<T> = T extends { _id?: infer U }
    ? U extends any
        ? T & Required<{ _id: U }>
        : (T & { _id: Types.ObjectId })
    : T & { _id: Types.ObjectId };

[Starrah]

By the way, my Typescript version is 4.7.2. I surprisingly found that T extends any? true: false equals to never when T is never, because the following code:

type Test<T> = T extends any ? true : false
type A = Test<never>
const a: A = false

gives error output

Error:(3, 7) TS2322: Type 'boolean' is not assignable to type 'never'.

The behaviour is the same in Typescript 4.6.3.

[Uzlopak]

Well, I guess, I should have implemented appropriate typings tests.

[Starrah]

@mohammad0-0ahmad @Uzlopak
Though with the discussion above, we (or just I) cannot still figure out what the codes here means, specifically, what does the U extends any mean.

However, I want to point out that the following test case written by me, got failed:

// index.test-d.ts
import {model, Schema} from "mongoose";
import {expectType} from "tsd";

const schema_with_string_id = new Schema({_id: String, nickname: String})
const theModel = model('test', schema_with_string_id)
const obj = new theModel()

expectType<string>(obj._id)

when testing it with tsd, gives the following error:

 index.test-d.ts:8:0
  ✖  8:0  Parameter type string is declared too wide for argument type string & ObjectId.

According to @mohammad0-0ahmad :

Hello @Starrah ! Firstly, you are not bothering, and you are totally right. I think this meant to be something like:

  export type Require_id<T> = T extends { _id?: infer U }
    ? IfEquals<U, any> extends true
      ? (T & { _id: Types.ObjectId })
      : T & Required<{ _id: U }>
    : T & { _id: Types.ObjectId };

Thanks for your review, I will make sure to refactor this. What are your thoughts @Uzlopak about that?

I think that the test case failure above indicates a bug. So, I decide to raise a new issue (#12070) to report and/or further discuss this bug.

[olawalejuwonm]

Does this work for method too?