[RFC] JTDDataType

[erikbrinkman]

What issue does this pull request resolve?
Adds JTDDataType that infers the validated type from a JTDSchema.

What changes did you make?
Adds the type and adds tests around it.

Is there anything that requires more attention while reviewing?
The diff is actually pretty simple, it's much simpler than the JTDSchemaType. There are three things that caught my eye:

1. this night not work well as a schema validator. Due to the permissive nature of typescript and empty schema definition, anything that is basically an object, should be a valid schema to validate unknown. I don't really know a good way to do this. Thus JTDDataType<InvalidSchema> will just evaluate to unknown. That will probably throw errors somewhere if you're expecting a type to be validated, but it's maybe not the best.
2. The handling of timestamp is a bit odd. Because of the way the schema works, if you have JTDDataType<{ type: "timestamp" }> then the result type is string | Date which is probably not the type you want. Is this how Ajv currently works, e.g. if you have { type: "timestamp" } either will be validated? Is the the desired behavior? If so, it might make sense to have "options" for JTDDataType that tell it which choice to choose from as the union is probably not super helpful. This type of option will probably be useful when Ajv allows Map<string, ...> for values types as well, as the union is also probably not super helpful.
3. This currently doesn't do anything special to handle omitting literals, and without the const keyword it could be hard to debug. I don't have any good ideas here, but it could be very frustrating for people who omit const to always get unknown validation.

Note: this is stacked on a few commits, so only look at the last one

[epoberezkin]

Is this how Ajv currently works, e.g. if you have { type: "timestamp" } either will be validated?

yes

Is the the desired behavior?

I'd wait for more usage/feedback to decide it... It seems correct to allow both:

1. if you are validating the result of JSON parsing - which is the most common use case, then yes - this is indeed desired to allow strings.
2. if validation happens before serialising (also common use case) then allowing Date also makes sense.

[epoberezkin]

If so, it might make sense to have "options" for JTDDataType that tell it which choice to choose from as the union is probably not super helpful.

In most practical settings the application would have some semantic validation step before passing to the application logic, so this type would be used in a narrow context - only after JTD but before semantic validation...
I am planning some "best practises" doc that would explain that - many people try to conflate structural and semantic validation in one step, that led to overcomplicating of JSON Schema and still doesn't solve all the problem.

I also think that parsing rather than validating can be a better approach - just in the process of making compiled parsers from the schemas (where these types would also be used)...

I'd keep a union for now until there is some feedback.

[epoberezkin]

This currently doesn't do anything special to handle omitting literals, and without the const keyword it could be hard to debug. I don't have any good ideas here, but it could be very frustrating for people who omit const to always get unknown validation.

Can it maybe detect and alert if "string" type is used in "type" or "enum" and result in never for data?

I am yet to review the actual code, a bit behind :)

Thank you!

[erikbrinkman]

No worries on the quick review, I don't specifically need to use this, so take your time in parsing through it.

You make a good case for the union, so that will stay as is.

I actually did already set it up so that enums without const become never instead of unknown. I just added some tests for this. As addressed elsewhere, there's not a good way to trigger a better user facing error, so this is basically it :/

[epoberezkin]

@erikbrinkman reviewed it - it seems null changes (#1456) should be merged first, then master merged into this one and conflicts resolved - then I will check again...

[epoberezkin]

The only comment is to allow additionalProperties in JTDDataType if the schema allows it (unless it does allow them and I missed it:)

[erikbrinkman]

I took this off the branch with the nulls so they can be addressed separately. I think the additionalProperties was your own comment, which should be resolved now.

[epoberezkin]

I am going to merge - anything that is missing here before it can be released?

[erikbrinkman]

Outside of the one comment, which should be a non breaking change if done later, I think it's ready!

[epoberezkin]

Thank you!

[epoberezkin]

Added to the docs: https://ajv.js.org/guide/typescript.html#utility-type-for-jtd-data-type

I don't think type inference would work though without adding type signatures - will do it tomorrow.

[erikbrinkman]

The new theme looks great! I tried blindly adding the type to validate as a test and started getting typescript errors because the types are getting too complex, which makes sense, the JTDDataType is complex. I started looking into AJV core to fix it, but it proved more difficult than I expected. It feels like it should be possible to limit how far typescript is trying to verify signatures.

I was able to get it work by manually telling the compiler what the signatures of compile and verify were. Potentially it makes sense just to do this in jtd.ts, and that will prevent this problem. It'll also potentially fix the issue with type inference. I'm still looking into it though.

[epoberezkin]

I think to use JTDDataType with type inference we need some generic type that all JTD schemas extend. In the ideal world it would be JTDSchemaType without parameters.

Then the additional type signature for compile and others would be something like:

compile(schema: JTDSchemaType): (data: JTDDataType<typeof schema>) => boolean

The advantage of having a generic type for some JTD schema that it would also validate schema in place, not when it is used in JTDDataType

Maybe it's possible to make inference work without this type, but I didn't manage to

[erikbrinkman]

Interesting. I honestly had not thought ahead to how this would work in the function signature. Taking any type as I wrote the type transformer doesn't make sense to help people type things. I guess I ultimately imagined the JTD compile to look like:

compile<T>(schema: JTDSchemaType<T>): ValidateFunction<T>;
compile<T extends JTDSchema>(schema: T): ValidateFunction<JTDDataType<T>>;
compile<T = unknown>(schema: JTDSchema): ValidateFunction<T> {
compile(schema: JTDSchema): ValidateFunction<unknown> {
    ...
}

ideally for the purposes of both of these functions, the JTDDataType is the only signature necessary, but but I see two reasons to have multiple overloading:

1. the ability to say I trust my schema, which seems bad, but maybe there's a reason for it?
2. for complex types, the type created by JTDDataType should be compatible, but might not look as nice as the manual type / interface

[epoberezkin]

Yes, I was thinking there will be multiple signatures too, what’s currently missing is the 2nd (and some generic JTD schema type to use in it).