Automatic schema generation tool?

[twpayne]

Would you be interested in functionality to generate a CUE schema from example JSON documents?

The idea would be that you throw a load of example JSON files that you want to generate at the tool and the tools spits out a minimal CUE schema which unifies with all the input documents. Such a generated schema could be useful for people getting started with CUE.

I've written such a tool to generate Go structs from multiple JSON documents or multiple XML documents. Would you be interested in a contribution of similar functionality for CUE?

Apologies if you already have a tool like this, or if I've mixed something up. I'm a CUE n00b.

[mpvl]

The algorithm for this would be "anti unification". The idea has been around for a long time. See for instance:
#7
#15

This is certainly something that would be very useful. There are some design issues that make this tricky in the general case, though:

1. does {a:1} and {a:3} generalize to {a: int}, {a: uint}, {a: int & >=1 & <= 3}, {a: 1 | 3}?
2. do we want to "uninperpolate" strings: {a: "foo", b: "foofoo"} and {a: "bar", b: "barbar"} -> {a: string, b: "\(a)\(a)"}?

In case 1, the "minimal" schema, where minimal is defined as the least upper bound is {a: 1 | 3}, but that probably doesn't capture the general intent of this operation.

In other words, to design this properly, we really need to know how people intend to use this and what they typically expect. Input very welcome! If we take a simple stance towards the before-mentioned points, this doesn't have to be all that complicated. Anti unification is a fairly simple algorithm, generally speaking.

[twpayne]

Thanks for the detailed response and pointers :)

For intended use, I would suggest that this tool generates an initial schema for users looking to adopt CUE to their existing project. The goal would be that this schema unifies with all their existing documents and is a good starting point for further refinement. It should get the overall "shape" of the data correct (e.g. generate int and the more specific uint types where possible), without the precise refinement.

This means:

1. {a:1} and {a:3} generalizes to {a: uint}.
2. No string uninterpolation. I have implemented this in the past for text/template, but any autogenerated solution is likely to contain unwanted artefacts.

As you say, with a simple stance this is straightforward to implement: walk the input documents, gather statistics on the values observed, and then generate the appropriate type based on the observed values.

[sivukhin]

Hi, I'm in the process of CUE adoption and for internal tasks I write simple tool which converts bunch of JSON values in the CUE definitions.
Check this out: https://github.com/sivukhin/cuebootstrap

[twpayne]

Nice! This is exactly what I had in mind, and the bootstrap in the name describes very well what it does.

[sivukhin]

@twpayne, FYI, I added feature to cuebootstrap which allow you to predefine basic skeleton of the final schema in order to make bootstrap output even more useful. For example, you can process JSON with following simple skeleton:

#item: type: string @discriminative()
#array: [...#item] @root()

And got following schema for the given JSON:

#item: {
    #insert: {
        id:       string
        value:    string
        timeout?: number
    }
    #delete: {
        id: string
    }
    type: "insert" | "delete"
    if type == "insert" {
        #insert
    }
    if type == "delete" {
        #delete
    }
}
#array: [...#item]

[{"type":"insert","id":"1","value":"json"},{"type":"delete","id":"2"},{"type":"insert","id":"1","value":"cue","timeout":1000}]

[twpayne]

Ooh, that's really sweet.

For info, I gave up trying to do this automatically in go-jsonstruct: twpayne/go-jsonstruct#17.