cuelang.org: Documentation and content ideas, requests, and feedback.

[Lyoness]

Following up on the first CUE Docs and Content Call #1964. Please add your content and documentation ideas, discussions, requests, and feedback for cuelang.org here.

This is intended as a meta issue that "starts the ball rolling".

Prioritizing ease of submission over structure, so keeping this open-ended for now. This can come in any form: a friction log, a "brain dump", or in the form of a question.

The goal of this discussion is provide a place for people to voice what their needs or wishlist are for learning and onboarding to CUE: What you need, what you wish you had, etc., and see if we start to see trends and patterns emerging.

[disintegrator]

Cross-posting my thoughts from Slack…

Some general thoughts about content/docs:

• I spend a lot of time in MDN. Ironically, I never traverse the documentation site to find what I need. I always start at google with terms like "mdn aria tooltip" and I am reliably sent to where I need to be. This is probably a sign of how SEO can play an important role in discovery.
• TypeScript's regular blog posts around releases are fantastic:
  https://devblogs.microsoft.com/typescript/announcing-typescript-4-8-beta/
  https://devblogs.microsoft.com/typescript/announcing-typescript-4-8-rc/
  https://devblogs.microsoft.com/typescript/announcing-typescript-4-8/
• Counterpoint: The discoverability of the TypeScript docs is not great. I look through the release blog posts more often for detailed explanations of a language feature!
• The TypeScript and Go playgrounds are invaluable for exploring language features. I like the TypeScript playground more because it's a richer experience with the monaco editor and some LSP features.
• The in-line interactive examples in Go docs are fantastic if I want a deeper understanding of an API. I can tweak the example code a little and rerun to see the output. I don't know if this is a common behaviour but it's useful if you're feeling like a tinkerer.
• The CUE language spec is one of the few specs that I read fairly completely. I confess I don't spend time on the EBNF blocks but I appreciate that they're there. It's more that when I can't find something in the curated docs, I can drop down to the well laid-out language spec. I was very happy to hear that Marcel is looking at updating the language in the spec to be more approachable to folks without a background in graph theory.
• I haven't used Django in a lifetime but I recall that the tutorial was an incredible onboarding experience way back when I was a graduate software engineer.
• At work, we follow the Divio documentation system to explain how to use our infra platform (built on k8s). For certain pages, we've done well to keep them fresh and relevant but some other pages may fall behind.

[disintegrator]

In addition, I shared some ideas on literate programming on Slack. For example, I'd love it if we can write CUE in Markdown:

<!-- pricing.cue.md -->

# Our pricing scheme

This is a contrived example of how a payments provider might price the usage of their product.

We start out by defining a basic structure for product usage.

```cue
#UsageType: "payment" | "refund" | "verification"

usage: {
  type: #UsageType
  amount: >= 0.0
}
```

After that we can define a set of fee types that apply to different usage types.

When we process payments successfully we change a fixed fee on top of a percent of the original payment amount.

```cue
#PaymentFee: {
    amount: usage.amount * 0.1 + 0.2
}
```

Other fee types are a fixed rate only:

```cue
#RefundFee: #Fee & {
    amount: 0.3
}

#VerificationFee: #Fee & {
    amount: 0.3
}
```

The following definition catalogs all our fee types so that we can easily access them when calculating pricing for some usage.

```cue
#AllFees: #UsageType: #Fee
#AllFees: {
    payment: #PaymentFee
    refund: #RefundFee
    verification: #VerificationFee
}
```

At this point, we have the necessary types defined to evaluate feed for concrete usage values.

```cue
usage: { type: "payment", amount: 10.00 }
fee: (#AllFees[usage.type]).amount
```

This value of `fee` should come out to `1.200` given the usage above.

Running cue export --out json pricing.cue.md should work by extract the CUE that is defined in ```cue fences and evaluate these as if they belonged in one .cue file.

Interleaving documentation and code like this might be a powerful educational tool... It kind of resembles the format seen on learnxinyminutes.com.

[myitcv]

Very interesting though about extracting from markdown. Presumably this wouldn't work in the general case though, where you want to deliberately introduce errors etc?

[disintegrator]

Very interesting though about extracting from markdown. Presumably this wouldn't work in the general case though, where you want to deliberately introduce errors etc?

It might be possible to have a different type of fence for adding assertions or evaluations that are for documentation purposes only similar to Go's checked examples.

[djosephsen]

👋 Hello,
Great seeing you all in the call, and I'm really excited that the project is putting some investment into a docs redesign. I'm also thrilled the project has settled on diataxis for organization.

I think I'm in the fairly common situation of being the person on their respective team who has discovered cue and is using it for something, and would really love to advocate for its use internally. So my personal priority with respect to the documentation is in the diataxis tutorial dimension. I would like my teamies to think what's this cue thing dave keeps droning on about? and google the project, encounter a learn more button, and click it, to find a conversationally-toned introduction that begins with running cue at the command line, and progresses through import and export, showing off the language semantics in the context of simple operations on a trivial yaml config file, whose complexity grows as the tutorial progresses.

For example, first we learn about default json output and using -out to change it, which progresses to -out cue which progresses to import, which then sort of, re-anchors us in cue as the base language, and explains why that's a better thing by introducing improvements like comments, trailing commas, and terse shortcuts, Then we keep building and eventually find ourselves in a gentle introduction to underscores for un-exported values, definitions and ultimately constraints.

Totally understand we're still in the planning phase, but fwiw I think a proper tutorial as defined by the diataxis standard would super help me get other engineers not only up and running, but excited about the possibilities of the tool, and I'm happy to pitch in and help write it if that sounds like a thing the project is interested in.

kcoolbai

[myitcv]

This is great, thanks @djosephsen. Wonderful to see you and others on the call.

Digging a bit more on your situation.

What tags/terms would you use to describe the context in which you and those you are trying to persuade about CUE find yourselves? Do you have a pre-existing system of use like JSON Schema for validation? Or language X for policy? Or YAML for configuration? I'm trying to see if there is a specific angle you want covered in your "controversially-toned" introduction, to help ground certain concepts or drive certain points home.

Your references to cmd/cue suggest to me that you're currently using CUE as a Swiss-army knife for wrangling data, schema, validation etc. Is this building to a specific use-case, where you use CUE via the Go API? I ask because not everyone will actually use cmd/cue from day 1. They might instead learn the parts of the language they require and jump straight into using CUE with the Go (or other when they are created) language bindings. Hence the language guide will likely only talk about the cmd/cue tooling in passing. That said, there will likely need to be tutorials (which have overlap with the language guide) which talk about how to get stuff done with cmd/cue. It's one of the most powerful ways to become productive with CUE, and a story we hear very often.

[djosephsen]

My use cases currently revolve around data validation. I maintain a herd of telemetry systems that provide input to various data-pipelines, and it is sometimes the case that an innocuous-seeming change to one of these daemons will subtly affect the data quality downstream. To account for this sort of thing I wrote a tool in go to assert constraints like this date field should never be empty or there should be at least n records where transport == foo . The tool makes a 100 most-recent record query against the current production data-set and vets the constraints against the data, to provide a unit-test flavored data-validation. I am currently in the process of replacing that tool with Cue.

I use the go bindings for my use case, but IMO for an entry-point into getting acquainted with what cuelang is, and performing the kind of playful, tinkering, experimentation of the sort most engineers prefer when first encountering a new tool, cmd/cue is the place to start. Totally agree the bindings are in dire need of the same tutorial treatment, but I also think there's a very organic progression from wow this is cli is cool, cue seems very relevant to several problems I have into I wonder if there are language bindings for this, and I think that direction is definitely something the tutorial I'm describing above could build into.

[Lyoness]

I also think there's a very organic progression from wow this is cli is cool, cue seems very relevant to several problems I have into I wonder if there are language bindings for this, and I think that direction is definitely something the tutorial I'm describing above could build into.

Excellent suggestion for a learning progression. This helps us prioritize which tooling-related tutorials should come first, and what to build on.

[aakarim]

My thoughts from the call:

• Diataxis looks like a great way to organise technical documentation for users already convinced that CUE is a worthwhile solution to their problems. However, my biggest hurdle in managing multiple contracting teams is to get them to understand the value of CUE in the first place. They also need help in finding where it fits into their existing workflows. A ‘story-orientated’ style of documentation will do well to get them to this point, where they can look at stories from across the industry as to how CUE has helped solve their problems. Once this is in place it sets the stage for an ‘AHA’ moment where they then start moving onto more concrete tutorials and guides.
• As a heavy Go user I expected the package system to be similar to the Go package system. After struggling for a little while I realised that it’s almost the inverse, where files are combined with ancestor package files to form ‘instances’. I now see the value in this approach because you can build clean schema and policy abstractions, but it is counter-intuitive for programmers coming from other languages, so the more content that exists explaining this the better.

[Lyoness]

A ‘story-orientated’ style of documentation will do well to get them to this point, where they can look at stories from across the industry as to how CUE has helped solve their problems. Once this is in place it sets the stage for an ‘AHA’ moment where they then start moving onto more concrete tutorials and guides.

Thank you for the feedback. We are planning on adding user stories and case studies on cuelang.org

[package management] ... is counter-intuitive for programmers coming from other languages, so the more content that exists explaining this the better.

Thank you for this! Will add to the shortlist for content.

[cedricgc]

Hi all, I took the time to review Diataxis yesterday and I think it is a great option. Before reading I lacked a theory of knowledge on what each type of documentation can solve which is cleared up nicely. I intend to compile a set of tutorials/how-to we can write to get developers productive in CUE.

One realization going through the categories is that it is hard for end users to connect without examples of end-to-end workflows that result in some useful outcome. Much is this is because:

1. We are focused on building out the primitives like the language stability, package management, and engine improvements
2. Configuration is so much less opinionated than general purpose code (there is no main function) that people do not know how to tie everything together.

I figure we take a common problem like expanding CUE configuration to file outputs and build a generalized solution that is a complete answer to how to solve a problem. Naturally, this will require a more opinionated approach to config and have users learn some of the primitives of package management, importing, and automation

[Lyoness]

I intend to compile a set of tutorials/how-to we can write to get developers productive in CUE.

Looking forward to seeing the list.

Agree that examples of end-to-end workflows are needed, and the challenges (and opportunities!) for the more complex workflows.

[cedricgc]

Idea that fits in the explanation quadrant: We need to create a clear argument on why using a configuration language (with its restrictions) is the better approach than using general purpose languages for config.

I have my own ideas on that, mainly that the restrictions we place on the language make it simpler to analyze, transform, and create new kinds of tools to work with data. The long term vision is great, but the issue today is we have the restrictions but the innovative capabilities are not available yet. We want to create the stories as @aakarim said that show how using CUE opens up new ways of development not available before.

[aakarim]

Dagster has some really good examples of 'this vs that': https://dagster.io/blog/dagster-airflow and https://dagster.io/blog/rebundling-the-data-platform . Those articles were very helpful in bringing us to adopt Dagster, because we understood clearly where it fit into, and where it deviated from our existing mental models for data pipelines.

Really, they're thought-leadership pieces, because ultimately its goal is to change the way you think about the problem rather than suggesting a direct alternative. I see this as analogous with the problems that CUE solves for.

[cedricgc]

Explanation Quadrant: Have some documentation that explains the differences between structs and definitions that is SEO friendly. I can see understanding the semantics of each something that people will need to look up multiple times before it becomes intuitive.

Right now you can find the details in the language spec: https://cuelang.org/docs/references/spec/#structs

A separate breakdown should be an article once the language is hammered out

[hlindberg]

I would like to see simple examples of typical constraints and normalisations. As always documentation tends to do the "step 1: draw a line, step2: draw three lines forming a triangle, step 3: draw lots of lines and triangles to make a photo perfect tiger"...

• required property
• optional property
• mutually exclusive properties
• unique property value (for example all X.ID in a context)
• property value is a reference and must exist as the value of a property at a specified place in the document
• invariants ; if color == "blue", then temperature < hot, etc. x.min < x.max
• scaling; a property is measured in megabyte and I want to allow user input to be strings with value and scale Kb, Mb, Gb... and convert the value to an int (in megabyte).
• default value for a property
• assigning unique values like "n1", "n2" to a property as default value (property is an id and must exist, and could be filled in automatically where missing).
• no duplicate objects in a list
• min/ max number of elements in a list
• given an Elevator rated at max: 1000kg, make sure sum of people's weights in a list of People is < 1000kg
• a property regNbr that references a Car (with this regNbr), and the car must have color: "blue"
• A property that determines the data type of another property {type: "string", value "blah"}, {type: "int", value: 42}

I hope this is of use to you. These are all things I need to do in my project (although in a different problem domain than Cars, Elevators and People :-) )

[Lyoness]

Very helpful @hlindberg! Thanks!

[fab29p]

In the process of using and explaining Cue to others, I had to write a lot of markdown with cue examples. As I missed mkdocs code block annotations (among other useful stuff), I took the time to create a pygments lexer for Cue. I guess it's not perfect but it's good enough for my needs. I don't have time to invest in a proper publication in pygments. If anybody is interested to start from there, I will gladly share my work.

[myitcv]

Thanks @fab29p - would be delighted to help in this respect because it has overlap with the work that we are doing as part of https://alpha.cuelang.org/. For that we are using https://github.com/alecthomas/chroma which is the highlighter behind https://gohugo.io/. Is there somewhere you want to share your work thus far?

[fab29p]

I sent you an email with the file.