[POC] Outputless Pulumi and Async Components

[AaronFriel]

This PR implements two orthogonal features and proves that they compose in the Node SDK and helps evaluate trade-offs in developer experience.

User code examples:

• Outputless
• @Component class decorator
• Functional component wrapper

Outputless Pulumi

Adds a method on Output<T> to await the value as a Promise<ResolvedOutput<T>>. The resolved promise value is a subclass of Output<T> but it provides a type safe .value property that can be inspected and used in iteration and conditional logic safely while tracking dependencies.

Dependencies are tracked by global context. Using async components, described next, or a global forkPulumiContext function allows isolating these dependency trees.

Summary

Good:

• These were quite nice to work with. It does not feel like they will often be misused.
• The implementation does not use many Node.js-specific capabilities.
  • ResolvedOutput is a subclass of Output with an extra phantom type.
  • In TypeScript, a type guard is the easiest way to runtime assert that the value can be used, in other languages we could offer type coersion or reify the resolved output to one without an "unknown/void/any" typed value.
  • Languages with explicitly passed contexts are "easy", but all supported languages have either async local context facilities (Node.js, Python, .NET, Java) or use explicit context (Go).
  • This context management is a prerequisite for inline automation API support.

Bad:

• Secret values can be accessed through .value after checking if the value is known.
  • We could add warnings when resolving secret values.
• Using apply() on a ResolvedOutput returns an Output again, which necessitates await (...).asPromise().
  • This could be fixed in some languages, but I don't think the typing would be great.
  • Wouldn't be as bad if every language used Rust style postfix await. We merely need to lobby and implement postfix await in each language's spec.

Ugly:

• Forking outputless state is the easy part. Joining outputless states is harder if a user awaits a forked outputless state. Semantics could become unclear.
  • In Node.js constructors must run synchronously and there is no clear way to bracket the user defined constructor with a context. Requires something like Async Components, below, to ensure precise tracking of dependencies.
  • Joining semantics could be non-obvious.
• Resource condemnation via delete-before-replace would necessitate design changes to dependsOn, or this behavior of implicit dependencies could cause cascading resource deletes very easily.
  • Is delete-before-replace causing a cascading delete really better than allowing a resource to be temporarily invalid?

Async Components

This adds two new mechanisms for defining components to understand the ergonomics of each. Both of these mechanisms use a ComponentResource underneath, and use async hooks to precisely track any asynchronous tasks spawned in the constructors. This allows both components to safely use asynchrony - including Outputless asynchrony - in spawned promises, emitters, intervals, and other forms of Node.js concurrency. Both of these implementations offer the same set of features:

• Automatic logical naming, with an overridable namingConvention component resource option. By default the logical names of all child resources are prefixed.
• Automatic parenting, overridable for advanced use.
• Precise awaiting on child resources.
• Automatic registerOutputs on completion of child resources.

Not implemented, but made possible by this implementation, we could precisely track inputs and outputs of the components in stack files. This would provide greatly increased visibility in Pulumi Cloud via Resource Search / Insights.

The two implementations are very similar, differing mostly in the developer experience and not in implementation:

• @Component is a class decorator, using TC-39 stage-3 compatible decorators. The user implements a class (which should not extend pulumi.ComponentResource) and applies the decorator. The decorator creates an anonymous subclass of the user declared class, wrapping the constructor to run with the correct async context tracking, modifies the opts, and then instantiates its parent class.
• FunctionalComponent is a wrapper for declaring a component as a plain function. It takes a callback which resembles a resource constructor's arguments (name, inputs, opts), and provides all of the same facilities as the decorator.

Summary

Good

• Both of these approaches are enormous wins for component dx.
• Tracking inputs and outputs in state would be huge for resource search.

Bad

• In most of our languages, constructors must be synchronous, this makes authoring components with asynchrony pretty clumsy.
• Functional components have their own downside, are unfamiliar.

Ugly:

• Hard to see a path for anything like Outputless and class-based components:

While the Component seems more familiar, the proof of concept evaluation found two major downsides:

First, there were significant challenges in making pulumi up work. The class decorator syntax requires TypeScript 5.x, and the in-box ts-node and tsconfig for Pulumi projects is very far from supporting @ syntax. Ordinarily, as when writing Pulumi projects with ESModule support, we would use a --loader. However the ESBuild and the tsx loader does not yet support decorator transpilation (evanw/esbuild#104). As a result, this only worked by installing a side-by-side version of TypeScript 5.x and down-transpiling to CommonJS and ES2022 syntax.

Second, constructors in TypeScript must be synchronous. While these wrapped constructors do precisely track any tasks spawned during the constructor call, it's very awkward to do asynchronous work inside a constructor as it necessitates spawning a promise, and though registerOutputs is called at the correct time, the constructor must synchronously prepare all properties on the class, for example by creating promise resolvers and passing those to any spawned tasks.

[pulumi-bot]

Changelog

[uncommitted] (2024-04-22)