Foundational CLI library for new Plugin SDK

[aaronsteers]

This came up in a call with @pandemicsyn... and it was previously proposed for the Singer SDK by @edgarrmondragon.

We used Click for singer-sdk and also for Meltano Core but there appear to be some clear advantages to using Typer as our starting point when we create the new Plugin SDK.

Potential Advantages:

1. We get terminal auto-complete out of box.
2. We get Rich support out of box.
3. It's built on top of Click.
4. Design-wise it's more similar to FastAPI, which we are also starting to leverage on the Meltano side.

Thoughts, suggestions, alternatives?

[WillDaSilva]

I support using Typer (and FastAPI), but will caution that the stability and future of those projects is largely in the hands of Sebastián Ramírez, a.k.a. tiangolo. It probably won't be a problem, but there's a low bus-factor. Fortunately since Typer is built off of Click, we always have the option of migrating to Click if absolutely necessary without too much difficulty.

Also, regarding this point:

We get terminal auto-complete out of box.

That's only true if the user installs the shell completions for typer-cli, and then uses typer-cli to invoke our program. In practice I'm not sure any significant number of people do this. Unfortunately shell completions will remain a difficult problem for Python-based CLIs, even if we use Typer.

[aaronsteers]

the stability and future of those projects is largely in the hands of Sebastián Ramírez, a.k.a. tiangolo.

Thanks for calling this out. As you call out, there's an exit strategy in Click if we need it - and if it became a serious issue, the community may also create a fork or viable alternative. I double checked and Typer uses the MIT license, which is permissive for adoption and continued iteration.

To @rabidaudio's point, there's not a ton of impact to actual behaviors themselves, and this is more about a dev-friendly experience and streamlining the creation of initial SDK scaffolding. While Meltano (and supporting contributors) might need to scramble to migrate off Typer (or any other library) it would have little if any real functional impact to already built plugins.

Worst case scenario, if we choose to expose Typer interfaces directly to developers, removing or migrating off of Typer later would require some minimal refactoring between SDK version bumps. (Not ideal but manageable.)

[rabidaudio]

Maybe I'm missing something here, why does it matter what cli library plugins use? In my mind, plugins would export class implementations (e.g. dagster_meltano.PluginInvokerFactory) that would just get called by Meltano if the plugin supports it, defaulting to the native implementations otherwise. If the plugin has an executable, it doesn't matter what cli library (if any) it uses. In fact that executable could even be written in a language other than Python.

[aaronsteers]

@rabidaudio - You make good points and I want to highlight a few things in response.

If the plugin has an executable, it doesn't matter what cli library (if any) it uses. In fact that executable could even be written in a language other than Python.

Agreed on both points. This specific discussion is more focused on what foundation is used for the new SDK. Developers would be able to use any language and framework they'd like, just as they can do with Singer taps and targets, with or without the Singer SDK.

Maybe I'm missing something here, why does it matter what cli library plugins use?

First, I should say again, this is just for the new Plugin SDK itself - and individual plugin definitions would not need to use this framework.

While we are making a fresh start, now is a time to re-evaluate the foundations we'll build on for our first POC extensions/plugins and for the new SDK itself. Migrating from Click to Type on the Singer SDK was enough of a change and/or risk that we haven't taken the plunge on that yet. Also, in the Meltano codebase itself, we still have a dependency on the Click library. An argument could be made that migrating Meltano from Click to Typer would have some benefits in code maintainability and/or UX - and the experience we would gain could potentially inform future migrations (or not!) for Meltano core as well.

[rabidaudio]

Got it, thanks for the explanation!

I haven't used Typer before but I'm sure it's good. And I think there's some value in preferring modern python tools and design patterns, as most data folks aren't software folks and are maybe not as familiar with the most accepted way to do things. My background is software but not python and I can say I've learned a lot of modern python conventions and tooling by following Meltano

[z3z1ma]

So I am gonna riff but I think its valuable

State/Persistence

Plugins need state so we need a way for Meltano to persist and return that state. That is hugely beneficial IMO. Think of SSH where we want to cache PID of the tunnel after we spin it up. Or a Slack alert plugin which needs to know when it last sent a message. We have a Meltano db interface which already handles state in (I presume) an abstracted manner. Can we leverage that interface?
In the most decoupled way:

utility:
  capabilities:
    - state

Expectation:

1. Plugin outputs a state json to stdout (or some fd) abstracted through SDK
2. Meltano core writes state to wherever (database, state backend, whatever)
3. State is passed in via --state in the dynamically generated cli and available in context

Plugin Interface

I struggled to find a right way to interface with a plugin. The SDK needs to make this clear. For example, when we set config in meltano.yml, a plugin has no standardized way to accept the config (a la catalog.json). So a plugin if forced to use env vars which are implicitly loaded by Meltano into the executable env. Both typer and click support using env vars for defaults.

So I think it would make sense to dynamically generate cli (with click or typer) as we do with taps/targets but to set options + env dynamically based on plugin settings / config in order to generate a unified interface that is intuitive both with Meltano config and as a standalone invocation. That is slightly different than Singer today where we no the options are --state --catalog etc.

Expectation:

def main(name: str = typer.Argument("name", envvar="NAMESPACE_UTILITY_NAME")):
    print(f"Hello Mr. {name}")

This would make my_sdk_utility --name=Alex "just work" but so too would a meltano yaml file with name: Alex in a config entry.

Processes

I found psutil very useful in creating a unified abstract for subprocesses which were either invoked by the wrapper or picked up from a cached PID. 🤷

Thoughts

Forcing devs into a standard interface like up/down/status/restart makes sense for services which can then be further abstracted and utilized in interesting ways in Meltano core like (--with-service). Where is the line between services/plugins?

[pandemicsyn]

👍 nice, this is mostly pretty spot on to what I've been thinking as well (and @aaronsteers I think) and the direction im working towards. One additional piece is that, every plugin will be passed a context that describes basically, what plugins are being invoked, in what order, and what their current status is (e.g. if they've already run and whether it appears to have completed successfully or failed).

[pandemicsyn]

@aaronsteers tl;dr wonder if we shouldn't just roll with click ?

Potential Advantages:

We get terminal auto-complete out of box.

That would be handy for meltano, but seems a lot less useful for a plugin.

We get Rich support out of box.

@aaronsteers I rolled with the typer for my experimentation but fwiw the output style when rich is present isn't my cup of tea and I'm not sure it brings much to table over vanilla click.

Mainly because resizing your terminal really breaks help output formatting, and the exception formatting. This mostly looks ok in meltano invoke , but capturing and blindly bubbling up this sort of output in meltano run as it would today is just gonna make our error presentation situation...even more gnarly.

Design-wise it's more similar to FastAPI, which we are also starting to leverage on the Meltano side.

Honestly it just happens to be from the FastAPI author, im not sure there's design similarities that translate.

---

fwiw - really easy to see what I mean be terminal "resizes" causes issue by just messing with the demo's in the online docs:

[aaronsteers]

@pandemicsyn - Thanks for this feedback and for sharing your learnings here.

I don't know if there are options to mitigate the formatting issues but you make a good case to keep with Click in the Plugin SDK for now at least.