Engine and Golang support for shimless providers #10916

Frassle · 2022-10-04T09:02:17Z

Description

This allows the pulumi-language-go plugin to start up providers directly from .go source files.

The other language providers will be extended to support this as well in time.

Checklist

I have added tests that prove my fix is effective or that my feature works

I have run make changelog and committed the changelog/pending/<file> documenting my change

Yes, there are changes in this PR that warrants bumping the Pulumi Service API version

pulumi-bot · 2022-10-04T09:03:11Z

Changelog

[uncommitted] (2022-11-14)

Features

[engine] Engine and Golang support for language plugins starting providers directly.
#10916

Frassle · 2022-10-04T11:28:26Z

.vscode/settings.json

@@ -1,5 +1,5 @@
 {
-    "go.buildTags": "all smoke",


This stopped "run test" widgets working in non-smoke test files

I have a PR to fix this - I think I've just forgotten to merge it!

Queued: #10879

If that fails to merge for any reason, I'll pull the change into a separate PR.

AaronFriel · 2022-10-04T18:50:46Z

sdk/go/common/util/rpcutil/writer.go

+func MakeInstallDependenciesStreams(
+	server pulumirpc.LanguageRuntime_InstallDependenciesServer,
+	isTerminal bool) (io.Closer, io.Writer, io.Writer, error) {
+
+	return makeStreams(
+		func(b []byte) error {
+			return server.Send(&pulumirpc.InstallDependenciesResponse{Stdout: b})
+		},
+		func(b []byte) error {
+			return server.Send(&pulumirpc.InstallDependenciesResponse{Stderr: b})
+		},
+		isTerminal)
+}


Will npm continue to render a progress bar (re: #10901) on unix-y platforms, at least? It looks like this indirection shouldn't change the behavior.

It seems like it would be a pain to write a test to verify, so I'll just ask to confirm. (Not sure what we could test, could we snoop the stream and detect if it's sending VT commands?)

Yeh this shouldn't change the current InstallDependencies behavior, I just ended up with a slightly different proto-layout. Given we update all these things in lock-step I could probably do a sanitize pass over this to bring everything into the same system, just need to do the dance with the yaml and java repos so wouldn't want to do it just before a release day.

sdk/go/common/resource/plugin/langruntime_plugin.go

AaronFriel · 2022-10-13T20:18:00Z

proto/pulumi/language.proto

@@ -39,6 +39,9 @@ service LanguageRuntime {

    // GetProgramDependencies returns the set of dependencies required by the program.
    rpc GetProgramDependencies(GetProgramDependenciesRequest) returns (GetProgramDependenciesResponse) {}
+
+    // RunPlugin executes a plugin program and returns its result asynchronously.
+    rpc RunPlugin(RunPluginRequest) returns (stream RunPluginResponse) {}


Frustrating that we can't have a tuple of immediate data and a trailing stream in a single RPC. The response API will look really funny if we want to add a message type for communicating a structured failure starting the plugin outside of the context of running it.

Yeh I really wish grpc streams were a three type message of (immediate response message, many stream messages, final terminating message), but we work with what we've got.

AaronFriel

Looks good to me, but I'm curious about the test skip.

AaronFriel · 2022-10-13T21:21:10Z

tests/integration/integration_test.go

@@ -663,6 +663,9 @@ func testComponentProviderSchema(t *testing.T, path string) {
 		test := test
 		t.Run(test.name, func(t *testing.T) {
 			t.Parallel()
+
+			t.Skip("This needs to use a plugin host to deal with non-native-binary providers")


Why is this test being skipped when it previously passed?

So this actually only matters for Go so I'll move the skip to just that test (in tests/integration/integration_go_test.go) but it's because this method is attempting to exec the plugin binary directly and then manually wrapping the grpc machinery around it, but I've changed component_setup.sh so we don't actually build the go component anymore so there is no native binary for it to exec.

I'll just skip the go test for now, and I'll rewrite this test to use the plugin host machinery properly before the node and python components are similarly stripped down.

This allows the pulumi-language-go plugin to start up providers directly from .go source files. The other language providers will be extended to support this as well in time.

AaronFriel

I think I'm happy with these changes.

Frassle · 2022-11-17T09:08:31Z

bors merge

bors · 2022-11-17T09:40:55Z

Build succeeded:

bors-ok

Frassle force-pushed the fraser/shimlessGo branch from 0c5d7f1 to e7b4767 Compare October 4, 2022 09:14

Frassle requested review from AaronFriel and Zaid-Ajaj October 4, 2022 09:16

Frassle force-pushed the fraser/shimlessGo branch 3 times, most recently from 02735df to dee259a Compare October 4, 2022 10:14

Frassle commented Oct 4, 2022

View reviewed changes

Frassle force-pushed the fraser/shimlessGo branch 2 times, most recently from 9893164 to aa4cd71 Compare October 4, 2022 12:22

AaronFriel reviewed Oct 4, 2022

View reviewed changes

sdk/go/common/resource/plugin/langruntime_plugin.go Show resolved Hide resolved

AaronFriel mentioned this pull request Oct 5, 2022

Run plugins from language plugin #10847

Closed

3 tasks

AaronFriel force-pushed the fraser/shimlessGo branch from 46c22f3 to aa4cd71 Compare October 5, 2022 23:53

Frassle force-pushed the fraser/shimlessGo branch from aa4cd71 to ff38be7 Compare October 13, 2022 16:36

AaronFriel reviewed Oct 13, 2022

View reviewed changes

Frassle force-pushed the fraser/shimlessGo branch from ff38be7 to b19be5f Compare October 13, 2022 21:40

Frassle requested a review from kpitzen October 28, 2022 15:13

AaronFriel self-requested a review November 11, 2022 17:44

Frassle force-pushed the fraser/shimlessGo branch from b19be5f to f7ccf38 Compare November 14, 2022 10:04

Engine and Golang support for shimless providers

9e5f1cc

This allows the pulumi-language-go plugin to start up providers directly from .go source files. The other language providers will be extended to support this as well in time.

Frassle force-pushed the fraser/shimlessGo branch from f7ccf38 to 9e5f1cc Compare November 14, 2022 11:25

AaronFriel approved these changes Nov 17, 2022

View reviewed changes

bors bot merged commit 3e98035 into master Nov 17, 2022

bors bot deleted the fraser/shimlessGo branch November 17, 2022 09:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Engine and Golang support for shimless providers #10916

Engine and Golang support for shimless providers #10916

Frassle commented Oct 4, 2022 •

edited

pulumi-bot commented Oct 4, 2022 •

edited

Frassle Oct 4, 2022

AaronFriel Oct 4, 2022

AaronFriel Oct 4, 2022

AaronFriel Oct 4, 2022

AaronFriel Oct 4, 2022

Frassle Oct 4, 2022

AaronFriel Oct 13, 2022

Frassle Oct 13, 2022

AaronFriel left a comment

AaronFriel Oct 13, 2022

Frassle Oct 13, 2022

AaronFriel left a comment

Frassle commented Nov 17, 2022

bors bot commented Nov 17, 2022

Engine and Golang support for shimless providers #10916

Engine and Golang support for shimless providers #10916

Conversation

Frassle commented Oct 4, 2022 • edited

Description

Checklist

pulumi-bot commented Oct 4, 2022 • edited

Changelog

[uncommitted] (2022-11-14)

Features

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AaronFriel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AaronFriel left a comment

Choose a reason for hiding this comment

Frassle commented Nov 17, 2022

bors bot commented Nov 17, 2022

Frassle commented Oct 4, 2022 •

edited

pulumi-bot commented Oct 4, 2022 •

edited