Merge #11310 #11341 #11345

11310: feat: sign pulumi binaries with cosign r=AaronFriel a=dirien

<!--- 
Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. -->

This PR adds cosign to the build process of the Pulumi binaries. I changed the pipeline so cosign can sign without keys by authenticating with OIDC. GitHub supports this.

## Checklist

<!--- Please provide details if the checkbox below is to be left unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the Pulumi Service,
then the service should honor older versions of the CLI where this change would not exist.
You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Service API version
  <!-- `@Pulumi` employees: If yes, you must submit corresponding changes in the service repo. -->


11341: Add tfconvert to pulumi convert r=Frassle a=Frassle

<!--- 
Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. -->

This adds tfconvert to `pulumi convert`, inline with the intention that all conversions will be mediated by the CLI/engine.

There are two reasons for adding other conversion systems to pulumi convert:
1. It means only the engine needs to worry about calling into all the language code generators, tfconvert now just has to concern itself with returning PCL output.
2. It means only the engine needs to worry about plugin and schema loading, tfconvert now just gets an interface passed to it for accessing schemas.

The longer term intention is that we shouldn't have a direct build link to tfconvert here, but instead "pluginify" converters in some way. Hopefully yaml, terraform, arm, helm, etc could all be handled by individual plugins.

With that in mind, we should name this such that it fits into our plugin system. I've currently set the `--from` argument as "terraform" but we probably don't want to use that as a plugin name because that would map to github.com/pulumi/pulumi-terraform which already exists and is a provider for terraform state.

I think we _probably_ want to go with "hcl", but open to other suggestions.

## Checklist

<!--- Please provide details if the checkbox below is to be left unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the Pulumi Service,
then the service should honor older versions of the CLI where this change would not exist.
You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Service API version
  <!-- `@Pulumi` employees: If yes, you must submit corresponding changes in the service repo. -->


11345: Do not reference Pulumi nuget if a project reference to Pulumi.csproj already exists r=Zaid-Ajaj a=Zaid-Ajaj

When generating dotnet SDKs, if a schema doesn't specify a package reference to Pulumi nuget, we add it automatically by default. However, when generating test dotnet sdks, we also use project references that refer to the local Pulumi SDK and it is not correct to have either (although usually it compiles if you don't use latest SDK changes) 

This PR makes it so that if we are already referencing a local Pulumi SDK via a project reference, then we don't add a package reference to Pulumi



Co-authored-by: Engin Diri <engin.diri@ediri.de>
Co-authored-by: Fraser Waters <fraser@pulumi.com>
Co-authored-by: Zaid Ajaj <zaid.naom@gmail.com>

.github/workflows/ci-build-binaries.yml

@@ -48,6 +48,9 @@ jobs:
     env:
       PULUMI_VERSION: ${{ inputs.version }}
 
+    permissions:
+      id-token: write
+
     steps:
       - name: "Windows cache workaround"
         # https://github.com/actions/cache/issues/752#issuecomment-1222415717

.github/workflows/ci-prepare-release.yml

@@ -3,6 +3,8 @@ name: Prepare
 permissions:
   # To create a draft release
   contents: write
+  # To sign artifacts.
+  id-token: write
 
 on:
   workflow_call:
@@ -47,29 +49,47 @@ jobs:
         with:
           ref: ${{ inputs.ref }}
 
-      - name: Install rsign2
-        uses: baptiste0928/cargo-install@bf6758885262d0e6f61089a9d8c8790d3ac3368f # v1.3.0
-        with:
-          crate: rsign2
-          version: 0.6.1
-
       - name: Install b3sum
         uses: baptiste0928/cargo-install@bf6758885262d0e6f61089a9d8c8790d3ac3368f # v1.3.0
         with:
           crate: b3sum
           version: 1.3.0
 
+      - uses: sigstore/cosign-installer@9becc617647dfa20ae7b1151972e9b3a2c338a2b # v2.8.1
+
       - name: Download all artifacts
         uses: actions/download-artifact@v2
         with:
           path: artifacts.tmp
-
+      - name: Rename SDKs
+        # This step must match the rename SDKs step in the "publish" job below.
+        run: |
+          (
+            cd artifacts.tmp/artifacts-dotnet-sdk
+            for file in *.nupkg ; do
+              mv -vT "$file" "sdk-dotnet-$file"
+            done
+          )
+          (
+            cd artifacts.tmp/artifacts-python-sdk
+            for file in *.whl ; do
+              mv -vT "$file" "sdk-python-$file"
+            done
+          )
+          (
+            cd artifacts.tmp/artifacts-nodejs-sdk
+            for file in *.tgz ; do
+              mv -vT "$file" "sdk-nodejs-$file"
+            done
+          )
       - name: Flatten artifact directories
         run: |
           mkdir -p ./artifacts
-          mkdir -p ./sums.tmp
           mv ./artifacts.tmp/artifacts-*/* ./artifacts
 
+      - name: Create sums.tmp
+        run: mkdir -p ./sums.tmp
+
         # Each of these commands strips the ./ prefix to match existing (<=3.39) formatting.
       - name: Checksums with SHA256
         working-directory: artifacts
@@ -85,30 +105,19 @@ jobs:
         working-directory: artifacts
         run: sha512sum ./* | sed 's/.\///' | tee ../sums.tmp/SHA512SUMS
 
-      - name: Sign checksums
-        working-directory: sums.tmp
-        # Requires a signing key to be configured.
-        if: false
+      - name: Sign binaries and checksums
         shell: bash
         env:
-          # RELEASE_KEY: ${{ secrets.RELEASE_KEY }}
           version: ${{ inputs.version }}
         run: |
-          set -u
-          releaseKey="$(mktemp -d)/release.key"
-          echo "$RELEASE_KEY" > "${releaseKey}"
-          set -x
-          for file in *; do
-            echo | rsign sign \
-              -p "${GITHUB_WORKSPACE}/.github/workflows/release.pub" \
-              -s "${releaseKey}" \
-              -t "${{ inputs.project }} v$version signed with automated key" \
-              -c 'see website for signing information' \
-              -x "${file}.auto.minisig" \
-              "${file}"
+          ls -la
+          # Sign all artifacts and checksums:
+          for file in ./{artifacts,sums.tmp}/*; do
+              echo "$file"
+              COSIGN_EXPERIMENTAL=1 cosign sign-blob \
+                --bundle="./sums.tmp/${file}".sig  \
+                "${file}"
           done
-          rm "${releaseKey}"
-          cat ./*.auto.minisig
 
       - uses: actions/upload-artifact@v2
         with:
@@ -136,6 +145,7 @@ jobs:
         with:
           path: artifacts.tmp
       - name: Rename SDKs
+        # This step must match the rename SDKs step in the "sign" job above.
         run: |
           (
             cd artifacts.tmp/artifacts-dotnet-sdk

.github/workflows/ci.yml

@@ -2,6 +2,7 @@ name: CI
 
 permissions:
   contents: read
+  id-token: write
 
 on:
   workflow_call:

.github/workflows/on-merge.yml

@@ -3,6 +3,8 @@ name: Merge
 permissions:
   # To create a draft release.
   contents: write
+  # To sign artifacts.
+  id-token: write
 
 on:
   push:
@@ -37,6 +39,8 @@ jobs:
     uses: ./.github/workflows/ci.yml
     permissions:
       contents: read
+      # To sign artifacts.
+      id-token: write
     with:
       ref: ${{ github.ref }}
       version: ${{ needs.info.outputs.version }}
@@ -49,6 +53,8 @@ jobs:
     uses: ./.github/workflows/ci-prepare-release.yml
     permissions:
       contents: write
+      # To sign artifacts.
+      id-token: write
     with:
       ref: ${{ github.ref }}
       version: ${{ needs.info.outputs.version }}

.github/workflows/on-pr.yml

@@ -1,8 +1,12 @@
 name: Pull Request
 
 permissions:
+  # To create a draft release.
   contents: write
+  # To comment on PRs.
   pull-requests: write
+  # To sign artifacts.
+  id-token: write
 
 on:
   pull_request:
@@ -42,6 +46,8 @@ jobs:
     uses: ./.github/workflows/ci.yml
     permissions:
       contents: read
+      # To sign artifacts.
+      id-token: write
     with:
       ref: ${{ github.ref }}
       version: ${{ needs.info.outputs.version }}
@@ -70,6 +76,8 @@ jobs:
     uses: ./.github/workflows/ci-prepare-release.yml
     permissions:
       contents: write
+      # To sign artifacts.
+      id-token: write
     with:
       ref: ${{ github.ref }}
       version: ${{ needs.info.outputs.version }}

.github/workflows/pr-test-acceptance-on-dispatch.yml

@@ -12,6 +12,8 @@ permissions:
   contents: read
   # Only the 'changelog-comment' job should use this permission.
   pull-requests: write
+  # To sign artifacts.
+  id-token: write
 
 concurrency:
   group: ${{ github.workflow }}-${{ github.event.client_payload.pull_request.number }}
@@ -51,6 +53,8 @@ jobs:
     uses: pulumi/pulumi/.github/workflows/ci.yml@master
     permissions:
       contents: read
+      # To sign artifacts.
+      id-token: write
     with:
       ref: refs/pull/${{ github.event.client_payload.pull_request.number }}/merge
       version: ${{ needs.info.outputs.version }}

changelog/pending/20221109--ci--sign-pulumi-binaries-with-cosign.yaml

@@ -0,0 +1,4 @@
+changes:
+- type: feat
+  scope: ci
+  description: GitHub release artifacts are now signed using [cosign](https://github.com/sigstore/cosign) and signatures are uploaded to the [Rekor transparency log](https://rekor.tlog.dev/).

changelog/pending/20221114--cli--added-from-terraform-to-pulumi-convert.yaml

@@ -0,0 +1,4 @@
+changes:
+- type: feat
+  scope: cli
+  description: Added "--from=tf" to pulumi convert.

pkg/cmd/pulumi/convert.go

@@ -25,6 +25,7 @@ import (
 	"github.com/spf13/cobra"
 
 	javagen "github.com/pulumi/pulumi-java/pkg/codegen/java"
+	tfgen "github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tf2pulumi/convert"
 	yamlgen "github.com/pulumi/pulumi-yaml/pkg/pulumiyaml/codegen"
 	"github.com/pulumi/pulumi/pkg/v3/codegen/dotnet"
 	gogen "github.com/pulumi/pulumi/pkg/v3/codegen/go"
@@ -209,6 +210,11 @@ func runConvert(cwd string, from string, language string, outDir string, generat
 		} else {
 			return result.FromError(fmt.Errorf("unrecognized source %s", from))
 		}
+	} else if from == "tf" {
+		proj, program, err = tfgen.Eject(cwd, loader)
+		if err != nil {
+			return result.FromError(fmt.Errorf("could not load terraform program: %w", err))
+		}
 	} else {
 		return result.FromError(fmt.Errorf("unrecognized source %s", from))
 	}

pkg/codegen/dotnet/gen.go

@@ -2182,9 +2182,25 @@ func genProjectFile(pkg *schema.Package,
 	if packageReferences == nil {
 		packageReferences = map[string]string{}
 	}
+
+	// if we don't have a package reference to Pulumi SDK from nuget
+	// we need to add it, unless we are referencing a local Pulumi SDK project via a project reference
 	if _, ok := packageReferences["Pulumi"]; !ok {
-		packageReferences["Pulumi"] = "[3.23.0,4)"
+		referencedLocalPulumiProject := false
+		for _, projectReference := range projectReferences {
+			if strings.HasSuffix(projectReference, "Pulumi.csproj") {
+				referencedLocalPulumiProject = true
+				break
+			}
+		}
+
+		// only add a package reference to Pulumi if we're not referencing a local Pulumi project
+		// which we usually do when testing schemas locally
+		if !referencedLocalPulumiProject {
+			packageReferences["Pulumi"] = "[3.23.0,4)"
+		}
 	}
+
 	w := &bytes.Buffer{}
 	err := csharpProjectFileTemplate.Execute(w, csharpProjectFileTemplateContext{
 		XMLDoc:            fmt.Sprintf(`.\%s.xml`, assemblyName),

pkg/codegen/testing/test/testdata/azure-native-nested-types/dotnet/Pulumi.AzureNative.csproj

@@ -45,7 +45,6 @@
   </ItemGroup>
 
   <ItemGroup>
-    <PackageReference Include="Pulumi" Version="[3.23.0,4)" />
     <PackageReference Include="System.Collections.Immutable" Version="5.0.0" />
   </ItemGroup>
 

pkg/codegen/testing/test/testdata/cyclic-types/dotnet/Pulumi.Example.csproj

@@ -45,7 +45,6 @@
   </ItemGroup>
 
   <ItemGroup>
-    <PackageReference Include="Pulumi" Version="[3.23.0,4)" />
   </ItemGroup>
 
   <ItemGroup>

pkg/codegen/testing/test/testdata/dash-named-schema/dotnet/Pulumi.FooBar.csproj

@@ -45,7 +45,6 @@
   </ItemGroup>
 
   <ItemGroup>
-    <PackageReference Include="Pulumi" Version="[3.23.0,4)" />
     <PackageReference Include="Pulumi.Aws" Version="4.20" ExcludeAssets="contentFiles" />
     <PackageReference Include="Pulumi.Kubernetes" Version="3.7" ExcludeAssets="contentFiles" />
     <PackageReference Include="Pulumi.Random" Version="4.2" ExcludeAssets="contentFiles" />

pkg/codegen/testing/test/testdata/dashed-import-schema/dotnet/Pulumi.Plant.csproj

@@ -45,7 +45,6 @@
   </ItemGroup>
 
   <ItemGroup>
-    <PackageReference Include="Pulumi" Version="[3.23.0,4)" />
   </ItemGroup>
 
   <ItemGroup>

pkg/codegen/testing/test/testdata/functions-secrets/dotnet/Pulumi.Mypkg.csproj

@@ -45,7 +45,6 @@
   </ItemGroup>
 
   <ItemGroup>
-    <PackageReference Include="Pulumi" Version="[3.23.0,4)" />
   </ItemGroup>
 
   <ItemGroup>

pkg/codegen/testing/test/testdata/hyphen-url/dotnet/Pulumi.Registrygeoreplication.csproj

@@ -45,7 +45,6 @@
   </ItemGroup>
 
   <ItemGroup>
-    <PackageReference Include="Pulumi" Version="[3.23.0,4)" />
     <PackageReference Include="Pulumi.AzureNative" Version="1.28.*" ExcludeAssets="contentFiles" />
   </ItemGroup>
 

pkg/codegen/testing/test/testdata/naming-collisions/dotnet/Pulumi.Example.csproj

@@ -45,7 +45,6 @@
   </ItemGroup>
 
   <ItemGroup>
-    <PackageReference Include="Pulumi" Version="[3.23.0,4)" />
   </ItemGroup>
 
   <ItemGroup>

pkg/codegen/testing/test/testdata/nested-module-thirdparty/dotnet/Pulumi.FooBar.csproj

@@ -45,7 +45,6 @@
   </ItemGroup>
 
   <ItemGroup>
-    <PackageReference Include="Pulumi" Version="[3.23.0,4)" />
   </ItemGroup>
 
   <ItemGroup>

pkg/codegen/testing/test/testdata/output-funcs-edgeorder/dotnet/Pulumi.Myedgeorder.csproj

@@ -45,7 +45,6 @@
   </ItemGroup>
 
   <ItemGroup>
-    <PackageReference Include="Pulumi" Version="[3.23.0,4)" />
   </ItemGroup>
 
   <ItemGroup>

pkg/codegen/testing/test/testdata/output-funcs-tfbridge20/dotnet/Pulumi.Mypkg.csproj

@@ -50,7 +50,6 @@
     <PackageReference Include="Moq" Version="4.13.1" />
     <PackageReference Include="NUnit" Version="3.12.0" />
     <PackageReference Include="NUnit3TestAdapter" Version="4.2.1" />
-    <PackageReference Include="Pulumi" Version="[3.23.0,4)" />
   </ItemGroup>
 
   <ItemGroup>

pkg/codegen/testing/test/testdata/output-funcs/dotnet/Pulumi.Mypkg.csproj

@@ -50,7 +50,6 @@
     <PackageReference Include="Moq" Version="4.13.1" />
     <PackageReference Include="NUnit" Version="3.12.0" />
     <PackageReference Include="NUnit3TestAdapter" Version="4.2.1" />
-    <PackageReference Include="Pulumi" Version="[3.23.0,4)" />
   </ItemGroup>
 
   <ItemGroup>

pkg/codegen/testing/test/testdata/plain-and-default/dotnet/Pulumi.FooBar.csproj

@@ -45,7 +45,6 @@
   </ItemGroup>
 
   <ItemGroup>
-    <PackageReference Include="Pulumi" Version="[3.23.0,4)" />
   </ItemGroup>
 
   <ItemGroup>

pkg/codegen/testing/test/testdata/plain-object-defaults/dotnet/Pulumi.Example.csproj

@@ -45,7 +45,6 @@
   </ItemGroup>
 
   <ItemGroup>
-    <PackageReference Include="Pulumi" Version="[3.23.0,4)" />
   </ItemGroup>
 
   <ItemGroup>

pkg/codegen/testing/test/testdata/plain-object-disable-defaults/dotnet/Pulumi.Example.csproj

@@ -45,7 +45,6 @@
   </ItemGroup>
 
   <ItemGroup>
-    <PackageReference Include="Pulumi" Version="[3.23.0,4)" />
   </ItemGroup>
 
   <ItemGroup>

pkg/codegen/testing/test/testdata/plain-schema-gh6957/dotnet/Pulumi.Xyz.csproj

@@ -45,7 +45,6 @@
   </ItemGroup>
 
   <ItemGroup>
-    <PackageReference Include="Pulumi" Version="[3.23.0,4)" />
     <PackageReference Include="Pulumi.Aws" Version="4.*" ExcludeAssets="contentFiles" />
   </ItemGroup>
 

pkg/codegen/testing/test/testdata/regress-8403/dotnet/Pulumi.Mongodbatlas.csproj

@@ -45,7 +45,6 @@
   </ItemGroup>
 
   <ItemGroup>
-    <PackageReference Include="Pulumi" Version="[3.23.0,4)" />
   </ItemGroup>
 
   <ItemGroup>

pkg/codegen/testing/test/testdata/regress-node-8110/dotnet/Pulumi.My8110.csproj

@@ -45,7 +45,6 @@
   </ItemGroup>
 
   <ItemGroup>
-    <PackageReference Include="Pulumi" Version="[3.23.0,4)" />
   </ItemGroup>
 
   <ItemGroup>