AzureNative.App.ContainerApp failure to add referenced Azure Key Vault secret.

[mentallabyrinth]

What happened?

We're trying to add a secret which references Azure Key Vault into our Azure Container App. However, attempts to update fail with this error message:

azure-native:app:ContainerApp (dev-spa-eus2-aca0001):
    error: Code="ContainerAppSecretInvalid" Message="Invalid Request: Container app secret(s) with name(s) 'testing123' are invalid: value or keyVaultUrl and identity should be provided."

Example

Here's the code that adds the secret into the container app.

new AzureNative.App.Inputs.SecretArgs
{
      Name = "testing123",
      KeyVaultUrl = "https://dev-api-kv01.vault.azure.net/secrets/AZUREADCLIENT--CLIENTSECRET",
      Identity = "/subscriptions/<redacted>/resourcegroups/dev-app-eus2-rg0001fdfde73a/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dev-api-eus2-mi0001a15f69b1s"
}

The failure occurs after running pulumi -s dev up

We were successful in adding a key vault referenced secret manually using the same configuration as above in the Azure portal. Inspecting the resource JSON we see the following:

{
      "name": "testing",
      "keyVaultUrl": "https://dev-api-kv01.vault.azure.net/secrets/AZUREADCLIENT--CLIENTSECRET",
      "identity": "/subscriptions/<redacted>/resourcegroups/dev-app-eus2-rg0001fdfde73a/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dev-api-eus2-mi0001a15f69b1"
}

Output of pulumi about

CLI
Version 3.90.1
Go Version go1.21.9
Go Compiler gc

Plugins
NAME VERSION
azure-native 2.38.0
azuread 5.48.0
azuredevops 3.0.0
dotnet unknown
random 4.16.0
twingate 0.0.48

Host
OS darwin
Version 13.5.2
Arch arm64

This project is written in dotnet: executable='/nix/store/d38z3dpvqpaq8az30nnlp2lq37fjprbi-dotnet-sdk-8.0.101/bin/dotnet' version='8.0.101'

Dependencies:
NAME VERSION
Azure.Containers.ContainerRegistry 1.1.1
Pulumi 3.63.0
Pulumi.AzureAD 5.48.0
Pulumi.AzureDevOps 3.0.0
Pulumi.AzureNative 2.38.0
Pulumi.Random 4.16.0

Pulumi locates its logs in /var/folders/mz/vmtwh2n56nzbm7nshzkhfngh0000gn/T/ by default

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[justinvp]

This looks specific to Azure Native, so moving to the appropriate repo and someone will take a look soon.

[thomas11]

I've written a complete program to reproduce this issue, see below.

This upstream issue suggests that the error value or keyVaultUrl and identity should be provided can be misleading and the actual problem might be an input validation error. In that light, I've experimented with several modifications of secret naming, identity, etc.; but to no avail.

Via verbose logging I verified that the request Pulumi sends to Azure matches the correct one from the portal, yet the request is denied.

It seems like either an Azure issue to me at this point, or a misconfiguration in the identity and Key Vault access. Pulumi seems to be sending the program input to Azure faithfully.

We could try to reproduce the issue with an ARM template or a similar deployment without Pulumi.

using Pulumi;
using AzureNative = Pulumi.AzureNative;

const string sub = "redacted";
const string tenant = "redacted";
var identity = "redacted";
var identityFull =  Output.Format($"/subscriptions/{sub}/resourceGroups/pulumi-dev-shared/providers/Microsoft.ManagedIdentity/userAssignedIdentities/test-identity");

// Repro for https://github.com/pulumi/pulumi-azure-native/issues/3243
return await Deployment.RunAsync(() =>
{
    var resourceGroup = new AzureNative.Resources.ResourceGroup("rg", new()
    {
        Location = "East US",
    });

    var kv = new AzureNative.KeyVault.Vault("kv", new()
    {
        Location = "East US",
        ResourceGroupName = resourceGroup.Name,
        Properties = new AzureNative.KeyVault.Inputs.VaultPropertiesArgs
        {
            TenantId = tenant,
            Sku = new AzureNative.KeyVault.Inputs.SkuArgs
            {
                Family = "A",
                Name = AzureNative.KeyVault.SkuName.Standard,
            },
            AccessPolicies = new[]
            {
                new AzureNative.KeyVault.Inputs.AccessPolicyEntryArgs
                {
                    TenantId = tenant,
                    ObjectId = identity,
                    Permissions = new AzureNative.KeyVault.Inputs.PermissionsArgs
                    {
                        Secrets = {Union<string, AzureNative.KeyVault.SecretPermissions>.FromT1(AzureNative.KeyVault.SecretPermissions.All)}
                    },
                },            
            },
        },
    });

    var secret = new AzureNative.KeyVault.Secret("secret", new()
    {
        SecretName = "testing",
        VaultName = kv.Name,
        ResourceGroupName = resourceGroup.Name,
        Properties = new AzureNative.KeyVault.Inputs.SecretPropertiesArgs
        {
            Value = "supersecret",
        },
    });

    var managedEnvironment = new AzureNative.App.ManagedEnvironment("managedEnvironment", new()
    {
        EnvironmentName = "testcontainerenv",
        Location = "East US",
        ResourceGroupName = resourceGroup.Name,
        Sku = new AzureNative.App.Inputs.EnvironmentSkuPropertiesArgs
        {
            Name = AzureNative.App.SkuName.Consumption,
        },
    });

    var containerApp = new AzureNative.App.ContainerApp("containerApp", new()
    {
        Configuration = new AzureNative.App.Inputs.ConfigurationArgs
        {
            Secrets = new[]
            {
                new AzureNative.App.Inputs.SecretArgs
                {
                    Name = "testing",
                    KeyVaultUrl = Output.Format($"https://{kv.Name}.azure.net/secrets/{secret.Name}"),
                    Identity = identityFull
                }
            },
        },
        ContainerAppName = "testcontainerapp0",
        EnvironmentId = managedEnvironment.Id,
        Identity = new AzureNative.App.Inputs.ManagedServiceIdentityArgs
        {
            Type = AzureNative.App.ManagedServiceIdentityType.UserAssigned,
            UserAssignedIdentities = { { identityFull } }
        },
        Location = "East US",
        ResourceGroupName = resourceGroup.Name,
        Template = new AzureNative.App.Inputs.TemplateArgs
        {
            Containers = new[]
            {
                new AzureNative.App.Inputs.ContainerArgs
                {
                    Image = "mcr.microsoft.com/deployment-environments/runners/core:latest",
                    Name = "testcontainerapp0",
                    Probes = new[]
                    {
                        new AzureNative.App.Inputs.ContainerAppProbeArgs
                        {
                            HttpGet = new AzureNative.App.Inputs.ContainerAppProbeHttpGetArgs
                            {
                                HttpHeaders = new[]
                                {
                                    new AzureNative.App.Inputs.ContainerAppProbeHttpHeadersArgs
                                    {
                                        Name = "Custom-Header",
                                        Value = "Awesome",
                                    },
                                },
                                Path = "/health",
                                Port = 8080,
                            },
                            InitialDelaySeconds = 3,
                            PeriodSeconds = 3,
                            Type = AzureNative.App.Type.Liveness,
                        },
                    },
                },
            },
            InitContainers = new[]
            {
                new AzureNative.App.Inputs.InitContainerArgs
                {
                    Args = new[]
                    {
                        "-c",
                        "while true; do echo hello; sleep 10;done",
                    },
                    Command = new[]
                    {
                        "/bin/sh",
                    },
                    Image = "mcr.microsoft.com/deployment-environments/runners/core:latest",
                    Name = "testinitcontainerapp0",
                    Resources = new AzureNative.App.Inputs.ContainerResourcesArgs
                    {
                        Cpu = 0.5,
                        Memory = "1Gi",
                    },
                },
            },
            Scale = new AzureNative.App.Inputs.ScaleArgs
            {
                MaxReplicas = 5,
                MinReplicas = 1,
                Rules = new[]
                {
                    new AzureNative.App.Inputs.ScaleRuleArgs
                    {
                        Custom = new AzureNative.App.Inputs.CustomScaleRuleArgs
                        {
                            Metadata =
                            {
                                { "concurrentRequests", "50" },
                            },
                            Type = "http",
                        },
                        Name = "httpscalingrule",
                    },
                },
            },
        },
        WorkloadProfileType = "GeneralPurpose",
    }, new CustomResourceOptions { DependsOn = { secret } } );
});

[MatteoCalabro-TomTom]

In case you haven't resolved, it was a bug in Azure API. Using container apps api version v20230501 or v20240301 (these two aren't preview) has no issue with KV secrets

[thomas11]

The added docs are live now. I hope that helps.

[MatteoCalabro-TomTom]

The added docs are live now. I hope that helps.

That is sql server ☺️

[thomas11]

Gah, I commented on the wrong issue. Sorry!

[mikhailshilkov]

@thomas11 Have you tried #3243 (comment)? Maybe we can close the issue?

[thomas11]

@mikhailshilkov I'm happy to take @MatteoCalabro-TomTom's word for it :)

However, other users could still run into this. I'd like us to

1. Add docs to this resource that warn of this caveat
2. File an issue to consider this when picking the v3 default version

[MatteoCalabro-TomTom]

@mikhailshilkov I'm happy to take @MatteoCalabro-TomTom's word for it :)

However, other users could still run into this. I'd like us to

1. Add docs to this resource that warn of this caveat
2. File an issue to consider this when picking the v3 default version

There should be a new version coming still, which will probably become stable, bringing Otel support to ACA.
At the moment, v20230501 seems to be the one used by Azure Portal.