Refreshing project involving machine learning services managed network settings causes resources to be replaced #3225
Comments
pierskarsenbarg
added
kind/bug
|
This looks like a similar problem to #611 and #1112 - where certain sub-property collections are also represented as standalone resources. One oddity here is that the standalone resource gets removed on refresh. We'd need to identify why this isn't returned correctly by the API as there might be something else going on. Otherwise, the existing mechanism should be sufficient to handle maintaining the existing property transparently if it wasn't originally set. |
|
It looks like there's an issue with this resource's GET function. Running the repro myself showed this warning in the diagnostics: Here's the repro, modified to include the missing Vaultusing Pulumi;
using Pulumi.AzureNative.Resources;
using Pulumi.AzureNative.Storage;
using OperationalInsights = Pulumi.AzureNative.OperationalInsights;
using Pulumi.AzureNative.OperationalInsights.Inputs;
using Insights = Pulumi.AzureNative.Insights;
using Pulumi.AzureNative.KeyVault;
using MLS = Pulumi.AzureNative.MachineLearningServices;
using Pulumi.AzureNative.MachineLearningServices.Inputs;
using System;
return await Pulumi.Deployment.RunAsync(() =>
{
var resourceGroup = new ResourceGroup("pl-cs-ml-rg", new ResourceGroupArgs
{
Location = "uksouth"
});
var storageAccount = new StorageAccount("mlsa", new Pulumi.AzureNative.Storage.StorageAccountArgs
{
ResourceGroupName = resourceGroup.Name,
Sku = new Pulumi.AzureNative.Storage.Inputs.SkuArgs
{
Name = Pulumi.AzureNative.Storage.SkuName.Standard_LRS
},
Kind = Kind.StorageV2
});
var appInsights = new OperationalInsights.Workspace("appInsights", new()
{
Features = new WorkspaceFeaturesArgs
{
EnableLogAccessUsingOnlyResourcePermissions = true
},
PublicNetworkAccessForIngestion = OperationalInsights.PublicNetworkAccessType.Enabled,
PublicNetworkAccessForQuery = OperationalInsights.PublicNetworkAccessType.Enabled,
ResourceGroupName = resourceGroup.Name,
RetentionInDays = 30,
Sku = new WorkspaceSkuArgs
{
Name = OperationalInsights.WorkspaceSkuNameEnum.PerGB2018
},
WorkspaceCapping = new WorkspaceCappingArgs
{
DailyQuotaGb = -1
},
WorkspaceName = "pkmlcsopworkspace"
});
var insightsComponent = new Insights.Component("component", new()
{
ApplicationType = Insights.ApplicationType.Web,
FlowType = "Redfield",
IngestionMode = Insights.IngestionMode.LogAnalytics,
Kind = "web",
PublicNetworkAccessForIngestion = Insights.PublicNetworkAccessType.Enabled,
PublicNetworkAccessForQuery = Insights.PublicNetworkAccessType.Enabled,
RequestSource = "IbizaMachineLearningExtension",
ResourceGroupName = resourceGroup.Name,
ResourceName = "pkmlcsinworkspace",
RetentionInDays = 90,
WorkspaceResourceId = appInsights.Id
});
var vault = new Vault("vault", new VaultArgs
{
ResourceGroupName = resourceGroup.Name,
Properties = new Pulumi.AzureNative.KeyVault.Inputs.VaultPropertiesArgs
{
TenantId = "xxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxx",
Sku = new Pulumi.AzureNative.KeyVault.Inputs.SkuArgs
{
Family = "A",
Name = Pulumi.AzureNative.KeyVault.SkuName.Standard,
},
}
});
var mlWorkspace = new MLS.V20231001.Workspace("mlworkspace", new()
{
ApplicationInsights = insightsComponent.Id,
FriendlyName = "pk-ml-cs-workspace",
Description = "",
DiscoveryUrl = "https://uksouth.api.azureml.ms/discovery",
HbiWorkspace = false,
Identity = new MLS.V20231001.Inputs.ManagedServiceIdentityArgs
{
Type = MLS.V20231001.ManagedServiceIdentityType.SystemAssigned
},
KeyVault = vault.Id,
ManagedNetwork = new MLS.V20231001.Inputs.ManagedNetworkSettingsArgs
{
IsolationMode = "AllowInternetOutbound",
OutboundRules = new InputMap<object>()
},
PublicNetworkAccess = MLS.V20231001.PublicNetworkAccess.Enabled,
ResourceGroupName = resourceGroup.Name,
Sku = new MLS.V20231001.Inputs.SkuArgs
{
Name = "Basic",
Tier = MLS.V20231001.SkuTier.Basic
},
StorageAccount = storageAccount.Id,
WorkspaceName = "pk-ml-cs-workspace"
});
var rule = new MLS.ManagedNetworkSettingsRule("mlsrule", new()
{
ResourceGroupName = resourceGroup.Name,
WorkspaceName = mlWorkspace.Name,
Properties = new PrivateEndpointOutboundRuleArgs
{
Destination = new PrivateEndpointDestinationArgs
{
SparkEnabled = false,
ServiceResourceId = storageAccount.Id,
SubresourceTarget = "blob"
},
Category = MLS.RuleCategory.UserDefined,
Type = "PrivateEndpoint"
},
});
});Running this with verbose logs shows that it's attempting to perform the GET against According to the specification, this endpoint should be Running this under a debugger shows that the URL is being constructed from the resource's Side note: I don't seem able to delete this rule resource within the portal either - nothing happens when I click the delete button: |
|
No you have to delete the ml workspace to delete the rule in the portal. |
|
When performing the create, we calculate the The Azure response contains an pulumi-azure-native/provider/pkg/provider/provider.go Lines 909 to 911 in d422a42 However, on this API, Azure is returning an identifier which is missing the subscription and resource group context: Original ID: "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/pl-cs-ml-rgxxxxxx/providers/Microsoft.MachineLearningServices/workspaces/pk-ml-cs-workspace-2/outboundRules/mlsrule"
Azure's ID: "pk-ml-cs-workspace-2/outboundRules/mlsrule"This incorrect identifier is then used for subsequent requests, which then fail. |
|
It knows it’s attached to the workspace because it attaches the rule to the workspace which is weird. I thought it was like that vnet/subnet issue from before |
|
Manually fixing the Therefore there's two separate issues at play here:
|
|
As it stands, I would advise against using the standalone rule resource until it's fixed by Azure as it's not currently workable given the broken API. At that point, it would be worth considering the enhancement to make it easier to manage rules just as external resources similar to VNets/Subnets. |
|
Yeah I’ve already advised them not to but they were only doing it because MS told them to. I don’t know how keen they are to start pushing MS to fix it but I’ll find out |
|
Ok, marking as blocked as there's no reason to start work on the ability to use standalone rule resources until we can successfully rules as resources without errors from the API. |
|
@danielrbradley which API endpoint is returning the wrong id? We are engaging with MS on this from our side. |
|
@Werner-Swart-83 the incorrect ID is being returned after awaiting the result of the e.g. |
|
@danielrbradley MS has confirmed that the fix for that endpoint has been applied to WestEurope. |
|
That's great! Thank you for the update @Werner-Swart-83 |
What happened?
When creating a
MachineLearningService.Workspaceit's possible to create the managednetwork outbound rules as a separate resource (and apparently customers are encouraged to do so by Microsoft). However, when you do this and runpulumi refreshit updates the workspace resource with the values set in the network settings resource and then deletes the network settings rule resource.Then when the update happens, Pulumi recreates the settingsrule resource (because it no longer exists in Azure) and clears out the managed settings rule in the workspace and then the same thing happens all over again.
Example
Code can be found here: https://github.com/pulumi/customer-engineering/tree/master/customer-support/5019 (private repo)
Output of
pulumi aboutAdditional context
If you look at state export that I've taken before and after the refresh, you can see here that the
outboundRulesinput is empty and we havenetworksettingsruleresource here: https://github.com/pulumi/customer-engineering/blob/60d8ce2e2835bab670bce807e276934186ebb1fd/customer-support/5019/stack-before-refresh.json#L421-L479but after the refresh the ML workspace now has the network settings in the
outboundRulesection: https://github.com/pulumi/customer-engineering/blob/60d8ce2e2835bab670bce807e276934186ebb1fd/customer-support/5019/stack-after-refresh.json#L324-L334 and the setting rule resource is gone.I've also added https://github.com/pulumi/customer-engineering/blob/master/customer-support/5019/out.txt which are verbose logs and if you search for
HTTP/2.0 404 Not Foundyou can see that it's the rules that can't be found.Contributing
Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).
The text was updated successfully, but these errors were encountered: