The first example is misleading.

[philiplai]

This line of code is simply not working.
var linearPredictor = model.LastTransformer;
There is no LastTransformer property.

Can someone post a working example?

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: 5ec45442-8e5f-a544-cc9a-58483a383513
• Version Independent ID: 040169f9-d169-cbbd-099e-f04a7a5a6b3f
• Content: PermutationFeatureImportanceExtensions.PermutationFeatureImportance Method (Microsoft.ML)
• Content Source: dotnet/xml/Microsoft.ML/PermutationFeatureImportanceExtensions.xml
• Product: dotnet-ml-api
• GitHub Login: @natke
• Microsoft Alias: nakersha

[frank-dong-ms-zz]

@philiplai Thanks for using ML.NET. This sample works for me, see the screenshot below:

Please make sure you copy the full code sample, add latest version of ML.NET from nuget as reference and try again. If that still not working please share your repro project and a screenshot indicating problem so we can investigate.

[antoniovs1029]

@philiplai if your problem is that you're trying to use .LastTransformer in your own code, and NOT in the sample you've referenced above, then this situation might be related to: dotnet/docs#19006

In general this might be solved by using something like the following:

            var lastTransformer = ((TransformerChain <ITransformer>)model).LastTransformer;
            var linearPredictor = (ISingleFeaturePredictionTransformer<object>)lastTransformer;

            var permutationMetrics = mlContext.MulticlassClassification
                .PermutationFeatureImportance(linearPredictor, transformedData,
                permutationCount: 30);

but as explained in the issue I linked above, the precise castings would depend on your particular scenario.

[philiplai]

Thanks! I made a mistake. Sorry!

[frank-dong-ms-zz]

Feel free to reopen or raise new issue if you have any further question, close this issue now.

[philiplai]

Thanks, Frank. I got an error on var model = pipeline.Fit(data) after I plugged in my data.

System.InvalidOperationException: 'The asynchronous operation has not completed.'

Any ideas?

[frank-dong-ms-zz]

Could you please share your code and data so I can take a look?

[philiplai]

My code is almost same as the example. It seems it is an known issue:

https://stackoverflow.com/questions/62976568/system-invalidoperationexception-the-asynchronous-operation-has-not-completed

[philiplai]

@philiplai if your problem is that you're trying to use .LastTransformer in your own code, and NOT in the sample you've referenced above, then this situation might be related to: dotnet/docs#19006

In general this might be solved by using something like the following:

            var lastTransformer = ((TransformerChain <ITransformer>)model).LastTransformer;
            var linearPredictor = (ISingleFeaturePredictionTransformer<object>)lastTransformer;

            var permutationMetrics = mlContext.MulticlassClassification
                .PermutationFeatureImportance(linearPredictor, transformedData,
                permutationCount: 30);

but as explained in the issue I linked above, the precise castings would depend on your particular scenario.

I tried this method. But the program was stuck on this line of code below:

        var permutationMetrics = mlContext.BinaryClassification
            .PermutationFeatureImportance(linearPredictor, predictions,
            permutationCount: 30);

I have tried everything possible since the beginning of this year. Just can never get this feature work.

[frank-dong-ms-zz]

@philiplai this looks like an bug introduced at latest 1.5.1 version of ML.NET, see similar issue for more context:
dotnet/machinelearning-samples#833
#5312

so change the ML.NET version back to 1.5.0 should fix the asynchronous operation has not completed issue, please have a try, thanks.

[frank-dong-ms-zz]

I have tried everything possible since the beginning of this year. Just can never get this feature work.

What are other issue that prevent you from using this feature? This async operation issue should only occurs at 1.5.1

[antoniovs1029]

Hi, @philiplai . It's not clear to me either if you can't run your code because of the System.InvalidOperationException: 'The asynchronous operation has not completed.' which @frank-dong-ms has already said will be fixed in upcoming release, or if you're (also) having this problem:

I tried this method. But the program was stuck on this line of code below:
var permutationMetrics = mlContext.BinaryClassification
.PermutationFeatureImportance(linearPredictor, predictions,
permutationCount: 30);

For this problem, please notice that PFI is a resource intensive algorithm, so it usually takes some time to execute and this isn't a bug, but the nature of the algorithm itself. It might also be the case that you might have several columns and this also affects the time it takes for PFI to run. You can take a look at what I've answered to another user who was having a similar problem with waiting time for PFI:
#5247 (comment)

notice that, as I explained there, some transformers create several (even thousands) of columns, and for these transformers PFI isn't suitable as it will take A LOT of time (several hours) and it wouldn't provide much useful information anyway.

If you're able to share your code and/or model, along with your dataset, it could be easier to assess if this is causing your problem with PFI.

[philiplai]

Thank you, @frank-dong-ms @antoniovs1029 . After I downloaded the latest version, PFI was running.