How to Deal with Missed Predictions in Confusion Matrix for binary Classification

[Eldad27]

I'm trying to generate confusion matrix for a binary classification task. After a model has predicted results, for example in object detection, some detections are missed. A saved model will therefore have a different shape (prediction values) compared with the ground truths shape (target/expected). That is '''y_trues > y_preds'''. Using confusion matrix such as below, How do you handle this to avoid discrepancies in classification report while maintaining same shape for both y_trues and y_preds?

[jnothman]

Specify all possible classes using the `labels` parameter

[Eldad27]

@jnothman . That has been done already as you see in the code '''labels=[1,0]'''. There are two classes. How does the metric handle missed detections? How does python represent/recognize the missed the missed detections in the code? How does the metric compute the '''y_trues and y_preds''' while representing missed predictions in y_preds? Sounds funny but I have tried NaN , None , " " and "-" already in place of missing values while including it as part of the possible labels parameter but none of these worked. For e.g. let's say we have two unpredicted values, I replace the two unpredicted values with Nan

actual = [1,0,0,1,0,0,1,0,0,1]
predicted = [1,0,0,1,0,0,1,0,NaN,NaN]

matrix = confusion_matrix(actual, predicted, labels=[1,0,NaN]

[jnothman]

Sorry, I had not seen your example, and I still find your use case a bit perplexing. What do you mean by a missed prediction? Rather, how did you construct your prediction array such that some of them don't align with the true array?

but none of these worked.

Indeed, what would it look like if it "worked" in the way you expect?

[Eldad27]

@jnothman To clarify 'missed predictions' = objects not detected by the model during testing phase. The task is actually an instance segmentation task using Mask rcnn model. Inference results were saved as a .pth file. To determine detected objects classes/categories, I follow facebookresearch/maskrcnn-benchmark#1300 (comment) . Results were extracted (as a np array) from the .pth file using the code below.

import torch
import numpy as np

predictions = torch.load("predictions.pth")
y_preds = [ ]
for box in predictions:
   scores = box.get_field("scores")
   labels = box.get_field("labels")
   mask = box.get_field("mask")
   mask_scores = box.get_field("mask_scores")
   y_preds = np.concatenate((y_preds, labels.numpy()), axis=None)
y_preds = np.array(y_preds, dtype=np.int32)
print(shape.y_preds)
print(y_preds)

From my little understanding, because there were missed detections, the code above prints y_preds with shape < y_trues. This means y_preds count is less than y_trues count. How does scikit-learn confusion matrix handle this please?

Please let me know where I got it wrong.

[anshu957]

@Eldad27 , I'm having the same issue and I've resolved this by replacing Nan with an arbitrary value {-1, 10, ....} other than 0,1 (for binary classification):

actual = [1,0,0,1,0,0,1,0,0,1]
predicted = [1,0,0,1,0,0,1,0,-1,-1]

matrix = confusion_matrix(actual, predicted, labels=[1,0])