all_gather produces garbage result

[gfx73]

Can you please help me with all_gather function? It somehow produces negative values.
Here is the code I have in on_train_epoch_end function:

        if torch.any(self.query_labels < 0):
            print('ahtung')
            raise Exception('ahtung')

        print('query_labels before all gather:', self.query_labels[0], self.query_labels.shape, self.query_labels.dtype)
        
        feature_bank = self.all_gather(self.feature_bank).view(-1, emb_dim)
        target_bank = self.all_gather(self.target_bank).view(-1)

        query_embeddings = self.all_gather(self.query_embeddings).view(-1, emb_dim)
        query_labels = self.all_gather(self.query_labels).reshape(-1)
        print('query_labels after gather:', query_labels[0], query_labels.shape, query_labels.dtype)

I explicitly check that self.query_labels doesn't have negative values. But the prints are as follows:

query_labels before all gather: tensor(3, device='cuda:1') torch.Size([21551]) torch.int64
query_labels after gather: query_labels before all gather: tensor(2, device='cuda:0') torch.Size([21955]) torch.int64
query_labels after gather: tensor(-4759082983527105483, device='cuda:1') torch.Size([43102]) torch.int64

What are the possible reasons for such behavior?

[quant-1-30]

negative torch dimension may due to precision overflow , you can use LongTensor instead of default IntTensor