Bug with filtered indices

[albertvillanova]

Describe the bug

As reported by @partiallytyped (and by @Muennighoff):

• map and filter not working properly in multiprocessing with the new release 2.6.0 #5111 (comment)

There is an issue with the indices of a filtered dataset.

Steps to reproduce the bug

ds = Dataset.from_dict({"num": [0, 1, 2, 3]})
ds = ds.filter(lambda num: num % 2 == 0, input_columns="num", batch_size=2)
assert all(item["num"] % 2 == 0 for item in ds)

Expected results

The indices of the filtered dataset should correspond to the examples with "language" equals to "english".

Actual results

Indices to items with other languages are included in the filtered dataset indices

Preliminar investigation

It seems a bug introduced by:

• Fast dataset iter #5030

[albertvillanova]

The issue is here:

datasets/src/datasets/arrow_dataset.py

Line 2964 in 3ad9644

for i, batch in enumerate(pbar):

[albertvillanova]

@partiallytyped, @Muennighoff: the issue is fixed.

We are planning to make a patch release today.

[m-rph]

Thanks a lot for the swift response! For a brief moment yesterday I thought I had gone insane 🤣On 14 Oct 2022, at 15:44, Albert Villanova del Moral ***@***.***> wrote: @partiallytyped, @Muennighoff: the issue is fixed. We are planning to make a patch release today. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>