Indexing for Distributed Mode

[vibhatha]

I want to know how indexes are handled in dask distributed?

Issue is when data is being re-shuffled, the indexes are lost. So when I index a particular dataframe, how this is managed?

[jsignell]

It might be helpful to have a concrete example. If the index is explicitly set (using set_index), the indexes should not be lost on reshuffle operations.

[vibhatha]

Yes this is the case. So, once the shuffle happened, is there a re-indexing taken place after the shuffle operation to generate new indices for shuffled values? Think of a scenario where hash indexing is used.

[jsignell]

The index is still timestamp after the shuffle. Dask.dataframe does not require the data be indexed along the same column that it is partitioned on.

[vibhatha]

When shuffling takes place, data could move across different partitions, so when that move takes place, does the data move with the index value? This is possible if index is kept as an array. But in the case of a hash index, the index values are in a data structure like multimap. So in such a case, when data needs to be moved, once it is moved, data has to be re-indexed? Is this wrong?

Isn't this the case when consider a distributed mode? Even for sequential mode, after a shuffle, data either has to shuffle along with the corresponding index value (if the index kept as an array) or re-indexed if the index is kept as a multimap.

Please correct me if I am wrong.

[jsignell]

My understanding is that the data and the index values travel together. I am looking at https://docs.dask.org/en/latest/dataframe-groupby.html#shuffle-methods for instance and that's how I read it. I don't understand what you mean by a hash index or an index stored as a multimap. I really do think an example would help.

[vibhatha]

I will try to compile an example. I am just thinking in a generic way before writing my application. I wanted to broadly understand how indexing can be used with Dask in a case of data being shuffled frequently.