You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When the buffer reaches distributed.protocol.serialize.pickle_loads, buffers[0] is a bytes object.
This causes pickle_loads to deep-copy the buffer in order to honour the writeable flag of the original.
then the numpy object is no longer deserialized by distributed.protocol.serialize.pickle_loads, but it's instead processed by distributed.protocol.numpy.deserialize_numpy_array, which receives a writeable buffer
then we are using again distributed.protocol.serialize.pickle_loads, which receives a read-only buffer but this time the writeable flag is False so no deep copy happens.
The text was updated successfully, but these errors were encountered:
Ok, found the difference. the deep-copy is NOT tripped on the transfer of the task output from a to b; it's the random seed that's sent from the client to scheduler. 🤦
crusaderky
changed the title
Buffer is deep-copied in weird edge case
random seed is deep-copied in edge case
Apr 3, 2024
crusaderky
changed the title
random seed is deep-copied in edge case
np.random.RandomState buffer is deep-copied in edge case
Apr 3, 2024
crusaderky
changed the title
np.random.RandomState buffer is deep-copied in edge case
The buffer of embedded numpy variables is deep-copied from client to scheduler
Apr 3, 2024
crusaderky
changed the title
The buffer of embedded numpy variables is deep-copied from client to scheduler
The buffer of embedded numpy variables is deep-copied in client->scheduler comms
Apr 4, 2024
When the buffer reaches
distributed.protocol.serialize.pickle_loads
,buffers[0]
is a bytes object.This causes pickle_loads to deep-copy the buffer in order to honour the writeable flag of the original.
To verify, add at the top of
pickle_loads
:What's causing me a migraine is:
with
then the numpy object is no longer deserialized by
distributed.protocol.serialize.pickle_loads
, but it's instead processed bydistributed.protocol.numpy.deserialize_numpy_array
, which receives a writeable bufferthen we are using again
distributed.protocol.serialize.pickle_loads
, which receives a read-only buffer but this time the writeable flag is False so no deep copy happens.The text was updated successfully, but these errors were encountered: