Compatibility Issue with joblib during serialization of scipy object in Python 3.11.5
#1544
Comments
|
I can't reproduce in my environment. Could you please provide the full traceback you have with your minimal reproducer and give more details on your environment (OS/kernel version/cpu architecture) ? I wonder if it could be related to #1545 . |
|
Hi, thanks for following up the issue! I just noticed this issue might be related to the I created a new env in anaconda3 called Then with If I just run the script in a from joblib import Parallel, delayed
import scipy.stats
def compute_scipy_function(input_data):
# Simple use of a scipy function in the function to be parallelized
result = scipy.stats.norm.pdf(input_data, loc=0, scale=1)
return result
if __name__ == "__main__":
input_data_list = [1, 2, 3, 4, 5]
results = Parallel(n_jobs=2)(delayed(compute_scipy_function)(i) for i in input_data_list)
print("Results:", results)Output: Then I installed the Then if I run the cell in VS Code Jupyter notebook or the Fatal Python error: init_sys_streams: can't initialize sys standard streams
Python runtime state: core initialized
OSError: [Errno 9] Bad file descriptor
Current thread 0x00007ff853081100 (most recent call first):
<no Python frame>
Fatal Python error: init_sys_streams: can't initialize sys standard streams
Python runtime state: core initialized
OSError: [Errno 9] Bad file descriptor
Current thread 0x00007ff853081100 (most recent call first):
<no Python frame>
exception calling callback for <Future at 0x103898b10 state=finished raised BrokenProcessPool>
joblib.externals.loky.process_executor._RemoteTraceback:
"""
Traceback (most recent call last):
File "/Users/kang/opt/anaconda3/envs/test2/lib/python3.11/site-packages/joblib/externals/loky/process_executor.py", line 391, in _process_worker
call_item = call_queue.get(block=True, timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/kang/opt/anaconda3/envs/test2/lib/python3.11/multiprocessing/queues.py", line 122, in get
return _ForkingPickler.loads(res)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/kang/opt/anaconda3/envs/test2/lib/python3.11/site-packages/joblib/externals/cloudpickle/cloudpickle.py", line 649, in subimport
__import__(name)
File "/Users/kang/opt/anaconda3/envs/test2/lib/python3.11/site-packages/scipy/__init__.py", line 67, in <module>
from scipy._lib import _pep440
File "/Users/kang/opt/anaconda3/envs/test2/lib/python3.11/site-packages/scipy/_lib/__init__.py", line 12, in <module>
from scipy._lib._testutils import PytestTester
File "/Users/kang/opt/anaconda3/envs/test2/lib/python3.11/site-packages/scipy/_lib/_testutils.py", line 21, in <module>
_tags = list(sys_tags())
^^^^^^^^^^^^^^^^
File "/Users/kang/opt/anaconda3/envs/test2/lib/python3.11/site-packages/packaging/tags.py", line 543, in sys_tags
yield from cpython_tags(warn=warn)
File "/Users/kang/opt/anaconda3/envs/test2/lib/python3.11/site-packages/packaging/tags.py", line 212, in cpython_tags
platforms = list(platforms or platform_tags())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/kang/opt/anaconda3/envs/test2/lib/python3.11/site-packages/packaging/tags.py", line 400, in mac_platforms
version_str = subprocess.run(
^^^^^^^^^^^^^^^
File "/Users/kang/opt/anaconda3/envs/test2/lib/python3.11/subprocess.py", line 571, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/Users/kang/opt/anaconda3/envs/test2/bin/python', '-sS', '-c', 'import platform; print(platform.mac_ver()[0])']' returned non-zero exit status 1.
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/kang/opt/anaconda3/envs/test2/lib/python3.11/site-packages/joblib/externals/loky/_base.py", line 26, in _invoke_callbacks
callback(self)
File "/Users/kang/opt/anaconda3/envs/test2/lib/python3.11/site-packages/joblib/parallel.py", line 385, in __call__
self.parallel.dispatch_next()
File "/Users/kang/opt/anaconda3/envs/test2/lib/python3.11/site-packages/joblib/parallel.py", line 834, in dispatch_next
if not self.dispatch_one_batch(self._original_iterator):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/kang/opt/anaconda3/envs/test2/lib/python3.11/site-packages/joblib/parallel.py", line 901, in dispatch_one_batch
self._dispatch(tasks)
File "/Users/kang/opt/anaconda3/envs/test2/lib/python3.11/site-packages/joblib/parallel.py", line 819, in _dispatch
job = self._backend.apply_async(batch, callback=cb)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/kang/opt/anaconda3/envs/test2/lib/python3.11/site-packages/joblib/_parallel_backends.py", line 556, in apply_async
future = self._workers.submit(SafeFunction(func))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/kang/opt/anaconda3/envs/test2/lib/python3.11/site-packages/joblib/externals/loky/reusable_executor.py", line 176, in submit
return super().submit(fn, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/kang/opt/anaconda3/envs/test2/lib/python3.11/site-packages/joblib/externals/loky/process_executor.py", line 1129, in submit
raise self._flags.broken
joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed to un-serialize. Please ensure that the arguments of the function are all picklable.
joblib.externals.loky.process_executor._RemoteTraceback:
"""
Traceback (most recent call last):
File "/Users/kang/opt/anaconda3/envs/test2/lib/python3.11/site-packages/joblib/externals/loky/process_executor.py", line 391, in _process_worker
call_item = call_queue.get(block=True, timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/kang/opt/anaconda3/envs/test2/lib/python3.11/multiprocessing/queues.py", line 122, in get
return _ForkingPickler.loads(res)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/kang/opt/anaconda3/envs/test2/lib/python3.11/site-packages/joblib/externals/cloudpickle/cloudpickle.py", line 649, in subimport
__import__(name)
File "/Users/kang/opt/anaconda3/envs/test2/lib/python3.11/site-packages/scipy/__init__.py", line 67, in <module>
from scipy._lib import _pep440
File "/Users/kang/opt/anaconda3/envs/test2/lib/python3.11/site-packages/scipy/_lib/__init__.py", line 12, in <module>
from scipy._lib._testutils import PytestTester
File "/Users/kang/opt/anaconda3/envs/test2/lib/python3.11/site-packages/scipy/_lib/_testutils.py", line 21, in <module>
_tags = list(sys_tags())
^^^^^^^^^^^^^^^^
File "/Users/kang/opt/anaconda3/envs/test2/lib/python3.11/site-packages/packaging/tags.py", line 543, in sys_tags
yield from cpython_tags(warn=warn)
File "/Users/kang/opt/anaconda3/envs/test2/lib/python3.11/site-packages/packaging/tags.py", line 212, in cpython_tags
platforms = list(platforms or platform_tags())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/kang/opt/anaconda3/envs/test2/lib/python3.11/site-packages/packaging/tags.py", line 400, in mac_platforms
version_str = subprocess.run(
^^^^^^^^^^^^^^^
File "/Users/kang/opt/anaconda3/envs/test2/lib/python3.11/subprocess.py", line 571, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/Users/kang/opt/anaconda3/envs/test2/bin/python', '-sS', '-c', 'import platform; print(platform.mac_ver()[0])']' returned non-zero exit status 1.
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/kang/Downloads/MyPlayground/2024/04/20240403/joblib testing.py", line 11, in <module>
results = Parallel(n_jobs=2)(delayed(compute_scipy_function)(i) for i in input_data_list)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/kang/opt/anaconda3/envs/test2/lib/python3.11/site-packages/joblib/parallel.py", line 1098, in __call__
self.retrieve()
File "/Users/kang/opt/anaconda3/envs/test2/lib/python3.11/site-packages/joblib/parallel.py", line 975, in retrieve
self._output.extend(job.get(timeout=self.timeout))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/kang/opt/anaconda3/envs/test2/lib/python3.11/site-packages/joblib/_parallel_backends.py", line 567, in wrap_future_result
return future.result(timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/kang/opt/anaconda3/envs/test2/lib/python3.11/concurrent/futures/_base.py", line 456, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/Users/kang/opt/anaconda3/envs/test2/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/Users/kang/opt/anaconda3/envs/test2/lib/python3.11/site-packages/joblib/externals/loky/_base.py", line 26, in _invoke_callbacks
callback(self)
File "/Users/kang/opt/anaconda3/envs/test2/lib/python3.11/site-packages/joblib/parallel.py", line 385, in __call__
self.parallel.dispatch_next()
File "/Users/kang/opt/anaconda3/envs/test2/lib/python3.11/site-packages/joblib/parallel.py", line 834, in dispatch_next
if not self.dispatch_one_batch(self._original_iterator):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/kang/opt/anaconda3/envs/test2/lib/python3.11/site-packages/joblib/parallel.py", line 901, in dispatch_one_batch
self._dispatch(tasks)
File "/Users/kang/opt/anaconda3/envs/test2/lib/python3.11/site-packages/joblib/parallel.py", line 819, in _dispatch
job = self._backend.apply_async(batch, callback=cb)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/kang/opt/anaconda3/envs/test2/lib/python3.11/site-packages/joblib/_parallel_backends.py", line 556, in apply_async
future = self._workers.submit(SafeFunction(func))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/kang/opt/anaconda3/envs/test2/lib/python3.11/site-packages/joblib/externals/loky/reusable_executor.py", line 176, in submit
return super().submit(fn, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/kang/opt/anaconda3/envs/test2/lib/python3.11/site-packages/joblib/externals/loky/process_executor.py", line 1129, in submit
raise self._flags.broken
joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed to un-serialize. Please ensure that the arguments of the function are all picklable.And the traceback from the Jupyter notebook: {
"name": "BrokenProcessPool",
"message": "A task has failed to un-serialize. Please ensure that the arguments of the function are all picklable.",
"stack": "---------------------------------------------------------------------------
_RemoteTraceback Traceback (most recent call last)
_RemoteTraceback:
\"\"\"
Traceback (most recent call last):
File \"/Users/kang/opt/anaconda3/envs/test2/lib/python3.11/site-packages/joblib/externals/loky/process_executor.py\", line 391, in _process_worker
call_item = call_queue.get(block=True, timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File \"/Users/kang/opt/anaconda3/envs/test2/lib/python3.11/multiprocessing/queues.py\", line 122, in get
return _ForkingPickler.loads(res)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File \"/Users/kang/opt/anaconda3/envs/test2/lib/python3.11/site-packages/joblib/externals/cloudpickle/cloudpickle.py\", line 649, in subimport
__import__(name)
File \"/Users/kang/opt/anaconda3/envs/test2/lib/python3.11/site-packages/scipy/__init__.py\", line 67, in <module>
from scipy._lib import _pep440
File \"/Users/kang/opt/anaconda3/envs/test2/lib/python3.11/site-packages/scipy/_lib/__init__.py\", line 12, in <module>
from scipy._lib._testutils import PytestTester
File \"/Users/kang/opt/anaconda3/envs/test2/lib/python3.11/site-packages/scipy/_lib/_testutils.py\", line 21, in <module>
_tags = list(sys_tags())
^^^^^^^^^^^^^^^^
File \"/Users/kang/opt/anaconda3/envs/test2/lib/python3.11/site-packages/packaging/tags.py\", line 543, in sys_tags
yield from cpython_tags(warn=warn)
File \"/Users/kang/opt/anaconda3/envs/test2/lib/python3.11/site-packages/packaging/tags.py\", line 212, in cpython_tags
platforms = list(platforms or platform_tags())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File \"/Users/kang/opt/anaconda3/envs/test2/lib/python3.11/site-packages/packaging/tags.py\", line 400, in mac_platforms
version_str = subprocess.run(
^^^^^^^^^^^^^^^
File \"/Users/kang/opt/anaconda3/envs/test2/lib/python3.11/subprocess.py\", line 571, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/Users/kang/opt/anaconda3/envs/test2/bin/python', '-sS', '-c', 'import platform; print(platform.mac_ver()[0])']' returned non-zero exit status 1.
\"\"\"
The above exception was the direct cause of the following exception:
BrokenProcessPool Traceback (most recent call last)
Cell In[1], line 11
9 if __name__ == \"__main__\":
10 input_data_list = [1, 2, 3, 4, 5]
---> 11 results = Parallel(n_jobs=2)(delayed(compute_scipy_function)(i) for i in input_data_list)
12 print(\"Results:\", results)
File ~/opt/anaconda3/envs/test2/lib/python3.11/site-packages/joblib/parallel.py:1098, in Parallel.__call__(self, iterable)
1095 self._iterating = False
1097 with self._backend.retrieval_context():
-> 1098 self.retrieve()
1099 # Make sure that we get a last message telling us we are done
1100 elapsed_time = time.time() - self._start_time
File ~/opt/anaconda3/envs/test2/lib/python3.11/site-packages/joblib/parallel.py:975, in Parallel.retrieve(self)
973 try:
974 if getattr(self._backend, 'supports_timeout', False):
--> 975 self._output.extend(job.get(timeout=self.timeout))
976 else:
977 self._output.extend(job.get())
File ~/opt/anaconda3/envs/test2/lib/python3.11/site-packages/joblib/_parallel_backends.py:567, in LokyBackend.wrap_future_result(future, timeout)
564 \"\"\"Wrapper for Future.result to implement the same behaviour as
565 AsyncResults.get from multiprocessing.\"\"\"
566 try:
--> 567 return future.result(timeout=timeout)
568 except CfTimeoutError as e:
569 raise TimeoutError from e
File ~/opt/anaconda3/envs/test2/lib/python3.11/concurrent/futures/_base.py:456, in Future.result(self, timeout)
454 raise CancelledError()
455 elif self._state == FINISHED:
--> 456 return self.__get_result()
457 else:
458 raise TimeoutError()
File ~/opt/anaconda3/envs/test2/lib/python3.11/concurrent/futures/_base.py:401, in Future.__get_result(self)
399 if self._exception:
400 try:
--> 401 raise self._exception
402 finally:
403 # Break a reference cycle with the exception in self._exception
404 self = None
File ~/opt/anaconda3/envs/test2/lib/python3.11/site-packages/joblib/externals/loky/_base.py:26, in Future._invoke_callbacks(self)
24 for callback in self._done_callbacks:
25 try:
---> 26 callback(self)
27 except BaseException:
28 LOGGER.exception(f'exception calling callback for {self!r}')
File ~/opt/anaconda3/envs/test2/lib/python3.11/site-packages/joblib/parallel.py:385, in BatchCompletionCallBack.__call__(self, out)
383 with self.parallel._lock:
384 if self.parallel._original_iterator is not None:
--> 385 self.parallel.dispatch_next()
File ~/opt/anaconda3/envs/test2/lib/python3.11/site-packages/joblib/parallel.py:834, in Parallel.dispatch_next(self)
826 def dispatch_next(self):
827 \"\"\"Dispatch more data for parallel processing
828
829 This method is meant to be called concurrently by the multiprocessing
(...)
832
833 \"\"\"
--> 834 if not self.dispatch_one_batch(self._original_iterator):
835 self._iterating = False
836 self._original_iterator = None
File ~/opt/anaconda3/envs/test2/lib/python3.11/site-packages/joblib/parallel.py:901, in Parallel.dispatch_one_batch(self, iterator)
899 return False
900 else:
--> 901 self._dispatch(tasks)
902 return True
File ~/opt/anaconda3/envs/test2/lib/python3.11/site-packages/joblib/parallel.py:819, in Parallel._dispatch(self, batch)
817 with self._lock:
818 job_idx = len(self._jobs)
--> 819 job = self._backend.apply_async(batch, callback=cb)
820 # A job can complete so quickly than its callback is
821 # called before we get here, causing self._jobs to
822 # grow. To ensure correct results ordering, .insert is
823 # used (rather than .append) in the following line
824 self._jobs.insert(job_idx, job)
File ~/opt/anaconda3/envs/test2/lib/python3.11/site-packages/joblib/_parallel_backends.py:556, in LokyBackend.apply_async(self, func, callback)
554 def apply_async(self, func, callback=None):
555 \"\"\"Schedule a func to be run\"\"\"
--> 556 future = self._workers.submit(SafeFunction(func))
557 future.get = functools.partial(self.wrap_future_result, future)
558 if callback is not None:
File ~/opt/anaconda3/envs/test2/lib/python3.11/site-packages/joblib/externals/loky/reusable_executor.py:176, in _ReusablePoolExecutor.submit(self, fn, *args, **kwargs)
174 def submit(self, fn, *args, **kwargs):
175 with self._submit_resize_lock:
--> 176 return super().submit(fn, *args, **kwargs)
File ~/opt/anaconda3/envs/test2/lib/python3.11/site-packages/joblib/externals/loky/process_executor.py:1129, in ProcessPoolExecutor.submit(self, fn, *args, **kwargs)
1127 with self._flags.shutdown_lock:
1128 if self._flags.broken is not None:
-> 1129 raise self._flags.broken
1130 if self._flags.shutdown:
1131 raise ShutdownExecutorError(
1132 'cannot schedule new futures after shutdown')
BrokenProcessPool: A task has failed to un-serialize. Please ensure that the arguments of the function are all picklable."
}I hope the information above could be of help. Thank you very much!
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hi all,
I recently noticed that there might be an issue for the latest version of
joblibto serializescipyobject. For example running the following code:will show error message:
Incompatible Environment:
If I switch back to older version of python (3.10), the script above could run without any issue.
Compatible Environment:
I haven't test the other versions of
scipy/joblibyet, but I will post further information if required.Just wondering if you have any idea if this is an issue with how
joblibserialize object or it is actually because of some changes inscipy? Any suggestions/solutions would be highly appreciated!The text was updated successfully, but these errors were encountered: