[PySpark] fix raw_prediction_col parameter and minor cleanup #8067
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@trivialfis Could you help to check the env issue? tests/python/test_with_modin.py::TestModin::test_modin 2022-07-13 08:17:52,808 WARNING services.py:2013 -- WARNING: The object store is using /tmp instead of /dev/shm because /dev/shm has only 67108864 bytes available. This will harm performance! You may be able to free up space by deleting files in /dev/shm. If you are inside a Docker container, you can increase /dev/shm size by passing '--shm-size=9.69gb' to 'docker run' (or add it to the run_options list in a Ray cluster config). Make sure to set this to more than 30% of available RAM.
[2022-07-13T08:17:55.032Z] 2022-07-13 08:17:54,528 ERROR worker.py:95 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): �[36mray::apply_func()�[39m (pid=129116, ip=172.17.0.2)
[2022-07-13T08:17:55.032Z] At least one of the input arguments for this task could not be computed:
[2022-07-13T08:17:55.032Z] ray.exceptions.RaySystemError: System error: cannot import name 'DataFrame' from 'pandas.core.frame' (/opt/python/envs/cpu_test/lib/python3.7/site-packages/pandas/core/frame.py)
[2022-07-13T08:17:55.032Z] traceback: Traceback (most recent call last):
[2022-07-13T08:17:55.032Z] File "/opt/python/envs/cpu_test/lib/python3.7/site-packages/ray/serialization.py", line 340, in deserialize_objects
[2022-07-13T08:17:55.032Z] obj = self._deserialize_object(data, metadata, object_ref)
[2022-07-13T08:17:55.032Z] File "/opt/python/envs/cpu_test/lib/python3.7/site-packages/ray/serialization.py", line 237, in _deserialize_object
[2022-07-13T08:17:55.032Z] return self._deserialize_msgpack_data(data, metadata_fields)
[2022-07-13T08:17:55.032Z] File "/opt/python/envs/cpu_test/lib/python3.7/site-packages/ray/serialization.py", line 192, in _deserialize_msgpack_data
[2022-07-13T08:17:55.032Z] python_objects = self._deserialize_pickle5_data(pickle5_data)
[2022-07-13T08:17:55.032Z] File "/opt/python/envs/cpu_test/lib/python3.7/site-packages/ray/serialization.py", line 180, in _deserialize_pickle5_data
[2022-07-13T08:17:55.032Z] obj = pickle.loads(in_band, buffers=buffers)
[2022-07-13T08:17:55.032Z] File "/opt/python/envs/cpu_test/lib/python3.7/site-packages/pandas/core/frame.py", line 129, in <module>
[2022-07-13T08:17:55.032Z] from pandas.core import (
[2022-07-13T08:17:55.032Z] File "/opt/python/envs/cpu_test/lib/python3.7/site-packages/pandas/core/generic.py", line 108, in <module>
[2022-07-13T08:17:55.032Z] from pandas.core import (
[2022-07-13T08:17:55.032Z] File "/opt/python/envs/cpu_test/lib/python3.7/site-packages/pandas/core/indexing.py", line 46, in <module>
[2022-07-13T08:17:55.032Z] import pandas.core.common as com
[2022-07-13T08:17:55.032Z] File "/opt/python/envs/cpu_test/lib/python3.7/site-packages/pandas/__init__.py", line 50, in <module>
[2022-07-13T08:17:55.032Z] from pandas.core.api import (
[2022-07-13T08:17:55.032Z] File "/opt/python/envs/cpu_test/lib/python3.7/site-packages/pandas/core/api.py", line 48, in <module>
[2022-07-13T08:17:55.032Z] from pandas.core.groupby import (
[2022-07-13T08:17:55.032Z] File "/opt/python/envs/cpu_test/lib/python3.7/site-packages/pandas/core/groupby/__init__.py", line 1, in <module>
[2022-07-13T08:17:55.032Z] from pandas.core.groupby.generic import (
[2022-07-13T08:17:55.032Z] File "/opt/python/envs/cpu_test/lib/python3.7/site-packages/pandas/core/groupby/generic.py", line 73, in <module>
[2022-07-13T08:17:55.032Z] from pandas.core.frame import DataFrame
[2022-07-13T08:17:55.032Z] ImportError: cannot import name 'DataFrame' from 'pandas.core.frame' (/opt/python/envs/cpu_test/lib/python3.7/site-packages/pandas/core/frame.py)
[2022-07-13T08:17:55.032Z] 2022-07-13 08:17:54,537 ERROR worker.py:95 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): �[36mray::apply_func()�[39m (pid=129116, ip=172.17.0.2)
[2022-07-13T08:17:55.032Z] At least one of the input arguments for this task could not be computed:
[2022-07-13T08:17:55.032Z] ray.exceptions.RaySystemError: System error: cannot import name 'DataFrame' from 'pandas.core.frame' (/opt/python/envs/cpu_test/lib/python3.7/site-packages/pandas/core/frame.py)
[2022-07-13T08:17:55.032Z] traceback: Traceback (most recent call last):
[2022-07-13T08:17:55.032Z] File "/opt/python/envs/cpu_test/lib/python3.7/site-packages/ray/serialization.py", line 340, in deserialize_objects
[2022-07-13T08:17:55.032Z] obj = self._deserialize_object(data, metadata, object_ref)
[2022-07-13T08:17:55.032Z] File "/opt/python/envs/cpu_test/lib/python3.7/site-packages/ray/serialization.py", line 237, in _deserialize_object
[2022-07-13T08:17:55.032Z] return self._deserialize_msgpack_data(data, metadata_fields)
[2022-07-13T08:17:55.032Z] File "/opt/python/envs/cpu_test/lib/python3.7/site-packages/ray/serialization.py", line 192, in _deserialize_msgpack_data
[2022-07-13T08:17:55.032Z] python_objects = self._deserialize_pickle5_data(pickle5_data)
[2022-07-13T08:17:55.032Z] File "/opt/python/envs/cpu_test/lib/python3.7/site-packages/ray/serialization.py", line 180, in _deserialize_pickle5_data
[2022-07-13T08:17:55.032Z] obj = pickle.loads(in_band, buffers=buffers)
[2022-07-13T08:17:55.032Z] File "/opt/python/envs/cpu_test/lib/python3.7/site-packages/pandas/core/frame.py", line 129, in <module>
[2022-07-13T08:17:55.032Z] from pandas.core import (
[2022-07-13T08:17:55.032Z] File "/opt/python/envs/cpu_test/lib/python3.7/site-packages/pandas/core/generic.py", line 108, in <module>
[2022-07-13T08:17:55.032Z] from pandas.core import (
[2022-07-13T08:17:55.032Z] File "/opt/python/envs/cpu_test/lib/python3.7/site-packages/pandas/core/indexing.py", line 46, in <module>
[2022-07-13T08:17:55.032Z] import pandas.core.common as com
[2022-07-13T08:17:55.032Z] File "/opt/python/envs/cpu_test/lib/python3.7/site-packages/pandas/__init__.py", line 50, in <module>
[2022-07-13T08:17:55.032Z] from pandas.core.api import (
[2022-07-13T08:17:55.032Z] File "/opt/python/envs/cpu_test/lib/python3.7/site-packages/pandas/core/api.py", line 48, in <module>
[2022-07-13T08:17:55.032Z] from pandas.core.groupby import (
[2022-07-13T08:17:55.032Z] File "/opt/python/envs/cpu_test/lib/python3.7/site-packages/pandas/core/groupby/__init__.py", line 1, in <module>
[2022-07-13T08:17:55.032Z] from pandas.core.groupby.generic import (
[2022-07-13T08:17:55.032Z] File "/opt/python/envs/cpu_test/lib/python3.7/site-packages/pandas/core/groupby/generic.py", line 73, in <module>
[2022-07-13T08:17:55.032Z] from pandas.core.frame import DataFrame
[2022-07-13T08:17:55.032Z] ImportError: cannot import name 'DataFrame' from 'pandas.core.frame' (/opt/python/envs/cpu_test/lib/python3.7/site-packages/pandas/core/frame.py) |
|
@WeichenXu123 Could you help to review? |
|
You have different Python environments in your executors and driver. |
|
@WeichenXu123 @trivialfis could you help to review it? |
trivialfis
approved these changes
Jul 15, 2022
WeichenXu123
approved these changes
Jul 15, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is to fix raw_prediction_col parameter instead of raw_prediction_ol and clean up some comments