Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-43348][PYTHON] Support Python 3.8 in PyPy3 #41024

Closed
wants to merge 3 commits into from
Closed

[SPARK-43348][PYTHON] Support Python 3.8 in PyPy3 #41024

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
8dfee0a
[SPARK-43348][INFRA] Upgrade PyPy to Python 3.8
dongjoon-hyun May 2, 2023
d0eea7b
Fix
dongjoon-hyun May 3, 2023
7f96222
Create SPARK-43354
dongjoon-hyun May 3, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
8 changes: 4 additions & 4 deletions dev/infra/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,10 +38,10 @@ RUN apt update
RUN $APT_INSTALL gfortran libopenblas-dev liblapack-dev
RUN $APT_INSTALL build-essential

RUN mkdir -p /usr/local/pypy/pypy3.7 && \
curl -sqL https://downloads.python.org/pypy/pypy3.7-v7.3.7-linux64.tar.bz2 | tar xjf - -C /usr/local/pypy/pypy3.7 --strip-components=1 && \
ln -sf /usr/local/pypy/pypy3.7/bin/pypy /usr/local/bin/pypy3.7 && \
ln -sf /usr/local/pypy/pypy3.7/bin/pypy /usr/local/bin/pypy3
RUN mkdir -p /usr/local/pypy/pypy3.8 && \
curl -sqL https://downloads.python.org/pypy/pypy3.8-v7.3.11-linux64.tar.bz2 | tar xjf - -C /usr/local/pypy/pypy3.8 --strip-components=1 && \
ln -sf /usr/local/pypy/pypy3.8/bin/pypy /usr/local/bin/pypy3.8 && \
ln -sf /usr/local/pypy/pypy3.8/bin/pypy /usr/local/bin/pypy3

RUN curl -sS https://bootstrap.pypa.io/get-pip.py | pypy3

Expand Down
12 changes: 2 additions & 10 deletions python/pyspark/cloudpickle/compat.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,5 @@
import sys


if sys.version_info < (3, 8):
try:
import pickle5 as pickle # noqa: F401
from pickle5 import Pickler # noqa: F401
except ImportError:
import pickle # noqa: F401
from pickle import _Pickler as Pickler # noqa: F401
else:
import pickle # noqa: F401
from _pickle import Pickler # noqa: F401
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From PyPy Python3.8, _pickle is removed.

import pickle # noqa: F401
from pickle import Pickler # noqa: F401
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the same with the upstream.

3 changes: 2 additions & 1 deletion python/pyspark/sql/tests/test_dataframe.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1454,7 +1454,8 @@ def test_create_dataframe_from_pandas_with_dst(self):
os.environ["TZ"] = orig_env_tz
time.tzset()

@unittest.skipIf(not have_pandas, pandas_requirement_message) # type: ignore
# TODO(SPARK-43354): Re-enable test_create_dataframe_from_pandas_with_day_time_interval
@unittest.skip("Fails in PyPy Python 3.8, should enable.")
def test_create_dataframe_from_pandas_with_day_time_interval(self):
# SPARK-37277: Test DayTimeIntervalType in createDataFrame without Arrow.
import pandas as pd
Expand Down