Update python libraries and tests from CPython 3.11

[youknowone]

Though it will be sometimes blocked by rust side issues, don't worry!
Please just create PR about the broken test and and create issues related to the new bugs.

If you are new to updating cpython libraries, this guide will be useful:
https://github.com/RustPython/RustPython/wiki/How-to-update-test-files

When updating module Lib/X, please don't forget to also update Lib/test/test_X

A few libraries doesn't need to be updated because CPython also kept it same as previous version. In that case, please leave a comment to let us mark it as done. Thank you!

Tips

• To get specific version of CPython source code, try git checkout v3.11.2 from CPython repository.
• Please check Update libraries and tests from CPython 3.10.6 #4064 for example. Not the entire changes but the each commits. One of the middle size example is 17e12de . Sometimes it can be really simple like 9571a68
• Please create one PR for one library+test pair whenever possible. Once you stuck by dependencies, go to one of the dependency first and finish it before restarting the stuck one unless they become simpler to be updated at the same time. Normally adding more libraries make passing tests (a lot) harder.
• To run tests, try cargo run --features ssl -- -m test -v <test_name>. e.g. If you changed test_unicode.py, cargo run --features ssl -- -m test -v test_unicode
  • If it runs too slow, try to add --release like cargo run --release --features ssl -- -m test -v test_unicode

List of libraries

• __future__.py
  • maybe related to allow from __future__ import generator_stop #4060?
• __hello__.py Update __hello__ from cpython #4570
• __phello__ Update __hello__ from cpython #4570
• _collections_abc.pyupdated _collections_abc.py and test_collections.py to CPython 3.11.2 #5016
• _compat_pickle.py
• _compression.py Update _compression, gzip, and test_gzip for CPython v3.11.2 #4688
• _markupbase.py Update _markupbase, html/parser for CPython v3.11.2 #4703
• _osx_support.py unchanged
  • Update test__osx_support.py from CPython 3.11 #4670
• _py_abc.py
• _pycodecs.py
• _pydecimal.py
• _pyio.py
• _sitebuiltins.py
• _threading_local.py
• _weakrefset.py
• abc.py Update abc.py from Python 3.11 #4985
• aifc.py
• antigravity.py — feat(antigravity): update to 3.11 #4591
• argparse.py — feat(argparse): update to 3.11 #4592
• ast.py
• asynchat.py
• asyncio
  • Implement ascii codec in Rust #3118
  • sys.get_asyncgen_hooks and sys.get_coroutine_origin_tracking_depth #3859
• asyncore.py
• base64.py Commit 185111d
• bdb.py
• binhex.py Deprecated.
• bisect.py Update bisect.py and test_bisect.py from CPython v3.11.2 #4774
• bz2.py Unchanged
• calendar.py Update calendar.py from CPython 3.11 #4669
• cgi.py
• cgitb.py
• chunk.py
• cmd.py
• code.py
• codecs.py
  • See also io.IncrementalNewlineDecoder and codecs.IncrementalNewlineDecoder #3861
• codeop.py
• collections
  • See also Introduce _collections.OrderedDict type (a.k.a. collections.OrderedDict) #3418
• colorsys.py
• compileall.py Update compileall to CPython 3.11.5 #5070
• concurrent
• configparser.py feat(configparser): update to 3.11 #4595
• contextlib.py Update test/test_contextlib.py from CPython 3.11.2 #4649 Update contextlib.py and test_contextlib from CPython 3.11.2 #4659
• contextvars.py
  • [RFC] Implement PEP 567 (contextvars) #3188
• copy.py Update copy.py from CPython 3.11 #4674
• copyreg.py
• csv.py
• ctypes
  • Implement the ctypes module #1217
  • See also Ctypes implementation #2364
• dataclasses.py pending match statement support.
• datetime.py
• dbm
• decimal.py
• difflib.py update test_difflib from CPython3.11.2 #5063
• dis.py
  • see also Add missing dis features for modulefinder #3846
• distutils
• doctest.py
• email
• encodings
• ensurepip Update ensurepip from cpython 3.11.2 #4590
• enum.py Update enum to CPython 3.11.5 #5074
• filecmp.py
• fileinput.py Update fileinput to CPython 3.11.5 #5071
• fnmatch.py
• formatter.py Deprecated
• fractions.py Update fractions to CPython 3.11.5 #5072
• ftplib.py Update ftplib to CPython 3.11.5 #5073
• functools.py
• gc.py
  • [RFC] GC: Both Stop-the-World and On-the-fly #4158
• genericpath.py
• getopt.py
• getpass.py
• gettext.py
• glob.py Update glob.py and test/test_glob.py from CPython v3.11.2 #4744
• graphlib.py
• gzip.py Update _compression, gzip, and test_gzip for CPython v3.11.2 #4688
• hashlib.py
• heapq.py
• hmac.py unchanged
• html
  • html/parser Update _markupbase, html/parser for CPython v3.11.2 #4703
  • test_html is unchanged
• http
• imghdr.py
• imp.py
• importlib Update importlib from CPython 3.11.2 #4561
  • Add (fake) frozenmodule __origname__ and _imp.find_frozen #4559
  • Add proper __origname__ to frozen module #4565
• inspect.py
• io.py
  • Handle windows test proper way #3960
  • io.TextIOWrapper.__init__ encoding="locale" support #4662
  • See also TextIOWrapper.detach #4072
• ipaddress.py
• json
• keyword.py unchanged
• linecache.py
• locale.py Update test.support from CPython 3.11.2 #4537
  • See also locale support including _locale and time.tzname #3850
• logging
• mailbox.py
  • TextIOWrapper.detach #4072
• mimetypes.py
• multiprocessing
  • See also _multiprocessing_SemLock #3965
• netrc.py
• nntplib.py
• ntpath.py
• nturl2path.py
• numbers.py
• opcode.py
• operator.py unchanged
• optparse.py unchanged
• os.py
  • Handle windows test proper way #3960
  • See also Tracking issue of os.utime regression #4053
• pathlib.py
  • Handle windows test proper way #3960
• pdb.py
• pickle.py
  • See also replace array.__reduce__ to array.__reduce_ex__ #3876
• pickletools.py
• pkgutil.py
• platform.py
• plistlib.py
• posixpath.py
• pprint.py
• pty.py
• py_compile.py
• pydoc.py
  • pydoc_data
• queue.py
  • See also [RFC] _queue is missing #3608
• quopri.py
• random.py Fix int.from_bytes and Update random.py and test/test_random.py from CPython v3.11.2 #4748
• re.py
• reprlib.py
• rlcompleter.py
• runpy.py
• sched.py
• secrets.py
• selectors.py
• shelve.py
• shlex.py
• shutil.py
  • Handle windows test proper way #3960
• signal.py
• site.py
  • Add test_site from CPython 3.11.2 #4789
  • io.TextIOWrapper.__init__ encoding="locale" support #4662
  • Add sys.winver #4550
• smtpd.py
• smtplib.py
• sndhdr.py
• socket.py
• socketserver.py
• sqlite3
• sre_compile.py
• sre_constants.py
• sre_parse.py
• ssl.py
  • See also ssl.SSLContext miss post_handshake_auth #3927
• stat.py
• statistics.py
• string.py
• stringprep.py
• struct.py
• subprocess.py Update subprocess to CPython 3.11 #4981
• sunau.py
• sysconfig.py Update sysconfig, test_sys and test_syslog from CPython 3.11 #4569
• tabnanny.py
• tarfile.py
• telnetlib.py
• tempfile.py
• textwrap.py
• this.py
• threading.py
• timeit.py
• token.py
• tokenize.py
• tomllib
• trace.py Update trace from CPython 3.11.2 #4586
• traceback.py
• tty.py
• types.py
• typing.py
• unittest Update unittest from CPython 3.11 #4560
• urllib
• uu.py
• uuid.py
• venv
  • Handle windows test proper way #3960
• warnings.py
  • implement warnings more #4013
• weakref.py
• webbrowser.py
• wsgiref
• xdrlib.py
• xml
• xmlrpc
• zipapp.py
• zipfile.py
• zipimport.py
• test.support Update test.support from CPython 3.11.2 #4537
  • Add _imp._override_frozen_modules_for_tests #4540
  • Check our wasm32-unknown-unknown compatible with CPython emscripten build #4538
  • sys.platform of wasm32-wasi build to wasi instead of unknown #4539
  • sys.safe_path with Command line parameter -P and environment variable PYTHONSAFEPATH #4541

List of tests without python libraries

• test/test___all__.py
• test/test__locale.py
• test/test__opcode.py
• test/test_abstract_numbers.py
• test/test_array.py Update test_array.py from Cpython v3.11.2 #4745
  • See also replace array.__reduce__ to array.__reduce_ex__ #3876
• test/test_asdl_parser.py
• test/test_asyncgen.py
  • sys.get_asyncgen_hooks and sys.get_coroutine_origin_tracking_depth #3859
• test/test_atexit.py Update test_atexit from cpython 3.11.2 #4621
• test/test_audioop.py
• test/test_audit.py
• test/test_augassign.py unchanged
• test/test_baseexception.py — Update test_baseexception.py from cpython 3.11.2 #4624
• test/test_bigaddrspace.py
• test/test_bigmem.py Update test_bigmem from cpython 3.11.2 #4625
• test/test_binascii.py
  • See also Implement strict_mode keyword parameter in binascii.a2b_base64() #4374
• test/test_binop.py unchanged
• test/test_bool.py Update test/test_bool from cpython 3.11.2 #4631
• test/test_buffer.py Update test/test_buffer from CPython 3.11.2 #4632
• test/test_bufio.py unchanged
• test/test_builtin.py update test_builtin.py from cpython v3.11.2 #4765
• test/test_bytes.py Update test_bytes.py from CPython v3.11.2 #4746
• test/test_bz2.py
• test/test_c_locale_coercion.py
• test/test_call.py
• test/test_charmapcodec.py unchanged
• test/test_check_c_globals.py
• test/test_class.py
• test/test_clinic.py
• test/test_cmath.py unchanged
• test/test_cmd_line.py Update test_cmd_line from Cpython v3.11.2 #4801
• test/test_cmd_line_script.py Update test_cmd_line_script from Cpython v3.11.2 #4802
• test/test_code_module.py unchanged
• test/test_codeccallbacks.py unchanged
• test/test_compare.py Update test_compare from Cpython v3.11.2 #4803
• test/test_compile.py Update test_compile from Cpython v3.11.2 #4804
• test/test_complex.py update test_complex.py from cpython 3.11.2 #4755
• test/test_contains.py unchanged
• test/test_context.py Update test_context from Cpython v3.11.2 #4805
• test/test_contextlib_async.py
• test/test_coroutines.py
• test/test_crashers.py Update test_crashers from Cpython v3.11.2 #4807 CPython specific
• test/test_csv.py
• test/test_decorators.py Update test_decorators from Cpython v3.11.2 #4806
• test/test_defaultdict.py unchanged
• test/test_deque.py updated test_deque.py from CPython v3.11.2 #4761
• test/test_descr.py unchanged
• test/test_descrtut.py
• test/test_devpoll.py Update test_devpoll from Cpython v3.11.2 #4808
• test/test_dict.py Update test_dict.py from CPython v3.11.2 #4763
• test/test_dict_version.py Update test_dict_version.py from Cpython v3.11.2 #4809 Not implemented in RustPython
• test/test_dictcomps.py unchanged
• test/test_dictviews.py Update test_dictviews.py from Cpython v3.11.2 #4810
• test/test_docxmlrpc.py Update test_docxmlrpc.py from Cpython v3.11.2 #4811
• test/test_dtrace.py Update test_dtrace.py from Cpython v3.11.2 #4812
• test/test_dynamic.py Update test_dynamic.py from Cpython v3.11.2 #4813
• test/test_dynamicclassattribute.py unchanged
• test/test_eintr.py Update test_eintr.py from Cpython v3.11.2 #4814
• test/test_embed.py Update test_embed.py from Cpython v3.11.2 #4815 CPython specific.
• test/test_enumerate.py
• test/test_eof.py Add test_eof.py from Cpython v3.11.2 #4753
• test/test_epoll.py Update test_epoll.py from Cpython v3.11.2 #4816
• test/test_errno.py unchanged
• test/test_except_star.py
• test/test_exception_group.py Add test_exception_group.py from Cpython v3.11.2 #4817
• test/test_exception_hierarchy.py Update test_exception_hierarchy.py from Cpython v3.11.2 #4818
• test/test_exception_variations.py
• test/test_exceptions.py Update test_exceptions.py from Cpython v3.11.2 #4819
• test/test_extcall.py
• test/test_faulthandler.py Update test_faulthandler.py from Cpython v3.11.2 #4820
• test/test_fcntl.py unchanged
• test/test_file.py unchanged
• test/test_file_eintr.py
• test/test_fileio.py Update test_fileio.py from Cpython v3.11.2 #4821
• test/test_fileutils.py Add test_fileutils.py from Cpython v3.11.2 #4822 CPython specific
• test/test_finalization.py
• test/test_float.py Update test_float.py from Cpython v3.11.2 #4823
• test/test_flufl.py Add test_flufl.py from Cpython v3.11.2 #4824 CPython specific
• test/test_fork1.py
• test/test_format.py Update test_format.py from Cpython v3.11.2 #4825
• test/test_frame.py
• test/test_frozen.py
• test/test_fstring.py
• test/test_funcattrs.py Add test_funcattrs.py from Cpython v3.11.2 #4829
• test/test_future.py Update test_future.py from Cpython v3.11.2 #4830
• test/test_future3.py unchanged
• test/test_future4.py unchanged
• test/test_future5.py unchanged
• test/test_gdb.py
• test/test_generator_stop.py
• test/test_generators.py
• test/test_genericalias.py Update test_genericclass.py from Cpython v3.11.2 #4831
• test/test_genericclass.py
• test/test_genexps.py
• test/test_getpath.py
• test/test_global.py Update test_global.py from Cpython v3.11.2 #4832
• test/test_grammar.py
• test/test_grp.py Update test_grp.py from Cpython v3.11.2 #4833
• test/test_hash.py Update test_hash.py from Cpython v3.11.2 #4834
• test/test_hashlib.py Update test_hashlib.py from Cpython v3.11.2 & refactor hashlib #4835
• test/test_index.py unchanged
• test/test_int.py
• test/test_int_literal.py unchanged
• test/test_interpreters.py
• test/test_ioctl.py unchanged
• test/test_isinstance.py unchanged
• test/test_iter.py
  • Handle windows test proper way #3960
• test/test_iterlen.py unchanged
• test/test_itertools.py Update test_itertools.py from Cpython v3.11.2 #4836
• test/test_keywordonlyarg.py unchanged
• test/test_kqueue.py Add test_kqueue.py from Cpython v3.11.2 #4837
• test/test_largefile.py Add test_largefile.py from Cpython v3.11.2 #4838
• test/test_launcher.py CPython specific
• test/test_list.py
  • See also list panic when size > isize::MAX #1750, Update test_list.py from Cpython v3.11.2 #4839
• test/test_listcomps.py update test_listcomps.py from cpython 3.11.2 #4757
• test/test_lltrace.py
• test/test_long.py Update test_long.py from Cpython v3.11.2 #4840
• test/test_longexp.py unchanged
• test/test_marshal.py
  • See also marshal module is imcomplete #3458
• test/test_math.py update test_math.py from cpython 3.11.2 #4754
• test/test_memoryio.py unchanged
• test/test_memoryview.py unchanged
• test/test_metaclass.py
• test/test_minidom.py
• test/test_mmap.py
  • See also mmap module for windows #3847, Update test_mmap.py from Cpython v3.11.2 #4841
• test/test_module.py Update test_module.py from Cpython v3.11.2 #4842
• test/test_multibytecodec.py
• test/test_named_expressions.py unchanged
• test/test_nis.py
• test/test_numeric_tower.py Update test_numeric_tower.py from Cpython v3.11.2 #4843
• test/test_opcache.py Update test_opcache.py from Cpython v3.11.2 #4844
• test/test_openpty.py unchanged
• test/test_ordered_dict.py Update test_ordered_dict.py from Cpython v3.11.2 #4846
• test/test_ossaudiodev.py
• test/test_osx_env.py
• test/test_patma.py
• test/test_peepholer.py
• test/test_pep646_syntax.py
• test/test_picklebuffer.py
• test/test_pkg.py
• test/test_poll.py Add test_poll.py from Cpython v3.11.2 #4888
• test/test_popen.py Update test_popen.py from Cpython v3.11.2 #4889
• test/test_positional_only_arg.py
• test/test_posix.pyUpdate test_posix.py from Cpython v3.11.2 #4890
  • See also Enhance compatibility of os.pathconf_names #4494
• test/test_pow.py Update test_pow.py from Cpython v3.11.2 #4891
• test/test_print.py added test_print.py and marked TODOs #4778
• test/test_property.py Update test_property.py from Cpython v3.11.2 #4892
  • See also support property name #4067
• test/test_pulldom.py Add test_pulldom.py from Cpython v3.11.2 #4893
• test/test_pwd.py
• test/test_pyexpat.py
  • See also etree.XML raises TypeError on valid XML #3430
• test/test_raise.py
• test/test_range.py
• test/test_readline.py
• test/test_regrtest.py
• test/test_repl.py
• test/test_resource.py
• test/test_richcmp.py
• test/test_robotparser.py
• test/test_sax.py
• test/test_scope.py
• test/test_script_helper.py
• test/test_select.py
• test/test_set.py
  • See also &= and -= operator between Set types do not work correctly. #3992
• test/test_setcomps.py
• test/test_slice.py
• test/test_smtpnet.py
• test/test_sort.py updated test_sort from cpython 3.11.2 #4749
• test/test_source_encoding.py
• test/test_spwd.py
• test/test_stable_abi_ctypes.py
• test/test_startfile.py
• test/test_strftime.py
• test/test_string.py Update test_string.py from Cpython v3.11.2 #4751
• test/test_string_literals.py
• test/test_strptime.py
• test/test_strtod.py
• test/test_structseq.py
• See also __match_args__ for structseq #4063
• test/test_subclassinit.py
• test/test_sundry.py
• test/test_super.py
  • See also Need compare between metaclass and internal contents of classcell in __build_class__ #3865
• test/test_support.py
  • Check our wasm32-unknown-unknown compatible with CPython emscripten build #4538
  • sys.platform of wasm32-wasi build to wasi instead of unknown #4539
• test/test_syntax.py Update test_syntax from CPython 3.11.0 #4602
  • Two CPython 3.11 syntax changes involving * operator are unsupported #4479
• test.test_sys Update sysconfig, test_sys and test_syslog from CPython 3.11 #4569
  • _imp.extension_suffixes #4123
  • See also sys.get_asyncgen_hooks and sys.get_coroutine_origin_tracking_depth #3859
  • See also sys.safe_path with Command line parameter -P and environment variable PYTHONSAFEPATH #4541
• test/test_sys_setprofile.py
• test/test_sys_settrace.py
• test/test_syslog.py Update sysconfig, test_sys and test_syslog from CPython 3.11 #4569
• test/test_tcl.py
• test/test_thread.py
• test/test_threadedtempfile.py
• test/test_threadsignals.py
• test/test_time.py
  • See also locale support including _locale and time.tzname #3850
  • See also time.strftime(arg) fails when arg is not valid format string #4157
• test/test_timeout.py
• test/test_tix.py
• test/test_tuple.py
• test/test_type_annotations.py
• test/test_type_cache.py
• test/test_type_comments.py
• test/test_typechecks.py
• test/test_types.py Update test/test_types from CPython 3.11.2 #4683
• test/test_ucn.py
• test/test_unary.py
• test/test_unicode.py
• test/test_unicode_file.py
• test/test_unicode_file_functions.py Update test_unicode_file_functions.py from CPython 3.11.2 #4741
• test/test_unicode_identifiers.py
• test/test_unicodedata.py Update test_unicodedata from CPython 3.11.2 #4678
• test/test_univnewlines.py unchanged
• test/test_unpack.py Update test/test_unpack.py from CPython 3.11.2 #4676
• test/test_unpack_ex.py
• test/test_unparse.py
• test/test_userdict.py Update test/test_userlist.py and test_userdict.py from CPython 3.11.2 #4666
• test/test_userlist.py Update test/test_userlist.py and test_userdict.py from CPython 3.11.2 #4666
• test/test_userstring.py Update test/test_userstring.py from CPython 3.11.2 #4658
• test/test_utf8_mode.py unchanged
• test/test_utf8source.py unchanged
• test/test_wait3.py
• test/test_wait4.py
• test/test_weakset.py Update test/test_weakset.py from CPython 3.11.2 #4657
• test/test_winconsoleio.py
• test/test_winreg.py
• test/test_winsound.py
• test/test_with.py Update test/test_with.py from CPython 3.11.2 #4648
• test/test_yield_from.py unchanged
• test/test_zlib.py Update test/test_zlib.py from CPython 3.11.2 #4647
• test/test_asyncio
• test/test_capi
• test/test_import Add test_import from CPython 3.11.2 #4599
• test/test_tools

Not yet added libraries

These libraris are not added yet. Pure python one will be possible while others are not.

• _bootsubprocess.py
• _pycodecs.py
• cProfile.py
• crypt.py
• curses
• idlelib
• imaplib.py
• imp.py
  • See also correct impl _imp.source_hash  #4075
• lzma.py
• mailcap.py
• modulefinder.py
  • Add missing dis features for modulefinder #3846
• msilib
• pipes.py
• poplib.py
• pstats.py
• pyclbr.py
• symtable.py
• tkinter
• tracemalloc.py
• turtle.py
• wave.py
• zoneinfo

[DimitrisJim]

Pasting the comment (#4592 (review)) here too for anyone else who might need it:

You can use the following command to test your changes locally (in a manner similar to RustPython's CI):

# this runs all of the tests (not necessary if you know which test is affected)
cargo run --release --features ssl,jit -- -m test -j 1 -u all --slowest --fail-env-changed -v
# this runs only the test suite named "test_whatever" (usually located at `Lib/test/test_whatever.py`)
cargo run --release --features ssl,jit -- -m test -j 1 -u all --slowest --fail-env-changed -v test_whatever

For a list of all resources and more command-line options, you can execute cargo run --release --features ssl,jit -- -m test -h

[youknowone]

Thanks to everyone who participated in this series of patches! I am going to close this issue in favor of #5104