bpo-40958: Avoid buffer overflow in the parser when indexing the current line

[pablogsal]

https://bugs.python.org/issue40958

[pablogsal]

Before this PR

venv ❯ LSAN_OPTIONS="suppressions=asan-suppression.txt,print_suppressions=0" ./python -m test test_eof
0:00:00 load avg: 0.85 Run tests sequentially
0:00:00 load avg: 0.85 [1/1] test_eof
test test_eof failed -- Traceback (most recent call last):
  File "/home/pablogsal/github/python/master/Lib/test/test_eof.py", line 54, in test_line_continuation_EOF_from_file_bpo2180
    self.assertIn(b'unexpected EOF while parsing', err)
AssertionError: b'unexpected EOF while parsing' not found in b'Parser/tokenizer.c:978:50: runtime error: pointer index expression with base 0x625000016900 overflowed to 0xbebebebebebee6be\n====
=============================================================\n==48535==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x606000027c51 at pc 0x55ba1fba57d4 bp 0x7fffc21fe430 sp 0x7fffc
21fe420\nREAD of size 1 at 0x606000027c51 thread T0\n    #0 0x55ba1fba57d3 in ascii_decode Objects/unicodeobject.c:4941\n    #1 0x55ba1fca4f4a in unicode_decode_utf8 Objects/unicodeobject.c:499
9\n    #2 0x55ba203fb1bd in byte_offset_to_character_offset Parser/pegen.c:148\n    #3 0x55ba203fb1bd in _PyPegen_raise_error_known_location Parser/pegen.c:412\n    #4 0x55ba203fbe4d in _PyPege
n_raise_error Parser/pegen.c:373\n    #5 0x55ba203ff981 in tokenizer_error Parser/pegen.c:321\n    #6 0x55ba203ff981 in _PyPegen_fill_token Parser/pegen.c:638\n    #7 0x55ba2040277f in _PyPegen
_expect_token Parser/pegen.c:753\n    #8 0x55ba2041317a in _tmp_15_rule Parser/parser.c:16184\n    #9 0x55ba204024f9 in _PyPegen_lookahead (/home/pablogsal/github/python/master/python+0x1c344f9
)\n    #10 0x55ba20478e2c in compound_stmt_rule Parser/parser.c:1860\n    #11 0x55ba204815c2 in statement_rule Parser/parser.c:1224\n    #12 0x55ba204815c2 in _loop1_11_rule Parser/parser.c:159
54\n    #13 0x55ba204815c2 in statements_rule Parser/parser.c:1183\n    #14 0x55ba204854b7 in file_rule Parser/parser.c:716\n    #15 0x55ba204854b7 in _PyPegen_parse Parser/parser.c:24401\n
#16 0x55ba20405768 in _PyPegen_run_parser Parser/pegen.c:1077\n    #17 0x55ba204063ef in _PyPegen_run_parser_from_file_pointer Parser/pegen.c:1137\n    #18 0x55ba1feb53c6 in PyRun_FileExFlags P
ython/pythonrun.c:1057\n    #19 0x55ba1feb572c in PyRun_SimpleFileExFlags Python/pythonrun.c:400\n    #20 0x55ba1f8cbdbb in pymain_run_file Modules/main.c:369\n    #21 0x55ba1f8cbdbb in pymain_
run_python Modules/main.c:553\n    #22 0x55ba1f8ce154 in Py_RunMain Modules/main.c:632\n    #23 0x55ba1f8ce154 in pymain_main Modules/main.c:662\n    #24 0x55ba1f8ce154 in Py_BytesMain Modules/
main.c:686\n    #25 0x7fa4b6ba7001 in __libc_start_main (/usr/lib/libc.so.6+0x27001)\n    #26 0x55ba1f8c848d in _start (/home/pablogsal/github/python/master/python+0x10fa48d)\n\n0x606000027c51
is located 0 bytes to the right of 49-byte region [0x606000027c20,0x606000027c51)\nallocated by thread T0 here:\n    #0 0x7fa4b78db459 in __interceptor_malloc /build/gcc/src/gcc/libsanitizer/as
an/asan_malloc_linux.cpp:145\n    #1 0x55ba1fbbaa1d in PyUnicode_New Objects/unicodeobject.c:1437\n    #2 0x55ba1fcee24b in _PyUnicode_Init Objects/unicodeobject.c:15535\n    #3 0x55ba1fe895c3
in pycore_init_types Python/pylifecycle.c:599\n    #4 0x55ba1fe895c3 in pycore_interp_init Python/pylifecycle.c:724\n    #5 0x55ba1fe93b4b in pyinit_config Python/pylifecycle.c:765\n    #6 0x55
ba1fe93b4b in pyinit_core Python/pylifecycle.c:926\n    #7 0x55ba1fe95b6c in Py_InitializeFromConfig Python/pylifecycle.c:1136\n    #8 0x55ba1f8c8752 in pymain_init Modules/main.c:66\n    #9 0x
55ba1f8ce10a in pymain_main Modules/main.c:653\n    #10 0x55ba1f8ce10a in Py_BytesMain Modules/main.c:686\n    #11 0x7fa4b6ba7001 in __libc_start_main (/usr/lib/libc.so.6+0x27001)\n\nSUMMARY: A
ddressSanitizer: heap-buffer-overflow Objects/unicodeobject.c:4941 in ascii_decode\nShadow bytes around the buggy address:\n  0x0c0c7fffcf30: 00 00 00 00 00 00 00 00 fa fa fa fa 00 00 00 00\n
0x0c0c7fffcf40: 00 00 00 07 fa fa fa fa 00 00 00 00 00 00 00 00\n  0x0c0c7fffcf50: fa fa fa fa 00 00 00 00 00 00 00 05 fa fa fa fa\n  0x0c0c7fffcf60: 00 00 00 00 00 00 00 00 fa fa fa fa 00 00 0
0 00\n  0x0c0c7fffcf70: 00 00 00 00 fa fa fa fa 00 00 00 00 00 00 00 01\n=>0x0c0c7fffcf80: fa fa fa fa 00 00 00 00 00 00[01]fa fa fa fa fa\n  0x0c0c7fffcf90: 00 00 00 00 00 00 00 00 fa fa fa fa
 00 00 00 00\n  0x0c0c7fffcfa0: 00 00 05 fa fa fa fa fa 00 00 00 00 00 00 00 fa\n  0x0c0c7fffcfb0: fa fa fa fa 00 00 00 00 00 00 00 00 fa fa fa fa\n  0x0c0c7fffcfc0: fd fd fd fd fd fd fd fd fa
fa fa fa fd fd fd fd\n  0x0c0c7fffcfd0: fd fd fd fd fa fa fa fa 00 00 00 00 00 00 00 fa\nShadow byte legend (one shadow byte represents 8 application bytes):\n  Addressable:           00\n  Par
tially addressable: 01 02 03 04 05 06 07 \n  Heap left redzone:       fa\n  Freed heap region:       fd\n  Stack left redzone:      f1\n  Stack mid redzone:       f2\n  Stack right redzone:
 f3\n  Stack after return:      f5\n  Stack use after scope:   f8\n  Global redzone:          f9\n  Global init order:       f6\n  Poisoned by user:        f7\n  Container overflow:      fc\n
Array cookie:            ac\n  Intra object redzone:    bb\n  ASan internal:           fe\n  Left alloca redzone:     ca\n  Right alloca redzone:    cb\n  Shadow gap:              cc\n==48535==
ABORTING\n'

test_eof failed

== Tests result: FAILURE ==

1 test failed:
    test_eof

Total duration: 355 ms
Tests result: FAILURE

~/github/python/master heads/master*
venv ❯ LSAN_OPTIONS="suppressions=asan-suppression.txt,print_suppressions=0" ./python -c '"Ṕýţĥòñ" +'
  File "<string>", line 1
    "Ṕýţĥòñ" +
              ^
SyntaxError: invalid syntax
venv ❯ LSAN_OPTIONS="suppressions=asan-suppression.txt,print_suppressions=0" ./python -c 'yield from'
  File "<string>", line 1
    yield from
              ^
SyntaxError: invalid syntax

After this PR

venv ❯ LSAN_OPTIONS="suppressions=asan-suppression.txt,print_suppressions=0" ./python -m test test_eof
0:00:00 load avg: 1.13 Run tests sequentially
0:00:00 load avg: 1.13 [1/1] test_eof

== Tests result: SUCCESS ==

1 test OK.

Total duration: 493 ms
Tests result: SUCCESS
^[[A
~/github/python/master [bpo-40958](https://bugs.python.org/issue40958)*
venv ❯ LSAN_OPTIONS="suppressions=asan-suppression.txt,print_suppressions=0" ./python -c '"Ṕýţĥòñ" +'
  File "<string>", line 1
    "Ṕýţĥòñ" +
              ^
SyntaxError: invalid syntax

~/github/python/master [bpo-40958](https://bugs.python.org/issue40958)*
venv ❯ LSAN_OPTIONS="suppressions=asan-suppression.txt,print_suppressions=0" ./python -c 'yield from'
  File "<string>", line 1
    yield from
              ^
SyntaxError: invalid syntax

[tiran]

A test is failing:

 ======================================================================
FAIL: testSyntaxErrorOffset (test.test_exceptions.ExceptionTests) (source=b'Python = "\xcf\xb3\xf2\xee\xed" +', lineno=1, offset=18)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\a\cpython\cpython\lib\test\test_exceptions.py", line 188, in check
    self.assertEqual(cm.exception.offset, offset)
AssertionError: 13 != 18

[bedevere-bot]

When you're done making the requested changes, leave the comment: I have made the requested changes; please review again.

[pablogsal]

Closing this as this is missing something important and unfortunately, I don't have time to investigate today :(

[pablogsal]

@lysnikolaou Feel free to investigate if you have some time

[lysnikolaou]

@lysnikolaou Feel free to investigate if you have some time

I'll do as soon as I get home. Sorry for not being able to help out now!