CIFuzz has started failing

[radarhere]

CIFuzz has started failing. As a demo, with the meaningless change from radarhere@510a5a3, I get https://github.com/radarhere/Pillow/runs/2084858912

I think the relevant part is

2021-03-11T09:33:27.0510333Z 2021-03-11 09:33:27,047 - root - INFO - Removing unaffected fuzz targets.
2021-03-11T09:33:27.0520068Z 2021-03-11 09:33:27,047 - root - DEBUG - Diffing against base_commit: 0000000000000000000000000000000000000000.
2021-03-11T09:33:27.0521040Z 2021-03-11 09:33:27,050 - root - INFO - Diffing against 0000000000000000000000000000000000000000.
2021-03-11T09:33:27.0545199Z 2021-03-11 09:33:27,053 - root - DEBUG - Stderr of command 'git diff --name-only 0000000000000000000000000000000000000000 --' is fatal: bad object 0000000000000000000000000000000000000000
2021-03-11T09:33:27.0545663Z .
2021-03-11T09:33:27.0546281Z 2021-03-11 09:33:27,053 - root - ERROR - Git diff failed with error message fatal: bad object 0000000000000000000000000000000000000000

But strangely, looking back for a passing run - https://github.com/python-pillow/Pillow/runs/2049950863

2021-03-07T09:59:39.8378646Z 2021-03-07 09:59:39,833 - root - INFO - Removing unaffected fuzz targets.
2021-03-07T09:59:39.8411899Z 2021-03-07 09:59:39,833 - root - DEBUG - Diffing against base_commit: a95fee0.
2021-03-07T09:59:39.8468825Z 2021-03-07 09:59:39,836 - root - INFO - Diffing against a95fee0.
2021-03-07T09:59:40.0008638Z 2021-03-07 09:59:40,000 - root - INFO - Files changed in PR: ['Tests/test_map.py', 'src/PIL/ImageFile.py', 'src/_imaging.c', 'src/map.c']
2021-03-07T09:59:40.0911302Z 2021-03-07 09:59:40,090 - root - ERROR - HTTP error with url https://storage.googleapis.com/oss-fuzz-coverage/latest_report_info/pillow.json.
2021-03-07T09:59:40.0918231Z 2021-03-07 09:59:40,090 - root - ERROR - Could not get the coverage report json from url: https://storage.googleapis.com/oss-fuzz-coverage/latest_report_info/pillow.json.
2021-03-07T09:59:40.0919750Z 2021-03-07 09:59:40,090 - root - ERROR - Could not find latest coverage report.

[wiredfool]

They must have just added code coverage support for python.

I'll see what it takes to support that, in the meantime I wonder if we can make this a soft fail in GHA?

[wiredfool]

I'm not seeing anything upstream that would indicate code coverage for python is working yet.

It actually looks like this is the issue:

BAD BUILD: /tmp/not-out/fuzz_pillow seems to have either startup crash or exit:
/tmp/not-out/fuzz_pillow -rss_limit_mb=2560 -timeout=25 -seed=1337 -runs=4 < /dev/null
Traceback (most recent call last):
  File "fuzz_pillow.py", line 23, in <module>
  File "/usr/local/lib/python3.8/site-packages/PyInstaller/loader/pyimod03_importers.py", line 493, in exec_module
  File "PIL/Image.py", line 94, in <module>
ImportError: libxcb.so.1: cannot open shared object file: No such file or directory

[radarhere]

In that case, I wonder if adding --add-binary /usr/local/lib/libxcb.so.1:. \ to https://github.com/google/oss-fuzz/blob/master/projects/pillow/build.sh fixes it.

[wiredfool]

Possibly -- but it's not failing on my local builds where I'm running the fuzzer.

It looks like it's failing with that on the oss-fuzz level as well @ google, so I'll look into it today.

[radarhere]

I see that #5328 had a fix for this, so we're waiting on google/oss-fuzz#5522.

But now valgrind is failing in master?

[wiredfool]

Yes, valgrind is erroring, and it's failing on a few images in the fuzz testing. It's possible that this is another run of the same issues that we've got ignored elsewhere, and it's possible that it's something new.

[radarhere]

CIFuzz is now passing again.

[wiredfool]

Huh. I wonder why that is., because my pr hasn’t been merged.

[radarhere]

I think this is why - google/oss-fuzz#5552

[radarhere]

For the record, the valgrind issues were solved by #5397