Pillow: Coverage build failure

[radarhere]

Hi. Pillow has been failing to complete the coverage build for a few weeks - https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=53281&sort=-opened&can=1&q=proj%3Apillow

https://oss-fuzz-build-logs.storage.googleapis.com/log-b333ca6d-aae4-4097-b134-efb9d533f5c8.txt

Starting Step #9
Step #9: Already have image (with digest): gcr.io/cloud-builders/gsutil
Step #9: CommandException: No URLs matched. Do the files you're operating on exist?
Finished Step #9
ERROR
ERROR: build step 9 "gcr.io/cloud-builders/gsutil" failed: step exited with non-zero status: 1

Looking for changes just before that, I found #8960. Any thoughts?

[jonathanmetzman]

@Navidem can you look into this?

[Navidem]

Looking at the coverage build logs, the failure started on Nov 7th with this PR merge. But looking closer reveals the coverage build was not generating any meaningful report even before Nov 7th and the mentioned PR just caused the silent failure to surface.

Looks like it cannot collect coverage because of some Atheris error. @DavidKorczynski does it look familiar to you?

Step #5: Already have image (with digest): gcr.io/oss-fuzz-base/base-runner
Step #5: Entering python fuzzing
Step #5: Error happened getting coverage of fuzz_pillow
Step #5: This is likely because Atheris did not exit gracefully
Step #5: Entering python fuzzing
Step #5: ls: cannot access '/corpus/fuzz_pillow.pkg': No such file or directory
Step #5: Error happened getting coverage of fuzz_pillow.pkg
Step #5: This is likely because Atheris did not exit gracefully
Step #5: Entering python fuzzing
Step #5: ls: cannot access '/corpus/fuzz_font.pkg': No such file or directory
Step #5: Error happened getting coverage of fuzz_font.pkg
Step #5: This is likely because Atheris did not exit gracefully
Step #5: Entering python fuzzing
Step #5: mv: cannot stat '.coverage': No such file or directory
Step #5: unzip:  cannot find or open /workspace/out/libfuzzer-coverage-x86_64/fuzz_pillow.pkg.deps.zip, /workspace/out/libfuzzer-coverage-x86_64/fuzz_pillow.pkg.deps.zip.zip or /workspace/out/libfuzzer-coverage-x86_64/fuzz_pillow.pkg.deps.zip.ZIP.
Step #5: rsync: link_stat "/workspace/out/libfuzzer-coverage-x86_64/medio" failed: No such file or directory (2)
Step #5: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1207) [sender=3.1.3]
Step #5: mv: cannot stat '/workspace/out/libfuzzer-coverage-x86_64/.coverage_fuzz_pillow': No such file or directory
Step #5: Translating the coverage
Step #5: Done with path walk
Step #5: cp: cannot stat '.new_coverage': No such file or directory
Step #5: cp: cannot stat '.new_coverage': No such file or directory
Step #5: unzip:  cannot find or open /workspace/out/libfuzzer-coverage-x86_64/fuzz_pillow.pkg.pkg.deps.zip, /workspace/out/libfuzzer-coverage-x86_64/fuzz_pillow.pkg.pkg.deps.zip.zip or /workspace/out/libfuzzer-coverage-x86_64/fuzz_pillow.pkg.pkg.deps.zip.ZIP.
Step #5: rsync: link_stat "/workspace/out/libfuzzer-coverage-x86_64/medio" failed: No such file or directory (2)
Step #5: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1207) [sender=3.1.3]
Step #5: mv: cannot stat '/workspace/out/libfuzzer-coverage-x86_64/.coverage_fuzz_pillow.pkg': No such file or directory
Step #5: Translating the coverage
Step #5: Done with path walk
Step #5: cp: cannot stat '.new_coverage': No such file or directory
Step #5: cp: cannot stat '.new_coverage': No such file or directory
Step #5: unzip:  cannot find or open /workspace/out/libfuzzer-coverage-x86_64/fuzz_font.pkg.pkg.deps.zip, /workspace/out/libfuzzer-coverage-x86_64/fuzz_font.pkg.pkg.deps.zip.zip or /workspace/out/libfuzzer-coverage-x86_64/fuzz_font.pkg.pkg.deps.zip.ZIP.
Step #5: rsync: link_stat "/workspace/out/libfuzzer-coverage-x86_64/medio" failed: No such file or directory (2)
Step #5: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1207) [sender=3.1.3]
Step #5: mv: cannot stat '/workspace/out/libfuzzer-coverage-x86_64/.coverage_fuzz_font.pkg': No such file or directory
Step #5: Translating the coverage
Step #5: Done with path walk
Step #5: cp: cannot stat '.new_coverage': No such file or directory
Step #5: cp: cannot stat '.new_coverage': No such file or directory
Step #5: unzip:  cannot find or open /workspace/out/libfuzzer-coverage-x86_64/fuzz_font.pkg.deps.zip, /workspace/out/libfuzzer-coverage-x86_64/fuzz_font.pkg.deps.zip.zip or /workspace/out/libfuzzer-coverage-x86_64/fuzz_font.pkg.deps.zip.ZIP.
Step #5: rsync: link_stat "/workspace/out/libfuzzer-coverage-x86_64/medio" failed: No such file or directory (2)
Step #5: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1207) [sender=3.1.3]
Step #5: mv: cannot stat '/workspace/out/libfuzzer-coverage-x86_64/.coverage_fuzz_font': No such file or directory
Step #5: Translating the coverage
Step #5: Done with path walk
Step #5: cp: cannot stat '.new_coverage': No such file or directory
Step #5: cp: cannot stat '.new_coverage': No such file or directory
Step #5: Couldn't combine from non-existent path '.coverage_*'
Step #5: No data to report.
Step #5: No data to report.
Step #5: cp: cannot stat 'htmlcov/status.json': No such file or directory
Step #5: mv: cannot stat 'htmlcov/*': No such file or directory
Step #5: mv: cannot stat '.coverage_*': No such file or directory
Finished Step #5

[DavidKorczynski]

@DavidKorczynski does it look familiar to you?

Yes, I took at short look at this just now, but will need some more time find the root cause! Am on it.

[DavidKorczynski]

@radarhere the coverage build is not working because you're not using compile_python_fuzzer to build the Python fuzzers.

compile_python_fuzzer has the necessary logic to create coverage builds for Python, and also of compiling regular fuzzers (ASAN) in the necessary wrapper bash script. See these lines:

oss-fuzz/infra/base-images/base-builder/compile_python_fuzzer

Lines 55 to 93 in 87a914a

	if [[ $SANITIZER = *coverage* ]]; then
	cat <<EOF > coverage_wrapper.py
	###### Coverage stub
	import atexit
	import coverage
	cov = coverage.coverage(data_file='.coverage', cover_pylib=True)
	cov.start()
	# Register an exist handler that will print coverage
	def exit_handler():
	cov.stop()
	cov.save()
	atexit.register(exit_handler)
	####### End of coverage stub
	EOF
	# Prepend stub and create tmp file
	cat coverage_wrapper.py $fuzzer_path > tmp_fuzzer_coverage.py
	# Overwrite existing fuzzer with new fuzzer that has stub
	mv tmp_fuzzer_coverage.py $fuzzer_path
	fi
	rm -rf $PYFUZZ_WORKPATH
	mkdir $PYFUZZ_WORKPATH $FUZZ_WORKPATH
	pyinstaller --distpath $OUT --workpath=$FUZZ_WORKPATH --onefile --name $fuzzer_package "$@" $fuzzer_path
	# Disable executable bit from package as OSS-Fuzz uses executable bits to
	# identify fuzz targets. We re-enable the executable bit in wrapper script
	# below.
	chmod -x $OUT/$fuzzer_package
	# In coverage mode save source files of dependencies in pyinstalled binary
	if [[ $SANITIZER = *coverage* ]]; then
	rm -rf /medio/
	python3 /usr/local/bin/python_coverage_helper.py $FUZZ_WORKPATH "/medio"
	zip -r $fuzzer_package.deps.zip /medio
	mv $fuzzer_package.deps.zip $OUT/
	fi

I've tested locally that the changes in this PR #9127 fixes the coverage build, and, indeed that it shows the code coverage of the Python code as well.

If you change https://github.com/python-pillow/Pillow/blob/main/Tests/oss-fuzz/build.sh to have the logic in the PR I linked, then the coverage build will start working.

[radarhere]

Thanks very much! Those changes have been applied, and the build is now succeeding in https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=53281&sort=-opened&can=1&q=proj%3Apillow

[DavidKorczynski]

Awesome @radarhere !

[radarhere]

The coverage build failure for Pillow has returned - https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=59870&sort=-opened&can=1&q=proj%3Apillow

I've tried reproducing it at radarhere@3408f10 with python infra/helper.py coverage pillow, but it actually does generate coverage_fuzz_ files, unlike the failing build log.

Any ideas what has gone wrong now? Or what I'm doing incorrectly in my attempt to reproduce the problem and figure out the error myself?

[jonathanmetzman]

@alan32liu can you take a look at this since Navid can't help with this.

[DonggeLiu]

I was unable to reproduce this locally with the latest images either, but here is what I found:

1. The error existed before June 20.
2. Corpora were still generated after the build failure (gsutil ls -l gs://pillow-corpus.clusterfuzz-external.appspot.com/libFuzzer/pillow_fuzz_pillow)
3. Recent changes in Pillow's Tests directory do not seem to cause the failure.

@DavidKorczynski would you happen to know something else we could test to locate the bug? I would really appreciate that!

[DavidKorczynski]

The problem is that when the fuzzers were run with the corpus to collect the code coverage the exit code was non-zero, meaning likely a bug/exception happened when running through the corpus. This means no code coverage was collected as a clean exit is needed for Coverage.py to succeed.

The telling sign here is the following output from the logs:

Step #5: Entering python fuzzing
Step #5: Error happened getting coverage of fuzz_font
Step #5: This is likely because Atheris did not exit gracefully
Step #5: Entering python fuzzing
Step #5: Error happened getting coverage of fuzz_pillow

See here in the coverage module for where the error is reported

oss-fuzz/infra/base-images/base-runner/coverage

Lines 177 to 181 in d591611

	$OUT/$target $corpus_real -atheris_runs=$(ls -la $corpus_real | wc -l) > $LOGS_DIR/$target.log 2>&1
	if (( $? != 0 )); then
	echo "Error happened getting coverage of $target"
	echo "This is likely because Atheris did not exit gracefully"
	return 0

It's quite hidden away from the logs, I'll see to coming up with something that has more clear messaging.

There are a couple of possible solutions:

1. It need not be a code issue as such, but sometimes it's the coverage instrumentation of the native code that causes this (somewhat related: Enable code coverage of native code for Python projects #10084). If this is the case, then you can remove instrumentation of native code for coverage builds using a similar approach to pandas:
   

   oss-fuzz/projects/pandas/build.sh

   Lines 18 to 21 in d591611

   	if [ $SANITIZER == "coverage" ]; then
   	export CFLAGS=""
   	export CXXFLAGS=""
   	fi

2. Did OSS-Fuzz report some bug recently that somehow got into the corpus? If so, this would also cause it to exit improperly and break the code coverage.

I'd probably try (1) first and see if this solves it.

[DonggeLiu]

Thanks for the detailed explanation, @DavidKorczynski!
A follow-up silly question:
The cloud build failed due to missing `.coverage_pillow', but I could not reproduce it locally. Is that because the step was running on the cloud but not executed in local tests? Is there a way to manually trigger that in local tests? Thanks!

[radarhere]

For no reason that I know of, it's started working again - https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=59870#c3