fix syntax error reporting from stdin (#357)

[stevenkaras]

In b73a3d1, there was an assumption that text is None only if there was an
encoding error with the file. However this was the case for all pythons before
3.9 when reading code from stdin.

This takes care to correctly report as much context as possible, so errors
aren't silently dropped with the unhelpful "problem decoding source" message.

[asottile]

could you add a test for this?

[stevenkaras]

Added explicit tests of the new behavior and the stdin example as described in the issue.

[asottile]

this should generate an actual syntax error and validate it so we can know when to delete this behaviour (3.9+)

[stevenkaras]

This is testing the reporting of a syntax error. Specifically, the behavior of what to do when the offsets are nonsensical (which they are for actual decoding errors in all versions of python including 3.9+). I'll attempt to clarify this in the test comment.

[asottile]

ah I see -- also I think the None fits on the line above within the character limit too if you want to move that as well! ideally we should set up some auto-formatting

[asottile]

but that's probably for another PR!

[asottile]

I don't think this actually tests the behaviour -- and we already have a SyntaxError test

[stevenkaras]

The behavior is triggered when ast.parse throws a SyntaxError from code with a filepath that doesn't exist (stdin).

I'll see if I can trigger this with a different syntax error that has more consistent messaging to avoid the rats nest of expected messages by version. The offset behavior is a bit concerning though. It looks like pypy has special handling that is objectively wrong in certain situations (such as this one).

[asottile]

pypy is usually pretty attentive to fixes for syntax error offsets (with a little help from me they inspired the improvements to cpython 3.10's error messages)

[stevenkaras]

I meant pyflakes handling of pypy's offsets as here:

pyflakes/pyflakes/api.py

Lines 46 to 56 in 44ef321

	if checker.PYPY:
	if text is None:
	lines = codeString.splitlines()
	if len(lines) >= lineno:
	text = lines[lineno - 1]
	if isinstance(text, bytes):
	try:
	text = text.decode('ascii')
	except UnicodeDecodeError:
	text = None
	offset -= 1

It's probably something that used to be an issue with pypy and was fixed since on the other end. I tested with pypy3.7-7.3.9, but tracking down which version of pypy changed offsets in syntax errors is out of scope for here.

[asottile]

oh yeah we should totally kill that -- forgot pyflakes had special cases (probably from like 2015-era pypy)