-
Notifications
You must be signed in to change notification settings - Fork 236
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Remove imprecise T_IDENTIFIER
capture groups, leading to simpler token sequences
#258
Remove imprecise T_IDENTIFIER
capture groups, leading to simpler token sequences
#258
Conversation
A `T_IDENTIFIER` represents: * a constant * a function name * a function name prefixed with `\` * a class name * a class name prefixed with `\` * a class constant * a class constant, with class name prefixed with `\` The current capturing regular expression is too simplistic, and it leads to edge cases where `Foo\42.5` are captured as separate tokens. In this patch, we deprecate `DocLexer::T_NAMESPACE_SEPARATOR` (which shouldn't be looked up directly anymore), and tell the parser to work with wider `T_IDENTIFIER` tokens instead. Note that a test is also being removed. That is intentional, since the test verifies a parser issue when parsing `@Foo\3.42`: * previously, the tokens would be `[@ Foo\3.42]` * now they are `[@, Foo, \, 3.42]` This also means that the above will be parsed as `@Foo`.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Makes sense (+ would fix a deprecation that blocks Symfony's CI on PHP 7.4 :) )
@@ -988,18 +988,7 @@ private function Identifier() | |||
|
|||
$this->lexer->moveNext(); | |||
|
|||
$className = $this->lexer->token['value']; | |||
|
|||
while ($this->lexer->lookahead['position'] === ($this->lexer->token['position'] + strlen($this->lexer->token['value'])) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
because lookahead can be null here, this line triggers a deprecation on 7.4
that's one of the main remaining blockers we have to make Symfony's tests pass on PHP 7.4 :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@guilhermeblanco let's separate the concern of fixing 7.4 compat and the issue found with this patch: we should check for the lookahead to exist in this while()
, and have a test that triggers a deprecation warning on 7.4
Fixes #273 |
* @expectedException \Doctrine\Common\Annotations\AnnotationException | ||
* @expectedExceptionMessage [Syntax Error] Expected Doctrine\Common\Annotations\DocLexer::T_IDENTIFIER or Doctrine\Common\Annotations\DocLexer::T_TRUE or Doctrine\Common\Annotations\DocLexer::T_FALSE or Doctrine\Common\Annotations\DocLexer::T_NULL, got '3.42' at position 5. | ||
*/ | ||
public function testInvalidIdentifierInAnnotation() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Urgh, I shouldn't have removed this test. It should at least have a test declaring another sort of failure here.
As agreed, we decided to take a simplified approach and only resolve the 7.4 notice instead. |
Note: I just need a review - will forward-port after that.
Discovered while working on #257
A
T_IDENTIFIER
represents:\
\
\
The current capturing regular expression is too simplistic, and it
leads to edge cases where
Foo\42.5
are captured as separate tokens.In this patch, we deprecate
DocLexer::T_NAMESPACE_SEPARATOR
(whichshouldn't be looked up directly anymore), and tell the parser to work
with wider
T_IDENTIFIER
tokens instead.Note that a test is also being removed. That is intentional, since the
test verifies a parser issue when parsing
@Foo\3.42
:[@ Foo\3.42]
[@, Foo, \, 3.42]
This also means that the above will be parsed as
@Foo
.