Skip to content

Commit

Permalink
Handle the case where verbatim enters a None group
Browse files Browse the repository at this point in the history
Fixes #1235. As seen in that issue, a syntax node can actually cross the
border of a None-delimited group, due to such groups being transparent
to the parser in most cases. In the cases that this can occur, the
presence of the group is known to be semantically irrelevant, so we just
explicitly ignore the presence of the group.

It's important that we only ignore None-delimited groups when the
verbatim end requires entering the group, as such groups can have semantic
impact (such as grouping binary operator chains) in other positions.
  • Loading branch information
CAD97 committed Oct 20, 2022
1 parent 5c625cf commit 4051201
Showing 1 changed file with 17 additions and 1 deletion.
18 changes: 17 additions & 1 deletion src/verbatim.rs
@@ -1,5 +1,5 @@
use crate::parse::{ParseBuffer, ParseStream};
use proc_macro2::TokenStream;
use proc_macro2::{Delimiter, TokenStream};
use std::iter;

pub fn between<'a>(begin: ParseBuffer<'a>, end: ParseStream<'a>) -> TokenStream {
Expand All @@ -8,6 +8,22 @@ pub fn between<'a>(begin: ParseBuffer<'a>, end: ParseStream<'a>) -> TokenStream
let mut tokens = TokenStream::new();
while cursor != end {
let (tt, next) = cursor.token_tree().unwrap();

if next > end {
// In some edge cases, a syntax node can actually cross the border
// of a None-delimited group, due to such groups being transparent
// to the parser in most cases. In the cases that this can occur,
// the presence of the group is known to be semantically irrelevant,
// so we should just ignore the presence of the group. (Issue #1235)
if let Some((inside, _span, after)) = cursor.group(Delimiter::None) {
assert!(next == after);
cursor = inside;
continue;
} else {
panic!("verbatim end must not be inside a delimited group");
}
}

tokens.extend(iter::once(tt));
cursor = next;
}
Expand Down

0 comments on commit 4051201

Please sign in to comment.