Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement token-based handling of attributes during expansion #82608

Merged
merged 1 commit into from
Apr 11, 2021

Commits on Apr 11, 2021

  1. Implement token-based handling of attributes during expansion

    This PR modifies the macro expansion infrastructure to handle attributes
    in a fully token-based manner. As a result:
    
    * Derives macros no longer lose spans when their input is modified
      by eager cfg-expansion. This is accomplished by performing eager
      cfg-expansion on the token stream that we pass to the derive
      proc-macro
    * Inner attributes now preserve spans in all cases, including when we
      have multiple inner attributes in a row.
    
    This is accomplished through the following changes:
    
    * New structs `AttrAnnotatedTokenStream` and `AttrAnnotatedTokenTree` are introduced.
      These are very similar to a normal `TokenTree`, but they also track
      the position of attributes and attribute targets within the stream.
      They are built when we collect tokens during parsing.
      An `AttrAnnotatedTokenStream` is converted to a regular `TokenStream` when
      we invoke a macro.
    * Token capturing and `LazyTokenStream` are modified to work with
      `AttrAnnotatedTokenStream`. A new `ReplaceRange` type is introduced, which
      is created during the parsing of a nested AST node to make the 'outer'
      AST node aware of the attributes and attribute target stored deeper in the token stream.
    * When we need to perform eager cfg-expansion (either due to `#[derive]` or `#[cfg_eval]`),
    we tokenize and reparse our target, capturing additional information about the locations of
    `#[cfg]` and `#[cfg_attr]` attributes at any depth within the target.
    This is a performance optimization, allowing us to perform less work
    in the typical case where captured tokens never have eager cfg-expansion run.
    Aaron1011 committed Apr 11, 2021
    Configuration menu
    Copy the full SHA
    a93c4f0 View commit details
    Browse the repository at this point in the history