Add a macro to track a file

[GuillaumeGomez]

I recently released doc-comment 0.4 which converted declarative macros to proc-macros. I encountered a "funny" issue though: I needed the compiler to track files that were included using my proc-macros. In order to do so, I generated anonymous constants which looked like this:

const _: &'static str = include_str!("file.md");

However, I have no use for those constants and they make the source code heavier for just keeping track of a file. Therefore, I propose to add a track!("file") macro which would allow to track files like include_str! without requiring the usage of anonymous constants.

I can make the implementation if the rust teams agree to add this feature. :)

[Mark-Simulacrum]

IMO, the proper solution is to add proc_macro APIs that indicate "interest" in a file, rather than further expanding the set of macros.

[estebank]

Isn't this already supported using build.rs?

[ehuss]

The user of the proc-macro would need to add it to their build.rs, which is a bit tedious (and I suspect most people won't even know they should do that). It would also cause the entire build script to rerun, which could be quite expensive if it is doing other unrelated things. Build scripts can also be tricky for inexperienced users (such as requiring "rerun-if-changed=build.rs" to avoid full rebuilds).

I've seen it mentioned in several places where there is desire to register change detection for various things (including environment variables mentioned here).

Grepping through crates.io, I found about 10 proc-macros that use the include_str/include_bytes trick to force a dependency check.

[ehuss]

cc #74690, where the ability to register the usage of an environment variable was added. I would suspect adding support for tracking files could use a similar mechanism. I'm not sure if this specifically needs to use a macro, or if a function call would be sufficient (tracked_env::file(...)? tracked_env::path(...)?).

[petrochenkov]

The bare minimum would be ability to add file paths to dependencies.

However, it would be significantly more useful to also have ability to load the file into a token stream and create a source map for it, so all the spans are assigned properly.

[luser]

However, it would be significantly more useful to also have ability to load the file into a token stream and create a source map for it, so all the spans are assigned properly.

This is #55904

[abonander]

The documentation for proc_macro::tracked_path::path() says:

Track a file explicitly.

However, I just tested it in the process of fixing launchbadge/sqlx#681 and it seems to work for directories too. Is this intentional? Because if it's not, I would love for it to continue working as-is because it'll let us finally fix that bug.

And if it is intentional or it's decided that working for directories is a feature and not a bug, it should be mentioned in the documentation.

[abonander]

It's actually kind of an issue for us that path() takes impl AsRef<str> as we need to use Path::canonicalize() to get a path that this API can accept (I assume, see next paragraph), so we then need to do the fallible conversion back to &str which is a lot of boilerplate to use this function.

On that note, the function should probably document how it treats relative paths, if at all. Is it relative to the workspace root? If that's the case then we don't even need to canonicalize.

[petrochenkov]

it seems to work for directories too. Is this intentional?

The specified file path gets into the depinfo file written by rustc.
How it's treated further depends entirely on tools interpreting that depinfo.

IIRC, cargo recently supported directories cargo:rerun-if-changed=PATH (https://doc.rust-lang.org/cargo/reference/build-scripts.html#cargorerun-if-changedpath), so directories in depinfo may be interpreted in the same way, but it needs to be clarified cargo people.

[petrochenkov]

It's actually kind of an issue for us that path() takes impl AsRef<str>

+1, it should probably accept a &Path, if it's possible to pass through proc macro bridge (#84029 (comment)).

[ehuss]

Yea, this will be dependent on the build system. I think for Cargo it is safe to rely on that behavior, as it is documented in the link given.

Taking a &Path probably sounds like the right thing to do, but it will likely break in horrible ways if using non-utf8 paths, so it may give a false sense of support.

[luser]

Taking a &Path probably sounds like the right thing to do, but it will likely break in horrible ways if using non-utf8 paths, so it may give a false sense of support.

This is a sensible position, and is exactly why @sunshowers created camino. It's annoying, but perhaps the API should take AsRef<Path>, and be fallible so it can reject non -UTF-8 paths? I'm not sure if that would make things more usable or just move the pain elsewhere.

Taking a Path would also allow sidestepping the concerns @abonander raised, in that the implementation could be responsible for serializing the path in the necessary way (canonicalized, absolute, crate-relative, whatever).

[sunshowers]

yeah, depinfo is an instance of what I call the "makefile problem". Given that rustc uses these files, should it support non-utf8 paths at all? cargo already doesn't.

I guess you don't need cross-platform support for depinfo files, so the case is a bit less strong for rustc than for cargo.

[abonander]

There's a couple orthogonal concerns here:

The API really needs to document what it does with relative paths. If it's purely up to whatever is consuming the depinfo then that should be documented as well, though it would also be very helpful to describe how Cargo treats it in that case. SQLx's proc macros are already written with the assumption that Cargo is being used, although that has also drawn a few complaints.

If we don't want to specify the behavior with relative paths and just tell the user to always canonicalize the path, that's okay. We just need to say one way or the other.

As for non-UTF-8 paths, I'm torn. In SQLx's case the paths are taken from string literals in the macro input so they should always be UTF-8.

If I knew I didn't need to manipulate the paths at all, I'd be perfectly fine with the function taking impl AsRef<str>.

If the treatment of relative dirs was specified but differed from my expectation, say for example, I'm expecting the paths to be relative to the crate root whereas Cargo interprets the paths relative to the workspace root, then I would still need to do path manipulation and impl AsRef<Path> would be nicer.

For sanity if we take impl AsRef<Path> then the function should be fallible and return an error on non-UTF-8 paths. That's just the sanest way to do it and lets the proc macro keep control over what to do next. However, that's an unnecessary annoyance if you're always dealing with paths as &strs.

Do we have anything against having two different functions? Say, path_str(impl AsRef<str>) and path(impl AsRef<Path>), where the former is infallible?

[pksunkara]

Tangential question: What's the path forward to stabilize this? I don't see any other PRs which are tracking this feature.

[drahnr]

@pksunkara the PR #87173 is probably the next step.

@abonander Adding two versions seems excessive compared to the feature provided, #87173 (comment) resolves this by a steering meeting discussion outcome, to use a Path based argument in order to not limit the API to the limitations of today's implemention.

I am not aware of any further issues at this point.

[abonander]

@drahnr please see my concerns above about specifying the semantics regarding relative paths and directories, that doesn't appear to have been discussed at all in the linked PR.

[drahnr]

The API really needs to document what it does with relative paths. If it's purely up to whatever is consuming the depinfo then that should be documented as well, though it would also be very helpful to describe how Cargo treats it in that case. SQLx's proc macros are already written with the assumption that Cargo is being used, although that has also drawn a few complaints.

If we don't want to specify the behavior with relative paths and just tell the user to always canonicalize the path, that's okay. We just need to say one way or the other.

@abonander indeed, I'll document the behaviour on relative paths and which base will be used in detail.

[m-ou-se]

This is now tracked in #99515, together with the tracked_env feature.