interpret: better control over whether we read data with provenance #97684

RalfJung · 2022-06-03T11:47:11Z

The resolution in rust-lang/unsafe-code-guidelines#286 seems to be that when we load data at integer type, we implicitly strip provenance. So let's implement that in Miri at least for scalar loads. This makes use of the fact that Scalar layouts distinguish pointer-sized integers and pointers -- so I was expecting some wild bugs where layouts set this incorrectly, but so far that does not seem to happen.

This does not entirely implement the solution to rust-lang/unsafe-code-guidelines#286; we still do the wrong thing for integers in larger types: we will copy_op them and then do validation, and validation will complain about the provenance. To fix that we need mutating validation; validation needs to strip the provenance rather than complaining about it. This is a larger undertaking (but will also help with rust-lang/miri#845 since we can reset padding to Uninit).

The reason this is useful is that we can now implement addr as a transmute from a pointer to an integer, and actually get the desired behavior of stripping provenance without exposing it!

rust-highfive · 2022-06-03T11:47:14Z

Some changes occured to the CTFE / Miri engine

cc @rust-lang/miri

Some changes occured to the CTFE / Miri engine

cc @rust-lang/miri

rust-highfive · 2022-06-03T11:47:15Z

r? @nagisa

(rust-highfive has picked a reviewer for you, use r? to override)

RalfJung · 2022-06-03T11:55:25Z

r? @oli-obk

RalfJung · 2022-06-03T12:11:41Z

src/test/ui/consts/const-eval/ub-enum.rs

@@ -25,10 +24,12 @@ const BAD_ENUM: Enum = unsafe { mem::transmute(1usize) };
 //~^ ERROR is undefined behavior

 const BAD_ENUM_PTR: Enum = unsafe { mem::transmute(&1) };
-//~^ ERROR is undefined behavior
+//~^ ERROR any use of this value will cause an error
+//~| WARN this was previously accepted by the compiler but is being phased out


We now error earlier here (already during evaluation, not just during validation), which leads to a different error message.

RalfJung · 2022-06-03T12:45:09Z

The reason this is useful is that we can now implement addr as a transmute from a pointer to an integer, and actually get the desired behavior of stripping provenance without exposing it!

I have now tested that this is the case. :) But I will leave actually changing addr to a future PR (since that is also a libs thing and should probably involve other people).

oli-obk · 2022-06-03T13:14:29Z

compiler/rustc_const_eval/src/interpret/operand.rs

+            let scalar = alloc.read_scalar(
+                alloc_range(Size::ZERO, size),
+                s.is_ptr() || (number_may_have_provenance && size == self.pointer_size()),
+            )?;


I think there should be an InterpCx method for this (it's repeated 3x here after all)

For what part exactly?
This should just be s.is_ptr(); all the rest is just to support the Miri flag that allows ptr-int transmutation...

sure, but just from a code perspective, these 3 duplications could be deduplicated with a method. The miri flag support won't go away after all.

Well I am actually considering removing that Miri flag, given how complicated the provenance story is anyway and how we don't know if there even is demand for such a flag.

Making it into a method has the problem that it'd take Size and Scalar and require both to match; it's not a great API. I can try making it a local closure so at least nobody will try to use it anywhere else.

Okay, I made it a closure. Does that work for you?

This should not be used outside of read_immediate_from_mplace_raw so a method would send a wrong signal IMO.

compiler/rustc_middle/src/mir/interpret/allocation.rs

oli-obk · 2022-06-03T13:22:45Z

compiler/rustc_const_eval/src/interpret/memory.rs

+        &self,
+        range: AllocRange,
+        read_pointer: bool,
+    ) -> InterpResult<'tcx, ScalarMaybeUninit<Tag>> {
        let range = self.range.subrange(range);


General note on AllocRef: what bugged me a few times before is that all these methods take an AllocRange. Maybe we should move to a scheme where we add a slice method to AllocRef and thus require chaining of the sort of alloc.slice(range).read_pointer() instead of alloc.read_pointer(range).

I mostly added the AllocRange to avoid having two Size parameters whose meaning is unclear. But yeah I guess in practice this did not work out as well as I had hoped...

That's a topic for a different PR though, I think.

oli-obk · 2022-06-03T14:59:06Z

@bors r+

bors · 2022-06-03T14:59:08Z

📌 Commit ecf34dd720223c5f6e6c682313268a76dfb4d9b6 has been approved by oli-obk

bjorn3 · 2022-06-05T10:15:53Z

src/test/ui/consts/const-eval/ref_to_int_match.32bit.stderr

-           }
+   = note: `#[deny(const_err)]` on by default
+   = warning: this was previously accepted by the compiler but is being phased out; it will become a hard error in a future release!
+   = note: for more information, see issue #71800 <https://github.com/rust-lang/rust/issues/71800>


This changes a hard error into a future compat lint that can be disabled.

Yes, that is what I noted above. Errors during const evaluation (as opposed to errors occurring during validation, which is after evaluation finished) are still future-compat lints (#71800).

bors · 2022-06-05T11:53:27Z

⌛ Testing commit ecf34dd720223c5f6e6c682313268a76dfb4d9b6 with merge 6ac75ce48f727d41c0560e056a8c27e1d080cde7...

bors · 2022-06-05T12:31:20Z

💔 Test failed - checks-actions

RalfJung · 2022-06-05T14:13:49Z

@bors r=oli-obk

bors · 2022-06-05T14:13:51Z

📌 Commit d208f80 has been approved by oli-obk

implement ptr.addr() via transmute As per the discussion in rust-lang/unsafe-code-guidelines#286, the semantics for ptr-to-int transmutes that we are going with for now is to make them strip provenance without exposing it. That's exactly what `ptr.addr()` does! So we can implement `ptr.addr()` via `transmute`. This also means that once rust-lang#97684 lands, Miri can distinguish `ptr.addr()` from `ptr.expose_addr()`, and the following code will correctly be called out as having UB (if permissive provenance mode is enabled, which will become the default once the [implementation is complete](rust-lang/miri#2133)): ```rust fn main() { let x: i32 = 3; let x_ptr = &x as *const i32; let x_usize: usize = x_ptr.addr(); // Cast back an address that did *not* get exposed. let ptr = std::ptr::from_exposed_addr::<i32>(x_usize); assert_eq!(unsafe { *ptr }, 3); //~ ERROR Undefined Behavior: dereferencing pointer failed } ``` This completes the Miri implementation of the new distinctions introduced by strict provenance. :) Cc `@Gankra` -- for now I left in your `FIXME(strict_provenance_magic)` saying these should be intrinsics, but I do not necessarily agree that they should be. Or if we have an intrinsic, I think it should behave exactly like the `transmute` does, which makes one wonder why the intrinsic should be needed.

… r=oli-obk interpret: better control over whether we read data with provenance The resolution in rust-lang/unsafe-code-guidelines#286 seems to be that when we load data at integer type, we implicitly strip provenance. So let's implement that in Miri at least for scalar loads. This makes use of the fact that `Scalar` layouts distinguish pointer-sized integers and pointers -- so I was expecting some wild bugs where layouts set this incorrectly, but so far that does not seem to happen. This does not entirely implement the solution to rust-lang/unsafe-code-guidelines#286; we still do the wrong thing for integers in larger types: we will `copy_op` them and then do validation, and validation will complain about the provenance. To fix that we need mutating validation; validation needs to strip the provenance rather than complaining about it. This is a larger undertaking (but will also help resolve rust-lang/miri#845 since we can reset padding to `Uninit`). The reason this is useful is that we can now implement `addr` as a `transmute` from a pointer to an integer, and actually get the desired behavior of stripping provenance without exposing it!

RalfJung · 2022-06-06T11:51:02Z

@bors p=1
Needed to get Miri back into shape

Dylan-DPC · 2022-06-06T13:15:18Z

@bors p=6

bors · 2022-06-06T13:29:01Z

⌛ Testing commit d208f80 with merge 9d20fd1...

bors · 2022-06-06T16:09:47Z

☀️ Test successful - checks-actions
Approved by: oli-obk
Pushing 9d20fd1 to master...

rust-highfive · 2022-06-06T16:10:14Z

📣 Toolstate changed by #97684!

Tested on commit 9d20fd1.
Direct link to PR: #97684

💔 miri on windows: test-fail → build-fail (cc @eddyb @oli-obk @RalfJung).
💔 miri on linux: test-fail → build-fail (cc @eddyb @oli-obk @RalfJung).

@eddyb

Tested on commit rust-lang/rust@9d20fd1. Direct link to PR: <rust-lang/rust#97684> 💔 miri on windows: test-fail → build-fail (cc @eddyb @oli-obk @RalfJung). 💔 miri on linux: test-fail → build-fail (cc @eddyb @oli-obk @RalfJung).

adjust for better provenance control This is the Miri side of rust-lang/rust#97684.

rust-timer · 2022-06-06T17:27:28Z

Finished benchmarking commit (9d20fd1): comparison url.

Instruction count

Primary benchmarks: 🎉 relevant improvements found
Secondary benchmarks: 🎉 relevant improvements found

	mean¹	max	count²
Regressions 😿 (primary)	N/A	N/A	0
Regressions 😿 (secondary)	N/A	N/A	0
Improvements 🎉 (primary)	-0.7%	-1.9%	8
Improvements 🎉 (secondary)	-5.5%	-10.5%	12
All 😿🎉 (primary)	-0.7%	-1.9%	8

Max RSS (memory usage)

Results

Primary benchmarks: 🎉 relevant improvement found
Secondary benchmarks: 🎉 relevant improvement found

	mean¹	max	count²
Regressions 😿 (primary)	N/A	N/A	0
Regressions 😿 (secondary)	N/A	N/A	0
Improvements 🎉 (primary)	-2.4%	-2.4%	1
Improvements 🎉 (secondary)	-3.8%	-3.8%	1
All 😿🎉 (primary)	-2.4%	-2.4%	1

Cycles

Results

Primary benchmarks: mixed results
Secondary benchmarks: 🎉 relevant improvements found

	mean¹	max	count²
Regressions 😿 (primary)	2.4%	2.4%	1
Regressions 😿 (secondary)	N/A	N/A	0
Improvements 🎉 (primary)	-2.9%	-2.9%	2
Improvements 🎉 (secondary)	-9.8%	-14.7%	9
All 😿🎉 (primary)	-1.1%	-2.9%	3

If you disagree with this performance assessment, please file an issue in rust-lang/rustc-perf.

@rustbot label: -perf-regression

the arithmetic mean of the percent change ↩ ↩² ↩³
number of relevant changes ↩ ↩² ↩³

RalfJung · 2022-06-06T17:38:17Z

Well that is unexpected, but I won't complain.^^

This effectively reverts rust-lang#97684 for CTFE

allow numbers with provenance within CTFE execution This effectively reverts rust-lang#97684 for CTFE. Undoes the diagnostic changes that are tracked in rust-lang#99923, only for beta. (On master this patch wouldn't apply any more, `enforce_number_no_provenance` is gone with rust-lang#99644 since the interpreter engine is not supposed to ever have provenance on integers.) The test changes are an exact un-do of rust-lang#97684. However there is still some risk here since this exact code is not what has been battle-tested. r? `@Mark-Simulacrum`

…=pnkfelix beta-backport of provenance-related CTFE changes This is all part of dealing with rust-lang#99923. The first commit backports the effects of rust-lang#101101. `@pnkfelix` asked for this and it turned out to be easy, so I think this is uncontroversial. The second commit effectively repeats rust-lang#99965, which un-does the effects of rust-lang#97684 and therefore means rust-lang#99923 does not apply to the beta branch. I honestly don't think we should do this; the sentiment in rust-lang#99923 was that we should go ahead with the change but improve diagnostics. But `@pnkfelix` seemed to request such a change so I figured I would offer the option. I'll be on vacation soon, so if you all decide to take the first commit only, then someone please just force-push to this branch and remove the 2nd commit.

implement ptr.addr() via transmute As per the discussion in rust-lang/unsafe-code-guidelines#286, the semantics for ptr-to-int transmutes that we are going with for now is to make them strip provenance without exposing it. That's exactly what `ptr.addr()` does! So we can implement `ptr.addr()` via `transmute`. This also means that once rust-lang/rust#97684 lands, Miri can distinguish `ptr.addr()` from `ptr.expose_addr()`, and the following code will correctly be called out as having UB (if permissive provenance mode is enabled, which will become the default once the [implementation is complete](rust-lang/miri#2133)): ```rust fn main() { let x: i32 = 3; let x_ptr = &x as *const i32; let x_usize: usize = x_ptr.addr(); // Cast back an address that did *not* get exposed. let ptr = std::ptr::from_exposed_addr::<i32>(x_usize); assert_eq!(unsafe { *ptr }, 3); //~ ERROR Undefined Behavior: dereferencing pointer failed } ``` This completes the Miri implementation of the new distinctions introduced by strict provenance. :) Cc `@Gankra` -- for now I left in your `FIXME(strict_provenance_magic)` saying these should be intrinsics, but I do not necessarily agree that they should be. Or if we have an intrinsic, I think it should behave exactly like the `transmute` does, which makes one wonder why the intrinsic should be needed.

rust-highfive assigned nagisa Jun 3, 2022

rustbot added the T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. label Jun 3, 2022

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jun 3, 2022

rust-highfive assigned oli-obk and unassigned nagisa Jun 3, 2022

RalfJung force-pushed the better-provenance-control branch from 9a3458f to 0900569 Compare June 3, 2022 12:03

RalfJung mentioned this pull request Jun 3, 2022

The plan for provenance rust-lang/miri#2133

Closed

6 tasks

RalfJung commented Jun 3, 2022

View reviewed changes

RalfJung mentioned this pull request Jun 3, 2022

adjust for better provenance control rust-lang/miri#2183

Merged

This comment has been minimized.

Sign in to view

RalfJung force-pushed the better-provenance-control branch from 0900569 to 7d5c833 Compare June 3, 2022 13:03

oli-obk requested changes Jun 3, 2022

View reviewed changes

RalfJung force-pushed the better-provenance-control branch 2 times, most recently from fa59ef1 to ecf34dd Compare June 3, 2022 14:26

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jun 3, 2022

RalfJung mentioned this pull request Jun 3, 2022

implement ptr.addr() via transmute #97710

Merged

bjorn3 reviewed Jun 5, 2022

View reviewed changes

bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Jun 5, 2022

This comment has been minimized.

Sign in to view

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jun 5, 2022

Dylan-DPC mentioned this pull request Jun 6, 2022

Rollup of 5 pull requests #97781

Closed

bors added the merged-by-bors This PR was explicitly merged by bors. label Jun 6, 2022

bors merged commit 9d20fd1 into rust-lang:master Jun 6, 2022

rustbot added this to the 1.63.0 milestone Jun 6, 2022

RalfJung deleted the better-provenance-control branch June 6, 2022 16:10

bors added a commit to rust-lang/miri that referenced this pull request Jun 6, 2022

Auto merge of #2183 - RalfJung:better-provenance-control, r=RalfJung

9376978

adjust for better provenance control This is the Miri side of rust-lang/rust#97684.

bors mentioned this pull request Jun 6, 2022

(Selectively) turn on validation in const eval #95377

Closed

bors added a commit to rust-lang/miri that referenced this pull request Jun 6, 2022

Auto merge of #2183 - RalfJung:better-provenance-control, r=RalfJung

3361eab

adjust for better provenance control This is the Miri side of rust-lang/rust#97684.

RalfJung mentioned this pull request Jun 11, 2022

[Merged by Bors] - unpin nightly and disable weak memory emulation bevyengine/bevy#4988

Closed

Manishearth mentioned this pull request Jul 29, 2022

Regression in consteval: error[E0080]: could not evaluate static initializer (unable to turn pointer into raw bytes) #99923

Closed

RalfJung added a commit to RalfJung/rust that referenced this pull request Jul 30, 2022

allow numbers with provenance within CTFE execution

29ce4d5

This effectively reverts rust-lang#97684 for CTFE

RalfJung mentioned this pull request Jul 30, 2022

allow numbers with provenance within CTFE execution #99965

Merged

RalfJung mentioned this pull request Sep 2, 2022

beta-backport of provenance-related CTFE changes #101320

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

interpret: better control over whether we read data with provenance #97684

interpret: better control over whether we read data with provenance #97684

RalfJung commented Jun 3, 2022 •

edited

rust-highfive commented Jun 3, 2022

rust-highfive commented Jun 3, 2022

RalfJung commented Jun 3, 2022

RalfJung Jun 3, 2022

RalfJung commented Jun 3, 2022

This comment has been minimized.

oli-obk Jun 3, 2022

RalfJung Jun 3, 2022

oli-obk Jun 3, 2022

RalfJung Jun 3, 2022

RalfJung Jun 3, 2022

oli-obk Jun 3, 2022

RalfJung Jun 3, 2022

oli-obk Jun 3, 2022

oli-obk commented Jun 3, 2022

bors commented Jun 3, 2022

bjorn3 Jun 5, 2022 •

edited

RalfJung Jun 5, 2022 •

edited

bors commented Jun 5, 2022

bors commented Jun 5, 2022

This comment has been minimized.

RalfJung commented Jun 5, 2022

bors commented Jun 5, 2022

RalfJung commented Jun 6, 2022

Dylan-DPC commented Jun 6, 2022

bors commented Jun 6, 2022

bors commented Jun 6, 2022

rust-highfive commented Jun 6, 2022

rust-timer commented Jun 6, 2022

RalfJung commented Jun 6, 2022

interpret: better control over whether we read data with provenance #97684

interpret: better control over whether we read data with provenance #97684

Conversation

RalfJung commented Jun 3, 2022 • edited

rust-highfive commented Jun 3, 2022

rust-highfive commented Jun 3, 2022

RalfJung commented Jun 3, 2022

Choose a reason for hiding this comment

RalfJung commented Jun 3, 2022

This comment has been minimized.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oli-obk commented Jun 3, 2022

bors commented Jun 3, 2022

bjorn3 Jun 5, 2022 • edited

Choose a reason for hiding this comment

RalfJung Jun 5, 2022 • edited

Choose a reason for hiding this comment

bors commented Jun 5, 2022

bors commented Jun 5, 2022

This comment has been minimized.

RalfJung commented Jun 5, 2022

bors commented Jun 5, 2022

RalfJung commented Jun 6, 2022

Dylan-DPC commented Jun 6, 2022

bors commented Jun 6, 2022

bors commented Jun 6, 2022

rust-highfive commented Jun 6, 2022

rust-timer commented Jun 6, 2022

Footnotes

RalfJung commented Jun 6, 2022

RalfJung commented Jun 3, 2022 •

edited

bjorn3 Jun 5, 2022 •

edited

RalfJung Jun 5, 2022 •

edited