Strict Provenance MVP #95241

Gankra · 2022-03-23T15:44:10Z

This patch series examines the question: how bad would it be if we adopted
an extremely strict pointer provenance model that completely banished all
int<->ptr casts.

The key insight to making this approach even vaguely pallatable is the

ptr.with_addr(addr) -> ptr

function, which takes a pointer and an address and creates a new pointer
with that address and the provenance of the input pointer. In this way
the "chain of custody" is completely and dynamically restored, making the
model suitable even for dynamic checkers like CHERI and Miri.

This is not a formal model, but lots of the docs discussing the model
have been updated to try to the concept of this design in the hopes
that it can be iterated on.

See #95228

Gankra · 2022-03-23T15:47:47Z

Hmm I probably should remove my changes to ptr examples, they should stick to "official" APIs.

Gankra · 2022-03-23T15:54:55Z

Note that a lot of the meat of this PR is just an enormous treatise being appended to core::ptr's top-level docs, discussing the high level "concept".

Gankra · 2022-03-23T15:56:07Z

cc @RalfJung for if you feel this new enormous treatise is acceptable to include given that it prefixed with the caveat that it is extremely overly strict, experimental, and non-normative.

Gankra · 2022-03-23T16:00:40Z

library/core/src/fmt/mod.rs

+        // formatter takes an &Opaque. Rust understandably doesn't think we should compare
+        // the function pointers if they don't have the same signature, so we cast to
+        // pointers to convince it that we know what we're doing.
+        if self.formatter as *mut u8 == USIZE_MARKER as *mut u8 {


i should probably revert this to not make AVR sad

Is this caused by an address space mismatch in the cast since avr uses AS1 for function pointers? I.e. an invalid bitcast error?

I played around with the datalayout code a few weeks ago to improve handling of address spaces. I can try to upload that as a PR later this week and see what else needs to be done to use an addrspacecast instead of a bitcast so you don't need the inttoptr/ptrtoint workaround. Should hopefully also help for CHERI support (which was my original motivation for looking at that code).

See: avr-rust#143

And:

rust/library/core/src/ptr/mod.rs

Line 1390 in 9280445

// HACK: The intermediate cast as usize is required for AVR

(fwiw validating that these changes help for AVR is probably blocked by https://reviews.llvm.org/D114611)

library/core/src/intrinsics.rs

library/core/src/ptr/mut_ptr.rs

library/std/src/backtrace.rs

Gankra · 2022-03-23T16:05:19Z

library/std/src/sys/windows/fs.rs

@@ -57,6 +57,9 @@ pub struct DirEntry {
    data: c::WIN32_FIND_DATAW,
 }

+unsafe impl Send for OpenOptions {}
+unsafe impl Sync for OpenOptions {}
+


There was some argument as to whether this is a serious Robustness To Refactoring regression. If so, I can replace this with a wrapper type for just c::LPSECURITY_ATTRIBUTES -- I don't know if such a wrapper already exists.

Robustness To Refactoring

I searched for this term on Google and there are a total of two results, none of which have any relation to the term itself. What does this mean?

It's not a specific term, just Capitalization for Emphasis. (i.e. this is a potential regression to correctness in the face of refactoring in the future, such as the addition of another !Send/!Sync field in the future.)

library/std/src/sys/unix/weak.rs

Gankra · 2022-03-23T16:11:54Z

lmk know what y'all wanna do about the "AVR is weird" situation which might also be a "wasm is weird" situation

library/std/src/sys/unix/thread.rs

Gankra · 2022-03-23T16:26:39Z

library/std/src/sync/once.rs


        // Try to slide in the node at the head of the linked list, making sure
        // that another thread didn't just replace the head of the linked list.
        let exchange_result = state_and_queue.compare_exchange(
            current_state,
-            me | RUNNING,
+            me.with_addr(me.addr() | RUNNING),


Yeah map_addr was added to the set of APIs we want to go with after I wrote all this so it's most only using with_addr, which isn't necessarily a problem.

If we get really serious with this approach, I also like to imagine further conveniences, like abbreviating p.with_addr(p.addr() | TAG) as p.tag_addr(TAG) and similar for masking, which saves writing a map_addr closure.

Agreed, let's be exuberantly maximalist about encapsulating these patterns. :)

(My really maximalist thought is wondering how much raw pointer usage that's just for tagging could be replaced with completely safe wrappers like Tagged<'a, T> that consider how much alignment you get from T and handle masking on access automatically. But this can happen outside the standard library, I hope.)

Edit: And I feel like I would have felt like such an interface was vaguely ~dangerous~ in the past, but the strict provenance mental model and with_addr make it way more obvious to me when something is correct.

(My really maximalist thought is wondering how much raw pointer usage that's just for tagging could be replaced with completely safe wrappers like Tagged<'a, T> that consider how much alignment you get from T and handle masking on access automatically. But this can happen outside the standard library, I hope.)

(Is this considered bad practice to link crates which happen to be yours on rust-lang discussion? Anyway, I have a crate that provides tagged pointer support, ptr-union, though it doesn't automatically detect guaranteed alignment yet, due to stable const generic limitations. I'm currently working on making it work under strict provenance / not use inttoptr.)

workingjubilee · 2022-03-23T16:29:00Z

We'll want to do a bunch of nonsense with try builds, so:
~~@bors delegate=Gankra~~
Grr.

solson

A couple more trivial nitpicks. (Well, and a typo that looks like it should fail to compile?)

library/std/src/sys/unix/thread.rs

rust-highfive · 2022-03-23T16:30:06Z

r? @Mark-Simulacrum

(rust-highfive has picked a reviewer for you, use r? to override)

workingjubilee · 2022-03-23T16:30:52Z

Hmm, apparently I don't understand bors syntax as well as I thought.
@bors delegate+

workingjubilee · 2022-03-23T16:31:41Z

...Weird. Is the bot dead?
@bors try

Gankra · 2022-03-23T16:35:18Z

Yeah this PR seems... cursed? the CI isn't even running.

bstrie · 2022-03-23T16:37:21Z

Github is having trouble it seems: https://www.githubstatus.com/

solson · 2022-03-23T16:46:24Z

library/alloc/src/rc.rs

    }
 }

 pub(crate) fn is_dangling<T: ?Sized>(ptr: *mut T) -> bool {
-    let address = ptr as *mut () as usize;
-    address == usize::MAX
+    (ptr as *mut ()).addr() == usize::MAX


This isn't a vital question for this PR, but it makes me wonder if .addr() should work on ?Sized directly, making this just ptr.addr() == usize::MAX despite ptr being potentially wide.

I assume historically as usize isn't allowed on DST pointers because it's a footgun in a world where we claim this roundtrips. With a clear addr method it might be fine to do this, but I'd like to be conservative against people just abusing these APIs and roundtripping anyways? Not sure I'm convinced by that argument.

I could definitely live without it, wasn't really sure what to think.

bors · 2022-03-23T16:51:57Z

✌️ @Gankra~~ can now approve this pull request

bors · 2022-03-23T16:52:32Z

✌️ @Gankra can now approve this pull request

bors · 2022-03-30T10:09:14Z

⌛ Testing commit e3a3afe with merge e50ff9b...

bors · 2022-03-30T12:28:44Z

☀️ Test successful - checks-actions
Approved by: workingjubilee
Pushing e50ff9b to master...

Gankra · 2022-03-30T12:37:31Z

YESSSSS

ok i will properly post a public announcement for this when it hits nightly and I can link the docs

rust-timer · 2022-03-30T13:49:00Z

Finished benchmarking commit (e50ff9b): comparison url.

Summary: This benchmark run did not return any relevant results.

If you disagree with this performance assessment, please file an issue in rust-lang/rustc-perf.

@rustbot label: -perf-regression

Fix a minor typo from rust-lang#95241 which prevented compilation on x86_64-unknown-openbsd.

…, r=dtolnay Fix library/std compilation on openbsd. Fix a minor typo from rust-lang#95241 which prevented compilation on x86_64-unknown-openbsd.

Gankra · 2022-04-05T20:41:44Z

I have published an article "properly" announcing this and further discussing the ideas behind the project and "The Tower Of Weakening": https://gankra.github.io/blah/tower-of-weakenings/

(Just wrapping up loose threads on my work)

Gankra changed the title ~~Cleaned provenance~~ Strict Provenance MVP Mar 23, 2022

Gankra force-pushed the cleaned-provenance branch from e172082 to 8c1bdc6 Compare March 23, 2022 15:53

Gankra commented Mar 23, 2022

View reviewed changes

library/core/src/intrinsics.rs Outdated Show resolved Hide resolved

Gankra commented Mar 23, 2022

View reviewed changes

library/core/src/ptr/mut_ptr.rs Outdated Show resolved Hide resolved

Gankra commented Mar 23, 2022

View reviewed changes

library/std/src/backtrace.rs Outdated Show resolved Hide resolved

Gankra commented Mar 23, 2022

View reviewed changes

library/std/src/sys/unix/weak.rs Outdated Show resolved Hide resolved

workingjubilee mentioned this pull request Mar 23, 2022

Rewrite Gankra's provenance draft to be lib-only #95229

Closed

bstrie reviewed Mar 23, 2022

View reviewed changes

RalfJung mentioned this pull request Mar 23, 2022

Should format!("{:p}", ptr) leak provenance? rust-lang/unsafe-code-guidelines#322

Closed

solson reviewed Mar 23, 2022

View reviewed changes

library/std/src/sys/unix/thread.rs Outdated Show resolved Hide resolved

library/std/src/sys/unix/thread.rs Outdated Show resolved Hide resolved

rust-highfive assigned Mark-Simulacrum Mar 23, 2022

rustbot added the T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. label Mar 23, 2022

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Mar 23, 2022

Gankra force-pushed the cleaned-provenance branch from 8e1015b to b4e12ad Compare March 23, 2022 16:44

solson reviewed Mar 23, 2022

View reviewed changes

Gankra mentioned this pull request Mar 23, 2022

Tracking Issue for strict_provenance #95228

Open

13 tasks

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Mar 30, 2022

bors added the merged-by-bors This PR was explicitly merged by bors. label Mar 30, 2022

bors merged commit e50ff9b into rust-lang:master Mar 30, 2022

rustbot added this to the 1.61.0 milestone Mar 30, 2022

bors mentioned this pull request Mar 30, 2022

Clean up, categorize and sort unstable features in std. #95032

Merged

This was referenced Mar 30, 2022

Specialize len in ExactSizeIterator implementations #91998

Closed

Add slice methods for indexing via an array of indices. #83608

Merged

rustbot removed the perf-regression Performance regression. label Mar 30, 2022

Gankra mentioned this pull request Mar 30, 2022

[strict provenance] Close The Gap Between AtomicUsize and AtomicPtr #95492

Closed

Gankra added the A-strict-provenance Area: Strict provenance for raw pointers label Mar 30, 2022

sunfishcode added a commit to sunfishcode/rust that referenced this pull request Mar 31, 2022

Fix library/std compilation on openbsd.

c89f11e

Fix a minor typo from rust-lang#95241 which prevented compilation on x86_64-unknown-openbsd.

sunfishcode mentioned this pull request Mar 31, 2022

Fix library/std compilation on openbsd. #95505

Merged

dtolnay mentioned this pull request Mar 31, 2022

Handle rustc_const_stable attribute in library feature collector #95354

Merged

lopopolo mentioned this pull request Apr 2, 2022

Resync Rc fork from upstream std artichoke/cactusref#103

Open

Gankra mentioned this pull request Apr 4, 2022

Add convenience byte offset/check align functions to pointers #95643

Merged

lqd mentioned this pull request Sep 15, 2022

internal compiler error: opaque type with non-universal region substs #101852

Closed

joboet mentioned this pull request Sep 30, 2022

std: use futex in Once #99505

Merged

Amanieu mentioned this pull request Dec 29, 2022

Lift T: Sized bounds from some strict_provenance pointer methods #103702

Merged

matthiaskrgr mentioned this pull request Apr 20, 2023

ICE: "no errors encountered even though delay_span_bug issued" in recursive impl #110623

Closed

lqd mentioned this pull request Jun 8, 2023

unchecked region constraints for opaque types in dead code #112417

Open

lqd mentioned this pull request Aug 11, 2023

RPIT hidden types can be ill-formed #114728

Closed

This was referenced Mar 28, 2024

ICE: {OpaqueTypeKey { def_id: DefId .. #123157

Open

ICE: None in compiler/rustc_mir_build/src/build/matches/mod.rs #119786

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Strict Provenance MVP #95241

Strict Provenance MVP #95241

Gankra commented Mar 23, 2022

Gankra commented Mar 23, 2022

Gankra commented Mar 23, 2022

Gankra commented Mar 23, 2022

Gankra Mar 23, 2022

arichardson Mar 23, 2022

Gankra Mar 23, 2022

leo60228 Mar 24, 2022

Gankra Mar 23, 2022

LoganDark May 25, 2022

CAD97 May 25, 2022

Gankra commented Mar 23, 2022

This comment was marked as resolved.

Gankra Mar 23, 2022

solson Mar 23, 2022

bstrie Mar 23, 2022

solson Mar 23, 2022 •

edited

CAD97 Mar 25, 2022

workingjubilee commented Mar 23, 2022 •

edited

solson left a comment •

edited

rust-highfive commented Mar 23, 2022

workingjubilee commented Mar 23, 2022

workingjubilee commented Mar 23, 2022 •

edited

Gankra commented Mar 23, 2022

bstrie commented Mar 23, 2022

solson Mar 23, 2022

Gankra Mar 23, 2022

solson Mar 23, 2022

bors commented Mar 23, 2022

bors commented Mar 23, 2022

bors commented Mar 30, 2022

bors commented Mar 30, 2022

Gankra commented Mar 30, 2022

rust-timer commented Mar 30, 2022

Gankra commented Apr 5, 2022

Strict Provenance MVP #95241

Strict Provenance MVP #95241

Conversation

Gankra commented Mar 23, 2022

Gankra commented Mar 23, 2022

Gankra commented Mar 23, 2022

Gankra commented Mar 23, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Gankra commented Mar 23, 2022

This comment was marked as resolved.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

solson Mar 23, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

workingjubilee commented Mar 23, 2022 • edited

solson left a comment • edited

Choose a reason for hiding this comment

rust-highfive commented Mar 23, 2022

workingjubilee commented Mar 23, 2022

workingjubilee commented Mar 23, 2022 • edited

Gankra commented Mar 23, 2022

bstrie commented Mar 23, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bors commented Mar 23, 2022

bors commented Mar 23, 2022

bors commented Mar 30, 2022

bors commented Mar 30, 2022

Gankra commented Mar 30, 2022

rust-timer commented Mar 30, 2022

Gankra commented Apr 5, 2022

solson Mar 23, 2022 •

edited

workingjubilee commented Mar 23, 2022 •

edited

solson left a comment •

edited

workingjubilee commented Mar 23, 2022 •

edited