Deprecate the unstable ptr_to_from_bits feature

[scottmcm]

I propose that we deprecate the (unstable!) to_bits and from_bits methods on raw pointers. (With the intent to remove them once addr has been around long enough to make the transition easy on people -- maybe another 6 weeks remove them fairly soon after, as the strict and expose versions have been around for a while already.)

The APIs that came from the strict provenance explorations (#95228) are a more holistic version of these, and things like .expose_addr() work for the "that cast looks sketchy" case even if the full strict provenance stuff never happens. (As a bonus, addr is even shorter than to_bits, though it is only applicable if people can use full strict provenance! addr is not a direct replacement for to_bits.) So I think it's fine to move away from the {to|from}_bits methods, and encourage the others instead.

That also resolves the worry that was brought up (I forget where) that q.to_bits() and (*q).to_bits() both work if q is a pointer-to-floating-point, as they also have a to_bits method.

Tracking issue #91126
Code search: https://github.com/search?l=Rust&p=1&q=ptr_to_from_bits&type=Code

For potential pushback, some users in case they want to chime in

• @RSSchermer https://github.com/RSSchermer/ARWA/blob/365bb68541447453fc44f6fbcc5d394bb94c14e9/arwa/src/html/custom_element.rs#L105
• @strax https://github.com/strax/pbr/blob/99616d1dbf42f93ec8dd668d05b3180649558180/openexr/src/core/alloc.rs#L36
• @MiSawa https://github.com/MiSawa/pomelo/blob/577c6223588d539295a71ff125d8f249e59f4146/crates/kernel/src/timer.rs#L50

[rust-highfive]

r? @joshtriplett

(rust-highfive has picked a reviewer for you, use r? to override)

[MiSawa]

Made my thing use .addr() instead of .to_bits() MiSawa/pomelo@c769ad0 :)

[RSSchermer]

My use case linked above concerns a "WASM in the browser" context. It essentially consists of handing ownership of a value completely to the browser host, by attaching it to a custom element object. At a high level the steps are something like this:

• As part of the custom element constructor, allocate a Box with some data of type T: 'static which we to attach the the object. Turn said box into a raw *mut dyn Any. Serialize that *mut dyn Any (in this case as a byte slice that gets copied to a browser owned Javascript Uint8Array) and store that serialization on the custom element object.
• Whenever the user obtains a custom element object, they may obtain a shared reference to the attached data. To do so we read back the "address" part of the serialized data and construct a *const T.
• When the browser is completely done with the object - there are no remaining references to the object, neither from our WASM program, nor anywhere else (e.g. it's no longer attached to the DOM) - the browser will notify us via a FinalizationRegistry callback where it passes us our serialized pointer data. We deserialize the complete *mut dyn Any, then use it to reconstruct the Box<dyn Any> and let it drop.

(I don't use this process only for custom elements, I use essentially this same concept for "custom events" as well.)

In my understanding of "provenance", I think that in this case the browser becomes part of the provenance chain of custody. Our "box" can continue to live owned solely by the host with no remaining references from the Rust-WASM program (e.g. the custom element that owns our box is attached to the DOM). The user may reobtain a reference to the custom element object at a later point (e.g. via a Rust wrapper around Javascript's document.querySelector("#my_custom_element")); accessing the attached data then means rebuilding a pointer from a serialization. Finally, properly cleaning up the data at the end of its live again means reconstructing a pointer from a serialization, then reconstituting our "box" so that it's destructors can run.

I don't think the browser currently can, or will ever be able to, give the required provenance guarantees. My current understanding of "strict provenance" is that if it we're to become enforced by the compiler, this would mean that this use-case becomes impossible (because it goes beyond just removing from_bits; integer-pointer casting and transmuting would also stop working), is that correct? If that indeed is the case, then I would really like an escape hatch, because I personally consider this a high value use-case.

[RalfJung]

Made my thing use .addr() instead of .to_bits() MiSawa/pomelo@c769ad0 :)

Also note #95588 -- the intent (EDIT: or at least, my intent) with addr is that you must use with_addr instead of usize as ptr if you ever want to cast the address back to a pointer.

[eddyb]

@RSSchermer I believe this can be as simple as "you should pass Rust pointers to FFI" (and let the other side do the serialization/deserialization by accessing arbitrary parts of wasm linear memory).

Once you leave the wasm VM, the "host" side of that VM (JS code in the browser) has the equivalent access of a CHERI kernel (which can synthesize any possible userspace capability - this is needed for swapping RAM to disk then bringing it back into RAM later) or a very permissive DDC (default data capability).

In fact, I believe that on CHERI, a wasm VM would explicitly have a single pointer (capability) that covers the entire wasm linear memory (effectively a Rust &mut [u8]), with all wasm linear memory accesses being relative to that pointer (capability), no matter whether it's interpreted or JIT'd.

Provenance semantics actually go away a bit sooner, as wasm itself allows any code access to any part of the linear memory with no UB (lack of UB being part of the design of wasm), but only something like asm! could exploit that - anything higher-up would run into Rust or LLVM provenance rules which differ from wasm.

---

integer-pointer casting and transmuting would also stop working

AFAIK what's up for debate is "integer-pointer casting", whereas transmuting between pointers and integers has been on much shakier ground already - I think that's because even if ptr2int2ptr is supported, it's helpful to have that explicit "escape to integer" step to opt into it, whereas transmute is much more opaque.

[eddyb]

@MiSawa I believe you need to be careful in the physical address to pointer mapping.
I see 4 options for that kind of code:

• use some sort of MMIO-specific usize -> *mut T constructor (which AFAIK we don't yet have?)
  • actually, coming back to this after writing the rest, this is probably wrong if the data is populated by firmware (as opposed to being MMIO) and doesn't need volatile accesses
• change the acpi crate to pass a pointer through instead of a "physical address"
  • this is probably wrong - "physical" here can be different than a pointer because the latter may be virtual
• add a virtual_base_for_physical: *mut u8 field to your Handler that you can build a pointer from with virtual_base_for_physical.wrapping_add(physical_address) - not sure what to initialize it with though, 0 as *mut u8?
• hold off on switching to .addr() and continue to use as usize (not sure why .to_bits() was used)

---

It's tricky to fully understand the scope of this, but IMO if something is supposed to be a "physical address" that needs to be adjusted for a virtual window, then it shouldn't even be read from memory as a pointer.

If it is a pointer in the same address space, then the acpi crate should provide for that, e.g.:

trait AcpiHandler {...}
impl<H: AcpiHandler> AcpiTables<H> {
    pub unsafe fn from_rsdp(
        handler: H,
        rsdp_address: usize
    ) -> Result<AcpiTables<H>, AcpiError> {...}
}

needs to become:

trait AcpiHandler {
    type PhysicalAddr: Copy;
    ...
}
impl<H: AcpiHandler> AcpiTables<H> {
    pub unsafe fn from_rsdp(
        handler: H,
        rsdp_address: H::PhysicalAddr
    ) -> Result<AcpiTables<H>, AcpiError> {...}
}

[RalfJung]

@RSSchermer I can't quite imagine what that all looks like from a Rust perspective, but if (as @eddyb indicated) you pass opaque data as an array of MaybeUninit to some FFI, and it later returns back the same data to you, you can assume that provenance was preserved. Then you don't even ever have to cast pointers to/from an integer.

If it must be an integer then indeed you are outside the realm of strict provenance. As long as you explicitly cast (and do not transmute) between pointers and integers, the intent is that this is allowed, but it might not work with things like CHERI or Miri.

[RSSchermer]

@eddyb If I'm understanding you correctly, then - using my case as an example - if you wanted to do this serialization/deserializaiton with minimal context switches, it might look something like this:

struct CustomElementData {
    address_ptr: *const (),
    vtable_ptr: *const (),
    type_id: u64
}

#[wasm_bindgen]
fn serialize_in_js(linear_memory_offset: *mut CustomElementData, size: u32) {}

#[wasm_bindgen]
fn deserialize_in_js(linear_memory_offset: *mut CustomElementData) {}

let some_custom_element_data = Box::new(CustomElementData { ... });

let linear_memory_offset = some_custom_element_data.into_raw();
let size = mem::size_of::<CustomElementData>();

serialize_in_js(linear_memory_offset, size as u32);

unsafe {
     mem::drop(Box::from_raw(linear_memory_offset));
}

// Then later...

let uninit_custom_element_data = Box::new(CustomElementData {...});

let linear_memory_offset = uninit_custom_element_data.into_raw();

deserialize_in_js(linear_memory_offset);

// `uninit_custom_element_data` is now initialized with the deserialized data!
let init_custom_element_data = unsafe { Box::from_raw(linear_memory_offset) };

And in JS:

function serialize_in_js(linear_memory_offset, size) {
    // Create a view the relevant region of the WASM linear memory buffer. I'm not currently sure how 
    // one would obtain the reference to the correct WASM linear memory buffer.
    let view = new Uint8Array(wasm_linear_memory, linear_memory_offset, size);

    // Copy it to a new non-view Uint8Array
    let serialized = new Uint8Array(view);

    // Store it somewhere...
}

function deserialize_in_js(linear_memory_offset) {
    // ... retrieve the serialized buffer from wherever it was stored
    let serialized = ...;

    // Overwrite the target region in WASM linear memory buffer with the serialized buffer
    wasm_linear_memory.set(serialized, linear_memory_offset);
}

I think something like that might work. I'm still left with the question of how to obtain the reference to the relevant linear memory buffer. Is there/could there be a special JsValue that represents the WASM linear memory in which the current Rust code is running, so that it could be passed along to the serialize_in_js/deserialize_in_js functions?

Alternatively, if I understand you correctly, if I could somehow obtain the "WASM base pointer" (the zero pointer?) for the current WASM linear memory in Rust, then I could deserialize "on the Rust side" using that base pointer as provenance?

// Lets assume there is some constant that represents this "WASM linear memory base pointer"
let my_address_ptr: *const () = WASM_BASE_POINTER.with_addr(my_address_ptr_bits);

At least it seems the provenance situation in WASM is much simpler than I initially feared in my limited understanding of the concept. My case would be addressed if either of these solutions could be made to work (or perhaps there is still a better solution altogether?).

As to removing to_bits/from_bits: as long as it remains possible to cast ptr->int->ptr for the time being, I have no issue with that. I don't care so much for the to_bits/from_bits functions specifically, as much as I care for this more general "pointer serialization/deserialization" concept, it's just that to_bits/from_bits seemed more appropriate than integer casting.

[eddyb]

I think something like that might work.

Yeah, that looks about right.

I'm still left with the question of how to obtain the reference to the relevant linear memory buffer. Is there/could there be a special JsValue that represents the WASM linear memory in which the current Rust code is running, so that it could be passed along to the serialize_in_js/deserialize_in_js functions?

It's part of the "wasm VM state". In the browser, according to the docs on MDN, it seems that you can only access the Memory object if it was in the wasm module's exports.
wasm-bindgen seems to be exporting that with the name memory, so that might work (if you can get the Instance object, not sure that's easy from the JS bindings).

Looking around I did find that wasm-bindgen can expose Rust memory to JS using typed array views (doc link).
It relies on wasm_bindgen::memory(), which does appear to be exactly what you were asking for in the first place.
(When I started looking at this, I didn't realize wasm-bindgen doesn't give you much direct access from JS)

[MiSawa]

@eddyb Ah hmm, yeah it's weird. It uses physical_address as a pointer because it currently has identity mapping only (and it's already mapped), so physical and virtual addresses are actually the same. I made it it use as usize for now, but I guess to do this properly, I should make the thing that maps physical memory to virtual memory somehow create a pointer for the "root" of the chain of custody, and perhaps actually maps the region when requested by map_physical_region.

[RalfJung]

At least it seems the provenance situation in WASM is much simpler than I initially feared in my limited understanding of the concept. My case would be addressed if either of these solutions could be made to work (or perhaps there is still a better solution altogether?).

Note that provenance is a Rust concept. x86 doesn't have provenance any more than WASM does. So I don't think the fact that you are on WASM changes anything here.

vtable metadata makes things more tricky so I'll just consider pointer-sized arbitrary data being handed by the 'outside world' for now. If you have an FFI like

extern {
  fn store_data(..., data: usize);
  fn register_callback(..., callback: fn(usize)); // gets called with the `data`
}

then you cannot use strict provenance since you are forced to turn the data into an integer before giving it away.

But you can still do store_data(..., ptr as usize) and later

register_callback(..., |addr| {
  let ptr = addr as *mut T;
  ...
});

because round-tripping a pointer through an integer still works under 'permissive provenance'. You just need to use a ptr2int operation that explicitly tells the compiler that you will later cast this integer back to a pointer and not give the language any help re: the provenance of that new pointer.

The intent is to add ptr.escape_addr() that complements ptr.addr() but also additionally allows casts back to integers. That might optimize less well than addr but lets you work with FFIs that require integers.

---

Of course, if you can actually adjust the FFI (since it seems to be partially under your control) to actually use pointer types on the FFI boundary, that's even better as then provenance will be preserved. :)

[RalfJung]

As to removing to_bits/from_bits: as long as it remains possible to cast ptr->int->ptr for the time being, I have no issue with that. I don't care so much for the to_bits/from_bits functions specifically, as much as I care for this more general "pointer serialization/deserialization" concept, it's just that to_bits/from_bits seemed more appropriate than integer casting.

The issue is, it is not possible to serialize a pointer into an integer without losing information. Specifically, the provenance gets lost. So there are two options

• Accept the loss of information. This requires being explicit about the conversion (using casts or the not-yet-existent escape_addr/from_escaped_addr) and it means you are outside the strict provenance fragment (so Miri and CHERI might work less well or not at all).
• Not use integer types, so you can preserve all information. A 'generic' type that can hold all data, including provenance, is MaybeUninit<$int>. Or in your last post it looks like you can even use a custom type like CustomElementData that is 'honest' about having pointers there, that's even better.

[RSSchermer]

Thanks @eddyb, @RalfJung!

wasm_bindgen::memory indeed turned out to be exactly what was needed, thank you for making me aware of its existance @eddyb. It returns a WebAssembly.Memory object that contains the linear memory buffer for the calling WASM code.

For reference, here's a modified version of the outline above that better reflects what I ended up doing:

struct CustomElementData {
    address: *mut (),
    metadata: DynMetadata<dyn Any>,
    type_id: TypeId
}

#[wasm_bindgen]
fn serialize_custom_element_data(
    wasm_memory: &JsValue, 
    offset: *mut CustomElementData, 
    size: u32
) -> js_sys::Uint8Array {}

#[wasm_bindgen]
fn deserialize_custom_element_data(
    wasm_memory: &JsValue, 
    offset: *mut CustomElementData, 
    serialized: &js_sys::Uint8Array
) {}

let mut custom_element_data = CustomElementData { ... };

let offset = &mut custom_element_data as *mut CustomElementData;
let size = mem::size_of::<CustomElementData>();

let serialized = serialize_custom_element_data(&wasm_bindgen::memory(), offset, size as u32);

// Explicitly drop here so it doesn't drop early...
mem::drop(custom_element_data);

// Then later...

let uninit_custom_element_data = MaybeUninit::<CustomElementData>::uninit();
let offset = uninit_custom_element_data.as_mut_ptr();

deserialize_custom_element_data(&wasm_bindgen::memory(), offset, &serialized);

// `uninit_custom_element_data` is now initialized with the deserialized data!
let init_custom_element_data = unsafe { uninit_custom_element_data.assume_init() };

And in JS:

function serialize_custom_element_data(
    wasm_memory,
    offset,
    size
) {
    // Create a view the relevant region of the WASM linear memory buffer.
    let view = new Uint8Array(wasm_memory.buffer, offset, size);

    // Copy it to a new non-view Uint8Array and return
    return new Uint8Array(view);
}

function deserialize_custom_element_data(
    wasm_memory,
    offset,
    serialized
) {
    // View WASM memory as bytes
    let buffer_view = new Uint8Array(wasm_memory.buffer);

    // Overwrite the memory at the pointer offset with the `serialized` data
    buffer_view.set(serialized, offset);
}

(Link to the actual commit in case someone really wants the nitty gritty.)

Note that earlier I had assumed you'd have to box the data that you want to "serialize", because I seemed to recall that stack memory was not part of the linear memory buffer, but my impressions seems either wrong or I'm misunderstanding something about what's really going on under the hood here, because empirically it does work without boxing. Also note that my CustomElementData type here just serves as an example; I would expect this to work for any Sized type.

NINJA EDIT: I came accross this post, which gives a high level explanation as to why this works without boxing, even though the actual stack memory is indeed separate (in short: a "shadow stack" is maintained in linear memory); figured I'd link it here for completeness.

[scottmcm]

I've updated the deprecation message to point at the new exposed methods from #95588 instead, so this is currently blocked waiting on that one to land.

That way the "I don't want to think about it" replacement is sound, but hopefully the mention of the strict provenance APIs (and the good docs on exposed_addr and such) will encourage people to do the bigger change to use those instead.

EDIT: #95588 landed, so this is unblocked now.

[eddyb]

Note that earlier I had assumed you'd have to box the data that you want to "serialize", because I seemed to recall that stack memory was not part of the linear memory buffer, but my impressions seems either wrong or I'm misunderstanding something about what's really going on under the hood here, because empirically it does work without boxing. Also note that my CustomElementData type here just serves as an example; I would expect this to work for any Sized type.

NINJA EDIT: I came accross this post, which gives a high level explanation as to why this works without boxing, even though the actual stack memory is indeed separate (in short: a "shadow stack" is maintained in linear memory); figured I'd link it here for completeness.

Heh, glad it's working, and that you found an explanation.
Only wanted to comment that I find it odd that it became known as "shadow stack".

The stack in linear memory (used for variables that have their address taken, i.e. that could stay hidden in CPU registers when compiling natively) is the kind of thing you'd call "the (data) stack", as it's accessible with normal pointers, and "shadow stack" would typically refer to managed/hidden stacks, like:

• the wasm call stack that is part of the wasm VM and not exposed to the wasm code at all
• whatever is holding wasm locals (which could be combined with the call stack, or not)
  • or even wasm globals - for example, the "stack pointer" inside wasm is implemented with a wasm global, which a wasm JIT could e.g. reserve a CPU register for
  • either way, a wasm local/global never overlaps with wasm linear memory and so is "protected" from wasm

(One of the places where I've seen "shadow" used is the AddressSanitizer "shadow memory", used to hold metadata about allocations)

[scottmcm]

@RalfJung This change now points nightly people who were using {to|from}_bits at the expose versions instead.

I thought that was ok, but after the meeting today I'm not so confident about it. Do you think we should still have nightly methods for "this does what as does" as a third option (in addition to the strict ones and the expose ones).

[saethlin]

I've looked through the examples above. I think ARWA is well-spoken-for already, which is awesome.

pbr should be using ptr::is_aligned. That's unstable, but if the code was already comfortable with using a different nightly feature 🤷. The more common way to do this currently in stable rust is as, though that's really only because ptr::addr is unstable (a world without the strict-provenance APIs is full of bad decisions like this).

pomelo is a bit beyond me, but as far as I can tell, that address eventually lands here: https://docs.rs/x86_64/latest/src/x86_64/instructions/port.rs.html#31-40 which is an inline assembly block with nomem, so if that is correct, I think it's likely that address has nothing to do with Rust memory. If that's the case ptr::addr is a valid replacement.

[RalfJung]

@scottmcm yeah the expose functions do have the semantics of int2ptr/ptr2int casts. I don't think we will ever have the expose functions behave different from the as casts, so pointing users of these functions there makes sense. (If we don't end up going with expose and angelic non-det semantics for as, we'll rename/deprecate the expose methods, but not make expose different from as.)

The OP in this thread still says to use addr(), that should probably be changed.

@saethlin

so if that is correct, I think it's likely that address has nothing to do with Rust memory. If that's the case ptr::addr is a valid replacement.

As in, that memory must already be exposed since it "comes from the outside"?
Expose could still be needed if there are any references involved though.

[saethlin]

The code in question is generally well beyond my understanding, but yes my best guess is that the referenced "memory" does "come from the outside" (whether there is even memory at that address is not clear to me). As far as I can tell, the address comes from the magic system-level entrypoint related to UEFI. Effectively, an argument to main.

Whether there are also references in play I have no idea. Miri would know, but I think getting Miri to execute programs like this won't happen for a while yet.

[scottmcm]

@RalfJung Good point. I've updated the OP to not (essentially) say "use strict provenance instead" like it did before. And the deprecation warnings on the methods point to the expose versions.

@joshtriplett I think this PR is good to be reviewed now that we've had the meeting.

[RalfJung]

@rust-lang/libs-api this PR has been waiting for review for a while, would be nice to see it move ahead. :)

[bors]

☔ The latest upstream changes (presumably #104655) made this pull request unmergeable. Please resolve the merge conflicts.

[rustbot]

Hey! It looks like you've submitted a new PR for the library teams!

If this PR contains changes to any rust-lang/rust public library APIs then please comment with @rustbot label +T-libs-api -T-libs to tag it appropriately. If this PR contains changes to any unstable APIs please edit the PR description to add a link to the relevant API Change Proposal or create one if you haven't already. If you're unsure where your change falls no worries, just leave it as is and the reviewer will take a look and make a decision to forward on if necessary.

Examples of T-libs-api changes:

• Stabilizing library features
• Introducing insta-stable changes such as new implementations of existing stable traits on existing stable types
• Introducing new or changing existing unstable library APIs (excluding permanently unstable features / features without a tracking issue)
• Changing public documentation in ways that create new stability guarantees
• Changing observable runtime behavior of library APIs

[dtolnay]

I pushed a rebase to resolve conflict with #104632.

[dtolnay]

@bors r+

[bors]

📌 Commit a9e92be has been approved by dtolnay

It is now in the queue for this repository.

[dtolnay]

• Bump deprecation from 1.62 to 1.67, which is the current version in nightly
• Resolve conflict with Remove #[rustc_deprecated] #95960

@bors r+

[bors]

📌 Commit 6d943af has been approved by dtolnay

It is now in the queue for this repository.

[scottmcm]

Thanks, @dtolnay !