Auto merge of #59464 - alexcrichton:wasi-pr, r=fitzgen

Add a new wasm32-unknown-wasi target

This commit adds a new wasm32-based target distributed through rustup,
supported in the standard library, and implemented in the compiler. The
`wasm32-unknown-wasi` target is intended to be a WebAssembly target
which matches the [WASI proposal recently announced][LINK]. In summary
the WASI target is an effort to define a standard set of syscalls for
WebAssembly modules, allowing WebAssembly modules to not only be
portable across architectures but also be portable across environments
implementing this standard set of system calls.

The wasi target in libstd is still somewhat bare bones. This PR does not
fill out the filesystem, networking, threads, etc. Instead it only
provides the most basic of integration with the wasi syscalls, enabling
features like:

* `Instant::now` and `SystemTime::now` work
* `env::args` is hooked up
* `env::vars` will look up environment variables
* `println!` will print to standard out
* `process::{exit, abort}` should be hooked up appropriately

None of these APIs can work natively on the `wasm32-unknown-unknown`
target, but with the assumption of the WASI set of syscalls we're able
to provide implementations of these syscalls that engines can implement.
Currently the primary engine implementing wasi is [wasmtime], but more
will surely emerge!

In terms of future development of libstd, I think this is something
we'll probably want to discuss. The purpose of the WASI target is to
provide a standardized set of syscalls, but it's *also* to provide a
standard C sysroot for compiling C/C++ programs. This means it's
intended that functions like `read` and `write` are implemented for this
target with a relatively standard definition and implementation. It's
unclear, therefore, how we want to expose file descriptors and how we'll
want to implement system primitives. For example should `std::fs::File`
have a libc-based file descriptor underneath it? The raw wasi file
descriptor? We'll see! Currently these details are all intentionally
hidden and things we can change over time.

A `WasiFd` sample struct was added to the standard library as part of
this commit, but it's not currently used. It shows how all the wasi
syscalls could be ergonomically bound in Rust, and they offer a possible
implementation of primitives like `std::fs::File` if we bind wasi file
descriptors exactly.

Apart from the standard library, there's also the matter of how this
target is integrated with respect to its C standard library. The
reference sysroot, for example, provides managment of standard unix file
descriptors and also standard APIs like `open` (as opposed to the
relative `openat` inspiration for the wasi ssycalls). Currently the
standard library relies on the C sysroot symbols for operations such as
environment management, process exit, and `read`/`write` of stdio fds.
We want these operations in Rust to be interoperable with C if they're
used in the same process. Put another way, if Rust and C are linked into
the same WebAssembly binary they should work together, but that requires
that the same C standard library is used.

We also, however, want the `wasm32-unknown-wasi` target to be
usable-by-default with the Rust compiler without requiring a separate
toolchain to get downloaded and configured. With that in mind, there's
two modes of operation for the `wasm32-unknown-wasi` target:

1. By default the C standard library is statically provided inside of
   `liblibc.rlib` distributed as part of the sysroot. This means that
   you can `rustc foo.wasm --target wasm32-unknown-unknown` and you're
   good to go, a fully workable wasi binary pops out. This is
   incompatible with linking in C code, however, which may be compiled
   against a different sysroot than the Rust code was previously
   compiled against. In this mode the default of `rust-lld` is used to
   link binaries.

2. For linking with C code, the `-C target-feature=-crt-static` flag
   needs to be passed. This takes inspiration from the musl target for
   this flag, but the idea is that you're no longer using the provided
   static C runtime, but rather one will be provided externally. This
   flag is intended to also get coupled with an external `clang`
   compiler configured with its own sysroot. Therefore you'll typically
   use this flag with `-C linker=/path/to/clang-script-wrapper`. Using
   this mode the Rust code will continue to reference standard C
   symbols, but the definition will be pulled in by the linker configured.

Alright so that's all the current state of this PR. I suspect we'll
definitely want to discuss this before landing of course! This PR is
coupled with libc changes as well which I'll be posting shortly.

[LINK]: https://hacks.mozilla.org/2019/03/standardizing-wasi-a-webassembly-system-interface/
[wasmtime]: https://github.com/cranestation/wasmtime-wasi

config.toml.example

@@ -477,6 +477,9 @@
 # linked binaries
 #musl-root = "..."
 
+# The root location of the `wasm32-unknown-wasi` sysroot.
+#wasi-root = "..."
+
 # Used in testing for configuring where the QEMU images are located, you
 # probably don't want to use this.
 #qemu-rootfs = "..."

src/bootstrap/bin/rustc.rs

@@ -122,8 +122,8 @@ fn main() {
             cmd.arg("-Cprefer-dynamic");
         }
 
-        // Help the libc crate compile by assisting it in finding the MUSL
-        // native libraries.
+        // Help the libc crate compile by assisting it in finding various
+        // sysroot native libraries.
         if let Some(s) = env::var_os("MUSL_ROOT") {
             if target.contains("musl") {
                 let mut root = OsString::from("native=");
@@ -132,6 +132,12 @@ fn main() {
                 cmd.arg("-L").arg(&root);
             }
         }
+        if let Some(s) = env::var_os("WASI_ROOT") {
+            let mut root = OsString::from("native=");
+            root.push(&s);
+            root.push("/lib/wasm32-wasi");
+            cmd.arg("-L").arg(&root);
+        }
 
         // Override linker if necessary.
         if let Ok(target_linker) = env::var("RUSTC_TARGET_LINKER") {

src/bootstrap/compile.rs

@@ -129,6 +129,13 @@ fn copy_third_party_objects(builder: &Builder<'_>, compiler: &Compiler, target:
                 &libdir.join(obj),
             );
         }
+    } else if target.ends_with("-wasi") {
+        for &obj in &["crt1.o"] {
+            builder.copy(
+                &builder.wasi_root(target).unwrap().join("lib/wasm32-wasi").join(obj),
+                &libdir.join(obj),
+            );
+        }
     }
 
     // Copies libunwind.a compiled to be linked wit x86_64-fortanix-unknown-sgx.
@@ -190,6 +197,12 @@ pub fn std_cargo(builder: &Builder<'_>,
                 cargo.env("MUSL_ROOT", p);
             }
         }
+
+        if target.ends_with("-wasi") {
+            if let Some(p) = builder.wasi_root(target) {
+                cargo.env("WASI_ROOT", p);
+            }
+        }
     }
 }
 

src/bootstrap/config.rs

@@ -169,6 +169,7 @@ pub struct Target {
     pub ndk: Option<PathBuf>,
     pub crt_static: Option<bool>,
     pub musl_root: Option<PathBuf>,
+    pub wasi_root: Option<PathBuf>,
     pub qemu_rootfs: Option<PathBuf>,
     pub no_std: bool,
 }
@@ -344,6 +345,7 @@ struct TomlTarget {
     android_ndk: Option<String>,
     crt_static: Option<bool>,
     musl_root: Option<String>,
+    wasi_root: Option<String>,
     qemu_rootfs: Option<String>,
 }
 
@@ -605,6 +607,7 @@ impl Config {
                 target.linker = cfg.linker.clone().map(PathBuf::from);
                 target.crt_static = cfg.crt_static.clone();
                 target.musl_root = cfg.musl_root.clone().map(PathBuf::from);
+                target.wasi_root = cfg.wasi_root.clone().map(PathBuf::from);
                 target.qemu_rootfs = cfg.qemu_rootfs.clone().map(PathBuf::from);
 
                 config.target_config.insert(INTERNER.intern_string(triple.clone()), target);

src/bootstrap/lib.rs

@@ -861,6 +861,13 @@ impl Build {
             .map(|p| &**p)
     }
 
+    /// Returns the sysroot for the wasi target, if defined
+    fn wasi_root(&self, target: Interned<String>) -> Option<&Path> {
+        self.config.target_config.get(&target)
+            .and_then(|t| t.wasi_root.as_ref())
+            .map(|p| &**p)
+    }
+
     /// Returns `true` if this is a no-std `target`, if defined
     fn no_std(&self, target: Interned<String>) -> Option<bool> {
         self.config.target_config.get(&target)

src/ci/docker/dist-various-2/Dockerfile

@@ -34,6 +34,9 @@ COPY dist-various-2/build-x86_64-fortanix-unknown-sgx-toolchain.sh /tmp/
 # Any update to the commit id here, should cause the container image to be re-built from this point on.
 RUN /tmp/build-x86_64-fortanix-unknown-sgx-toolchain.sh "53b586346f2c7870e20b170decdc30729d97c42b"
 
+COPY dist-various-2/build-wasi-toolchain.sh /tmp/
+RUN /tmp/build-wasi-toolchain.sh
+
 COPY scripts/sccache.sh /scripts/
 RUN sh /scripts/sccache.sh
 
@@ -66,6 +69,7 @@ ENV TARGETS=x86_64-fuchsia
 ENV TARGETS=$TARGETS,aarch64-fuchsia
 ENV TARGETS=$TARGETS,sparcv9-sun-solaris
 ENV TARGETS=$TARGETS,wasm32-unknown-unknown
+ENV TARGETS=$TARGETS,wasm32-unknown-wasi
 ENV TARGETS=$TARGETS,x86_64-sun-solaris
 ENV TARGETS=$TARGETS,x86_64-unknown-linux-gnux32
 ENV TARGETS=$TARGETS,x86_64-unknown-cloudabi
@@ -74,5 +78,6 @@ ENV TARGETS=$TARGETS,nvptx64-nvidia-cuda
 
 ENV X86_FORTANIX_SGX_LIBS="/x86_64-fortanix-unknown-sgx/lib/"
 
-ENV RUST_CONFIGURE_ARGS --enable-extended --enable-lld --disable-docs
+ENV RUST_CONFIGURE_ARGS --enable-extended --enable-lld --disable-docs \
+  --set target.wasm32-unknown-wasi.wasi-root=/wasm32-unknown-wasi
 ENV SCRIPT python2.7 ../x.py dist --target $TARGETS

src/ci/docker/dist-various-2/build-wasi-toolchain.sh

@@ -0,0 +1,20 @@
+#!/bin/sh
+#
+# ignore-tidy-linelength
+
+set -ex
+
+# Originally from https://releases.llvm.org/8.0.0/clang+llvm-8.0.0-x86_64-linux-gnu-ubuntu-14.04.tar.xz
+curl https://s3-us-west-1.amazonaws.com/rust-lang-ci2/rust-ci-mirror/clang%2Bllvm-8.0.0-x86_64-linux-gnu-ubuntu-14.04.tar.xz | \
+  tar xJf -
+export PATH=`pwd`/clang+llvm-8.0.0-x86_64-linux-gnu-ubuntu-14.04/bin:$PATH
+
+git clone https://github.com/CraneStation/wasi-sysroot
+
+cd wasi-sysroot
+git reset --hard 320054e84f8f2440def3b1c8700cedb8fd697bf8
+make -j$(nproc) INSTALL_DIR=/wasm32-unknown-wasi install
+
+cd ..
+rm -rf reference-sysroot-wasi
+rm -rf clang+llvm*

src/librustc_target/spec/mod.rs

@@ -444,6 +444,7 @@ supported_targets! {
     ("asmjs-unknown-emscripten", asmjs_unknown_emscripten),
     ("wasm32-unknown-emscripten", wasm32_unknown_emscripten),
     ("wasm32-unknown-unknown", wasm32_unknown_unknown),
+    ("wasm32-unknown-wasi", wasm32_unknown_wasi),
     ("wasm32-experimental-emscripten", wasm32_experimental_emscripten),
 
     ("thumbv6m-none-eabi", thumbv6m_none_eabi),

src/librustc_target/spec/wasm32_unknown_wasi.rs

@@ -0,0 +1,113 @@
+//! The `wasm32-unknown-wasi` target is a new and still (as of March 2019)
+//! experimental target. The definition in this file is likely to be tweaked
+//! over time and shouldn't be relied on too much.
+//!
+//! The `wasi` target is a proposal to define a standardized set of syscalls
+//! that WebAssembly files can interoperate with. This set of syscalls is
+//! intended to empower WebAssembly binaries with native capabilities such as
+//! filesystem access, network access, etc.
+//!
+//! You can see more about the proposal at https://wasi.dev
+//!
+//! The Rust target definition here is interesting in a few ways. We want to
+//! serve two use cases here with this target:
+//!
+//! * First, we want Rust usage of the target to be as hassle-free as possible,
+//!   ideally avoiding the need to configure and install a local
+//!   wasm32-unknown-wasi toolchain.
+//!
+//! * Second, one of the primary use cases of LLVM's new wasm backend and the
+//!   wasm support in LLD is that any compiled language can interoperate with
+//!   any other. To that the `wasm32-unknown-wasi` target is the first with a
+//!   viable C standard library and sysroot common definition, so we want Rust
+//!   and C/C++ code to interoperate when compiled to `wasm32-unknown-unknown`.
+//!
+//! You'll note, however, that the two goals above are somewhat at odds with one
+//! another. To attempt to solve both use cases in one go we define a target
+//! that (ab)uses the `crt-static` target feature to indicate which one you're
+//! in.
+//!
+//! ## No interop with C required
+//!
+//! By default the `crt-static` target feature is enabled, and when enabled
+//! this means that the the bundled version of `libc.a` found in `liblibc.rlib`
+//! is used. This isn't intended really for interoperation with a C because it
+//! may be the case that Rust's bundled C library is incompatible with a
+//! foreign-compiled C library. In this use case, though, we use `rust-lld` and
+//! some copied crt startup object files to ensure that you can download the
+//! wasi target for Rust and you're off to the races, no further configuration
+//! necessary.
+//!
+//! All in all, by default, no external dependencies are required. You can
+//! compile `wasm32-unknown-wasi` binaries straight out of the box. You can't,
+//! however, reliably interoperate with C code in this mode (yet).
+//!
+//! ## Interop with C required
+//!
+//! For the second goal we repurpose the `target-feature` flag, meaning that
+//! you'll need to do a few things to have C/Rust code interoperate.
+//!
+//! 1. All Rust code needs to be compiled with `-C target-feature=-crt-static`,
+//!    indicating that the bundled C standard library in the Rust sysroot will
+//!    not be used.
+//!
+//! 2. If you're using rustc to build a linked artifact then you'll need to
+//!    specify `-C linker` to a `clang` binary that supports
+//!    `wasm32-unknown-wasi` and is configured with the `wasm32-unknown-wasi`
+//!    sysroot. This will cause Rust code to be linked against the libc.a that
+//!    the specified `clang` provides.
+//!
+//! 3. If you're building a staticlib and integrating Rust code elsewhere, then
+//!    compiling with `-C target-feature=-crt-static` is all you need to do.
+//!
+//! You can configure the linker via Cargo using the
+//! `CARGO_TARGET_WASM32_UNKNOWN_WASI_LINKER` env var. Be sure to also set
+//! `CC_wasm32-unknown-wasi` if any crates in the dependency graph are using
+//! the `cc` crate.
+//!
+//! ## Remember, this is all in flux
+//!
+//! The wasi target is **very** new in its specification. It's likely going to
+//! be a long effort to get it standardized and stable. We'll be following it as
+//! best we can with this target. Don't start relying on too much here unless
+//! you know what you're getting in to!
+
+use super::wasm32_base;
+use super::{LinkerFlavor, LldFlavor, Target};
+
+pub fn target() -> Result<Target, String> {
+    let mut options = wasm32_base::options();
+
+    options
+        .pre_link_args
+        .entry(LinkerFlavor::Gcc)
+        .or_insert(Vec::new())
+        .push("--target=wasm32-unknown-wasi".to_string());
+
+    // When generating an executable be sure to put the startup object at the
+    // front so the main function is correctly hooked up.
+    options.pre_link_objects_exe_crt.push("crt1.o".to_string());
+
+    // Right now this is a bit of a workaround but we're currently saying that
+    // the target by default has a static crt which we're taking as a signal
+    // for "use the bundled crt". If that's turned off then the system's crt
+    // will be used, but this means that default usage of this target doesn't
+    // need an external compiler but it's still interoperable with an external
+    // compiler if configured correctly.
+    options.crt_static_default = true;
+    options.crt_static_respected = true;
+
+    Ok(Target {
+        llvm_target: "wasm32-unknown-wasi".to_string(),
+        target_endian: "little".to_string(),
+        target_pointer_width: "32".to_string(),
+        target_c_int_width: "32".to_string(),
+        target_os: "unknown".to_string(),
+        target_env: "wasi".to_string(),
+        target_vendor: "unknown".to_string(),
+        data_layout: "e-m:e-p:32:32-i64:64-n32:64-S128".to_string(),
+        arch: "wasm32".to_string(),
+        linker_flavor: LinkerFlavor::Lld(LldFlavor::Wasm),
+        options,
+    })
+}

src/libstd/Cargo.toml

@@ -18,8 +18,8 @@ alloc = { path = "../liballoc" }
 panic_unwind = { path = "../libpanic_unwind", optional = true }
 panic_abort = { path = "../libpanic_abort" }
 core = { path = "../libcore" }
-libc = { version = "0.2.44", default-features = false, features = ['rustc-dep-of-std'] }
-compiler_builtins = { version = "0.1.1" }
+libc = { version = "0.2.51", default-features = false, features = ['rustc-dep-of-std'] }
+compiler_builtins = { version = "0.1.8" }
 profiler_builtins = { path = "../libprofiler_builtins", optional = true }
 unwind = { path = "../libunwind" }
 rustc-demangle = { version = "0.1.10", features = ['rustc-dep-of-std'] }

src/libstd/os/mod.rs

@@ -51,6 +51,7 @@ cfg_if! {
 #[cfg(target_os = "emscripten")] pub mod emscripten;
 #[cfg(target_os = "fuchsia")]    pub mod fuchsia;
 #[cfg(target_os = "hermit")]     pub mod hermit;
+#[cfg(target_env = "wasi")]      pub mod wasi;
 #[cfg(all(target_vendor = "fortanix", target_env = "sgx"))] pub mod fortanix_sgx;
 
 pub mod raw;

src/libstd/os/wasi.rs

@@ -0,0 +1,6 @@
+//! WASI-specific definitions
+
+#![stable(feature = "raw_ext", since = "1.1.0")]
+
+#[stable(feature = "rust1", since = "1.0.0")]
+pub use crate::sys::ext::*;

src/libstd/sys/mod.rs

@@ -35,6 +35,9 @@ cfg_if! {
     } else if #[cfg(target_os = "redox")] {
         mod redox;
         pub use self::redox::*;
+    } else if #[cfg(target_env = "wasi")] {
+        mod wasi;
+        pub use self::wasi::*;
     } else if #[cfg(target_arch = "wasm32")] {
         mod wasm;
         pub use self::wasm::*;

src/libstd/sys/wasi/alloc.rs

@@ -0,0 +1,43 @@
+use crate::alloc::{GlobalAlloc, Layout, System};
+use crate::ptr;
+use crate::sys_common::alloc::{MIN_ALIGN, realloc_fallback};
+use libc;
+
+#[stable(feature = "alloc_system_type", since = "1.28.0")]
+unsafe impl GlobalAlloc for System {
+    #[inline]
+    unsafe fn alloc(&self, layout: Layout) -> *mut u8 {
+        if layout.align() <= MIN_ALIGN && layout.align() <= layout.size() {
+            libc::malloc(layout.size()) as *mut u8
+        } else {
+            libc::aligned_alloc(layout.size(), layout.align()) as *mut u8
+        }
+    }
+
+    #[inline]
+    unsafe fn alloc_zeroed(&self, layout: Layout) -> *mut u8 {
+        if layout.align() <= MIN_ALIGN && layout.align() <= layout.size() {
+            libc::calloc(layout.size(), 1) as *mut u8
+        } else {
+            let ptr = self.alloc(layout.clone());
+            if !ptr.is_null() {
+                ptr::write_bytes(ptr, 0, layout.size());
+            }
+            ptr
+        }
+    }
+
+    #[inline]
+    unsafe fn dealloc(&self, ptr: *mut u8, _layout: Layout) {
+        libc::free(ptr as *mut libc::c_void)
+    }
+
+    #[inline]
+    unsafe fn realloc(&self, ptr: *mut u8, layout: Layout, new_size: usize) -> *mut u8 {
+        if layout.align() <= MIN_ALIGN && layout.align() <= new_size {
+            libc::realloc(ptr as *mut libc::c_void, new_size) as *mut u8
+        } else {
+            realloc_fallback(self, ptr, layout, new_size)
+        }
+    }
+}