no_std support #528

justinmoon · 2020-12-01T19:46:09Z

Use the alloc create, which requires the user define a global allocator.

I ignored old rust versions thus far. More research required to determine MSRV implications.
Import from core and alloc instead of std
Create std feature
Create use-bare-io feature which adds a
bare-io dependency to polyfill
std::io features. This is an experimental feature and should be
used with extreme caution. Is using a 3rd party crate acceptable for this?
Created an io module to actually do ^^
Replace HashMap and HashSet with BTreeMap and BTreeSet just to
get something working. This needs to be fixed.
CI runs tests no_std code

Closes #409

We use the [`alloc`](https://doc.rust-lang.org/alloc/) create, which requires the user define a global allocator. * Import from `core` and `alloc` instead of `std` * Create `std` feature * Create `use-bare-io` feature which adds a [bare-io](https://github.com/bbqsrc/bare-io) dependency to polyfill `std::io` features. This is an experimental feature and should be used with extreme caution. * Created an `io` module to polyfill `std::io` with `bare-io` in `no_std`. * Replace `HashMap` and `HashSet` with `BTreeMap` and `BTreeSet` just to get something working. This needs to be fixed. * CI runs tests `no_std` code

justinmoon · 2020-12-01T19:47:07Z

Cargo.toml


 [dependencies]
 base64-compat = { version = "1.0.0", optional = true }
 bech32 = "0.7.2"
-bitcoin_hashes = "0.9.0"
+bitcoin_hashes = { git = "https://github.com/justinmoon/bitcoin_hashes.git", branch = "more_no_std", features = [], default-features = false }


Hacky fork to get things working

justinmoon · 2020-12-01T19:48:08Z

src/blockdata/block.rs

@@ -308,12 +309,14 @@ impl fmt::Display for Bip34Error {
    }
 }

-impl ::std::error::Error for Bip34Error {}
+#[cfg(feature = "std")]
+impl std::error::Error for Bip34Error {}


I don't quite understand what a lack of an Error impl in `no_std would mean here ...

Not too much. The whole Error concept is std-only usually. The Result type doesn't require the second generic parameter to be Error, so it has very few effects.

justinmoon · 2020-12-01T19:48:53Z

src/io.rs

+pub use bare_io::*;
+
+#[cfg(feature = "std")]
+pub use std::io::*;


Any ideas on a better way to polyfill std::io?

What things need to be polyfilled exactly?

I mean at some level we can just feature-gate all stuff that needs std things, but I suppose your concern is that the fraction that would remain is so small that a polyfill could make more things no-std-compatible?

How does a polyfill version of Vec work?

F.e. if on-stack Vec's have a hard length requirement, that may cause various errors in different places as we assume that Vec's have infinite space. (F.e. when writing/encoding into them, we always unwrap errors so a length limit would cause these to panic.)

The polyfill is currently only for std::io. We're using alloc::Vec, which is exactly the same as std::Vec.

justinmoon · 2020-12-04T18:29:57Z

I added another commit updating dependencies to get this running on real microcontrollers, as demonstrated in this repo.

As for MSRV,

The alloc crate didn't stabilize until 1.36.0. If we want rust-bitcoin to run on microcontrollers I believe we need to upgrade MSRV to 1.36.0.
The bare-io crate uses a non_exhaustive attribute which didn't stabilize until 1.40.0. I bet we could find a way around this one either by patching bare-io or implementing the std::io polyfill ourselves.

`std::collections::HashMap` and `std::collections::HashSet` don't have any analogue in `alloc` because they require a OS RNG. [Recently no_std-compatible implementations have been merged](rust-lang/rust#58623) but I think we'd need to massively increase our MSRV. So I opted to move any `HasMap` or `HasSet` usage in tests to `BTreeMap` or `BTreeSet`, and to gate the only non-test usage of `HashMap` or `HashSet` (in the merkle block code which won't be helpful on MCUs) behind a "std" feature flag. Public API doesn't change.

Upgraded bare-io to a branch which adds a feature to disable the `#[non_exhaustive]` attribute, which wasn't stabilized until 1.40.0.

justinmoon · 2020-12-12T03:15:16Z

Now this works on stable 1.36.0, which is where alloc stabilized.

Kixunil

Making modifications to tests is not needed IMO, running tests without std doesn't make sense.

For cases where HashMap, HashSet are needed outside of tests a trait can be used to avoid hard dependency on std.

Kixunil · 2020-12-15T14:14:04Z

src/blockdata/opcodes.rs

@@ -834,7 +834,7 @@ mod tests {

    #[test]
    fn str_roundtrip() {
-        let mut unique = HashSet::new();
+        let mut unique = BTreeSet::new();


Should not be needed in tests, there's no point in making tests no_std

Kixunil · 2020-12-15T14:15:45Z

src/blockdata/constants.rs

@@ -19,7 +19,8 @@
 //! single transaction
 //!

-use std::default::Default;
+use core::default::Default;
+use alloc::vec::Vec;


I believe it's possible to avoid bumping MSRV if use like this is conditional. Assuming it's fine to have diferent MSRV for different features (I think it's reasonable).

Kixunil · 2020-12-15T14:24:37Z

src/util/merkleblock.rs

@@ -52,8 +52,12 @@
 //! assert_eq!(1, index[0]);
 //! ```

+#[cfg(feature="std")]
 use std::collections::HashSet;


This is removed here but not anywhere below, how it's possible?

TheBlueMatt · 2021-01-13T16:33:53Z

It looks like this needs a rather significant rebase?

justinmoon · 2021-01-14T04:47:20Z

@TheBlueMatt Yea I'm going to tackle this over next week.

sgeisler · 2021-01-14T15:23:09Z

Before rebasing this it probably would be best to look which dependencies need PRs merged and updates published first, otherwise I see you doing the same again in a month or two once all the dependencies are updated to have a no_std option.

TheBlueMatt · 2021-03-13T15:25:59Z

Replace HashMap and HashSet with BTreeMap and BTreeSet just to get something working. This needs to be fixed.

I think it makes sense to take a hashbrown dependency for this, as @GeneFerneau did at lightningdevkit/rust-lightning#842

h4x3rotab · 2021-04-07T02:52:42Z

Any updates or help wanted? This would be very helpful for us.

RCasatta · 2021-04-07T07:46:31Z

src/lib.rs

@@ -23,6 +23,8 @@
 //! software.
 //!

+#![cfg_attr(not(feature = "std"), no_std)]


as suggested also by @Kixunil there is no need to prevent use of std in tests

#![cfg_attr(all(not(feature = "std"), not(test)), no_std)]

dr-orlovsky · 2021-08-09T21:15:03Z

After #603 got merged I assume this one can be closed

justinmoon commented Dec 1, 2020

View reviewed changes

justinmoon mentioned this pull request Dec 1, 2020

no_std support bitcoindevkit/bdk#205

Closed

3 tasks

Update dependencies to actually get rust-bitcoin working on real MCU

cab8669

justinmoon force-pushed the no_std branch from 64d0ca3 to cab8669 Compare December 4, 2020 18:13

justinmoon added 2 commits December 11, 2020 19:47

Compiles for stable 1.36.0

1e1f29a

Upgraded bare-io to a branch which adds a feature to disable the `#[non_exhaustive]` attribute, which wasn't stabilized until 1.40.0.

Remove redundant 'features = []' from some dependencies

be0d332

Kixunil reviewed Dec 15, 2020

View reviewed changes

dr-orlovsky mentioned this pull request Dec 15, 2020

Rust 2018 & MSRV bump 1.41.1 tracking issue #510

Closed

20 tasks

apoelstra mentioned this pull request Jan 18, 2021

no_std support rust-bitcoin/rust-miniscript#201

Closed

TheBlueMatt mentioned this pull request Mar 13, 2021

Initial support for no_std platforms lightningdevkit/rust-lightning#842

Closed

RCasatta reviewed Apr 7, 2021

View reviewed changes

devrandom mentioned this pull request May 5, 2021

no_std support, keeping MSRV #603

Merged

2 tasks

apoelstra closed this Sep 24, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

no_std support #528

no_std support #528

justinmoon commented Dec 1, 2020 •

edited

justinmoon Dec 1, 2020

justinmoon Dec 1, 2020

stevenroose Dec 2, 2020

justinmoon Dec 1, 2020

stevenroose Dec 2, 2020

stevenroose Dec 2, 2020

justinmoon Dec 4, 2020 •

edited

justinmoon commented Dec 4, 2020 •

edited

justinmoon commented Dec 12, 2020

Kixunil left a comment

Kixunil Dec 15, 2020

Kixunil Dec 15, 2020

Kixunil Dec 15, 2020

TheBlueMatt commented Jan 13, 2021

justinmoon commented Jan 14, 2021 •

edited

sgeisler commented Jan 14, 2021

TheBlueMatt commented Mar 13, 2021

h4x3rotab commented Apr 7, 2021

RCasatta Apr 7, 2021 •

edited

dr-orlovsky commented Aug 9, 2021

no_std support #528

no_std support #528

Conversation

justinmoon commented Dec 1, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

justinmoon Dec 4, 2020 • edited

Choose a reason for hiding this comment

justinmoon commented Dec 4, 2020 • edited

justinmoon commented Dec 12, 2020

Kixunil left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TheBlueMatt commented Jan 13, 2021

justinmoon commented Jan 14, 2021 • edited

sgeisler commented Jan 14, 2021

TheBlueMatt commented Mar 13, 2021

h4x3rotab commented Apr 7, 2021

RCasatta Apr 7, 2021 • edited

Choose a reason for hiding this comment

dr-orlovsky commented Aug 9, 2021

justinmoon commented Dec 1, 2020 •

edited

justinmoon Dec 4, 2020 •

edited

justinmoon commented Dec 4, 2020 •

edited

justinmoon commented Jan 14, 2021 •

edited

RCasatta Apr 7, 2021 •

edited