`no_std` support for the `url` crate #831

domenukk · 2023-04-08T15:32:34Z

This extends/rebases/fixes #717.
All checks seem to pass.
For no_std support, however, as mentioned in #717, this still needs a bunch of fixes. Specifically, It looks like we'll have to use the ip_in_core nightly feature, or use this crate: https://docs.rs/no-std-net. I'd personally opt for nightly, as it looks like it will be merged eventually, but I could do both.

Add default feature flag "std" that enables a `std::error::Error` impl for `Errors`.

domenukk · 2023-04-08T16:09:12Z

Went for ip_in_core now, so it needs nightly, but seems to work.

domenukk · 2023-04-08T16:26:44Z

url/src/lib.rs

    ///
-    /// # fn run() -> Result<(), Box<dyn Error>> {
+    /// # use url::ParseError;


This is not super pretty. If URL had a no_std error type that also supports strings, it could be nicer. However, it's not visible in docs so should be fine

domenukk · 2023-04-10T13:54:26Z

I spent some time trying to get serde no_std support working, but it doesn't play nicely with the ip_in_core nightly feature - it throws errors like

the trait `Serialize` is not implemented for `Ipv6Addr`

valenting · 2023-04-10T13:59:54Z

Went for ip_in_core now, so it needs nightly, but seems to work.

If it breaks stable, that's obviously a no-go.
Also, our msrv is 1.56.0, so preferably the no_std changes would also work on that version.

codecov · 2023-04-10T13:59:55Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (92f356e) 81.70% compared to head (316c868) 81.70%.

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #831   +/-   ##
=======================================
  Coverage   81.70%   81.70%           
=======================================
  Files          20       20           
  Lines        3547     3547           
=======================================
  Hits         2898     2898           
  Misses        649      649

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

domenukk · 2023-04-10T14:15:12Z

It should not break stable, it should only "break" no_std (which never worked in the first place).
Switching to no-std-net also doesn't fix the problem with serde - the project seems stale/would need this to be merged:
dunmatt/no-std-net#16

domenukk · 2023-04-10T14:32:05Z

Moving to no-std-net anyway, since it seems to fit your msrv better

domenukk · 2023-04-16T23:43:04Z

Since I do need the nightly version, but I can see why some people wouldn't, I added two features:

no_std_net uses the no_std_net crate, no nightly needed (but the extra dependency)
unstable enabled unstable net_in_core feature, and also the error_in_core feature. Due to its opt-in nature, I would assume this is an okay solution (?)

Else we can wait until the features get stabilised.
Feedback welcome.

domenukk · 2023-04-16T23:45:21Z

idna/src/uts46.rs

@@ -453,7 +453,7 @@ impl Idna {
            return Errors::default();
        }
        let mut errors = processing(domain, self.config, &mut self.normalized, out);
-        self.output = std::mem::replace(out, String::with_capacity(out.len()));
+        self.output = core::mem::replace(out, String::with_capacity(out.len()));


FYI, this fixes a bug in the current idna no_std support. Why is it not caught in CI, and should I open an extra PR for it?

I recently dabbled with no_std stuff as well and I think the sure-fire way to test no_std compatibility is to:

target a platform that does not have std (certain ARM platforms) OR

have a really simple no_std binary crate which uses the libraries

Yes I'm trying it with cargo +nightly build -Zbuild-std=core,alloc --target aarch64-unknown-none -v --release --no-default-features

Could you split this off into a separate PR?

…ml on older rust versions)

domenukk · 2023-05-31T13:01:33Z

Any news on this? :)

valenting · 2023-06-14T07:17:48Z

idna/src/uts46.rs

@@ -453,7 +453,7 @@ impl Idna {
            return Errors::default();
        }
        let mut errors = processing(domain, self.config, &mut self.normalized, out);
-        self.output = std::mem::replace(out, String::with_capacity(out.len()));
+        self.output = core::mem::replace(out, String::with_capacity(out.len()));


Could you split this off into a separate PR?

url/Cargo.toml

domenukk · 2023-06-14T17:00:30Z

Opened #843 to fix the idna no_std bug.

bors-servo · 2023-07-12T11:00:00Z

☔ The latest upstream changes (presumably #853) made this pull request unmergeable. Please resolve the merge conflicts.

bors-servo · 2023-07-14T21:12:00Z

☔ The latest upstream changes (presumably beb2cde) made this pull request unmergeable. Please resolve the merge conflicts.

domenukk · 2023-07-15T12:38:31Z

.github/workflows/main.yml

+        if: |
+          matrix.os == 'ubuntu-latest' &&
+          matrix.rust == 'nightly'
+        run: rustup target add aarch64-unknown-none && rustup component add rust-src --toolchain nightly-x86_64-unknown-linux-gnu && cargo +nightly build -Zbuild-std=core,alloc --target aarch64-unknown-none -v --release --no-default-features --features=alloc,unstable


@lucacasonato you could take over this check to the idna create

domenukk · 2023-07-16T19:44:39Z

Apparently building --no-default-features workspace-wide didn't work, moved that check to the url create only -> CI might turn green

domenukk · 2023-07-19T13:21:37Z

It's green 🎉

domenukk · 2023-11-23T00:12:41Z

Any news on this? @valenting

domenukk · 2023-11-23T13:29:52Z

Any hint on what to do with codecov CI?

madsmtm

I think the guiding principle for this feature should be: "Don't inconvenience people who don't care about no_std", which is... All of url's current users ;)

So, I really don't think we should bump MSRV for this feature - but a way to solve that would be to use rustc_version in a build script to detect when the feature is available. (This does come with a slight compile-time cost, but hopefully that's acceptable, and it will go away once the functionality is in url's MSRV).

madsmtm · 2023-12-22T06:22:11Z

url/Cargo.toml


 [features]
-default = []
+default = ["std"]


I fear this might be a fairly big breaking change?

When I added no_std support for the other libraries in this crate, it was more trivial, since there was only three crates that disabled default features - but that is not the case for url; there are many crates specifying default-features = false, so even though there was no default features previously, this change will affect all of those crates.

Python script for finding crates using `default-features = false`.

import json import requests import sys API_URL = "https://crates.io/api/v1/crates" crate_name = sys.argv[1] page = 1 while True: # print(f"page {page}") data = json.loads(requests.get(f"{API_URL}/{crate_name}/reverse_dependencies?per_page=100&page={page}").content) page += 1 if len(data['versions']) == 0: break versions = {x['id']: x for x in data['versions']} for dep in data['dependencies']: if not dep['default_features']: crate = versions[dep['version_id']] print(f"{crate['crate']} had `default-features = false`")

I don't see any other way?

Well... I see two:

Bundling this in with other breaking changes, and releasing url v0.3

Calling the feature unstable-no-std instead, to signify that it may break other crates in the dependency chain, and that it is a bad way of doing it which will be rectified once v0.3 is released.

madsmtm · 2023-12-22T06:27:23Z

url/Cargo.toml

+no_std_net = ["no-std-net"]
+# UNSTABLE FEATURES (requires Rust nightly)
+# For no_std: Use errors_in_core and net_in_core
+unstable = ["idna/unstable"]


rust-lang/rust#108443 is FCP-ing now, so it'll probably land in a few months - perhaps we should wait with this feature until then?

Right now this PR also uses error_in_core (rust-lang/rust#103765) but that part can be removed if we want to go for stable instead

The value in that feature is that this could become a non-breaking change (other than bumping MSRV, which is probably acceptable).

Update: core::net has landed in Rust 1.77.0

url/Cargo.toml

madsmtm and others added 8 commits February 17, 2022 13:58

Make form_urlencoded no_std compatible

35653b0

Make data-url no_std compatible

194122c

Make idna no_std compatible

422a260

Add default feature flag "std" that enables a `std::error::Error` impl for `Errors`.

Test no_std support

81ee074

Initial work at making url no_std compatible

67d2a4c

Merge branch 'master' into no_std

db92dd2

Fixed testcases for no_std

bb7955a

fix no_std for nightly

a98aef8

domenukk marked this pull request as ready for review April 8, 2023 16:09

Add no_std to CI

0d14d5a

domenukk commented Apr 8, 2023

View reviewed changes

Fix no_std for deps

315eb43

Moved to no-std-net

6d622ab

domenukk force-pushed the no_std branch from a2093b6 to 6d622ab Compare April 10, 2023 14:41

Added unstable, no_std_net options

3efe254

domenukk commented Apr 16, 2023

View reviewed changes

OverOrion mentioned this pull request Apr 18, 2023

Add support for no_std integritee-network/rust-url#1

Merged

domenukk added 4 commits April 27, 2023 15:28

Make CI happy by removing no-std-net/serde (not supported in Cargo.to…

36bc5b3

…ml on older rust versions)

Merge branch 'master' into no_std

29bfcf7

Only test aarch64 on ubuntu

e16eb2b

Fix CI some more

3dd87d8

OverOrion mentioned this pull request May 12, 2023

No std upstream working rust-nostr/nostr#90

Closed

3 tasks

valenting requested changes Jun 14, 2023

View reviewed changes

lucacasonato mentioned this pull request Jul 11, 2023

Roadmap #850

Open

4 tasks

domenukk and others added 3 commits July 12, 2023 15:21

Merge branch 'master' into no_std

fd98554

Merge branch 'master' into no_std

b0bb942

Merge branch 'master' into no_std

047d334

domenukk mentioned this pull request Jul 15, 2023

Fix no_std Support for idna #843

Merged

domenukk commented Jul 15, 2023

View reviewed changes

Merge branch 'master' into no_std

028424a

domenukk force-pushed the no_std branch from 7b9f57a to 028424a Compare July 16, 2023 19:24

Only checking crate for no_std

7073957

domenukk added 3 commits July 19, 2023 13:57

only build no_std aarch64 none on nightly

be123af

Only build for aarch64 on nightly

c145584

Merge branch 'master' into no_std

be73851

Merge branch 'master' into no_std

96e2a04

domenukk requested a review from valenting November 23, 2023 00:12

domenukk added 2 commits November 23, 2023 01:15

Fix test

268c17e

fmt

0c02cd9

Merge branch 'master' into no_std

316c868

domenukk mentioned this pull request Dec 5, 2023

no-std support for hickory-proto hickory-dns/hickory-dns#2104

Draft

madsmtm mentioned this pull request Dec 22, 2023

no_std support for all crates #717

Closed

madsmtm reviewed Dec 22, 2023

View reviewed changes

yukibtc mentioned this pull request Jan 21, 2024

Remove url-fork dep rust-nostr/nostr#258

Open

Kailai-Wang mentioned this pull request Mar 22, 2024

Prefer no-std url crate litentry/litentry-parachain#2611

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`no_std` support for the `url` crate #831

`no_std` support for the `url` crate #831

domenukk commented Apr 8, 2023

domenukk commented Apr 8, 2023

domenukk Apr 8, 2023

domenukk commented Apr 10, 2023

valenting commented Apr 10, 2023

codecov bot commented Apr 10, 2023 •

edited

domenukk commented Apr 10, 2023

domenukk commented Apr 10, 2023

domenukk commented Apr 16, 2023

domenukk Apr 16, 2023

OverOrion Apr 18, 2023

domenukk Apr 27, 2023

valenting Jun 14, 2023

domenukk commented May 31, 2023

valenting Jun 14, 2023

domenukk commented Jun 14, 2023 •

edited

bors-servo commented Jul 12, 2023

bors-servo commented Jul 14, 2023

domenukk Jul 15, 2023

domenukk commented Jul 16, 2023

domenukk commented Jul 19, 2023

domenukk commented Nov 23, 2023

domenukk commented Nov 23, 2023

madsmtm left a comment

madsmtm Dec 22, 2023

domenukk Dec 22, 2023

madsmtm Dec 22, 2023

madsmtm Dec 22, 2023

domenukk Dec 22, 2023

madsmtm Dec 22, 2023

madsmtm Mar 21, 2024

no_std support for the url crate #831

Are you sure you want to change the base?

no_std support for the url crate #831

Conversation

domenukk commented Apr 8, 2023

domenukk commented Apr 8, 2023

Choose a reason for hiding this comment

domenukk commented Apr 10, 2023

valenting commented Apr 10, 2023

codecov bot commented Apr 10, 2023 • edited

Codecov Report

domenukk commented Apr 10, 2023

domenukk commented Apr 10, 2023

domenukk commented Apr 16, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

domenukk commented May 31, 2023

Choose a reason for hiding this comment

domenukk commented Jun 14, 2023 • edited

bors-servo commented Jul 12, 2023

bors-servo commented Jul 14, 2023

Choose a reason for hiding this comment

domenukk commented Jul 16, 2023

domenukk commented Jul 19, 2023

domenukk commented Nov 23, 2023

domenukk commented Nov 23, 2023

madsmtm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

`no_std` support for the `url` crate #831

`no_std` support for the `url` crate #831

codecov bot commented Apr 10, 2023 •

edited

domenukk commented Jun 14, 2023 •

edited