Add alloc feature #77

philipc · 2018-02-01T06:44:47Z

This allows building the endian_fd feature without std, although it does still require the alloc crate and a nightly rustc.

Fixes #75

This allows building the `endian_fd` feature without `std`, although it does still require the `alloc` crate and a nightly rustc.

m4b

So overall this looks good, and a lot of work, thank you!

So, I don’t think it’s true, but just to be clear, this does not now mean goblin will only compile on nightly, right ?

Can use on stable unless you explicitly request the alloc feature ? Then required to use nightly, yes ?

If this forces nightly unconditionally I can’t merge these changes, but I don’t think it does.

However the feature flag switching compiler requirements does worries me a little since features are supposed to be additive otherwise it can break cargo and have weird effects at a distance, or so I’m told.

Could you clarify a bit ?

Also if anyone else is wants to chime in, now is a good time

m4b · 2018-02-01T07:10:16Z

src/elf/section_header.rs

    use error;
    use core::fmt;
    use core::result;
    use core::ops::Range;
    use scroll::ctx;
    use container::{Container, Ctx};

+    #[cfg(feature = "endian_fd")]


Should this be endian fd or alloc?

endian_fd, because it's only used by SectionHeader::parse.

m4b · 2018-02-01T07:10:59Z

src/elf/sym.rs

-            use std::io::{Read, Seek};
-            use std::io::SeekFrom::Start;
+            if_std! {
+                use error::Result;


Why is error in std but not alloc?

Because it's only used by from_fd, which is only enabled for std.

m4b · 2018-02-01T07:15:34Z

src/archive/mod.rs

@@ -334,9 +335,9 @@ pub struct Archive<'a> {
    sysv_name_index: NameIndex<'a>,
    // the array of members, which are indexed by the members hash and symbol index
    member_array: Vec<Member<'a>>,
-    members: HashMap<&'a str, usize>,
+    members: BTreeMap<&'a str, usize>,


Why did this change? Because of random requirements?

Because HashMap depends on std, but BTreeMap is available in alloc. An alternative is to try to split up the archive support so that the HashMap stuff is only enabled for std.

Meh it’s fine. I don’t think O(n) lookup important for members, since there’s usually only like 10

m4b · 2018-02-01T07:16:57Z

src/lib.rs

 extern crate scroll;

-#[cfg_attr(feature = "std", macro_use)]
-#[cfg(feature = "std")] extern crate log;
+#[cfg(feature = "log")]


Log isn’t a feature, will this work?

Yes, cargo adds features for optional dependencies.

Well that’s awesome, did not know this!

m4b · 2018-02-01T07:19:14Z

src/lib.rs


-pub mod strtab;
+#[cfg(feature = "std")]
+mod alloc {


Why is this a mod?

Nevremind, it’s getting reexportrd and used elsewhere

m4b · 2018-02-01T07:25:59Z

src/mach/fat.rs

@@ -58,8 +60,7 @@ impl FatHeader {
 }

 #[repr(C)]
-#[derive(Clone, Copy, Default)]
-#[cfg_attr(feature = "std", derive(Pread, Pwrite, SizeWith))]
+#[derive(Clone, Copy, Default, Pread, Pwrite, SizeWith)]


If my understanding is correct these are unconditionally derived because in order for the module to be compiled it must have mach and alloc feature, yes? In which case we are safe to derive them.

If so I almost prefer the explicit cfg feature since although it’s redundant it gives reader a clue what’s going on. Or in case of refactors it’s a second fail safe of sorts.

Yes, mach depends on alloc, and alloc depends on scroll/derive. I kind of agree with you, except that there's a ton of these in load_command.rs and I was lazy. Additionally, for the ones in load_command.rs we would also need cfg_attr(feature = "std", derive(IOread, IOwrite)) (I don't actually understand why they don't give errors currently...).

Yea be lazy, I forgot there’s like 40 of them ;)

m4b · 2018-02-01T07:29:10Z

src/lib.rs


-#[cfg(all(feature = "endian_fd", feature = "elf64", feature = "elf32", feature = "pe64", feature = "pe32", feature = "mach64", feature = "mach32", feature = "archive"))]


Why is it safe to remove the intersection of all these features here and in the impl ?

Couldn’t the user compile with elf, mach, and std and the get a missing symbol error in the impl parser since PE wasn’t compiled.

Or does endian_fd mean everything is enabled now ?

In #75 (comment), you said that the parser flag (aka endian_fd) should turn on every other flag. So that is now enforced in Cargo.toml. Since this is all wrapped in if_endian_fd, specifying the features here is redundant. As I said in that issue, I'm not sure if this is correct, but I actually kind of like not needing to list out all the features here, so I went with that, but I'm happy to change it.

philipc · 2018-02-01T08:25:49Z

Can use on stable unless you explicitly request the alloc feature ? Then required to use nightly, yes ?

Yes, you can use on stable as long as either you request std, or you don't request alloc (or something that depends on alloc). One change due to this PR though is that previously mach/pe/arcive required std, but now they only require alloc. So if you previously depended on only mach/pe/archive, then you'll need to add std to your features, so that's a breaking change, but you can still get it working on stable.

philipc · 2018-02-01T08:31:00Z

However the feature flag switching compiler requirements does worries me a little since features are supposed to be additive otherwise it can break cargo and have weird effects at a distance, or so I’m told.

The features are still additive, so I don't think it will break cargo. I think it's fine to have a feature that depends on nightly, plenty of crates do that, and this case isn't even as bad as some other uses of nightly because you can always request std in order to avoid the nightly requirement.

m4b · 2018-02-01T18:19:03Z

So I should have asked this before but all things were done in haste, but what is the general motivating usecase for having the alloc flag?

So originally I thought that object wanted a no_std parser because it was being considered/suggested for use in the rustc compiler in its backtrace mechanism. rust-lang/rust#44251

The first thing is I don’t understand precisely why the rustc compiler would require no std there, and furthermore why we couldn’t allocate.

Perhaps that explains why the alloc flag is just for allocation parts?

But I still don’t understand who the alloc flag is for, specifically its nightly requirement, etc.

So essentially I’m a little worried if the changes here are a solution in need of a problem.

philipc · 2018-02-02T01:39:56Z

The first thing is I don’t understand precisely why the rustc compiler would require no std there

I'm not clear on that either, since rust-lang/rust#44251 is modifying libstd and using std::io. @whitequark can you elaborate?

and furthermore why we couldn’t allocate.

But we can allocate. If we couldn't allocate then there would be no point adding an alloc feature.

Perhaps that explains why the alloc flag is just for allocation parts?

But I still don’t understand who the alloc flag is for, specifically its nightly requirement, etc.

The purpose of the alloc flag is for the parts that require allocation, but don't require libstd.

The nightly requirement is because the alloc crate is not stable.

m4b · 2018-02-02T03:15:43Z

I’ll address couple comments in a bit but also why are the tests failing ?

philipc · 2018-02-02T03:27:18Z

Tests are failing on rust 1.18 because of cyclic features, which is only supported in relatively recent rust. The fix for that depends on what we decide about the endian_fd feature changes that you commented on.

whitequark · 2018-02-02T04:06:04Z

@m4b For example, I would like to use goblin in the ARTIQ firmware so that it would symbolize backtraces on its own instead of requiring the user to manually run addr2line.

m4b · 2018-02-02T04:30:07Z

@philipc

But we can allocate. If we couldn't allocate then there would be no point adding an alloc feature.

I just meant in the context of the compiler, it seemed like no_std was required, unless I missed something, and I was trying to figure out why that would be required.

Is it correct to say that in order to print backtraces in rustc, no_std is required? This still seems really strange to me; rustc afaics uses std as expected.

And i'm sure the C libbacktrace lib allocates, and is a regular C/C++ library without any funny business.

The purpose of the alloc flag is for the parts that require allocation, but don't require libstd.

So my point in asking about who wants it was to try to get a feel for whether it's actually a real pressing usecase.

Specifically, I understand at this point how it works, but I'm trying to understand some usecases for it.

Which brings me to @whitequark :

For example, I would like to use goblin in the ARTIQ firmware so that it would symbolize backtraces on its own instead of requiring the user to manually run addr2line.

Ok, so this is start for a good motivating example. If I'm understanding right, on or1k firmware you're in a no_std environment + you use the alloc crate, is this correct? To be 100% clear: you still can allocate, which you provide using the alloc crate nightly stuff, but in terms of rust, you need the parser crate to be no_std, yes?

I'm still curious also how goblin/object would help you in terms of "not requiring the user to manually run addr2line" - are you saying that object would be embedded in the firmware itself, and when an exception is raised of some kind, it prints the backtrace automatically before termination? (if so, that's awesome)

Bringing it back to the original rustc pr for backtrace printing, @whitequark, are the changes here, which allow a no_std mode with the alloc crate enough to have the object crate be substituted for backtrace generation, or is there another piece I'm missing?

Sorry for all the questions, just trying to understand the precise problem that's being solved by adding no_std + nightly alloc crate; I was worried that these changes were for a very, very niche usecase that likely wouldn't even be seen, maybe that's still the case, but flexibility is also good.

m4b · 2018-02-02T04:31:32Z

Tests are failing on rust 1.18 because of cyclic features, which is only supported in relatively recent rust. The fix for that depends on what we decide about the endian_fd feature changes that you commented on.

@philipc

I'm not familiar with cyclic features; do you know offhand which rust? I would really like to avoid a rustc version bump if possible, and am ok with a little momentary pain/redundancy if it means not bumping, but it depends on what exactly needs to change in order for that to happen.

philipc · 2018-02-02T04:48:28Z

The cyclic features PR is rust-lang/cargo#4473, I don't which rust that is in. But my intention is to fix the cycle, so that shouldn't require a version bump. However, I don't want to spend effort on that until we decide which way the endian_fd dependency should be. Either way can work, what's in this PR is just my best effort based on my understanding of your comments in #75, but maybe I misunderstood.

philipc · 2018-02-02T04:50:21Z

And i'm sure the C libbacktrace lib allocates, and is a regular C/C++ library without any funny business.

It uses anonymous mmap to allocate memory instead of malloc/free.

m4b · 2018-02-02T04:51:45Z

It uses anonymous mmap to allocate memory instead of malloc/free.

So it has a posix libc... with an operating system that allocates memory for you? So not no_std at all

m4b · 2018-02-02T05:01:34Z

However, I don't want to spend effort on that until we decide which way the endian_fd dependency should be

So I'm not sure what we're deciding here, but it sounds like its just whether endian_fd means the union of every feature, i.e., endian_fd -> mach, elf, archive, std more or less?

I don't think its unreasonable to want just an Archive or just ELF or just Mach-o parser (and the crate was built this way), so I think its better to leave this.

If you want the Object parser, then you need all of the features enabled (the default); like it is now.

It sounds like the changes as they are now would force you to have the Object parser (and hence the archive, Elf, and PE parsers), even if you just wanted the Mach parser, yes?

I think keeping the way it is now, with composable parsers, and a cfg(all flag for everything in order to get the goblin::Object the better, and if i'm understanding, the diff will actually be smaller?

whitequark · 2018-02-02T05:06:23Z

To be 100% clear: you still can allocate, which you provide using the alloc crate nightly stuff, but in terms of rust, you need the parser crate to be no_std, yes?

That is correct. Of course, goblin would be better off as a piece of software in general if it didn't allocate, and goblin not allocating would let me print backtraces in case of out-of-memory errors (we have a 4MB RAM region currently dedicated to that CPU core), but goblin+alloc would still be a huge improvement.

I'm still curious also how goblin/object would help you in terms of "not requiring the user to manually run addr2line" - are you saying that object would be embedded in the firmware itself, and when an exception is raised of some kind, it prints the backtrace automatically before termination? (if so, that's awesome)

Correct. I already have an ELF loader for other reasons, so I would simply flash a statically linked ELF with debug symbols.

philipc · 2018-02-02T05:49:40Z

I think keeping the way it is now, with composable parsers, and a cfg(all flag for everything in order to get the goblin::Object the better, and if i'm understanding, the diff will actually be smaller?

Agreed. I've reverted the endian_fd dependency changes, and tests pass now.

m4b · 2018-02-02T22:12:44Z

Ok this is looking good to me. I will test this later, will likely merge over weekend.

If anyone has any comments speak now or forever hold your peace. This is pretty significant change.

Thanks for your hard work and guidance on this @philipc!

/cc @willglynn if you have any comments (and again anyone else)

m4b · 2018-02-07T05:10:50Z

This is looking, good, doesn't appear to be a breaking change at all!

Just tried this with dryad, which somehow miraculously had no other breaking changes, and works fine (besides segfaulting as expected)

So, one question I have is: what happens when user specifies alloc + std in feature flags, std is used instead right?

philipc · 2018-02-07T05:14:26Z

Yes, std already depends on alloc in Cargo.toml.

m4b · 2018-02-07T05:21:16Z

Alright let's merge this fella

m4b · 2018-02-07T05:22:08Z

Minor nit, I'll fix it later, but shouldn't:

mach32 = ["alloc", "endian_fd"]

just be:

mach32 = ["endian_fd"]

since endian_fd depends on alloc?

philipc · 2018-02-07T05:34:16Z

Doesn't matter to me, but I think it documents the dependency better: there's non-endian_fd code in mach that depends on alloc. If we later changed mach so that it didn't depend on endian_fd, there would still be code in mach that depended on alloc.

m4b · 2018-02-07T05:36:54Z

ya, agreed actually.

Add alloc feature

916bc1a

This allows building the `endian_fd` feature without `std`, although it does still require the `alloc` crate and a nightly rustc.

m4b reviewed Feb 1, 2018

View reviewed changes

Revert endian_fd dependency changes

b864469

m4b merged commit 28cadaa into m4b:master Feb 7, 2018

philipc deleted the alloc branch February 7, 2018 05:34


		#[cfg(all(feature = "endian_fd", feature = "elf64", feature = "elf32", feature = "pe64", feature = "pe32", feature = "mach64", feature = "mach32", feature = "archive"))]

Add alloc feature #77

Add alloc feature #77

Conversation

philipc commented Feb 1, 2018

m4b left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

philipc commented Feb 1, 2018

philipc commented Feb 1, 2018

m4b commented Feb 1, 2018

philipc commented Feb 2, 2018 • edited

m4b commented Feb 2, 2018

philipc commented Feb 2, 2018

whitequark commented Feb 2, 2018

m4b commented Feb 2, 2018

m4b commented Feb 2, 2018 • edited

philipc commented Feb 2, 2018

philipc commented Feb 2, 2018

m4b commented Feb 2, 2018

m4b commented Feb 2, 2018

whitequark commented Feb 2, 2018

philipc commented Feb 2, 2018

m4b commented Feb 2, 2018

m4b commented Feb 7, 2018

philipc commented Feb 7, 2018

m4b commented Feb 7, 2018

m4b commented Feb 7, 2018

philipc commented Feb 7, 2018

m4b commented Feb 7, 2018

philipc commented Feb 2, 2018 •

edited

m4b commented Feb 2, 2018 •

edited