Feature Request: Rust/Cargo Support for Building Native Extensions

[ianks]

Over the past few years, Rust has proven to be an incredibly stable and accessible option for those wishing to write native extensions for Ruby. Rust is particularly well-suited for Ruby for a few reasons:

1. Integration is straight-forward, since it is easy to expose a C compatible API using extern C.
2. Rust is a bit more high-level, and approachable for Ruby developers since it prevents many of the classic C footguns (use-after-free, etc)
3. The landing page can explain the benefits better than me. Check out https://www.rust-lang.org/ if you aren't fully convinced.

One of the biggest pain points of creating a Gem with a Rust extension is integrating with Gem::Ext::Builder. There are currently a couple of solutions, and the rely on the RakeBulider interface. I have personally used https://github.com/malept/thermite, which is a very good solution, but still requires and external dependency and does not integrate as well as native builders at the moment.

Currently, Gem::Ext::Builder supports four builder types: ExtConfBuilder, ConfigureBuilder, RakeBuilder, and CmakeBuilder. I propose we add a new builder, CargoBuilder, which will detect a Cargo.toml file and build the gem native extensions using cargo. This ease the burden of developing and publishing Rust extensions for Ruby users.

Potential Problems

1. If the user does not have cargo installed, building the extension will not work.
   This is an issue, but is also an issue for the CmakeBuilder. One way to mitigate this issue would be to make it easier to build static binaries for Rust extensions.

2. The lack of a standard ruby interface for Rust extensions might cause pain for developers.
   Currently, many folks use https://github.com/steveklabnik/ruby-sys to interface with the C ruby API. It might make sense to make this "officially" supported at some point, but I don't think it has to be done immediately.

Please let me know your thoughts. I would be very excited to see Ruby take this step. I don't think I know enough about the Ruby extension building process or the Rust compilation process to actually implement this, so hopefully we could find a volunteer to step up. Thanks for reading :)

Notable Gems using Rust

1. https://github.com/danielpclark/faster_path
2. https://github.com/ianks/fast_woothee -- shameless plug
3. https://github.com/tildeio/helix -- a framework for building Rust extensions
4. https://github.com/danielpclark/rutie -- ruby/rust glue
5. https://github.com/steveklabnik/ruby-sys -- low level bindings for ruby
6. https://github.com/malept/rusty_blank -- String#blank? with rust

[djberg96]

Seems like a nice idea.

Just curious, is this something that could be written as a plugin? Or is building too tightly integrated into the core?

If it can be made as a plugin, maybe it would be best to refactor all of the rubygems builders as plugins so they could be installed as needed, and more easily maintained, while reducing the burden on core rubygems itself.

[ianks]

@djberg96 That could potentially work. The problem I see is that if the plug-in is not bundled by default, it could be burdensome for the end-user (installer-of-gem) if they had to remember to do something like gem install rugygems-cargo-builder.

We could perhaps have a git-submodule which is hosted elsewhere if this proves to be too burdensome on rubygems maintainers. I'd be interested to hear input from someone from the Cargo team on this. Maybe @alexcrichton could chime in 😀

[alexcrichton]

I suspect that something like this would be best to start out in the community, and if it becomes large enough and is agreed it should become official, this seems reasonable to host in the rust-lang organization and have a team of maintainers there (not necessarily the same set as cargo maintainers!). I suspect though we'd want support for something like this to bake in the ecosystem a bit before adding to rust-lang though.

[djberg96]

@ianks I dunno, doesn't seem that burdensome to me, plus it would allow you full control over the code without having to get permission from the core rubygems team for every PR. Then once it's stable, we could consider merging into core.

[hsbt]

@ianks Hi, I welcome to cargo builder to rubygems. But I'm not familiar with rust-lang. Can you provide the patch for this? Thanks.

[ianks]

Yes @hsbt I will look into this ASAP

[johncallahan]

I think this is a great idea. I just had all sorts of issues publishing a Helix-powered gem to rubygems due to gemspec templating problems. The rindy gem is a Ruby wrapper for the Hyperledger Indy SDK. Testing the gem locally, then to github-based deployment (in Gemfile), then to rubygems.org was a headache.

[ianks]

I reached out to @malept to see if they are interested in porting parts of https://github.com/malept/thermite directly into rubygems. I think this would be the best way to tackle the v1. Let's give it a week or so to see if they respond.

[ianks]

It's been a couple of weeks now so I'm going to assume that @malept may be too busy to do this right now.

Does anyone want to tackle this issue? As mentioned earlier, https://github.com/malept/thermite is a great piece of prior art to reference, and we have support from some Rust developers to help review and make sure things are tidy.

If you are interested, or know someone who may be, drop a comment here 😁

[timmyjose]

Not directly apropos, but I have experience in Rust, and I've been using Ruby/Rails for a few months on the job now. I would like to participate and help out in this effort, whichever form it might take.

[pkondzior]

Maybe it would be easier to addd support of Rust into https://github.com/rake-compiler/rake-compiler, it seems that this gem already allows to easily build either C or Java extensions through coc approach.

[ianks]

@timmyjose Thank you for volunteering! 🙏🏻If you run into any road blocks, please post here and we’ll do best to make sure you get unblocked.

[tarcieri]

Long time Ruby user with several years of Rust experience (and an MRI commit) here, would love to help make this happen.

[ianks]

@tarcieri thank for volunteering!

Since multiple people have volunteered, and I would hate to see this work duplicated, can someone commit to PRing the initial integration for the extension builder?

After that, I think there are a few things we can do to make this experience more seamless for gem builders.

1. Make it easy to scaffold a Rusty gem with bundler, i.e. bundle gem my_gem --rust
2. Support in rake-compiler to easily compile binaries for multiple platforms
3. Help to iron out any open issues in rutie gem to make the experience for newbies as seamless as possible.

As you can see, there's plenty of exciting work here to go around. 😀

[tarcieri]

I'm happy to take a backseat to anyone wanting to jump in right away. I can help review PRs, though. If there's one of the above items that no one wants to take on, I'd be happy to work on one of those.

[qubitrenegade]

I'd love to take a crack at 1. Make it easy to scaffold a Rusty gem with bundler, i.e. bundle gem my_gem --rust.

Any suggestions where to get started? Which of the existing Rusty gem should I use as a sort of "template"? Any of them a "better" candidate than the others (and by "Better" I mean: more generic, more templatateable, etc.)?

[danielpclark]

If you'd like input on any WIP code you have for this I'm here. I'm the author of FasterPath and maintainer of Rutie.

If you look at older FasterPath versions in the git history you can find builds pre Thermite. It was more of a hacky work around just to get it to work as I initially had a lot of trouble going the RakeBuilder route.

And if I may suggest a different design pattern if Thermite code is being ported over… I would much prefer the composition pattern rather than the heavily coupled mixins currently being used.

[malept]

Obviously from my lack of response, I haven't had a lot of time to devote to Rust/Ruby integration. However, I am happy to provide observations from my experiences and/or code review.

I want to be clear that the primary use case for Thermite for me is to provide a transparent way to provide native extensions written in Rust for different platforms. I do not want to expect users of my Rust-written gems to have to install Rust/Cargo in order to utilize my gem in their app. If this sort of "binary cache" feature can be added to rubygems proper (or a way to be able to extend it so that feature can be added to any builder), that would be excellent.

And if I may suggest a different design pattern if Thermite code is being ported over… I would much prefer the composition pattern rather than the heavily coupled mixins currently being used.

This (and the initial comments about it on the Thermite issue tracker) are part of the reason why Thermite hasn't seen much development as of late.

[danielpclark]

@malept I apologize for my inconsiderate bikeshedding. Design patterns are very much subjective and I was inconsiderate towards you and the good work that you have done. Please forgive me.

[ianks]

If this sort of "binary cache" feature can be added to rubygems proper (or a way to be able to extend it so that feature can be added to any builder), that would be excellent.

This is incredibly important feature IMO. Thanks for highlighting that @malept. I'm hoping rake-compiler + s.platform = Gem::Platform::RUBY can help to solve this for Rust.

[qubitrenegade]

Ok, I did a quick mockup of what this might look like. (I also put together a quick demo gem)

You can test this yourself:

git clone git@github.com:qubitrenegade/rubygems.git
git clone git@github.com:qubitrenegade/cargo_example.git
cd cargo_example
gem build cargo_example.gemspec
cd ../rubygems
ruby -Ilib bin/gem install ../cargo_example/cargo_example-0.1.0.gem --verbose

Note that this will invoke cargo while installing the gem.

If I understand correctly, Gem::Ext::Builder is only invoked during the gem installation process?

Isn't something like rake-compiler aimed at the development process? Although, it looks like there's a RakeBuilder WIP, which perhaps could leverage rake-compiler?

I'll keep poking at Thermite but welcome any feedback in the meantime.

[ianks]

Great stuff @qubitrenegade! I'll test it out locally when I get a chance.

As a general question, how should we handle linking? Should we link all the libraries that are typically linked when compiling a C extension?

[tarcieri]

The happy path for Rust is having Cargo handle linking

The shortest path to victory for embedding Rust is having Cargo produce a static library

The latter is probably the best place to start

[ianks]

From @tarcieri’s comment, I think it’s safe to say we should instruct cargo to link RbConfig::LIBS, replacing the -l with -l (note the space) to make cargo happy.

[briankung]

What's the status on this so far?

[indirect]

If you have read this ticket, you know as much as anyone else does.

[benreyn]

It looks like the work from @qubitrenegade is a good starting place, even if it requires cargo be installed on the users system. Where would be go from here? Is there a plan to upstream this work?

[briankung]

I'm not really sure what I'm doing, but I added some tests for building a minimal cargo project. I based it off of @qubitrenegade's branch and the existing test_gem_ext_cmake_builder.rb file: https://github.com/rubygems/rubygems/compare/master...briankung:cargo_builder?expand=1

There were some complications with bundler overwriting the HOME environment variable for compatibility with windows, which makes it tricky since rustup and cargo look for RUSTUP_HOME and CARGO_HOME in the user's home folder, so I had to re-set those before running the CargoBuilder. I imagine this would affect Windows compatibility.

[Hywan]

Any updates on this front? It's an exciting project with a lot of promises!