Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Discussion: fpm, ruby, ffi version conflicts #1795

Closed
jordansissel opened this issue Jun 17, 2021 · 9 comments
Closed

Discussion: fpm, ruby, ffi version conflicts #1795

jordansissel opened this issue Jun 17, 2021 · 9 comments

Comments

@jordansissel
Copy link
Owner

jordansissel commented Jun 17, 2021 •
edited

The spread of ruby versions, platforms, and dependencies is making it harder and harder to provide a good user experience when installing and using fpm.

Problem situation:

In #1709, we pinned a dependency ffi to the 1.12.x branch to enable support for Ruby 2.2 and older. However, this causes problems for Apple M1 OSX users who need ffi 1.14 or newer. Further, some platforms distributing Ruby 3.0 will only make newer versions of ruby ffi available. If we allow any version newer than 1.12, these older rubies will likely still fetch the newer-and-incompatible ffi versions.

This is largely an impossible situation where a "simple" solution needlessly punishes some population of users. Do we make it harder for Mac users? or do we make it harder for CentOS 7 users? It's a tough situation just leaving me feeling sour.

This crossroads is giving me some pause. I'm not sure exactly what solution will come.

Focusing just on ffi, I wonder if we can remove ffi as a dependency?

If I look at the Gemfile.lock for the dependency tree, I see that fpm has dependencies on ruby-xz and childprocess both of which require ffi.

  • lib/fpm/package/freebsd.rb does require "xz"
  • lib/fpm/util.rb does require "childprocess"
  • lib/fpm/util.rb uses FFI to access mknod

If we can replace these, fpm's need for ffi is eliminated.
@jordansissel
Copy link
Owner Author

Related issues: #1772, #1649, #1709, #1783, #1770, #1742, #1713, #1708, #1638

@jordansissel
Copy link
Owner Author

jordansissel commented Jun 17, 2021 •
edited

In hindsight, removing ffi seems critical now given the previous comment listing issues caused by ffi incompatibilities with Ruby versions or difficult scenarios at installation time:

  • ruby version rejected by ffi
  • local system has ruby installed, but ffi fails to install due to missing certain development headers (like, a ruby-dev package is missing)

@jordansissel jordansissel mentioned this issue Jun 17, 2021
Closed
@jordansissel
Copy link
Owner Author

I asked upstream for guidance, if any, and got a somewhat confusing response -- maybe I didn't provide enough context... ffi/ffi#906

@jordansissel
Copy link
Owner Author

Removing xz from the freebsd.rb could be done by using external program xz. We have a way to check for a program's presence using program_exists? and program_in_path?. The test suite uses this as well as the pear package support:

fpm/lib/fpm/package/pear.rb

Lines 39 to 41 in 72a925b

if !program_in_path?("pear")
raise ExecutableNotFound.new("pear")
end

In particular for freebsd.rb, replacing this section with non-ffi code:

https://github.com/jordansissel/fpm/blob/master/lib/fpm/package/freebsd.rb#L85-L95

@jordansissel
Copy link
Owner Author

Research on mknod. FPM calls mknod when copying special files like unix sockets, named pipes, characterSpecial, and blockSpecial.

Until very recently, FileUtils.copy_entry() would fail on sockets and named pipes. If we look at the patch, we can look at how each file type was addressed:

Other file special types are not addressed.

I tested unix sockets in both stream and datagram modes, and both create the same exact kind of file, so it seems like the UNIXServer.new(...).close is a good solution for "copying" unix sockets.

I don't know how old File.mkfifo is in Ruby, and that'll need some more research. At present, I am unable to compile Ruby 2.2 on my machine due to a too-new version of OpenSSL (#include'ing asn1.h throws a compiler #error)

Because of the fileutils change, I think we can change how fpm copies these supported file types and additionally abandon the other file types (char dev, block dev) as unsupported for now.

@jordansissel
Copy link
Owner Author

jordansissel commented Jun 18, 2021 •
edited

Looks like File.mkfifo was only added quite recently, in Ruby 2.3.0 according to this commit: ruby/ruby@9289515 which shows earliest git tag of v2_3_0_preview1

exec'ing mkfifo as a command might be an acceptable alternative for older rubies

@jordansissel
Copy link
Owner Author

For safesystem() and safesystemout(), let's see ...

Both methods invoke execmd() which itself invokes ChildProcess. I think we can replace this with Process.spawn which has existed in Ruby since 2004 in some form or another.

  • safesystem() invocations seem all the same kind of call, run a program.
  • Though it supports it, I was unable to find any safesystem() calls that pass env vars as the first argument
  • safesystemout() only captures stdout.

@jordansissel jordansissel mentioned this issue Jun 18, 2021
Merged
3 tasks
jordansissel added a commit that referenced this issue Jun 18, 2021
@jordansissel
Remove ffi dependency. #1795
d06016d
@jordansissel
Copy link
Owner Author

jordansissel commented Jun 18, 2021 •
edited

For some other background on my thought process --

This scenario is a complex one due to the following:

  • The recommended way to install fpm is gem install fpm
  • Many folks install fpm this way
  • some fpm dependencies do not work on a subset of ruby versions
  • fpm exists to /enable/ you without causing undue burdens
  • If fpm required a newer version of ruby, that would qualify as an undue burden, in my opinion specific to fpm's goals.
  • Thus, fpm should be installable with relative ease on as many ruby versions as possible.
  • gem install fpm uses rubygems as the packaging source
  • fpm currently asks for ffi ~> 1.12.0 due to ffi 1.13 and newer requiring a minimum Ruby version of 2.3
  • ffi version 1.12.x does not work on Apple M1 Mac rubies.
  • I'd like to support ruby 2.2 and older, where practical.
  • rubygems cannot (to my knowledge) specify complex constraints such as "When on Ruby 2.2, install FFI 1.12. When on newer rubies, install the latest FFI"

The objective is to keep gem install fpm as the best method for installing fpm while also enabling users on a wide range of ruby versions.

jordansissel added a commit that referenced this issue Jun 19, 2021
@jordansissel
Begin replacing ChildProcess with Process.spawn()
2886ad1 
The childprocess library uses `ffi`. Historically, installing ffi has
brought challenges for fpm users. This change is an attempt to use
ruby standard methods to replace ChildProcess.

For #1795
jordansissel added a commit that referenced this issue Jun 19, 2021
@jordansissel
Remove the need to invoke mknod using FFI.
a9efa34 
This removes fpm's direct use of FFI and removes `ffi` as a direct
dependency. For #1795

Cases:
* A unix socket.
* A named pipe
* A charDev should now fail (like /dev/tty)
* A blockDev should now fail (like /dev/sda1)

NOTE: In this change, chardev and blockdev support have been removed.
These "copies" previously were just calling `mknod` with identical
mode, basically copying the `mode` from stat(2) to mknod(2).
Exceptions are now thrown for chardev and blockdev.

Test cases:

    # Try to package a named pipe.
    % mkfifo /tmp/z.pipe
    % bundle exec bin/fpm -s dir -t rpm -n example /tmp/z.pipe
    Created package {:path=>"example-1.0-1.x86_64.rpm"}

    % rpm -qlvp example-1.0-1.x86_64.rpm
    prw-rw-r--    1 root     root                        0 Jun 17 22:40 /tmp/z.pipe

    # Create the unix socket
    % nc -lU /tmp/z.sock

    # Package it into an rpm
    % bin/fpm -s dir -t rpm -n example /tmp/z.sock |& less
    {:timestamp=>"2021-06-17T22:33:27.780347-0700", :message=>"Created package", :path=>"example-1.0-1.x86_64.rpm"}

    # Verify the file is of socket type ('s' at beginning of file mode
    % rpm -qlvp example-1.0-1.x86_64.rpm
    srwxrwxr-x    1 root     root                        0 Jun 17 22:33 /tmp/z.sock
jordansissel added a commit that referenced this issue Jun 19, 2021
@jordansissel
FreeBSD package: Replace ruby-xz usage with alternative using tar
d44e104 
For #1795

This replaces another library which uses ffi with an implementation
that doesn't need ffi.

I am not certain this is an exact replacement, but for my casual tests,
comparing .txz files generated before/after this commit, things seem ok.
This would benefit from real freebsd testing, though.
jordansissel added a commit that referenced this issue Jun 19, 2021
@jordansissel
Begin replacing ChildProcess with Process.spawn()
e3b3f9d 
The childprocess library uses `ffi`. Historically, installing ffi has
brought challenges for fpm users. This change is an attempt to use
ruby standard methods to replace ChildProcess.

For #1795
jordansissel added a commit that referenced this issue Jun 19, 2021
@jordansissel
Remove the need to invoke mknod using FFI.
18c5adc 
This removes fpm's direct use of FFI and removes `ffi` as a direct
dependency. For #1795

Cases:
* A unix socket.
* A named pipe
* A charDev should now fail (like /dev/tty)
* A blockDev should now fail (like /dev/sda1)

NOTE: In this change, chardev and blockdev support have been removed.
These "copies" previously were just calling `mknod` with identical
mode, basically copying the `mode` from stat(2) to mknod(2).
Exceptions are now thrown for chardev and blockdev.

Test cases:

    # Try to package a named pipe.
    % mkfifo /tmp/z.pipe
    % bundle exec bin/fpm -s dir -t rpm -n example /tmp/z.pipe
    Created package {:path=>"example-1.0-1.x86_64.rpm"}

    % rpm -qlvp example-1.0-1.x86_64.rpm
    prw-rw-r--    1 root     root                        0 Jun 17 22:40 /tmp/z.pipe

    # Create the unix socket
    % nc -lU /tmp/z.sock

    # Package it into an rpm
    % bin/fpm -s dir -t rpm -n example /tmp/z.sock |& less
    {:timestamp=>"2021-06-17T22:33:27.780347-0700", :message=>"Created package", :path=>"example-1.0-1.x86_64.rpm"}

    # Verify the file is of socket type ('s' at beginning of file mode
    % rpm -qlvp example-1.0-1.x86_64.rpm
    srwxrwxr-x    1 root     root                        0 Jun 17 22:33 /tmp/z.sock
jordansissel added a commit that referenced this issue Jun 19, 2021
@jordansissel
FreeBSD package: Replace ruby-xz usage with alternative using tar
ab4eb18 
For #1795

This replaces another library which uses ffi with an implementation
that doesn't need ffi.

I am not certain this is an exact replacement, but for my casual tests,
comparing .txz files generated before/after this commit, things seem ok.
This would benefit from real freebsd testing, though.
@jordansissel
Copy link
Owner Author

Resolved by #1796. Released with fpm 1.13.0

This was referenced Jun 29, 2021
Closed
Closed
@jordansissel jordansissel mentioned this issue Oct 29, 2021
Open
@jordansissel jordansissel mentioned this issue Jul 27, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@jordansissel