Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable packaging of curl|bash and other wild stuff. #1957

Open
wants to merge 13 commits into
base: main
Choose a base branch
from

Conversation

jordansissel
Copy link
Owner

@jordansissel jordansissel commented Nov 7, 2022

For #1853

References: https://gist.github.com/jordansissel/0e0fc22344a17eadd1a3b9a1bbb7ed60

Attempts that worked out well enough:

  • ✅ rustup
  • ✅ sdkman
  • ✅ sdkman's sdk install java

@jordansissel
Copy link
Owner Author

jordansissel commented Nov 7, 2022

I picked Podman to start with because that's what I have right now.

Here's converting rustup's curl|bash to an rpm.

The rustup docs say the following:

Run the following in your terminal, then follow the onscreen instructions.

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

  • I'll run it with |sh -s -- -y to run the installer non-interactively (accepting "yes" to all questions)
  • To make it faster/smaller, we'll also use rustup-init's flag, --profile=minimal
  • Add --no-modify-path so the installer doesn't mess with any .bashrc/.profile
  • Set a custom install path RUSTUP_HOME=/usr/local/rustup
% fpm --verbose -f -s curlbash -t rpm  -n rustup \
  --depends gcc \
  --curlbash-setup "apt-get update && apt-get install -y curl" \
  "curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- --profile=minimal --no-modify-path -y"
... [ some time passes ] ...
Created package {:path=>"rustup-1.0-1.x86_64.rpm"}
% rpm -qlp rustup-1.0-1.x86_64.rpm| less
...
/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/cargo
/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/rust-gdb
/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/rust-gdbgui
/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/rust-lldb
/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/rustc
...

🫡 We've got an rpm. Let's try it.

# Start with a fresh fedora 36 container
% podman run -v "$PWD/rustup-1.0-1.x86_64.rpm:/rustup-1.0-1.x86_64.rpm" -it fedora:36

# Install the rpm.
[root@7bcb72eab5fa /]# dnf install rustup-1.0-1.x86_64.rpm 
...

Dependencies resolved.
==========================================================================================================
 Package                             Architecture    Version                  Repository             Size
==========================================================================================================
Installing:
 rustup                              x86_64          1.0-1                    @commandline          146 M
Installing dependencies:
...
 gcc                                 x86_64          12.2.1-2.fc36            updates                33 M
 glibc-devel                         x86_64          2.35-20.fc36             updates                70 k
 glibc-headers-x86                   noarch          2.35-20.fc36             updates               475 k
...

Transaction Summary
==========================================================================================================
Install  19 Packages

Total size: 206 M
Total download size: 59 M
Installed size: 695 M
Is this ok [y/N]: 
...
Installed:
  binutils-2.37-36.fc36.x86_64                 binutils-gold-2.37-36.fc36.x86_64                         
  cpp-12.2.1-2.fc36.x86_64                     elfutils-debuginfod-client-0.187-4.fc36.x86_64            
  gc-8.0.6-3.fc36.x86_64                       gcc-12.2.1-2.fc36.x86_64                                  
  glibc-devel-2.35-20.fc36.x86_64              glibc-headers-x86-2.35-20.fc36.noarch                     
  guile22-2.2.7-5.fc36.x86_64                  kernel-headers-6.0.5-200.fc36.x86_64                      
  libmpc-1.2.1-4.fc36.x86_64                   libpkgconf-1.8.0-2.fc36.x86_64                            
  libtool-ltdl-2.4.7-1.fc36.x86_64             libxcrypt-devel-4.4.28-1.fc36.x86_64                      
  make-1:4.3-7.fc36.x86_64                     pkgconf-1.8.0-2.fc36.x86_64                               
  pkgconf-m4-1.8.0-2.fc36.noarch               pkgconf-pkg-config-1.8.0-2.fc36.x86_64                    
  rustup-1.0-1.x86_64                         

Cool, eh?

Let's try running rustc?

[root@9b80162d68a3 /]# PATH=/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin:"$PATH"

[root@9b80162d68a3 /]# echo 'fn main() { println!("Hello world!"); }' > hello.rs
[root@9b80162d68a3 /]# rustc hello.rs
[root@9b80162d68a3 /]# ./hello 
Hello world!

I was originally using tar's --exclude (didn't get committed), but
'--exclude tmp' will exclude any path with "tmp" in the name when I
really want to exclude just the toplevel directory "/tmp".

Then I rememered fpm has its own --exclude which is based on Ruby's
File::fnmatch, and that works differently than tar's --exclude and
also does what I want.

This bug was found by trying to package sdkman:

```
% bundle exec bin/fpm --verbose -f -s curlbash -t rpm -n sdkman \
  --depends zip --depends unzip \
  --curlbash-setup "apt-get update" \
  --curlbash-setup "apt-get install -y curl unzip zip"  'curl -s "https://get.sdkman.io?rcupdate=false" |  bash'
```
@jordansissel
Copy link
Owner Author

sdkman --

% bundle exec bin/fpm --verbose -f -s curlbash -t rpm -n sdkman \
  --depends zip --depends unzip \
  --curlbash-setup "apt-get update" \
  --curlbash-setup "apt-get install -y curl unzip zip"  'curl -s "https://get.sdkman.io?rcupdate=false" |  bash'

And installing it...

[root@693dc2883f2c /]# dnf install sdkman-1.0-1.x86_64.rpm 
...
Installed:
  sdkman-1.0-1.x86_64             unzip-6.0-57.fc36.x86_64             zip-3.0-32.fc36.x86_64            

Complete!

[root@693dc2883f2c /]# source "$HOME/.sdkman/bin/sdkman-init.sh"
[root@693dc2883f2c /]# sdk install java
Downloading: java 17.0.5-tem

In progress...

#############################                                                                        29.3%
Repackaging Java 17.0.5-tem...

Done repackaging...

Installing: java 17.0.5-tem
Done installing!


Setting java 17.0.5-tem as default.

👍

@jordansissel
Copy link
Owner Author

Heh, what if we package what sdkman does instead?

% bundle exec bin/fpm --verbose -f -s curlbash -t rpm -n sdkman-java \
  --curlbash-setup "apt-get update" \
  --curlbash-setup "apt-get install -y curl unzip zip" \
  --curlbash-setup 'curl -s "https://get.sdkman.io?rcupdate=false" |  bash' \
  'bash -c ". $HOME/.sdkman/bin/sdkman-init.sh; sdk install java"'
[root@77d5815e9be6 /]# rpm -Uvh sdkman-java-1.0-1.x86_64.rpm 
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Updating / installing...
   1:sdkman-java-1.0-1                ################################# [100%]

[root@77d5815e9be6 /]# /root/.sdkman/candidates/java/17.0.5-tem/bin/java --version
openjdk 17.0.5 2022-10-18
OpenJDK Runtime Environment Temurin-17.0.5+8 (build 17.0.5+8)
OpenJDK 64-Bit Server VM Temurin-17.0.5+8 (build 17.0.5+8, mixed mode, sharing)

@jordansissel
Copy link
Owner Author

Chef's installer

% bundle exec bin/fpm --verbose -f -s curlbash -t rpm -n chef \
  --curlbash-container-image fedora:36 \
  'curl -L https://omnitruck.chef.io/install.sh | bash'
% podman run -v "$PWD/chef-1.0-1.x86_64.rpm:/chef-1.0-1.x86_64.rpm" -it fedora:36
[root@3a51dd0abbfc /]# rpm -i chef-1.0-1.x86_64.rpm 
[root@3a51dd0abbfc /]# ohai platform
[root@3a51dd0abbfc /]# ohai platform
[
  "fedora"
]

@jordansissel
Copy link
Owner Author

Trying ohmyzsh:

% bundle exec bin/fpm --verbose -f -s curlbash -t rpm -n ohmyzsh \
  --depends zsh --depends git \
  --curlbash-container-image fedora:36 \
  --curlbash-setup "dnf install -y zsh git" \
  'curl -fsSL https://raw.github.com/ohmyzsh/ohmyzsh/master/tools/install.sh | sh'

Created package {:path=>"ohmyzsh-1.0-1.x86_64.rpm"}
[root@4a8f07e439b5 /]# dnf install ohmyzsh-1.0-1.x86_64.rpm 
Dependencies resolved.
==========================================================================================================
 Package                        Architecture   Version                         Repository            Size
==========================================================================================================
Installing:
 ohmyzsh                        x86_64         1.0-1                           @commandline         2.1 M
Installing dependencies:
...
Transaction Summary
==========================================================================================================
Install  79 Packages

Total size: 24 M
Total download size: 22 M
Installed size: 94 M
Is this ok [y/N]: y


[root@4a8f07e439b5 /]# zsh -ls
➜  / echo $ZSH_THEME
robbyrussell

image

@jordansissel
Copy link
Owner Author

Trying nodesource. This script seems to add a apt repository on Ubuntu then offers apt-get install nodejs. Let's capture the result of installing nodejs instead.

% bundle exec bin/fpm --verbose -f -s curlbash -t deb -n nodesource -v 19 \
  --curlbash-setup 'apt-get update' \
  --curlbash-setup 'apt-get install -y curl' \
  --curlbash-setup 'curl -fsSL https://deb.nodesource.com/setup_19.x | bash -' \
  'apt-get install -y nodejs'
Created package {:path=>"nodesource_19_amd64.deb"}

This is fun because it actually captures the nodejs install instead of just updating your apt repos.

% podman run -v "$PWD/nodesource_19_amd64.deb:/nodesource_19_amd64.deb" -it ubuntu:latest 
root@28e3fc58ef68:/# dpkg -i nodesource_19_amd64.deb 
...
root@28e3fc58ef68:/# node 
Welcome to Node.js v19.0.1.
Type ".help" for more information.
> 

@jordansissel
Copy link
Owner Author

It's not perfect, but it is funny :)

I found that sometimes fpm would crash during the cleanup step because
some directories had their write permission removed.

I'm rather surprised I don't remember seeing this problem in the past,
though it's possible my memory is bad and I just forget having
experienced this.

The solution I came up with is to ensure all directories have both
execute and write permissions before attempting `Fileutils.rm_r`

As a cheat, I reuse a FileUtils internal class (`FileUtils::Entry_`)
and its `preorder_traversal` methods to walk all directories, ensuring
that top-level and parent directories are modified first.

I know it's taboo to use internal/non-public code. It works on Ruby 2.7,
and if we need to change implementations to make things work better on
other rubies, then we'll do that.

It works.
Example:

```
FROM fedora:36

RUN ls
RUN touch /usr/bin/hello-world
```

```
% bundle exec bin/fpm -f -s curlbash -t rpm -n example /tmp/Dockerfile
Created package {:path=>"example-1.0-1.x86_64.rpm"}

% rpm -qlvp example-1.0-1.x86_64.rpm
-rw-r--r--    1 root     root                        0 Nov 12 21:43 /usr/bin/hello-world
```
@jordansissel
Copy link
Owner Author

Added Dockerfile/Containerfile support

Also added some scaffholding for docker support whenever that gets
added to fpm.
and also fixed bugs found by the tests. :)
Also ensure we clean up the temporary container image we created
I feel like this better suits the purpose, as we could curlbash things
but this feature can also just do regular Dockerfile/Containerfile
things.
@jordansissel jordansissel marked this pull request as ready for review November 14, 2022 01:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant