Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Building an omnibus package requires write access to the install destination #841

Open
drbrain opened this issue Jun 12, 2018 · 4 comments

Comments

@drbrain
Copy link

drbrain commented Jun 12, 2018

Description

When I'm building the next version of an omnibus-packaged application omnibus obliterates the existing files I installed using a previous package. If the build fails I may have a corrupt or non-working application.

If I'm building for the first time omnibus expects to be able to write to the installed locations. If this directory is only root-writable the omnibus build will fail with Errno::EACCESS as it tries to delete or overwrite the installed files.

Additionally, a macOS .pkg requires root permissions to install the package, which omnibus should not need to build that package.

Instead omnibus should build the installer in a directory completely under its ownership.

Omnibus Version

v5.6.8

Platform Version

macOS 10.13.5: Darwin asx.local 17.6.0 Darwin Kernel Version 17.6.0: Tue May 8 15:22:16 PDT 2018; root:xnu-4570.61.1~1/RELEASE_X86_64 x86_64

ruby 2.5.0: ruby 2.5.0p0 (2017-12-25 revision 61468) [x86_64-darwin17]

Replication Case

This starts by following the Get Sarted steps in the omnibus README, then extends that to a scenario where you've installed your software and want to package when you have a previous package installed.

  1. omnibus new MY_PROJECT_NAME
  2. bundle install --binstubs
  3. bin/omnibus build MY_PROJECT_NAME
    1. Discover this doesn't work because /var/cache/omnibus is not writable
    2. File Building an omnibus package writes to the install destination #840 because this is not documented as a necessary step
    3. Find out you need to create /var/cache/omnibus and make it writable
    4. sudo mkdir /var/cache/omnibus
    5. sudo chown $USER /var/cache/omnibus
  4. Re-run bin/omnibus build MY_PROJECT_NAME
    1. Discover this also doesn't work because you are missing "software named `preparation'"
    2. Find and copy that file into the project
    3. Skip filing a bug that will probably get closed like Building an omnibus package writes to the install destination #840 did, where you tried to re-open it after following up with a repeated reproduction
  5. Re-run bin/omnibus build MY_PROJECT_NAME
    1. Discover this also doesn't work because you are missing "software named `version-manifest'"
    2. Find and copy that file into the project
    3. Skip filing a bug that will probably get closed like Building an omnibus package writes to the install destination #840 did, where you tried to re-open it after following up with a repeated reproduction
  6. Re-run bin/omnibus build MY_PROJECT_NAME
    1. Discover this doesn't work because it's trying to write to /opt/MY_PROJECT_NAME
    2. sudo mkdir /opt/MY_PROJECT_NAME
    3. sudo chown $USER /opt/MY_PROJECT_NAME/
  7. Re-run bin/omnibus build MY_PROJECT_NAME
    1. This completes with pkg/myprojectname-0.0.0+20180612202154-1.pkg being created
  8. sudo rm -r /opt/MY_PROJECT_NAME/
  9. open pkg/myprojectname-0.0.0+20180612202154-1.pkg
    1. Click yes/accept/etc. through all the prompts and enter your password such that the software would be installed
  10. Make some change to my omnibus project
  11. bin/omnibus build MY_PROJECT_NAME
    1. Discover this fails with Permission denied @ apply2files - /opt/MY_PROJECT_NAME/LICENSES/.gitkeep (Errno::EACCES)

Expected results

I want to be able to build a package without corrupting and replacing my currently installed project. If there is some issue with the package it may not complete replace my currently installed and working version.

I should not require write access by default to the package installation destination. Since the installer requires my password to install, and they're installed such that I can't mess them up by accident, then building the software should not touch that same location.

Step 6 must not require 6.ⅱ or 6.ⅲ and step 11 must work without changing the permissions of /opt/MY_PROJECT_NAME

Additionally, the Get Started steps are incomplete and the README generated in the omnibus-MY_PROJECT_NAME directory does not include any of the extra steps I had to perform to reproduce the bug I care about.

Actual results

The install location must be writable in order to build an omnibus package

Additionally, if you file a bug about how the Get Started steps are missing steps (#840) it gets closed and the steps are not updated for correctness.

Build Output

Step 6:

$ bin/omnibus build MY_PROJECT_NAME
                    [CLI] I | 2018-06-12T13:19:36-07:00 | Using config from 'omnibus.rb'
[2018-06-12T13:19:36-07:00] INFO: The plugin path /etc/chef/ohai/plugins does not exist. Skipping...
           [BuildVersion] W | 2018-06-12T13:19:37-07:00 | Could not extract version information from 'git describe'! Setting version to 0.0.0.
Building MY_PROJECT_NAME 0.0.0+20180612201937...
  [Software: preparation] I | 2018-06-12T13:19:37-07:00 | Resolving manifest entry for preparation
[NullFetcher: preparation] I | 2018-06-12T13:19:37-07:00 | Fetching `preparation' (nothing to fetch)
[Software: version-manifest] I | 2018-06-12T13:19:37-07:00 | Resolving manifest entry for version-manifest
[NullFetcher: version-manifest] I | 2018-06-12T13:19:37-07:00 | Fetching `version-manifest' (nothing to fetch)
Traceback (most recent call last):
	17: from bin/omnibus:29:in `<main>'
	16: from bin/omnibus:29:in `load'
	15: from /usr/local/lib/ruby/gems/2.5.0/gems/omnibus-5.6.8/bin/omnibus:16:in `<top (required)>'
	14: from /usr/local/lib/ruby/gems/2.5.0/gems/omnibus-5.6.8/lib/omnibus/cli.rb:42:in `execute!'
	13: from /usr/local/lib/ruby/gems/2.5.0/gems/thor-0.20.0/lib/thor/base.rb:466:in `start'
	12: from /usr/local/lib/ruby/gems/2.5.0/gems/omnibus-5.6.8/lib/omnibus/cli/base.rb:33:in `dispatch'
	11: from /usr/local/lib/ruby/gems/2.5.0/gems/thor-0.20.0/lib/thor.rb:387:in `dispatch'
	10: from /usr/local/lib/ruby/gems/2.5.0/gems/thor-0.20.0/lib/thor/invocation.rb:126:in `invoke_command'
	 9: from /usr/local/lib/ruby/gems/2.5.0/gems/thor-0.20.0/lib/thor/command.rb:27:in `run'
	 8: from /usr/local/lib/ruby/gems/2.5.0/gems/omnibus-5.6.8/lib/omnibus/cli.rb:89:in `build'
	 7: from /usr/local/lib/ruby/gems/2.5.0/gems/omnibus-5.6.8/lib/omnibus/project.rb:1073:in `build'
	 6: from /usr/local/lib/ruby/2.5.0/fileutils.rb:193:in `mkdir_p'
	 5: from /usr/local/lib/ruby/2.5.0/fileutils.rb:193:in `each'
	 4: from /usr/local/lib/ruby/2.5.0/fileutils.rb:208:in `block in mkdir_p'
	 3: from /usr/local/lib/ruby/2.5.0/fileutils.rb:208:in `reverse_each'
	 2: from /usr/local/lib/ruby/2.5.0/fileutils.rb:210:in `block (2 levels) in mkdir_p'
	 1: from /usr/local/lib/ruby/2.5.0/fileutils.rb:232:in `fu_mkdir'
/usr/local/lib/ruby/2.5.0/fileutils.rb:232:in `mkdir': Permission denied @ dir_s_mkdir - /opt/MY_PROJECT_NAME (Errno::EACCES)

Step 11:

$ bin/omnibus build MY_PROJECT_NAME
                    [CLI] I | 2018-06-12T13:23:57-07:00 | Using config from 'omnibus.rb'
[2018-06-12T13:23:57-07:00] INFO: The plugin path /etc/chef/ohai/plugins does not exist. Skipping...
           [BuildVersion] W | 2018-06-12T13:23:58-07:00 | Could not extract version information from 'git describe'! Setting version to 0.0.0.
Building MY_PROJECT_NAME 0.0.0+20180612202358...
  [Software: preparation] I | 2018-06-12T13:23:58-07:00 | Resolving manifest entry for preparation
[NullFetcher: preparation] I | 2018-06-12T13:23:58-07:00 | Fetching `preparation' (nothing to fetch)
[Software: version-manifest] I | 2018-06-12T13:23:58-07:00 | Resolving manifest entry for version-manifest
[NullFetcher: version-manifest] I | 2018-06-12T13:23:58-07:00 | Fetching `version-manifest' (nothing to fetch)
Traceback (most recent call last):
	18: from bin/omnibus:29:in `<main>'
	17: from bin/omnibus:29:in `load'
	16: from /usr/local/lib/ruby/gems/2.5.0/gems/omnibus-5.6.8/bin/omnibus:16:in `<top (required)>'
	15: from /usr/local/lib/ruby/gems/2.5.0/gems/omnibus-5.6.8/lib/omnibus/cli.rb:42:in `execute!'
	14: from /usr/local/lib/ruby/gems/2.5.0/gems/thor-0.20.0/lib/thor/base.rb:466:in `start'
	13: from /usr/local/lib/ruby/gems/2.5.0/gems/omnibus-5.6.8/lib/omnibus/cli/base.rb:33:in `dispatch'
	12: from /usr/local/lib/ruby/gems/2.5.0/gems/thor-0.20.0/lib/thor.rb:387:in `dispatch'
	11: from /usr/local/lib/ruby/gems/2.5.0/gems/thor-0.20.0/lib/thor/invocation.rb:126:in `invoke_command'
	10: from /usr/local/lib/ruby/gems/2.5.0/gems/thor-0.20.0/lib/thor/command.rb:27:in `run'
	 9: from /usr/local/lib/ruby/gems/2.5.0/gems/omnibus-5.6.8/lib/omnibus/cli.rb:89:in `build'
	 8: from /usr/local/lib/ruby/gems/2.5.0/gems/omnibus-5.6.8/lib/omnibus/project.rb:1075:in `build'
	 7: from /usr/local/lib/ruby/gems/2.5.0/gems/omnibus-5.6.8/lib/omnibus/licensing.rb:57:in `create_incrementally'
	 6: from /usr/local/lib/ruby/gems/2.5.0/gems/omnibus-5.6.8/lib/omnibus/licensing.rb:57:in `tap'
	 5: from /usr/local/lib/ruby/gems/2.5.0/gems/omnibus-5.6.8/lib/omnibus/licensing.rb:59:in `block in create_incrementally'
	 4: from /usr/local/lib/ruby/gems/2.5.0/gems/omnibus-5.6.8/lib/omnibus/licensing.rb:119:in `prepare'
	 3: from /usr/local/lib/ruby/2.5.0/fileutils.rb:1051:in `touch'
	 2: from /usr/local/lib/ruby/2.5.0/fileutils.rb:1051:in `each'
	 1: from /usr/local/lib/ruby/2.5.0/fileutils.rb:1054:in `block in touch'
/usr/local/lib/ruby/2.5.0/fileutils.rb:1054:in `utime': Permission denied @ apply2files - /opt/MY_PROJECT_NAME/LICENSES/.gitkeep (Errno::EACCES)
@stevendanna
Copy link
Contributor

stevendanna commented May 1, 2019

Apologies this sat for so long without a response. You've reported a number of issues here. I've tried to list them all below:

  1. /var/cache/omnibus needs to exist and be writable
  2. preparation software not included by default
  3. version_manifest software not included by default
  4. /opt/MY_PROJECT_NAME needs to exist and be writable
  5. Can't rebuild MY_PROJECT_NAME when it is already installed

1 and 4 should probably be fixed with a combination of better docs and better error messages. We could potentially default the omnibus cache dir to another location if not running as root.

2 and 3 should just be fixed. It isn't clear to me we need the version_manifest software anymore since omnibus can do that directly. I think this was broken here: 2d1d714 since these definitions are included in omnibus-software which was included in the bundle.

Unfortunately, issue 5 is much trickier. The issue is that building the software in any directory other than /opt/PACKAGE_NAME is likely to (1) require pretty drastic changes to how omnibus builds C software and (2) will break compatibility with existing plans. I think a better error message is definitely in order; but, at this stage in the project, I'm not sure it will be worth the trade-off in compatibility to completely fix.

stevendanna added a commit that referenced this issue May 1, 2019
Omnibus was previously generating an invalid project by default
because it:

- Didn't include a version-manifest software definition
- Didn't include the "preparation" software definition
- Had an invalid Gemfile

This fixes those issues, getting us a bit closer to the Getting
Starting directions working out of the box for people.

Fixes #887
Partially addresses #841

Signed-off-by: Steven Danna <steve@chef.io>
@drbrain
Copy link
Author

drbrain commented May 1, 2019

Would you like me to open separate issues for the multiple issues you've identified as combined into this one?

5 is disappointing, I've worked on other software packagers (RubyGems, ruby-packer) and have managed to make use of ./configure features to install into temporary directories that allow for proper linking to occur during the build phase. My understanding is that these features are what OS package managers use to create their packages, also using temporary directories.

@stevendanna
Copy link
Contributor

stevendanna commented May 3, 2019

Would you like me to open separate issues for the multiple issues you've identified as combined into this one?

I thinks this is OK. I've fixed 2 of the issues, and another 2 are directly related to the title of the ticket so this ticket is fine to track them.

5 is disappointing, I've worked on other software packagers (RubyGems, ruby-packer) and have managed to make use of ./configure features to install into temporary directories that allow for proper linking to occur during the build phase. My understanding is that these features are what OS package managers use to create their packages, also using temporary directories.

I agree. It isn't impossible to do, but it would require many people to update their software definitions potentially. Perhaps a separate ticket for this could be opened and we could consider it if we ever decide to do another major version of omnibus. Or if someone thinks of a way to do it without breaking compatibility.

@drbrain
Copy link
Author

drbrain commented May 6, 2019

I'd hate to leave an issue around to decay in the history and not be noticed if you do get around to a major version bump

tas50 pushed a commit that referenced this issue Jul 31, 2020
Omnibus was previously generating an invalid project by default
because it:

- Didn't include a version-manifest software definition
- Didn't include the "preparation" software definition
- Had an invalid Gemfile

This fixes those issues, getting us a bit closer to the Getting
Starting directions working out of the box for people.

Fixes #887
Partially addresses #841

Signed-off-by: Steven Danna <steve@chef.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants