Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixes #37383 - Add OSTree Include/Exclude Refs options #10980

Merged
merged 2 commits into from
May 10, 2024

Conversation

lumarel
Copy link
Contributor

@lumarel lumarel commented Apr 27, 2024

What are the changes introduced in this pull request?

Since Pulp OSTree plugin version 2.0.0 it's possible to use include_refs and exclude_refs parameters,
these are especially handy when you want to clone Fedora OSTree (Atomic Desktop) variants.

This PR adds theses 2 generic_remote_option parameters.

Considerations taken when implementing this change?

It was also necessary to do a db migration for already existing repos,
due to my very limited knowledge how to do that, my migration might look a bit bare minimum,
I would be very happy about a suggestion how to do that in a better way if there is one 馃檪

I also noticed that pulp takes the values on creation, but somehow in my test environment it only updated in the Katello db but not in pulp, couldn't find any errors or other code parts which would indicate that to happen. (will put in more time to find what's going on, but maybe it's already apparent to you what it is)

I saw that there is the possibility that OSTree repos get added by a subscription, these shouldn't have anything in the generic_remote_options column so the migration should also be fine there.

What are the testing steps for this pull request?

  • Setup environment with OSTree enabled
  • Add OSTree repo (I recommend using a small one, Fedora repos still take a long time, 9h, with the include filter)
  • Parameters should be carried over into the pulp config
  • And the sync should work as expected with the limited head ref filter

Looks like this when set up:
image

@lumarel lumarel force-pushed the feature/ostree-incl-excl-properties branch from 06c7369 to 091a824 Compare April 27, 2024 21:39
@sjha4
Copy link
Member

sjha4 commented May 3, 2024

The VCR failures are related. We will need to re-record those. I can check if a re-record makes the tests green and if any updates are needed there.

One question: Can we default the new remote options to null instead of [] and avoid the data migration on older remotes?

@lumarel
Copy link
Contributor Author

lumarel commented May 4, 2024

Thanks for your review @sjha4 !

Hmm I tried switching the default to null, but it seems it still wants a migration:

# foreman-installer 
2024-05-04 15:51:02 [NOTICE] [root] Loading installer configuration. This will take some time.
2024-05-04 15:51:05 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2024-05-04 15:51:05 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
2024-05-04 15:51:12 [NOTICE] [configure] Starting system configuration.
2024-05-04 15:51:19 [NOTICE] [configure] 250 configuration steps out of 1420 steps complete.
2024-05-04 15:51:21 [NOTICE] [configure] 500 configuration steps out of 1422 steps complete.
2024-05-04 15:51:25 [NOTICE] [configure] 750 configuration steps out of 1425 steps complete.
2024-05-04 15:51:25 [NOTICE] [configure] 1000 configuration steps out of 1431 steps complete.
2024-05-04 15:51:44 [ERROR ] [configure] '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
2024-05-04 15:51:44 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: change from 'notrun' to ['0'] failed: '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
2024-05-04 15:51:52 [NOTICE] [configure] 1250 configuration steps out of 1431 steps complete.
2024-05-04 15:52:04 [NOTICE] [configure] System configuration has finished.

Error 1: Puppet Exec resource 'foreman-rake-db:migrate' failed. Logs:
  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]
    Adding autorequire relationship with User[foreman]
    Starting to evaluate the resource (1179 of 1431)
    Evaluated in 17.12 seconds
  Exec[foreman-rake-db:migrate](provider=posix)
    Executing check '/usr/sbin/foreman-rake db:abort_if_pending_migrations'
    Executing '/usr/sbin/foreman-rake db:migrate'
  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/unless
    rake aborted!
    NameError: undefined local variable or method `null' for RepositoryType[ostree]:Katello::RepositoryType
    /usr/share/gems/gems/katello-4.12.0/lib/katello/repository_types/ostree.rb:21:in `block in <top (required)>'
    /usr/share/gems/gems/katello-4.12.0/app/services/katello/repository_type_manager.rb:21:in `instance_eval'
    /usr/share/gems/gems/katello-4.12.0/app/services/katello/repository_type_manager.rb:21:in `register'
    /usr/share/gems/gems/katello-4.12.0/lib/katello/repository_types/ostree.rb:3:in `<top (required)>'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/dependencies.rb:326:in `load'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/dependencies.rb:326:in `block in load'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/dependencies.rb:299:in `load_dependency'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/dependencies.rb:326:in `load'
    /usr/share/gems/gems/katello-4.12.0/lib/katello/repository_types.rb:2:in `block in <top (required)>'
    /usr/share/gems/gems/katello-4.12.0/lib/katello/repository_types.rb:1:in `each'
    /usr/share/gems/gems/katello-4.12.0/lib/katello/repository_types.rb:1:in `<top (required)>'
    <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:85:in `require'
    <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:85:in `require'
    /usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
    /usr/share/gems/gems/zeitwerk-2.6.13/lib/zeitwerk/kernel.rb:34:in `require'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/dependencies.rb:332:in `block in require'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/dependencies.rb:299:in `load_dependency'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/dependencies.rb:332:in `require'
    /usr/share/gems/gems/katello-4.12.0/lib/katello/plugin.rb:2:in `<top (required)>'
    <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:85:in `require'
    <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:85:in `require'
    /usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
    /usr/share/gems/gems/zeitwerk-2.6.13/lib/zeitwerk/kernel.rb:34:in `require'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/dependencies.rb:332:in `block in require'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/dependencies.rb:299:in `load_dependency'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/dependencies.rb:332:in `require'
    /usr/share/gems/gems/katello-4.12.0/lib/katello/engine.rb:84:in `block in <class:Engine>'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/initializable.rb:32:in `instance_exec'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/initializable.rb:32:in `run'
    /usr/share/foreman/config/initializers/0_print_time_spent.rb:45:in `block in run'
    /usr/share/foreman/config/initializers/0_print_time_spent.rb:17:in `benchmark'
    /usr/share/foreman/config/initializers/0_print_time_spent.rb:45:in `run'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/initializable.rb:61:in `block in run_initializers'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/initializable.rb:60:in `run_initializers'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/application.rb:391:in `initialize!'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/railtie.rb:207:in `public_send'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/railtie.rb:207:in `method_missing'
    /usr/share/foreman/config/environment.rb:5:in `<top (required)>'
    <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:85:in `require'
    <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:85:in `require'
    /usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
    /usr/share/gems/gems/zeitwerk-2.6.13/lib/zeitwerk/kernel.rb:34:in `require'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/dependencies.rb:332:in `block in require'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/dependencies.rb:299:in `load_dependency'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/dependencies.rb:332:in `require'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/application.rb:367:in `require_environment!'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/application.rb:533:in `block in run_tasks_blocks'
    /usr/share/gems/gems/rake-13.0.3/exe/rake:27:in `<top (required)>'
    Tasks: TOP => db:abort_if_pending_migrations => db:load_config => environment
    (See full trace by running task with --trace)
  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns
    rake aborted!
    NameError: undefined local variable or method `null' for RepositoryType[ostree]:Katello::RepositoryType
    /usr/share/gems/gems/katello-4.12.0/lib/katello/repository_types/ostree.rb:21:in `block in <top (required)>'
    /usr/share/gems/gems/katello-4.12.0/app/services/katello/repository_type_manager.rb:21:in `instance_eval'
    /usr/share/gems/gems/katello-4.12.0/app/services/katello/repository_type_manager.rb:21:in `register'
    /usr/share/gems/gems/katello-4.12.0/lib/katello/repository_types/ostree.rb:3:in `<top (required)>'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/dependencies.rb:326:in `load'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/dependencies.rb:326:in `block in load'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/dependencies.rb:299:in `load_dependency'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/dependencies.rb:326:in `load'
    /usr/share/gems/gems/katello-4.12.0/lib/katello/repository_types.rb:2:in `block in <top (required)>'
    /usr/share/gems/gems/katello-4.12.0/lib/katello/repository_types.rb:1:in `each'
    /usr/share/gems/gems/katello-4.12.0/lib/katello/repository_types.rb:1:in `<top (required)>'
    <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:85:in `require'
    <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:85:in `require'
    /usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
    /usr/share/gems/gems/zeitwerk-2.6.13/lib/zeitwerk/kernel.rb:34:in `require'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/dependencies.rb:332:in `block in require'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/dependencies.rb:299:in `load_dependency'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/dependencies.rb:332:in `require'
    /usr/share/gems/gems/katello-4.12.0/lib/katello/plugin.rb:2:in `<top (required)>'
    <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:85:in `require'
    <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:85:in `require'
    /usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
    /usr/share/gems/gems/zeitwerk-2.6.13/lib/zeitwerk/kernel.rb:34:in `require'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/dependencies.rb:332:in `block in require'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/dependencies.rb:299:in `load_dependency'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/dependencies.rb:332:in `require'
    /usr/share/gems/gems/katello-4.12.0/lib/katello/engine.rb:84:in `block in <class:Engine>'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/initializable.rb:32:in `instance_exec'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/initializable.rb:32:in `run'
    /usr/share/foreman/config/initializers/0_print_time_spent.rb:45:in `block in run'
    /usr/share/foreman/config/initializers/0_print_time_spent.rb:17:in `benchmark'
    /usr/share/foreman/config/initializers/0_print_time_spent.rb:45:in `run'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/initializable.rb:61:in `block in run_initializers'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/initializable.rb:60:in `run_initializers'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/application.rb:391:in `initialize!'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/railtie.rb:207:in `public_send'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/railtie.rb:207:in `method_missing'
    /usr/share/foreman/config/environment.rb:5:in `<top (required)>'
    <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:85:in `require'
    <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:85:in `require'
    /usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
    /usr/share/gems/gems/zeitwerk-2.6.13/lib/zeitwerk/kernel.rb:34:in `require'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/dependencies.rb:332:in `block in require'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/dependencies.rb:299:in `load_dependency'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/dependencies.rb:332:in `require'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/application.rb:367:in `require_environment!'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/application.rb:533:in `block in run_tasks_blocks'
    /usr/share/gems/gems/rake-13.0.3/exe/rake:27:in `<top (required)>'
    Tasks: TOP => db:migrate => db:load_config => environment
    (See full trace by running task with --trace)
    change from 'notrun' to ['0'] failed: '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]

1 error was detected during installation.
Please address the errors and re-run the installer to ensure the system is properly configured.
Failing to do so is likely to result in broken functionality.

The full log is at /var/log/foreman-installer/katello.log

I'm also very happy for your help with the VCR failures, I think I once did such one for the Ansible modules, but has been a while 馃檪

@lumarel
Copy link
Contributor Author

lumarel commented May 4, 2024

Btw! Also found out that the changing the properties in the UI and them not getting promoted to pulp thing is a non-issue, the next time a sync gets triggered, it's definitely in the pulp remote config 馃憤馃徎

@sjha4
Copy link
Member

sjha4 commented May 9, 2024

Ack..we refresh remote when a sync is triggered.

Screenshot from 2024-05-09 14-29-01

On the migration part, I am still not sure why we need it. Without that the remote options will be empty and that works for pulp. Do we need to set the include and exclude refs to blank at all.

@sjha4
Copy link
Member

sjha4 commented May 9, 2024

In my testing, I removed the migration and am able to sync older ostree repos as well as update the repo on the UI and sync it with refs etc.

Only one VCR failure: katello/test/services/katello/pulp3/repository_integration_test.rb
We can re-record it for the new remote options. I will push a commit with the updated VCR on the PR once we get to it.

@lumarel lumarel force-pushed the feature/ostree-incl-excl-properties branch from 091a824 to 6caa241 Compare May 9, 2024 19:41
@lumarel
Copy link
Contributor Author

lumarel commented May 9, 2024

@sjha4 Okay if you say it should work without the migration then I will of course remove it 馃檪
I just couldn't get past a successful run of foreman-installer without the migration, but you sure have a lot more knowledge about that than me ^^

(removed the migration)

@sjha4 sjha4 force-pushed the feature/ostree-incl-excl-properties branch from 55dd5e7 to fbefa46 Compare May 10, 2024 14:20
Copy link
Member

@sjha4 sjha4 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APJ 馃憤馃徏

@sjha4 sjha4 merged commit 846b651 into Katello:master May 10, 2024
20 of 22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants