Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

jinja2 3.1.0 breaks mkdocs #2799

Closed
Orbiter opened this issue Mar 24, 2022 · 5 comments · Fixed by #2800 or aeternity/aeproject#410
Closed

jinja2 3.1.0 breaks mkdocs #2799

Orbiter opened this issue Mar 24, 2022 · 5 comments · Fixed by #2800 or aeternity/aeproject#410

Comments

@Orbiter
Copy link

Orbiter commented Mar 24, 2022

since the jinja2 3.1.0 release mkdocs does not work any more:

admin@host ui % pip install jinja2==3.1.0
Collecting jinja2==3.1.0
  Using cached Jinja2-3.1.0-py3-none-any.whl (132 kB)
Installing collected packages: jinja2
  Attempting uninstall: jinja2
    Found existing installation: Jinja2 3.0.0
    Uninstalling Jinja2-3.0.0:
      Successfully uninstalled Jinja2-3.0.0
Successfully installed jinja2-3.1.0
admin@host ui % mkdocs build
Traceback (most recent call last):
  File "/usr/local/bin/mkdocs", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/mkdocs/__main__.py", line 187, in build_command
    build.build(config.load_config(**kwargs), dirty=not clean)
  File "/usr/local/lib/python3.9/site-packages/mkdocs/config/base.py", line 216, in load_config
    from mkdocs.config.defaults import get_schema
  File "/usr/local/lib/python3.9/site-packages/mkdocs/config/defaults.py", line 1, in <module>
    from mkdocs.config import config_options
  File "/usr/local/lib/python3.9/site-packages/mkdocs/config/config_options.py", line 8, in <module>
    from mkdocs import utils, theme, plugins
  File "/usr/local/lib/python3.9/site-packages/mkdocs/theme.py", line 6, in <module>
    from mkdocs.utils import filters
  File "/usr/local/lib/python3.9/site-packages/mkdocs/utils/filters.py", line 13, in <module>
    @jinja2.contextfilter
AttributeError: module 'jinja2' has no attribute 'contextfilter'

However, if I install jinja2 3.0.0:

admin@host ui % pip install jinja2==3.0.0
Collecting jinja2==3.0.0
  Using cached Jinja2-3.0.0-py3-none-any.whl (133 kB)
Requirement already satisfied: MarkupSafe>=2.0.0rc2 in /usr/local/lib/python3.9/site-packages (from jinja2==3.0.0) (2.1.1)
Installing collected packages: jinja2
  Attempting uninstall: jinja2
    Found existing installation: Jinja2 3.1.0
    Uninstalling Jinja2-3.1.0:
      Successfully uninstalled Jinja2-3.1.0
Successfully installed jinja2-3.0.0
admin@host ui % mkdocs build
INFO     -  Cleaning site directory
INFO     -  Building documentation to directory: /Users/admin/git/searchlab/ui/site
INFO     -  Documentation built in 0.33 seconds
  • mkdocs can be patched by explicitly installing jinja2 3.0.0.
  • maybe this is not a mkdocs bug, but a jinja2 bug; however, this could be patched in mkdocs as well.
@Orbiter
Copy link
Author

Orbiter commented Mar 24, 2022

cross-reference in jinja2: pallets/jinja#1631

@davidism
Copy link

davidism commented Mar 24, 2022

Duplicate of #2794, with an open PR #2795

@Orbiter
Copy link
Author

Orbiter commented Mar 24, 2022

another victim readthedocs/readthedocs.org#9037

wranders added a commit to wranders/docs that referenced this issue Mar 24, 2022
temporarily pinned the version of jinja2 to address mkdocs build errors
mkdocs/mkdocs#2799
timtroendle added a commit to timtroendle/euro-calliope that referenced this issue Mar 25, 2022
jinja2 3.1.0 breaks mkdocs (mkdocs/mkdocs#2799), therefore fix jinja2 version to <3.1.0.
jdoiro3 added a commit to jdoiro3/mkdocs-multirepo-plugin that referenced this issue Mar 25, 2022
@analog-cbarber
Copy link

FWIW, jinja2 3.0.3 still works.

adelosa added a commit to adelosa/z390 that referenced this issue Mar 26, 2022
DarrenOfficial added a commit to DarrenOfficial/lavalink-list that referenced this issue Mar 26, 2022
PKUFlyingPig added a commit to PKUFlyingPig/cs-self-learning that referenced this issue Mar 26, 2022
felbinger added a commit to felbinger/AdminGuide that referenced this issue Mar 26, 2022
@oprypin
Copy link
Contributor

oprypin commented Mar 26, 2022

MkDocs version 1.2.4 was just released with only a fix for this, as well as version 1.3.0 with more new features.

https://github.com/mkdocs/mkdocs/releases

abhiTronix added a commit to abhiTronix/vidgear that referenced this issue Jun 27, 2022
jinja2 >=3.1.0 breaks mkdocs (mkdocs/mkdocs#2799), therefore fixing jinja2 version to <3.1.0.
abhiTronix added a commit to abhiTronix/vidgear that referenced this issue Jul 5, 2022
📓 Complete Release Notes: https://abhitronix.github.io/vidgear/latest/changelog/#v026-2022-07-05

New Features ✨
- **Docs:**
    * Added new bonus example for RSTP/RTP Live-Streaming using WriteGear's Compression Mode.
    * Added "How to resolve zmq.error.ZMQError" FAQ for NetGear API.(PR by @iandol)
- **Maintenance:** 
    * Added new patterns to `.gitignore` to ignore pypi's `build` directory and `egg-info` files.
- **CI:**
    * Switched to new Issue GitHub's form schema using YAMLs.

Updates/Improvements ⚡️ 
- Setup.py:
    * Replaced `>=` comparsion operator with more flexible `~=`.
    * Replaced `distutils.version.LooseVersion` with `pkg_resources.parse_version`.
- Docs:
    * Updated Site Navigation.
        + Added new notices to inform users more effectively about bonus examples.
        + Added new `Bonus` section to navigation and moved suitable pages under it.
    * Redesigned and Rewritten Donation and Contribution section to README.md
    * Updated Zenodo badge and bibtex entry.
- API:
    * Updated depreciated `tostring()` to `tobytes()` for the purposes for clarity in Python 3.2. https://docs.python.org/3/library/array.html#array.array.tobytes
- CI:
    * Added more paths and files to skip commits.

Breaking Updates/Changes 💥
- `-input_framerate` parameter now accepts any positive value for WriteGear and StreamGear APIs.

Bug-fixes 🐛
- API: Fixed `-input_framerate` less than 5 does not get used in WriteGear and StreamGear APIs.(PR by @freol35241)
- CamGear: Fixed Yt-dlp generated HTTP DASH Segments URLs not supported by OpenCV's VideoCapture(PR by @DynamiteC)
- StreamGear: 
    * Fixed `hls_segment_type` not working bug. (PR by @enarche-ahn)
    * Fixed critical logging parameter bug
        + Fixed debug logs even when `logging=False` in StreamGear's Real-time Mode. (patch suggested by @enarche-ahn)
        + Added length check to `-video_source` attribute to correctly infers it as empty(or invalid).
- CI:
    * Xfailed RSTP CamGear CI test.
    * Fixed pinned version syntax bug in docs_deployer workflow.
    * Fixed typos in Github forms and its context.
    * Added missing dependency.
- Docs:
    * Fixed jinja2 `3.1.0` or above breaks mkdocs.
        + `jinja2>=3.1.0` breaks mkdocs (mkdocs/mkdocs#2799), therefore pinned jinja2 version to `<3.1.0`.
    * Fixed support for new `mkdocstring` versions
        + Replaced rendering sub-value with options.
        + Removed pinned `mkdocstrings==0.17.0` version.
    * Fixed Netgear+Webgear bonus example code bugs.(PR by @iandol)
        + Added a missing import.
        + Removed `self.` typo.
        + Replaced the `return` value with `break` in the async as it triggers an error. 
    * Fixed external bug that causing "Home" tab to irresponsive randomly when accessed from other tabs.
abhiTronix added a commit to abhiTronix/deffcode that referenced this issue Jul 8, 2022
- 📌 `jinja2>=3.1.0` breaks mkdocs (mkdocs/mkdocs#2799), therefore pinned jinja2 version to `<3.1.0`.
abhiTronix added a commit to abhiTronix/deffcode that referenced this issue Jul 14, 2022
### ✨ New Features
- Sourcer API:
  * Implemented support for extracting metadata from live input devices/sources.
  * Added new `source_demuxer` and `forced_validate` parameters to `validate_source` internal method.
  * Implemented logic to validate `source_demuxer` value against FFmpeg supported demuxers.
  * Rearranged metadata dict.
  * Updated Code comments.
- FFdecoder API: 
  * Implemented functionality to supported live devices by allowing device path and respective demuxer into pipeline.
  * Included `-f` FFmpeg parameter into pipeline to specify source device demuxer.
  * Added special case for discarding `-framerate` value with Nonetype.
- CI:
  * Added new unittest `test_camera_capture()` to test support for live Virtual Camera devices.
  * Added new `v4l2loopback-dkms`, `v4l2loopback-utils` and kernel related APT dependencies. 
- Bash Script:
  * Added new FFmpeg command to extract image datasets from given video on Linux envs.
  * Created live Virtual Camera devices through `v4l2loopback` library on Github Actions Linux envs. 
    * Added `v4l2loopback` modprobe command to setup Virtual Camera named `VCamera` dynamically at `/dev/video2`.
    * Added `v4l2-ctl --list-devices` command for debugging.
    * Implemented FFmpeg command through `nohup`(no hangup) to feed video loop input to Virtual Camera in the background.

### ⚡️ Updates/Improvements
- Sourcer API:
  * Only either `source_demuxer` or `source_extension` attribute can be present in metadata.
  * Enforced `forced_validate` for live input devices/sources in `validate_source` internal method.
- FFdecoder API:
  * Rearranged FFmpeg parameters in pipeline.
  * Removed redundant code.
  * Updated Code comments.
- FFhelper API:
  * Logged error message on metadata extraction failure.
- CI:
  * Restricted `test_camera_capture()` unittest to Linux envs only.
  * Removed `return_generated_frames_path()` method support for Linux envs. 
  * Pinned jinja2 `3.1.0` or above breaking mkdocs. 
    * `jinja2>=3.1.0` breaks mkdocs (mkdocs/mkdocs#2799), therefore pinned jinja2 version to `<3.1.0`.
- Bash Script:
  * Updated to latest FFmpeg Static Binaries links. 
    * Updated download links to abhiTronix/ffmpeg-static-builds * hosting latest available versions.
    * Updated date/version tag to `12-07-2022`.
    * Removed depreciated binaries download links and code.
- Setup:
  * Bumped version to 0.2.1.
- Docs:
  * Updated Roadmap in README.md

### 💥 Updates/Changes
- Implement support for live input devices/sources.
  * `source` parameter now accepts device name or path.
  * Added `source_demuxer` parameter to specify demuxer for live input devices/sources.
  * Implemented Automated inserting of `-f` FFmpeg parameter whenever `source_demuxer` is specified by the user.

### 🐛 Bug-fixes
- Sourcer API:
  * Fixed Nonetype value bug in `source_demuxer` assertion logic.
  * Fixed typos in parameter names.
  * Added missing import.
- FFhelper API:
  * Logged error message on metadata extraction failure.
  * Fixed bug with `get_supported_demuxers` not detecting name patterns with commas.
  * Removed redundant logging.
- CI:
  * Fixed critical permission bug causing  `v4l2loopback` to fail on Github Actions Linux envs. 
    * Elevated privileges to `root` by adding `sudo` to all commands(including bash scripts and python commands).
    * Updated vidgear dependency to pip install from its git `testing` branch with recent bug fixes.
    * Replaced relative paths with absolute paths in unit tests.
  * Fixed WriteGear API unable to write frames due to permission errors.
  * Fixed `test_source_playback()` test failing on Darwin envs with OLD FFmpeg binaries.
    * Removed `custom_ffmpeg` value for Darwin envs.
  * Fixed various naming typos.
  * Fixed missing APT dependencies.
abhiTronix added a commit to abhiTronix/deffcode that referenced this issue Aug 9, 2022
### ✨ New Features
- Sourcer API:
    * Added support for `-ffprefixes` attribute through Sourcer API's `sourcer_param` dictionary parameter _(similar to FFdecoder API)_.
- FFdecoder API: 
    * Added new `output_frames_pixfmt` metadata property to preview and handle output frames pixel-format.
- Docs:
    * Added separate "Basic" and "Advanced" Recipes markdowns files with self-explanatory text, related usage code, asset _(such as images, diagrams, GIFs, etc.)_, and UI upgrades for bringing standard quality to visual design. 
    * Added separate `index.md` for Basic and Advanced Recipes with introductory text and curated hyperlinks for quick references to various recipes _(separated with sub-categories "Decoding", "Transcoding", and "Extracting Video Metadata")_.
    * Added related admonitions to specify python dependencies as well as other requirements and relevant information required for each of these recipes.
    * Added new Basic Decoding Recipes:
        * Added Decoding Video files with various pixel formats recipes.
        * Added Decoding Live Feed Devices recipes with `source_demuxer` FFdecoder API parameter.
        * Added Decoding Image sequences recipes supporting Sequential, Glob pattern , Single (looping) image.
        * Added Decoding Network Streams recipes.
    * Added new Basic Transcoding Recipes:
        * Added Transcoding Live frames recipes with OpenCV and WriteGear.
        * Added Transcoding Live Simple Filtergraphs recipes with OpenCV.
        * Added Saving Key-frames as Image recipes with different image processing libraries.
    * Added new Basic Extracting Video Metadata Recipes:
        * Added Extracting Video Metadata recipes with FFdecoder and Sourcer APIs.
    * Added new Advanced Decoding Recipes:
        * Added Hardware-Accelerated Video Decoding recipe using NVIDIA's H.264 CUVID Video-decoder(`h264_cuvid`).
        * Added Decoding Live Virtual Sources recipes with many test patterns using `lavfi` input virtual device.
    * Added new Advanced Decoding Recipes:
        * Added lossless Hardware-Accelerated Video Transcoding recipe with WriteGear API.
        * Added Transcoding Live Complex Filtergraphs recipes with WriteGear API.
        * Added Transcoding Video Art with Filtergraphs recipes with WriteGear API for creating real-time artistic generative video art using simple and complex filtergraphs.
    * Added new Advanced Updating Video Metadata Recipes:
        * Added Updating Video Metadata recipes with user-defined as well as source metadata in FFdecoder API.
    * Added new dark and light theme logo support.
    * Added new recipes GIF assets to `gifs` folder.
    * Added new dark logo `deffcode-dark.png` asset to `images` folder.
    * Added new `ffdecoder.png` and `sourcer.png` Image assets to `images` folder.
    * Added new `navigation.tabs` feature.
    * Added Material Announcement-Bar notifying recent changes.

### ⚡️ Updates/Improvements 
- Sourcer API:
    * Implemented new validation checks to ensure given `source` has usable video stream available by checking availability of either `video bitrate` or both `frame-size` and `framerate`_ properties in the source metadata.
    * Improved `extract_resolution_framerate` method for making framerate extraction more robust by falling back to extracting `TBR` value when no framerate value available in the source metadata.
- FFdecoder API:
    * Updated `metadata` property object to validate and override source metadata properties directly by overloading same property object before formulating Frames Decoder Pipeline:
        * Implemented validation checks to verify each validate manually assigned source metadata property against specific datatype before overriding.
        * Updated logging to notify invalid datatype values when assigned through `metadata` property object.
        * Added support for overriding `source_video_resolution` source metadata property to control frame-size directly through metadata.
        * Added support for overriding `output_frames_pixfmt` metadata attribute to be used as default pixel-format, when `frame_format` parameter value is None-type.
        * Improved handling of source metadata keys in metadata property object.
    * Updated `metadata` property object to handle and assign User-defined metadata directly by overloading the same property object:
        * Added new internal `user_metadata` class variable to handle all User-defined metadata information separately.            
        * FFdecoder API's `metadata` property object now returns User-defined metadata information merged with Source Video metadata.
        * Added `tuple` value warning log to notify users `json` module converts Python `tuples` to JSON `lists`.
    * Improved logic to test validity of `-custom_resolution` attribute value through `ffparams` dictionary parameter.
    * Improved handling of FFmpeg pipeline framerate with both user-defined and metadata defined values.
    * Added `tuple` to exception in datatype check for `ffparams` dictionary parameter.
    * Added datatype validation check for `frame_format` parameter.
    * Improved handling of `-framerate` parameter. 
- Maintenance:
    * Reformatted all Core class and methods text descriptions:
        * Rewritten introductory each API class description.
        * Moved reference block from `index.md` to class description.
        * Fixed missing class and methods parameter description.
        * Fixed typos and context in texts.
        * Reformatted code comments.
    * Simplified `for` loop with `if` condition checking in metadata property object.
    * Updated logging comments.
- Setup:
      * Updated project description in metadata.
      * Bumped version to `0.2.2`.
- Docs:
    * Updated Introduction doc:
        * Added new text sections such as "Getting Started", "Installation Notes", "Recipes a.k.a Examples" and "API in a nutshell".
        * Rewritten Introduction(`index.md`) with recent Information, redefined context, UI changes, updated recipe codes, curated hyperlinks to various recipes(separated with categories), and relatable GIFs.
        * Updated spacing in `index.md` using `spacer` class within `<div>` tag and `&nbsp;`.
        * Reformatted and centered DeFFcode Introductory description.
        * Reformatted FFmpeg Installation doc and Issue & PR guidelines.
        * Updated static FFmpeg binaries download URLs in FFmpeg Installation doc.
        * Refashioned text contexts, icons, and recipes codes.
        * Updated Key Features section with reflecting new features.
    * Updated README.md:
        * Updated README.md w.r.t recent changes in Introduction(`index.md`) doc.
        * Simplified and Reformatted text sections similar to Introduction doc. 
        * Imported new "Contributions" and "Donations" sections from VidGear docs.
        * Added collapsible text and output section using `<summary>` and `<detail>` tags.
        * Added experimental note GitHub blockquote to simulate admonition in README.md.
        * Removed tag-line from README.md and related image asset.
        * Simplified and Grouped README URL hyperlinks.
        * Removed Roadmap section.
    * Updated Recipes docs:
        * Revamped DeFFcode Introduction `index.md` with new Information, Context and UI changes, Updated example codes and hyperlinks.
        * Updated Announcement Bar to fix `announcement_link` variable and text.
        * Updated footer note to notify users regarding `tuple` value warning in FFdecoder API.
        * Rewritten recipes w.r.t breaking changes in APIs.
    * Updated Reference docs:
        * Completely revamped API's parameter reference docs.
        * Added new Functional Block Diagrams to FFdecoder and Sourcer API References.
        * Rewritten and Reformatted FFdecoder and Sourcer API's parameter reference docs with new information w.r.t recent changes.
        * Implemented new admonitions explaining new changes, related warnings/errors, usage examples etc.
        * Removed redundant `advanced.md` and `basic.md` docs.
        * Added new abstracts to FFhelper and Utils docs.
    * Updated docs site navigation and titles:
        * Reformatted `index.md` and `installation/index.md`.
        * Renamed `help/index.md` to `help/help.md`.
        * Moved basic and advanced recipes from `example` to `recipes` folder.
        * Imported "Donations" sections from VidGear docs to `help.md`.
        * Added updated page-title and navigation hyperlinks in `mkdocs.yml` to new markdown files incorporated recently.
        * Updated internal navigation hyperlinks in docs and removed old redundant file links.
    * Updated docs UI:
        * Added custom `spacer` class in CSS for custom vertical spacing.
        * Imported new "New", "Advance", "Alert", "Danger" and "Bug" admonitions custom CSS UI patches from vidgear.
        * Updated all admonitions icons with new custom icon SVG+XML URLs.
        * Reformatted `custom.css` and added missing comments.
        * Updated docs fonts:
            * Updated text font to `Heebo`.
            * Updated code font to `JetBrains Mono`.
        * Updated primary and accent colors:
            * Updated primary light color to `light green`.
            * Updated primary dark color to `amber`.
            * Updated accent light color to `green`.
            * Updated accent dark color to `lime`.
        * Replaced admonitions with appropriate ones.
        * Changed Color palette toggle icons.
        * Updated icons in title headings.
    * Updated admonitions messages.
    * Updated `changelog.md`.
- CI:
    * Pinned `jinja2` version to `<3.1.0`, since `jinja2>=3.1.0` breaks mkdocs (mkdocs/mkdocs#2799).
    * Updated unittests w.r.t recent changes in APIs: 
        * Updated `test_frame_format` unittest to include manually assign output pixel-format via `metadata` property object.
        * Updated `test_metadata` unittest to include new `checks` parameter to decide whether to perform Assertion test on assigned `metadata` properties in FFdecoder API.
        * Added new parametrize attributes in `test_metadata` and `test_seek_n_save` unittests to cover every use-cases.
        * Replaced `IOError` with `ValueError` in Sourcer API unittests.
    * Updated `test_metadata` unittest to verify `tuple` value warning.
    * Updated unittests to increase code coverage significantly.
    

### 💥 Updates/Changes
- Sourcer API:
    - Sourcer API's  `retrieve_metadata()` method now returns parsed metadata either as JSON string or dictionary type.
        * Added new `pretty_json` boolean parameter to `retrieve_metadata()`, that is when `True`, returns metadata formatted as JSON string instead of default python dictionary.
    - Changed `IOError` to `ValueError` in Sourcer API, raised when source with no decodable audio or video stream is provided.
- FFdecoder API:
    - Rename `extraparams` dictionary parameter to `ffparams` in FFdecoder API. 
    - The `source` metadata value cannot be altered through `metadata` property object in FFdecoder API. 
    - Removed `-ffpostfixes` attribute support from `ffparams` dictionary parameter in FFdecoder API, since totally redundant in favor of similar `-ffprefixes` and `-clones` attributes.

### 🐛 Bug-fixes
- FFdecoder API:
    * Fixed `metadata` property object unable to process user-defined keys when any source metadata keys are defined.
    * Fixed `TypeError` bug with string type `-framerate` parameter values.
- Sourcer API:
    * Fixed Sourcer API throws `IOError` for videos containing streams without both source bitrate and framerate defined _(such as from `lavfi` input virtual device)_.
    * Fixed `AttributeError` bug due to typo in variable name.
- CI:
    * Fixed support for newer mkdocstring version in DeFFcode Docs Deployer workflow.
        * Added new `mkdocstrings-python-legacy` dependency.
        * Replaced `rendering` variable with `options`.
        * Removed pinned `mkdocstrings==0.17.0` version.
        * Removed redundant variables.
    * Updated `test_metadata` unittest to fix `AssertionError` Bug.
- Docs:
    * Fixed some admonitions icons not showing bug using `!important` rule in CSS.
    * Fixed `404.html` static page not showing up.
    * Fixed invalid internal navigation hyperlinks and asset paths.
    * Removed `quote/cite/summary` admonition custom UI patches.
    * Removed redundant information texts.
    * Fixed typos in code comments.
    * Fixed typos in example code.
kimetro pushed a commit to kimetro/hypershift that referenced this issue Sep 23, 2022
Signed-off-by: David Vossel <davidvossel@gmail.com>

Kubevirt Platform: Make HosteControlPlane be aware to KubeVirt platform

This code change "hcp.Spec.Platform.Type" to "Kubevirt"
In this case, change Infrastructure to None platform, in order to prevent error in machine-config-server
This Change is done in order to allow HostedControlPlane special behavior for KubeVirt platform
For example: will add support for "cloud-provider-kubevirt" and "kubevirt-csi-driver"

Default to using empty dir storage for registry with kubevirt platform

Signed-off-by: David Vossel <davidvossel@gmail.com>

Enable wildcard routes for OCP cluster when CNV tests are being run

Signed-off-by: David Vossel <davidvossel@gmail.com>

Enhance Kubevirt e2e test to ensure HostedCluster completes install

Signed-off-by: David Vossel <davidvossel@gmail.com>

Fix events message unit test flake

Events by reason are added to a map and their resulting order is unpredictable. Sorting result and expected removes differences in order

copy labels from HostedCluster to admin kubeconfig secret

This copies labels from the HostedCluster instance to the resources created in the hostedcluster namespace to allow for label queries on resources created in that namespace. The idea is to be able to have a query using label selectors to get all resources in the centralized hostedcluster namespace associated with a an individual hostedcluster. This is the one piece that is created in that namespace that the user does not have control over specifying labels on. This adds labels from the HostedCluster instance that can be utilized for the selector.

    ensure token minter, socks proxy, and availablity prober are versioned with the control plane operator to prevent large scale restarts on hypershift operator upgrades for IBM Cloud

    This PR removes passing the socks proxy, avalability prober, and token-minter images from the hypershift operator to all the downstream namespaces for IBM Cloud deployments. Instead: the version of the control-plane-operator is utilized. This prevents large scale deployment rollouts on any hypershift operator update (that then propogates across all hostedclusters it manages)

Resources: Improve runtime of test

The test is currently extremely slow and takes about 20 minutes with
`-race -count=25`. Improve that by pre-creating the secret that holds
the bcrypt hash of the kubeadm password, which makes the reconciliation
skip the calculation of said hash and brings the runtime with above
parameters down to ~20 seconds.

Add proxy support

This change adds proxy support. It adds a small controler into the
hypershift operator that makes it update its own deployement
based on the proxy resource in the cluster. It further extends the
hypershift operator to pass on its proxy env vars onto the capi provider
and the CPO and the CPO to pass them on to the KAS, KCM and HCCO and
ignition server.

Lastly, the konnektivity-socks5-proxy is extended to not route a
hardcoded list of cloudprovider domains into konnektivity but use its
own proxy configuration for that (if any).

Ref https://issues.redhat.com/browse/HOSTEDCP-331

Dump: Dump guest cluster nodes

So the question "Did nodes join this cluster" can be answered easily,
rather than having to infer it from "Is there a running pod somwhere".

Run unitests with racedetector and -count=25 to detect flakes

unify hypershift install make targets

dev: specify a more useful `ko` base image

Before this commit, images built with `ko` used a base image that included
no debugging/userland tools. This commit updates the base image to use the
"debug" variant of the base image which is nominally larger but includes
a busybox userland, which is far more useful by default for a low cost.

PKI: Use ECDSA keys by default

These are computationally cheaper to generate and use and more secure
and smaller. They are also FIPS-compliant[0].
Compatibility with exiting RSA CAs is retained.

Due to lacking fips-compliance we can not use ed25519, as of time of
writing they are only proposed for that: [1].

For comparison:

Previously:

```
go test -race -count=5 ./control-plane-operator/controllers/hostedcontrolplane/pki/
ok github.com/openshift/hypershift/control-plane-operator/controllers/hostedcontrolplane/pki 25.986s
```

With this change:

```
go test -race -count=5 ./control-plane-operator/controllers/hostedcontrolplane/pki/
ok  	github.com/openshift/hypershift/control-plane-operator/controllers/hostedcontrolplane/pki	6.939s
```

[0]: https://csrc.nist.gov/publications/detail/fips/186/4/final
[1]: https://csrc.nist.gov/publications/detail/fips/186/5/draft

Ensure that all control plane pods use the cluster's pull secret

Productized images will require the cluster's pull secret to be pulled
from the registry.

This commit updates the reconciliation of service accounts to include
the pull secret if they don't already do so.

Fix typo in how to pause reconciliation doc

Remove CAPI-provider-agent ClusterRole from hypershift Agent platform

Alow hypershift operator to grant RBAC permissions for clusterdeployments and agentClusterInstalls to the cpai-provider-agent

Add ClusterID to HostedCluster API

Adds ClusterID field to HostedClusterSpec. The field value is generated
if left blank.
Modifies CVO so it can be initialized with the HostedCluster's ClusterID
before generating one.
The ClusterID is needed to add a replace label to ServiceMonitors and
PodMonitors so that metrics belonging to a particular cluster can be
identified.

Uses the clusterID in the HostedControlPlane to add a replace label to
every ServiceMonitor and PodMonitor created in a control plane namespace.

Hypershift-operator: Increase worker count

This increase the workers for the hostedcluster operator to 10 (from 1)
and for the nodepool controller to 30 (also from 1). Otherwise the
workers become a bottleneck if more than one hostedcluster is changed at
a time.

Both the 10 workers and the 30 workers are relatively arbitrary chosen,
these numbers might need revisiting in the future or even warrant a
config option.

Make clusterID optional in HostedControlPlane clusterID

Previous versions of control plane operators have no knowledge of new
fields added to the HostedControlPlane resource. In order to maintain
backwards compatibility we cannot add any required fields to the
HostedControlPlane type.

This change makes the `clusterID` field optional to maintain
compatibility with previous control-plane-operators.

set initialDelaySeconds to 60s on olm operators

add required-api to availability check in olm components

e2e: adjust budgets

Introduce `OIDCConfigurationInvalid` condition for OIDC setup validation

This commit introduces a new `OIDCConfigurationInvalid` condition for
HostedCluster. If OIDC documents cannot be reconciled for a cluster,
`OIDCConfigurationInvalid=True`, otherwise `OIDCConfigurationInvalid` is
removed. The message associated with the condition tries to provide some more
helpful context about what might be wrong to help the user correct the issue.
More error interpretation can be added in the future.

Use patch instead of update when updating HCP status and finalizers

This makes it possible for the control plane operator to work with
future versions of the HostedControlPlane CRD that contain required
fields that it's not aware of.

Using MergeFromWithOptimisticLock option enables similar semantics to
the update operation, ensuring that we update a known resourceVersion
of the resource.

Delete Cluster API Cluster Role binding (openshift#1143)

* Delete Cluster API Cluster Role binding

* Cluster API Cluster Role binding

* Updates

* fix: address PR feedback

Co-authored-by: Hidematsu Sueki <hidematsu.sueki@ibm.com>

fix AWS HostedCluster fixture to set hostname in alignment with endpointAccess

KAS: Never set proxy

This seems to make the apiserver unable to reach anything over
konnektivity. Not completely sure why this happens, probably because it
ends up having the proxy ip address for everything (that is
speculation).

Just skip setting any proxy env vars, the few requests the KAS does can
go through the guest cluster.

Converge helper binaries and ignitions-server into CPO binary

This makes them part of the payload and thus avoids rotating most of the
controlplane when the hypershift operator gets updated.

Fix `ko` entries and update ignition-server dev docs

This commit removes outdated binary entries from `ko.yaml` for those
which have been collapsed into `control-plane-operator` and updates
the in-cluster ignition server developer documentation to reflect
those changes.

Trigger reconcile when paused time is up

add required-api to availablity prober for OLM and HCCO

Revert "PKI: Use ECDSA keys by default"

This reverts commit 464b572.

HO: Don't report NotFund for hostedcluster as error

This is completely expected during deletion, logging it at error level
makes it look like an issue, which it is not.

KAS: Set proxy, but exempt pod and service CIDR

The KAS needs the proxy settings to communicate with the cloud provider.
However, the egress transport it uses wraps another transport that
respects proxy settings which is why we need to excempt pod and service
CIDR of the guest cluster to not break Konnektivity.

I also tried to stop using the egress config and use the
konnektivity-socks5-proxy, but that breaks SPDY connections (exec,
port-forward).

Ref https://issues.redhat.com/browse/HOSTEDCP-333

add external-dns flags to CI install make target

increase MaxConcurrentReconciles on AWS PrivateLink controllers

enable external-dns registry

Registry configuration: reconcile only what we need to changes

In its current state, the hosted cluster config operator overwrites any
changes made by the guest cluster admin to the registry configuration.
This prevents changes like enabling a route or increasing the number of
replicas.

This commit limits what we change to things we need to change and leave
everything else as is.

Add default vxlan port for kubevirt clusters

Signed-off-by: David Vossel <davidvossel@gmail.com>

Update staticcheck to a version that works with go 1.18

The version we currently use can not compile anything and fails with
errors like this:

could not load export data: cannot import "math/bits" (unknown iexport format version 2), export data is newer version - update tool (compile)

Note that this doesn't mean staticcheck supports generics, it just means
it can be compiled with go 1.18.

Dump: Always create an archive

Currently, dump just drops a lot of files. This is useful for browsing
them in the CI job output, but terrible for downloading them for local
inspection, as downloading a lot of files is extremely slow, even if the
files aren't big.

This change makes us always create an archive of the dump to not require
extending every CI job to do this manually.

docs: Upgrade mkdocs/material to fix Netlify breakages

This upgrades mkdocs/material to fix Netlify docs compilation breakages
resulting from mkdocs/mkdocs#2799.

docs for DNS indirection

Add additionalTrustBundle to HostedCluster API

Copy HostedCluster additionalTrustBundle to HostedControlPlane

HostedCluster can optionally reference a configmap, in which case we copy
the configmap to the HostedControlPlane namespace (similar to SSHKey and
other fields).

Create HostedCluster user-ca-bundle configmap

When AdditionalTrustBundle is defined we create this ConfigMap to
align with the behavior of regular OCP clusters and enable
consumption of user-defined CA certs by the guest cluster.

Add AdditionalTrustBundle to MCS bootstrap params

When AdditionalTrustBundle is specified, we serialize the configmap and
pass to the MCO bootstrap command via the default user-ca-bundle-config.yaml
location - this means the MCO bootstrap will read the file when included,
(the code already ignores the case where the file doesn't exist, since
openshift/installer only conditionally creates the manifest)

Add create cli option for additional-trust-bundle

This can be used to reference a ConfigMap that contains a user CA
bundle.

Add trust bundle volumes to hostedcluster_controller

The CPO and ignition server need the user CA so the registryclient
can access a local registry with a self-signed cert

Add trust bundle to hosted-cluster-config-operator

Add install additional-trust-bundle CLI

Adds a CLI option and corresponding volume to the operator pod,
this is needed so the operator can look up release image metadata
when the release image specified is locally mirrored.

Note the mount path/filename were chosen to align with the expected
defaults ref https://go.dev/src/crypto/x509/root_linux.go (and also
current OCP docs for cert injection using operators)

read apiserver-network-proxy image from ocp payload

Fix CPO to work with 4.11

The single-hyphen flags do not work anymore due to
operator-framework/operator-lifecycle-manager#2362

Retry EIP tagging failures during infra creation

Before this commit, EIP tagging failures resulting from the EIP not
being found after the EIP was successfully created led to infra creation
failing overall because the tagging operation was not retried.

This commit adds retry logic to EIP tagging to account for the case when
EIP creation succeeds but tagging fails because the AWS tagging API doesn't
yet see the new EIP.

default AntiAffinity rules to spread KubeVirt VMs across mgmt nodes

Signed-off-by: David Vossel <davidvossel@gmail.com>

Update to referencing 4.10 disks and documentation for KV guide

Signed-off-by: David Vossel <davidvossel@gmail.com>

Document KubeVirt Platform Ingress/DNS options

Signed-off-by: David Vossel <davidvossel@gmail.com>

Get autoscaler/machine-approver images from the payload

These components watch both management cluster (Machine scalable resources) and guest cluster.
Originally we were pinning the images to a version that would cross any HostedCluster.
This PR let us pick them from each particular payload resulting in some benefits:

Each hostedCluster runs the component version that was tested with that particular kube/ocp version
No additional work needed to productise the images as they com from the payload.
Since CAPI CRDs should be backward compatible, having different controller versions shouldn't cause an issue.
Once the CAPI image is in the payload we can do the same for it.

Hypershift operator: Give a priority that is higher than any controlplane component

e2e: Don't fail test on transient recoverable API lookup

Before this commit, calls to `WaitForConditionsOnHostedControlPlane()` could
fail a test if an API lookup fails even though that lookup is recoverable and
retried automatically. This made the test flaky.

This commit fixes the code so that these retriable errors are logged but do
not fail the test.

This commit also moves a log message which was intended to emit during retries
but was instead placed at the exit point.

Fix priority class for olm cronjob and verify priorityclasses in e2e

The olm cronjob had a prioryClass of openshift-user-critical which has a
priority that is above all other controlplane components in the
management cluster. Downgrade it to the standard
hypershift-control-plane and add an e2e test that verifies that no pod
has a priority higher than the etcd priority.

e2e: Don't enable user workload monitoring on management clusters

Before this commit, UWM was enabled by the e2e `setup` command, which
was used in the past but is no longer used. The UWM stack is thus wasting
resources on management clusters used for e2e runs.

This commit removes UWM from the monitoring setup for e2e tests.

Remove `ValidAMI` AWS NodePool condition when AMI is user-defined

Before this commit, the `ValidAMI=True` condition was applied to AWS NodePools
which specified user-defined AMIs, which was misleading because those AMIs are
not actually validated by HyperShift.

This commit removes the `ValidAMI` condition entirely from AWS NodePools which
declare user-specified AMIs, since they can't be validated.

Add node troubleshooting documentation

Documents some tools that are available in our code base that can help
understanding why nodes are not able to join a cluster.

Also adds some fixes to the getting started documentation.

Introduce logic to accommodate in place upgrades

This is still gated by validateManagement which prevents this logic from ever running.
Upcoming PRs will drop the gating validation and will complete the in place upgrade logic to run pods in the guest cluster.

do not wait on capi clusterrolebinding delete

Add support to set up a http proxy for guest clusters

This change:
* Adds a new --enable-proxy flag to the create aws infra command
  which will create a http proxy instance if set
* It will also cause a proxy config pointing to said instance to be
  added to the HostedCluster
* Extends the globalconfig package to optionally be able to set the
  status of a proxy config, including additional no_proxy hosts
* Makes the nodepool controllers userdata secret generation consider a
  proxy config if preset
* Makes the ignition config rendering use a proxy config with status
  set, as otherwise the proxy config will be ignored
* Makes the aws infra destroy command also  clean up instances to not
  get blocked by the proxy instance

Ref https://issues.redhat.com/browse/HOSTEDCP-333

Create valid route names with long namespace names

When using a long name for a cluster namespace, the routes that the
control plane operator creates is invalid because a segment of the
domain name has a length > 63 chars.

This commit sets the host name for routes to ensure that the resulting
host name is valid. Modification of the route hostname only happens when
the resulting hostname will be invalid. The name is shortened by
trimming the original name and appending a hash of the entire name,
resulting in repeatable and distinct names every time the name is
shortened.

Enforce hostedcluster service route immutability constraints

Before this PR, the `route.spec.host` field associated with a given
`hostedcluster.spec.services[].route` value was being defaulted by OCP and then
later overwritten with an empty string on subsequent reconcile loops. This is
fundamentally a side effect of a violation of the immutability constraint on the
`hostname` field.

To fix the issue, this PR refactors the relevant Route reconciliation code to
compute and set the `route.spec.host` field only during the creation of the
Route resource, which enforces the immutability constraint and resolves the side
effect.

Signal parsing config failure in a condition

This PR signal invalid cluster config in a HC condition. It also do the same for NodePools and refactors to narrow down the reconcileUserdata signature and let it only use what it takes.

Disable PodSecurity admission in 4.11 as it breaks conformance

This is enabled by default in 4.11/kube 1.23 which breaks a bunch of
networking tests that want to create privileged pods which the default
`Restricted` policty doesn't allow.

Do the same as the kas-operator and disable it.

ensure imagePullPolicies are IfNotPresent for better tolerance of networking outages

AWS infra destroy, handle empty instanceIDs

In the case where the nodepool didn't create any instances this
list can be empty on destroy, which causes an error like:

missing required field, TerminateInstancesInput.InstanceIds

This causes the destroy to get stuck, so instead handle the
case where the list is empty.

Make HyperShift operator compatible with previous CPOs without utilities

Adds symlinks in HyperShift operator image for previously standalone
utilities and enables the control plane operator to invoke the right
command based on the symlink it was invoked with.

Checks whether the CPO for a given cluster includes subcommands for
utilities by checking the labels on the CPO image, and only uses the
CPO image for utilities if it does include the labels.

continue to allow support for specification of clusterAutoscalerImage and machineApprover image until no longer necessary

IBM's production release candidate still relies on this behavior to deploy the images specified in the BOM. We will need to have this support in place until we can confirm we have successfully upgraded all production clusters to a later BOM version that no longer needs this behavior and instead uses the release image

Fix kubernetes.default for public clusters with proxy

On public clusters that use a http proxy, the kubernetes.default service
currently doesn't work, because it is served by a HAProxy that tries to
directly connect to the KAS, which isn't possible.

This change:
* Adds a new `kubernetes-default-proxy` subcommand to the CPO which is a
  simple tcp proxy that initiates a connection over http connect, as it
  doesn't seem that HAProxy supports this
* Uses a static pod with said subcommend on clusters with a https proxy
  that are not private

ensure release image annotation added to deployment template metadata to enable proper detection in IBM Cloud deployment process

The release image annotation should also be added to the pod template annotation so it is reflected on the pods that are scheduled from the deployment. This is necessary from a monitoring perspective and for detection in IBM Cloud's deployment process to ensure rollouts complete on a release image update

add validating webhook for HostedClusters

reconcile oauth serving cert rbac to allow oauth proxies in the openshift-monitoring namespace to work

This adds rbac that allows authenticated users to fetch the oauth-serving-cert configmap. This is necessary for the oauth proxies in the openshift-monitoring namespace to work and allow users access to the prometheus/grafana/alertmanager dashboards.

remove version check for api system that is not in 4.9

disable reconcile of registry config in IBMCloud deployments

Currently there is a race condition in IBMCloud deployments where if hypershift initializes the registry config before the managed service operation has a chance to properly initialize the registry config: It will result in a bad initial config that will cause the cluster-image-registry operator to crash and lead to tickets. For IBMCloud deployments: the initialization of this config should be delegated to the managed service to do the initalization so this does not occur. From there: the user can edit however they like in line with upstream openshift

Use forked processes instead of pods to generate ignition payload

Before this commit, ignition payloads for nodepools were generated by
orchestrating MCO components using pods. This was a performance problem because
ignition server restarts (e.g. during upgrades) resulted in thundering herds of
ignition payload generation pods, limiting overall cluster capacity.

This commit refactors ignition payload generation away from pods and to a
mechanism that extracts MCO binaries (and supporting files) from release
payloads so they can be forked as subprocesses of the ignition server
controller, removing all pod scheduling overhead. This change should be
completely transparent to the end user and is a drop-in replacement for the old
implementation.

The current implementation has the following limitations:

* Extracted payload contents are not cached across executions per release image.
  This is possible, but not worth doing unless subsequent performance analysis
  justifies it.

* All payload generation executions are serial. Concurrent executions are
  possible, but not worth doing unless subsequent performance analysis justifies
  it.

This commit also enables metrics collection for the ignition server component and
adds the following metrics:

* ign_server_token_rotation_total
* ign_server_payload_cache_miss_total
* ign_server_payload_generation_seconds
* ign_server_payload_cache_total

Ignition server: Actually use workdir

The defaulting was incorrect

configure cipher suites to prevent using medium strength ssl ciphers

We should only be using strong ciphers in hypershift components. By default Go includes medieum strength ciphers in the default tls config. This adjusts to only be strong ciphers (TLS 1.2 and TLS 1.3)

move to ga apis for all components now that management clusters at minimum release boundary

fix(cpo): Scope down secrets access for olm collect profiles cj

Update config.go

Added shutdown-send-retry-after and updated shutdown-delay-duration

feat(cpo): adhere to upgrade order from kube version skew policy

fix(cpo): set tls cipher suites flags on kcm and scheduler

Ensure cache is set during token rotation before reconciling

Add missing control plane prometheus rules

Ensure that everything uses imagePullPolicy IfNotPResent for resiliency

Ref https://issues.redhat.com/browse/HOSTEDCP-330

Updated secret permissions to conform to kubernetes CIS benchmark

Hypershift Image Pull Policy to IfNotPresent

feat(cpo): Support disable profiling annotation

Set Recommended Leader Election Values

Update control-plane-operator/controllers/hostedcontrolplane/scheduler/config_test.go

Co-authored-by: Hidematsu Sueki <hsueki794@gmail.com>

mend

Add fallback set cache value from old token

When you run multiple pods, by design the first one doing a token rotation in the secret CR forces the other two pods caches to generate a new payload for that token because they are not aware of it yet. This attempts to mitigate that scenario.

feat(oauth): allow challenge override for OpenID

cache registry files

adopt existing immutable selectors to prevent errors reconciling components from roks toolkit clusters

This fixes an error in the control plane operator where it will fail to reconcile and properly adopt deployments based on roks toolkit clusters. This is due to the fact that the selectors has changed and that is an immutable field. Specifically, hypershift selectors add a duplicate selector hypershit.component that is not contained in roks toolkit clusters. Even with the single app selector: deployment rollout behavior is unchanged. This allows for zero downtime adoption of roks toolkit clusters. Associated issue: openshift#1042

Use non-strict mode when parsing global config

The HyperShift operator uses the latest version of the OpenShift API to
serialize global config in the HCP (for compatibility with older CPOs).
The issue is that if there's a new field in the latest API that the
older CPO does not understand, we currently produce an error decoding
the YAML because the YAML serializer we use is using strict mode.

This commit switches to a YAML serializer that does have strict mode for
parsing of the global configuration. Making it possible for older CPOs
to parse YAML from the latest HyperShift operator.

change secret 420 to416

updt secret to 416

chg scrts perm to 416

chg scrts perm to 416

chg scrts perm to 416

chg scrts perm to 416

add pointer for scrt416 P

add pointer for scrt416 P

add pointer for scrt416

defaultmode 416 mcs-tls

416 mode on scrt

416 scr mode

change scrt DefaultMode to 416

DefaultMode to 416

Scrt DefaultMode416

Scrt DefaultMode416

change secret Defaultmode 420 to 416
lander2k2 added a commit to qleet/user-docs that referenced this issue Nov 9, 2022
Latest version of jinja2 breaks mkdocs
mkdocs/mkdocs#2799

Signed-off-by: Rich Lander <lander2k2@protonmail.com>
praveenkumarct added a commit to commercetools/commercetools-sync-java that referenced this issue Nov 21, 2022
candlerb added a commit to candlerb/solis_exporter that referenced this issue Nov 27, 2022
candlerb added a commit to candlerb/solis_exporter that referenced this issue Nov 27, 2022
RostislavKreisinger pushed a commit to shopsys/shopsys that referenced this issue Jan 25, 2023
mladenzivo pushed a commit to mladenzivo/substrait that referenced this issue Mar 27, 2023
See: mkdocs/mkdocs#2799

* force older jinja until proto plugin is updated.
vladh added a commit to vladh/glad that referenced this issue Apr 8, 2023
vladh added a commit to vladh/glad that referenced this issue Apr 28, 2024
shark314 pushed a commit to shark314/Sta_bot_tele that referenced this issue Apr 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
4 participants