v5.1.1

v5.1.1

This release includes a number of improvements surrounding our CLI and how the test suite is ran.

There was a bug found in the RHCOS version compatibility test that was fixed in #2109 for OCP beta versions greater than 4.14.

Bug Fixes and Improvements

• feat : Add a link back to the test case that mentioned security categories by @bnshr in #2075
• Operators script: label pods under test after waiting for all pods to come up by @edcdavid in #2096
• Bug fix: adjust OCP beta slice by @sebrandon1 in #2109
• Batch operators test catalog source automation by @shirmoran in #2107
• log: rename global log file by @jmontesi in #2116

CLI Improvements

• cmd/tnf: add subcommand to run the test suite by @jmontesi in #2065
• cmd/certsuite: rename the "tnf"command to "certsuite" by @jmontesi in #2076
• gitignore: add certsuite binary by @jmontesi in https://github.com/test-network-function/cnf-certification-test/pull/208788
• Add a workflow task to verify the "certsuite run" command by @jmontesi in #2085
• Parse flags after env vars to override them if required by @jmontesi in #2092
• cli: update test suite banner by @jmontesi in #2097
• cmd/certsuite: add new flags by @jmontesi in #2098
• Makefile: add build info to the certsuite binary by @jmontesi in #2105
• cmd/certsuite: add flags for selecting the TNF debug image by @jmontesi in #2106
• cmd/certsuite: add shorthand flags for the most common flags by @jmontesi in #2115

CI Updates

• Add 10 minute timeout for waiting for all pods to be ready by @sebrandon1 in #2078
• Update ubuntu runners to 24.04 by @sebrandon1 in #2079
• Makefile: update local golangci to v.1.58.2 by @jmontesi in #2084
• Switch RHCOS job back to 22.04 by @sebrandon1 in #2086
• pre-main: add container workflow task to check the certsuite command by @jmontesi in #2103
• Test: Revert changes to ubuntu 24.04 for cancelled jobs by @sebrandon1 in #2112
• Add binary test to nightlies by @sebrandon1 in #2111

Dependency Updates

• build(deps): bump k8s.io/apiextensions-apiserver from 0.30.0 to 0.30.1 by @dependabot in #2072
• build(deps): bump helm.sh/helm/v3 from 3.14.4 to 3.15.0 by @dependabot in #2077
• build(deps): bump k8s.io/kubectl from 0.29.4 to 0.30.1 by @dependabot in #2071
• Update operator-sdk to v1.34.2 by @sebrandon1 in #2083
• Update preflight to v1.9.5 by @sebrandon1 in #2081
• build(deps): bump actions/checkout from 4.1.5 to 4.1.6 by @dependabot in #2082
• build(deps): bump ubi9/python-39 from 1-182.1714671018 to 1-182.1715774360 in /.github/actions/documentation by @dependabot in #2090
• build(deps): bump github/codeql-action from 3.25.5 to 3.25.6 by @dependabot in #2089
• build(deps): bump requests from 2.31.0 to 2.32.0 in /.github/actions/documentation by @dependabot in #2091
• build(deps): bump github.com/go-logr/logr from 1.4.1 to 1.4.2 by @dependabot in #2093
• Bump helm.sh/helm/v3 from 3.15.0 to 3.15.1 by @dependabot in #2099
• Bump github.com/operator-framework/api from 0.24.0 to 0.25.0 by @dependabot in #2100
• Bump github.com/hashicorp/go-version from 1.6.0 to 1.7.0 by @dependabot in #2102
• Bump docker/login-action from 3.1.0 to 3.2.0 by @dependabot in #2110
• Bump ubi9/python-39 from 1-182.1715774360 to 1-182.1716478612 in /.github/actions/documentation by @dependabot in #2108
• Bump ubi9/ubi-minimal from 9.4-949.1714662671 to 9.4-949.1716471857 by @dependabot in #2113
• Bump ubi9/ubi from 9.4-947.1714667021 to 9.4-947.1716476138 by @dependabot in #2114

Full Changelog: v5.1.0...v5.1.1

v5.1.0

v5.1.0

We have renamed the test suite from cnf-certification-test to Red Hat Best Practices Test Suite for Kubernetes.

Future Changes on the Roadmap:

• Further releases in v5.1.x will include more tests in the operator test suite.
• Modifications to how the test suite is ran/kicked off by removing the scripts and environment variables to provide a seamless way to run the binaries.
• Since v5.0.8 we are now building arm64 images (see #1943), we have been experimenting with how to better test our new arm64 images on a fully arm enabled CI. This is an ongoing initiative that probably will not be completed until Github Actions allows for fully arm64 runners.

Test Suite Renaming

• README: rename CNF Certification Test by @jmontesi in #2012

OSSF Additions

• [StepSecurity] Apply security best practices by @step-security-bot in #2011
• Adding OSSF score card to workflow by @edcdavid in #1961
• Fix OSSF scorecard badge by @edcdavid in #2010

Various Updates

• Group badges by @rdavid in #2029
• Add README badges for QE OCP nightlies by @sebrandon1 in #2022
• flags: add a new flag to provide a rhcos_version_map file by @jmontesi in #2047
• Bug fix: avoid runtime panic (SIGSEGV) when using the list flag by @jmontesi in #2066
• operatingsystem: embed rhcos_version_map file into the test suite binary by @jmontesi in #2052
• scripts: use the test suite binary directly in "run-tnf-container.sh" by @jmontesi in #2033
• style : Modify catalog generation to add 4 security contexts categories by @bnshr in #2044
• main: simplify and move some functions to the certsuite and log packages by @jmontesi in #2060

CI Improvements

• Switch QE nightlies to use unstable image by @sebrandon1 in #2006
• Remove macos runner from unit tests by @sebrandon1 in #2007
• Mark operator best practices tests QE test covered by @bnshr in #2019
• Switch 4.14 actions runner by @sebrandon1 in #2039

Dependency Updates

• build(deps): bump actions/upload-artifact from 3.pre.node20 to 4.3.3 by @dependabot in #2016
• build(deps): bump github/codeql-action from 3.24.9 to 3.25.3 by @dependabot in #2015
• build(deps): bump actions/checkout from 4.1.1 to 4.1.4 by @dependabot in #2014
• build(deps): bump ubi9/python-39 from 1-161 to 1-172.1712567222 in /.github/actions/documentation by @dependabot in #2013
• build(deps): bump golangci/golangci-lint-action from 4 to 5 by @dependabot in #2008
• Fix for missing frozen dependency by @edcdavid in #2020
• build(deps): bump golangci/golangci-lint-action from 5.0.0 to 5.1.0 by @dependabot in #2021
• build(deps): bump ubi9/python-39 from 1-172.1712567222 to 1-182 in /.github/actions/documentation by @dependabot in #2025
• build(deps): bump ubi9/ubi from 9.3-1610 to 9.4-947 by @dependabot in #2026
• Update machine-config-operator; update preflight to v1.9.2 by @sebrandon1 in #2024
• build(deps): bump ubi9/ubi-minimal from 9.3-1612 to 9.4-949 by @dependabot in #2027
• build(deps): bump actions/setup-go from 5.0.0 to 5.0.1 by @dependabot in #2030
• build(deps): bump jinja2 from 3.1.3 to 3.1.4 in /.github/actions/documentation by @dependabot in #2041
• build(deps): bump golangci/golangci-lint-action from 5.1.0 to 5.3.0 by @dependabot in #2038
• build(deps): bump jinja2 from 3.1.3 to 3.1.4 in /.github/workflows by @dependabot in #2040
• build(deps): bump ubi9/python-39 from 1-182 to 1-182.1714671018 in /.github/actions/documentation by @dependabot in #2037
• build(deps): bump ubi9/ubi-minimal from 9.4-949 to 9.4-949.1714662671 by @dependabot in #2036
• build(deps): bump ubi9/ubi from 9.4-947 to 9.4-947.1714667021 by @dependabot in #2035
• build(deps): bump actions/checkout from 4.1.4 to 4.1.5 by @dependabot in #2046
• build(deps): bump golangci/golangci-lint-action from 5.3.0 to 6.0.0 by @dependabot in #2045
• build(deps): bump github/codeql-action from 3.25.3 to 3.25.4 by @dependabot in #2049
• build(deps): bump golangci/golangci-lint-action from 6.0.0 to 6.0.1 by @dependabot in #2048
• Update Go to v1.22.3 by @sebrandon1 in #2053
• build(deps): bump github/codeql-action from 3.25.4 to 3.25.5 by @dependabot in #2061
• build(deps): bump github.com/fatih/color from 1.16.0 to 1.17.0 by @dependabot in #2062
• build(deps): bump ossf/scorecard-action from 2.3.1 to 2.3.3 by @dependabot in #2058
• build(deps): bump golang.org/x/term from 0.19.0 to 0.20.0 by @dependabot in #2056
• Update GolangCI-lint to v1.58.1 by @sebrandon1 in #2054
• build(deps): bump github.com/operator-framework/api from 0.23.0 to 0.24.0 by @dependabot in #2057
• Update preflight to v1.9.4 by @sebrandon1 in #2055

New Contributors

• @step-security-bot made their first contribution in #2011

Full Changelog: v5.0.8...v5.1.0

v5.0.8

This release is a quick fix to update our affiliated-certification online/offline database of certified images.

Documentation Updates

• docs: CNF Certification rename to Red Hat Best Practices for Kubernetes by @jmontesi in #1999
• Catalog rename by @jmontesi in #2003

Dependency Updates

• build(deps): bump github.com/test-network-function/oct from 0.0.7 to 0.0.8 by @dependabot in #1997
• Update oct to v0.0.10 by @sebrandon1 in #2004

CI Updates

• Clean openshift-marketplace namespaces in QE by @sebrandon1 in #2000

Full Changelog: v5.0.7...v5.0.8

v5.0.7

v5.0.7

This is the first release that has been QE tested against OCP 4.15 clusters with all tests passing. Prior to this we were only testing against 4.13 and 4.14 clusters.

New Tests

• operator: add new test case operator-single-crd-owner by @jmontesi in #1951
• feat : Operator CRD versioning by @bnshr in #1957

Note: We plan on making our v5.1.x minor release when all of our new operator tests are completed and added to the test suite.

Improvements

• log: replace the pattern log.Error() + os.Exit(1) with log.Fatal() by @jmontesi in #1964
• Fix for test case operator-install-status-no-privileges. by @greyerof in #1665
• Operator script: Improved log output with limited information on command line and color coding. by @edcdavid in #1965
• operator: delete very long debug log by @jmontesi in #1975
• Add 4.15 and 4.16 to RHCOS versions script by @sebrandon1 in #1979
• Add OCP 4.15 to lifecycle matrix by @sebrandon1 in #1984

Dependency Updates

• Bump parser tag by @edcdavid in #1962
• build(deps): bump peaceiris/actions-gh-pages from 3 to 4 by @dependabot in #1966
• build(deps): bump github.com/k8snetworkplumbingwg/network-attachment-definition-client from 1.6.0 to 1.7.0 by @dependabot in #1971
• build(deps): bump helm.sh/helm/v3 from 3.14.3 to 3.14.4 by @dependabot in #1972
• Update Go to v1.22.2 by @sebrandon1 in #1973
• build(deps): bump k8s.io/apiextensions-apiserver from 0.29.3 to 0.29.4 by @dependabot in #1985
• build(deps): bump k8s.io/kubectl from 0.29.3 to 0.29.4 by @dependabot in #1988

Housekeeping

• identifiers: unset the qe covered flag for some new tests by @jmontesi in #1968
• versions: group together functions for version handling by @jmontesi in #1969
• Update the expected results template by @jmontesi in #1970
• Change 4.13 QE nightly to 4.15 by @sebrandon1 in #1976
• Kick off self-hosted runner QE runs at different times by @sebrandon1 in #1977
• Update the expected results template by @jmontesi in #1978
• Add workflow_dispatch to RHCOS job by @sebrandon1 in #1980
• Change name of job in yaml by @sebrandon1 in #1982
• Fix copyright years by @rdavid in #1983
• Update expected results template by @jmontesi in #1991

Full Changelog: v5.0.6...v5.0.7

v5.0.6

With v5.0.6 we are now building our released images in both arm64 and amd64 architectures.

Suite Improvements

• feat : Check for semantic versioning of the installed application operators by @bnshr in #1920
• feat : Operator CRD has OpenAPI spec by @bnshr in #1949
• log: add fatal level by @jmontesi in #1958

Multi-Arch Images

• Enable multi-platform image builds by @sebrandon1 in #1942
• Switch the build-and-push action for multi-arch by @sebrandon1 in #1953

Dependency Updates

• build(deps): bump github.com/test-network-function/oct from 0.0.6 to 0.0.7 by @dependabot in #1941
• build(deps): bump github.com/operator-framework/api from 0.22.0 to 0.23.0 by @dependabot in #1950
• build(deps): bump golang.org/x/term from 0.18.0 to 0.19.0 by @dependabot in #1959

Documentation Updates

• Documentation for all the functions in Manageability by @shimritproj in #1932
• Documentation for all the functions in Access-control by @shimritproj in #1923

CI

• Temporarily disable typos linter by @sebrandon1 in #1948
• Fix tnf image build by @sebrandon1 in #1952
• Build image first for QE PR runs by @sebrandon1 in #1955
• Only build image if tests pass by @sebrandon1 in #1956

Full Changelog: v5.0.5...v5.0.6

v5.0.5

Improvements

• Remove unused funcs by @sebrandon1 in #1916
• remove collector's EIP by @shirmoran in #1911
• Fix integer conversions by @sebrandon1 in #1921
• clientsholder: change Kubeconfig precedence by @jmontesi in #1931
• Avoid unnecessary rebuild of the test environment by @jmontesi in #1919

Config Updates

• Run test without need to label resources under test by @shimritproj in #1871
• Add configuration field for collector's endpoint by @shirmoran in #1912

Documentation Updates

• Refactor Operator Requirement Doc Link by @bnshr in #1935

CI Updates

• Changed alert msg to be sent to a slack webhook. by @greyerof in #1929

Dependency Updates

• build(deps): bump github.com/test-network-function/privileged-daemonset from 1.0.21 to 1.0.22 by @dependabot in #1909
• build(deps): bump github.com/test-network-function/test-network-function-claim from 1.0.35 to 1.0.36 by @dependabot in #1910
• build(deps): bump github.com/mittwald/go-helm-client from 0.12.8 to 0.12.9 by @dependabot in #1917
• Update operator-sdk to v1.34.1 by @sebrandon1 in #1915
• build(deps): bump google.golang.org/protobuf from 1.32.0 to 1.33.0 by @dependabot in #1918
• build(deps): bump helm.sh/helm/v3 from 3.14.2 to 3.14.3 by @dependabot in #1922
• build(deps): bump k8s.io/apiextensions-apiserver from 0.29.2 to 0.29.3 by @dependabot in #1924
• build(deps): bump k8s.io/kubectl from 0.29.2 to 0.29.3 by @dependabot in #1927
• build(deps): bump github.com/docker/docker from 24.0.7+incompatible to 24.0.9+incompatible by @dependabot in #1934

Full Changelog: v5.0.4...v5.0.5

v5.0.4

Bug Fixes and Improvements

• autodiscover: modify Istio Service Mesh detection by @jmontesi in #1896
• Only return error when an actual error occurs during a CNF run by @jmontesi in #1901
• Bumping the parser version to v0.4.4 to fix SRI issue (javascript) by @edcdavid in #1905

CI Updates

• Run all the tests in the GitHub CI by @jmontesi in #1889

Dependency Updates

• Update Go to v1.22 by @sebrandon1 in #1902
• build(deps): bump github.com/mittwald/go-helm-client from 0.12.7 to 0.12.8 by @dependabot in #1887
• build(deps): bump github.com/test-network-function/oct from 0.0.5 to 0.0.6 by @dependabot in #1891
• Update GolangCI-lint to v1.56.2 by @sebrandon1 in #1892
• build(deps): bump github.com/stretchr/testify from 1.8.4 to 1.9.0 by @dependabot in #1894
• build(deps): bump github.com/test-network-function/test-network-function-claim from 1.0.34 to 1.0.35 by @dependabot in #1899
• build(deps): bump golang.org/x/term from 0.17.0 to 0.18.0 by @dependabot in #1900
• Update preflight to v1.9.1 by @sebrandon1 in #1898
• Update operator-sdk to v1.34.0 by @sebrandon1 in #1897
• build(deps): bump ubi9/ubi from 9.3-1552 to 9.3-1610 by @dependabot in #1903
• build(deps): bump ubi9/ubi-minimal from 9.3-1552 to 9.3-1612 by @dependabot in #1904

Full Changelog: v5.0.3...v5.0.4

v5.0.3

v5.0.3 is a patch level update with dependency updates and CI improvements.

Dependency Updates

• build(deps): bump golang.org/x/term from 0.16.0 to 0.17.0 by @dependabot in #1865
• Update Go to v1.21.7 by @sebrandon1 in #1873
• build(deps): bump k8s.io/kubectl from 0.29.1 to 0.29.2 by @dependabot in #1880
• build(deps): bump helm.sh/helm/v3 from 3.14.0 to 3.14.1 by @dependabot in #1875
• build(deps): bump github.com/k8snetworkplumbingwg/network-attachment-definition-client from 1.5.0 to 1.6.0 by @dependabot in #1877
• build(deps): bump k8s.io/apiextensions-apiserver from 0.29.1 to 0.29.2 by @dependabot in #1881
• build(deps): bump helm.sh/helm/v3 from 3.14.1 to 3.14.2 by @dependabot in #1884
• build(deps): bump github.com/test-network-function/privileged-daemonset from 1.0.20 to 1.0.21 by @dependabot in #1885

CI Improvements

• Github ci: fixed example label for operators autodiscovery. by @greyerof in #1869
• Add new tnf command to check the test results by @jmontesi in #1864
• build(deps): bump golangci/golangci-lint-action from 3 to 4 by @dependabot in #1867
• Add the expected results template for the GitHub CI by @jmontesi in #1870
• workflows: use the tnf check command instead of the gradetool by @jmontesi in #1872
• Delete gradetool leftovers by @jmontesi in #1874

Full Changelog: v5.0.2...v5.0.3

v5.0.2

v5.0.2 includes a number of bug fixes and improvements to the test suite.

The ICMP tests were previously not specifying what interface they would be pinging over so that is corrected.

The tainted kernel test was improved to only test workers that have CNF workloads deployed on them.

The base image test was improved to use podman that was actually installed on the node rather than the compiled podman from the debug container image.

Bug Fixes

• Fix for platform-alteration-base-image test case. by @greyerof in #1861
• Specify network interface for ICMP tests by @sebrandon1 in #1853
• Add filter for tainted node test by @sebrandon1 in #1847

Improvements

• checksdb: turn the labels evaluator into a global variable by @jmontesi in #1834
• Delete empty files by @jmontesi in #1845
• checksdb: delete unused field by @jmontesi in #1828
• flags: allow filtered checks listing by @jmontesi in #1827
• Fix copyright years by @rdavid in #1841
• Propagate errors from preflight in the claim file by @edcdavid in #1684
• certsuite: process flags refactor by @jmontesi in #1848

Dependency Updates

• build(deps): bump github.com/test-network-function/oct from 0.0.4 to 0.0.5 by @dependabot in #1830
• build(deps): bump github.com/test-network-function/test-network-function-claim from 1.0.33 to 1.0.34 by @dependabot in #1831
• Update preflight to v1.8.1 by @sebrandon1 in #1854
• build(deps): bump github.com/operator-framework/api from 0.21.0 to 0.22.0 by @dependabot in #1859

Image Updates

• Use UBI python image by @rdavid in #1832
• build(deps): bump ubi9/ubi-minimal from 9.3-1475 to 9.3-1552 by @dependabot in #1839
• build(deps): bump ubi9/ubi from 9.3-1476 to 9.3-1552 by @dependabot in #1840

TNF Config Updates

• set hard coded collector app endpoint by @shirmoran in #1836

Unit Tests

• Add unit tests for testhelper package by @sebrandon1 in #1835

Scripting Improvements

• Set executable mode initially to the script by @rdavid in #1833
• Simplify operations involving variables by @rdavid in #1849
• Add a new line to results.html by @rdavid in #1852

CI Updates

• Pass Github run ID into QE for collector by @sebrandon1 in #1862

• Github CI: Step to use /mnt for docker storage. by @greyerof in #1857

• Add functionality for running against local CRC instances by @sebrandon1 in #1843

• Fix cluster bootstrapping step. by @greyerof in #1837

• build(deps): bump nick-fields/retry from 2 to 3 by @dependabot in #1850

• build(deps): bump peter-evans/create-pull-request from 5 to 6 by @dependabot in #1851

• adjust collector's ci test to send claim file to prod collector by @shirmoran in #1829

• enable changing cnf suites timeout using env var by @shirmoran in #1846

• Prepare for v5.0.2 by @sebrandon1 in #1863

Full Changelog: v5.0.1...v5.0.2

v5.0.1

v5.0.1 is a patch-level update to bring improved resource allocation for the daemonset the test suite spawns as part of testing.

There are also a number of dependency updates.

Daemonset Resource Usage Improvements

• Adding configurable cpu requests/limit for the debug daemonset by @edcdavid in #1805

Logging Improvements

• log: review logs for autodiscover and provider packages by @jmontesi in #1811

CI Updates

• Add retries for 'make build-image-local' by @sebrandon1 in #1812
• Re-enable gradetool by @sebrandon1 in #1810
• Remove free disk space cleanup by @sebrandon1 in #1821

Dependency Updates

• Update preflight to v1.8.0 by @sebrandon1 in #1809
• build(deps): bump helm.sh/helm/v3 from 3.13.3 to 3.14.0 by @dependabot in #1816
• build(deps): bump k8s.io/kubectl from 0.29.0 to 0.29.1 by @dependabot in #1813
• build(deps): bump ubi8/ubi from 8.9-1107 to 8.9-1107.1705420509 by @dependabot in #1818
• build(deps): bump k8s.io/apiextensions-apiserver from 0.29.0 to 0.29.1 by @dependabot in #1817
• build(deps): bump ubi8/ubi-minimal from 8.9-1108 to 8.9-1108.1705420507 by @dependabot in #1819
• build(deps): bump github.com/test-network-function/privileged-daemonset from 1.0.19 to 1.0.20 by @dependabot in #1822
• build(deps): bump github.com/k8snetworkplumbingwg/network-attachment-definition-client from 1.4.0 to 1.5.0 by @dependabot in #1823
• Update images to ubi9/ubi9-minimal by @sebrandon1 in #1825

Full Changelog: v5.0.0...v5.0.1