deployKF - 0.1.5

Helpful Links

• Getting Started

• Full Changelog
• Upgrade Guide

Upgrade Notes

• We strongly recommend updating to this version for security reasons.
• As always, if your deployKF platform is critical to your organization, you should test this upgrade in a non-production cluster. We have done extensive testing, but you could encounter unexpected issues.
• Please update your sync_argocd_apps.sh script version BEFORE SYNCING 0.1.5.
• The sample values for 0.1.4 had some values that will conflict with 0.1.5, YOU MUST REMOVE THEM from your custom values when upgrading:
  • kubeflow_tools.pipelines.kfpV2.defaultPipelineRoot
  • kubeflow_tools.pipelines.kfpV2.minioFix
  • kubeflow_tools.pipelines.kfpV2.launcherImage
• We have updated the default embedded Istio version to 1.17.8. To make the process of updating the sidecar images easier, we now provide the update_istio_sidecars.sh script to restart pods with incorrect istio sidecar container versions. Warning, running this script will cause DISRUPTION, especially to Notebooks, so ensure your users have saved their work!
• If you have followed the Air-Gapped Clusters guide, you must mirror the new images/charts used in 0.1.5, and update the corresponding values. For an overview of which images have changed, see the diff of default_values.yaml from 0.1.4 to 0.1.5. Warning, DO NOT continue using the old images/charts with 0.1.5, as this will not work.

Important Notes

• deployKF Dashboard:
  • We have grouped the sidebar links from Kubeflow Pipelines into their own section.
  • Users can now see their profiles with "view" and "edit" access on the "Manage Contributors" page.
• Kubeflow Pipelines:
  • This release includes a patched version of Kubeflow Pipelines 2.1.0 which is specially designed to be backward compatible with all V1 and V2-compatible pipelines.
  • When doing an in-place upgrade, you will not automatically have the V2 tutorial pipelines added to your cluster. You may want to upload them manually as "shared" pipeline definitions (find the YAML files attached to the GitHub release).
• Kubeflow Notebooks:
  • We have updated the default Kubeflow Notebooks images to the ones shipped with upstream 1.8.0, these provide significant version bumps for all packages, including TensorFlow 2.13.0 and PyTorch 2.1.0. These images will be updated further in the next release.
  • Please note, we have only updated the DEFAULT IMAGES, which will not affect any existing notebooks. To update existing Notebooks, you must delete and recreate them (data stored in the home directory PVC will be persisted, and you can re-attach it to the new notebooks).
• ARM Support:
  • For those waiting on full ARM64 support, there are now only two remaining components preventing this. The Kubeflow Notebooks backend (which will be updated in the next release), and Kubeflow Pipelines (which needs some help upstream, see kubeflow/pipelines#10309 to help).
• Istio:
  • While the default version of Istio (1.17.8) is very old, you can easily update to a newer version that is supported by deployKF by updating the deploykf_dependencies.istio.charts and deploykf_core.deploykf_istio_gateway.charts.istioGateway values.
  • We provide the update_istio_sidecars.sh script to restart pods with incorrect Istio sidecar container versions. Warning, running this script will cause DISRUPTION, especially to Notebooks, so ensure your users have saved their work!
  • In the next minor release, we will do a significant update to the default Istio version, and drop out-of-the-box support for very old Kubernetes versions.
• Kyverno:
  • We still have a hard dependency on Kyverno 1.10.0 due to issues upstream. Hopefully, this will change in the next deployKF version as we test and implement support for the recently released Kyverno 1.12 (which is NOT supported in deployKF 0.1.5).
  • This means that you are still unable to bring your own Kyverno deployment (unless it happens to be the 1.10.0 version). Once this is not the case, we will release a proper "use existing Kyverno" guide like we have for Istio.

What's Changed

Significant Changes

• docs: add update_istio_sidecars.sh script by @thesuperzapper in #132
• feat: update to Kubeflow Pipelines 2.1.0 by @thesuperzapper in #122
• feat: update dashboard to 0.1.1 + update sidebar links by @thesuperzapper in #163
• feat: update default notebook images to 1.8.0 by @thesuperzapper in #164

New Features

• feat: update oauth2-proxy to 7.6.0 by @thesuperzapper in #152
• feat: update cert-manager to 1.12.10 by @thesuperzapper in #153
• feat: update dex to 2.39.1 by @thesuperzapper in #155
• feat: update kubectl container to 1.26.15 by @thesuperzapper in #156
• feat: update default istio to 1.17.8 by @thesuperzapper in #157
• feat: update default minio to RELEASE.2024-05-10T01-41-38Z by @thesuperzapper in #158
• feat: update default mysql to 8.0.37 by @thesuperzapper in #159
• feat: update profile-controller and kfam to 1.8.0 by @thesuperzapper in #162
• feat: update trust-manager to 0.9.2 by @thesuperzapper in #154

Improvements

• improve: support argocd.appNamePrefix in argocd sync script by @thesuperzapper in #108
• improve: add robots.txt to deny all user-agents by @thesuperzapper in #106

Bug Fixes

• fix: argocd sync script only seeing first app in each group by @thesuperzapper in #109
• fix: require pruning in sync script by @thesuperzapper in #123
• fix: require bash 4.4+ for sync script by @thesuperzapper in #126
• fix: script should sync apps that failed their last sync by @thesuperzapper in #151
• fix: minio not starting, upstream removed curl by @thesuperzapper in #165
• fix: stop embedded mysql log spam about mysql_native_password by @thesuperzapper in #167

Documentation

• docs: update default argocd to 2.10.4 by @thesuperzapper in #114
• docs: add argocd helm example for plugin by @thesuperzapper in #121
• docs: remove confusing sample values by @thesuperzapper in #160
• docs: update default argocd to 2.10.11 by @thesuperzapper in #166

Full Changelog: v0.1.4...v0.1.5

deployKF - 0.1.4

Helpful Links

• Getting Started

• Full Changelog
• Upgrade Guide

Upgrade Notes

• There will be some downtime for Kubeflow Pipelines and users will be forced to re-authenticate.
• You MUST sync with pruning enabled, as we have changed a number of resources.
• If you are using our automated ArgoCD Sync Script:
  • Update to the latest script version, found in the main branch.
  • Ensure you respond "yes" to all "Do you want to sync with PRUNING enabled?" prompts.
  • To prevent the need to sync twice, please manually delete this ClusterPolicy using the following command BEFORE syncing: kubectl delete clusterpolicy "kubeflow-pipelines--generate-profile-resources"
  • (otherwise, the first sync will time-out waiting for kf-tools--pipelines to be healthy)

Important Notes

• We no longer use Kyverno to generate resources in each profile for Kubeflow Pipelines, we now include these resources directly based on your profile values, this is due to Kyverno not scaling well for large numbers of profiles. However, we still use Kyverno for cloning Secrets across namespaces, triggering restarts of Deployments, and a few other things.
• We have resolved the compatibility issues with Azure AKS. To enable the Azure-specific fixes, please set the kubernetes.azure.admissionsEnforcerFix value to true.
• There have been significant changes to how authentication is implemented. These changes should allow you to bring your own Istio Gateway Deployment (Pods) without having other services end up behind deployKF's authentication system. However, please note that deployKF still manages its own Gateway Resource (CRD).
• For those experiencing "route not found" issues when using an external proxy to terminate TLS, you can now disable "SNI Matching" on the Istio Gateway by setting the deploykf_core.deploykf_istio_gateway.gateway.tls.matchSNI value to false.

What's Changed

Significant Changes

• feat: allow other istio gateways on ingress deployment by @thesuperzapper in #66
• feat: allow disabling SNI matching on gateway by @thesuperzapper in #83
• fix: issues preventing deployment on Azure AKS by @thesuperzapper in #85
• improve: stop using kyverno to provision kfp profile resources by @thesuperzapper in #102

New Features

• feat: disable default plugins and resource-quotas in specific profiles by @thesuperzapper in #67
• feat: allow custom external service ports by @thesuperzapper in #82
• feat: allow disabling HTTPS redirect by @thesuperzapper in #86
• feat: add pod-labels value for cert-manager controller by @thesuperzapper in #88
• feat: optional sign-in page to stop background request CSRF accumulation by @thesuperzapper in #100

Improvements

• improve: use __Secure- cookie prefix and remove domains config by @thesuperzapper in #87
• improve: increase kyverno resource limits and add values by @thesuperzapper in #93
• improve: use CRD-level "replace" for kyverno ArgoCD app by @thesuperzapper in #94
• improve: argocd sync script should only wait for app health once by @thesuperzapper in #104

Bug Fixes

• fix: prevent kyverno log spam on missing generate context by @thesuperzapper in #54
• fix: rstudio logo format for non-chrome browsers by @thesuperzapper in #56
• fix: using AWS IRSA with Kubeflow Pipelines by @thesuperzapper in #79
• fix: use 307 status for HTTP redirects by @thesuperzapper in #81
• fix: proxy protocol envoyfilter for istio gateway by @thesuperzapper in #80
• fix: disallow out-of-band KFP audience when disabled by @thesuperzapper in #89
• fix: support kyverno chart changes (but keep kyverno version) by @thesuperzapper in #92
• fix: annotate cloned imagePullSecrets to be ignored by ArgoCD by @dkhachyan in #90
• fix: add background filter to restart trigger policies by @thesuperzapper in #95
• fix: prevent CSRF cookie accumulation on auth expiry by @thesuperzapper in #99

Documentation

• docs: update example ArgoCD to 2.9.6 by @thesuperzapper in #91

New Contributors

• @dkhachyan made their first contribution in #90

Full Changelog: v0.1.3...v0.1.4

deployKF - 0.1.3

Helpful Links

• Getting Started

• Full Changelog
• Upgrade Guide

Important Notes

• For more information about using the new "browser login flow" with Kubeflow Pipelines SDK, please see the updated Access Kubeflow Pipelines API guide.

What's Changed

Significant Changes

• feat: browser-based KFP SDK auth by @thesuperzapper in #45

New Features

• feat: update oauth2-proxy to 7.5.1 by @thesuperzapper in #44
• feat: kyverno policy for image-pull-secrets by @thesuperzapper in #47
• feat: add values for kyverno replicas by @thesuperzapper in #50

Improvements

• improve: limit trigger operations for kyverno policies by @thesuperzapper in #49

Bug Fixes

• fix: don't mount trust bundles with own cert-manager by @thesuperzapper in #46
• fix: ensure kyverno has permission to manage PodDefaults by @thesuperzapper in #51

Documentation

• docs: update sync script to force update kyverno policies by @thesuperzapper in #40
• docs: add requirement checks to argocd sync script by @thesuperzapper in #42
• docs: update reference argocd version to 2.8.5 by @thesuperzapper in #52

Miscellaneous

• refactor: always use v1 kyverno resources by @thesuperzapper in #48

Full Changelog: v0.1.2...v0.1.3

deployKF - 0.1.2

Helpful Links

• Getting Started

• Full Changelog
• Upgrade Guide

Important Notes

• If you are using the deployKF ArgoCD Plugin, you MUST update to the latest version of the plugin BEFORE upgrading to this version (see: #29).

What's Changed

Significant Changes

• docs: add reference sync_argocd_apps.sh script by @thesuperzapper in #38

Bug Fixes

• fix: set kyverno webhook failure policy to ignore (fix uninstall deadlock) by @thesuperzapper in #26
• fix: resolve cert-manager race conditions by @thesuperzapper in #28
• fix: argocd plugin with "file://" dependencies (needed for helm forks) by @thesuperzapper in #29
• fix: create separate namespaces app, if destination is remote by @thesuperzapper in #30
• fix: ensure namespaces are never deleted or pruned by @thesuperzapper in #31
• fix: add sync waves to argocd apps (fix deletion) by @thesuperzapper in #32
• fix: resolve profile generator race condition by @thesuperzapper in #33
• fix: resolve race conditions with cloned secrets by @thesuperzapper in #34
• fix: app-of-apps should always target argocd cluster by @thesuperzapper in #35

Documentation

• docs: move guides to website by @thesuperzapper in #20
• docs: improve example app-of-apps for plugin by @thesuperzapper in #37
• docs: improve sample values, add reference overrides by @thesuperzapper in #36

Full Changelog: v0.1.1...v0.1.2

deployKF - 0.1.1

Helpful Links

• Getting Started

• Full Changelog

What's Changed

Significant Changes

• feat: create argocd plugin by @thesuperzapper in #16

New Features

• feat: allow custom documentation links in dashboard by @yankcrime in #12
• feat: allow a single ArgoCD to manage deployKF across multiple clusters by @thesuperzapper in #17

Bug Fixes

• fix: set securityContext.fsGroup on minio pods by @thesuperzapper in #14
• fix: minio-console user permissions (update minio) by @thesuperzapper in #18

Documentation

• docs: improve getting started formatting by @thesuperzapper in #8
• docs: add links to important values in readme by @thesuperzapper in #9
• docs: improve getting started guide by @thesuperzapper in #11
• docs: add link to youtube demo by @thesuperzapper in #13

New Contributors

• @yankcrime made their first contribution in #12

Full Changelog: v0.1.0...v0.1.1

deployKF - 0.1.0

Helpful Links

• Getting Started

• Full Changelog
• Upgrade Guide

What's Changed

Significant Changes

• initial release 🎉 🎉 🎉

Full Changelog: https://github.com/deployKF/deployKF/commits/v0.1.0