v1.23.3 to 1.23.4 Upgrade Guide
This guide describes the steps to follow to upgrade the Kubernetes Fury Distribution (KFD) from v1.23.3 to 1.23.4.
If you are running a custom set of modules, or different versions than the ones included with each release of KFD, please refer to each module's release notes.
Notice that the guide will not cover changes related to the cloud provider, ingresses or pod placement changes. Only changes related to KFD and its modules.
âī¸ IMPORTANT we strongly recommend reading the whole guide before starting the upgrade process to identify possible blockers.
â ī¸ WARNING the upgrade process involves downtime of some components.
Upgrade procedureâ
As a high-level overview, the upgrade procedure consists on:
- Upgrading KFD (all the core modules).
- Upgrading the Kubernetes cluster itself.
1. Upgrade KFDâ
The suggested approach to upgrade the distribution is to do it one module at a time, to reduce the risk of errors and to make the process more manageable.
Networking module upgradeâ
To upgrade the Networking module to the new version, update the version on the Furyfile.yml
file to the new version:
versions:
networking: v1.12.2
...
Then, download the new modules with furyctl
with the following command:
furyctl vendor -H
Apply your Kustomize project that uses Networking module packages as bases with:
âī¸ IMPORTANT you may want to limit the scope of the command to only the networking module, otherwise, the first time you apply with
--server-side
other pods may also be restarted.The same applies to the rest of the modules, we will not include this warning in every step for simplicity.
kustomize build <your-project-path/networking> | kubectl apply -f - --server-side
Wait until all Calico pods are restarted and running. You can check Calico's Grafana dashboard "General / Felix Dashboard (Calico)" and the "Networking / *" dashboards to make sure everything is working as expected.
Monitoring module upgradeâ
â ī¸ WARNING downtime for the Monitoring stack is expected during this process.
To upgrade the Monitoring module to the new version, update the version on the Furyfile.yml
file to the new version:
versions:
...
monitoring: v2.1.0
...
Then, download the new modules with furyctl
with the following command:
furyctl vendor -H
Then apply your Kustomize project that uses Monitoring module packages as bases with:
kustomize build <your-project-path/monitoring> | kubectl apply -f - --server-side
Wait a minute and check that you can see metrics in Grafana, both old and new, check that all Prometheus Targets are up and that Alertmanager is working as expected.
If using Thanosâ
â ī¸ WARNING this will cause downtime in Thanos components, some metrics and alerts could be missed while performing the upgrade.
- Since the release ships made changes in the architecture of
Thanos
, the upgrade process needs the deletion of all resources from the old version of the module.
To upgrade thanos
core module from v0.24.0
to v0.30.2
, execute the following:
- Delete the old deployment, for example if you were using thanos-with-store:
kustomize build katalog/thanos/thanos-with-store -n monitoring | kubectl delete -f -
- Add the right base to your kustomize, for example, Thanos using MinIO as storage and pointing to a single Prometheus:
resources:
...
- "../../vendor/katalog/monitoring/thanos/thanos-minio/single"
Note:
prometheus-operator
andprometheus-operated
are already included inside thanos package as base
- Finally, apply the new deployment
Logging module upgradeâ
â ī¸ WARNING downtime of the Logging stack is expected during this process.
To upgrade the Logging module to the new version, update the version on the Furyfile.yml
file to the new version:
versions:
...
logging: v3.1.3
...
Then, download the new modules with furyctl
with the following command:
furyctl vendor -H
Since this upgrade changes the major version, there are some manual steps involving breaking changes that you need to do before applying the project:
Remove the old minio
stack:
kubectl delete sts minio -n logging
Apply your Kustomize project that uses Logging module packages as bases with:
kustomize build <your-project-path/logging> | kubectl apply -f - --server-side
Ingress module upgradeâ
â ī¸ WARNING some downtime of the NGINX Ingress Controller is expected during the upgrade process.
To upgrade the Ingress module to the new version, update the version on the Furyfile.yml
file to the new version:
versions:
...
ingress: v1.14.1
...
Then, download the new modules with furyctl
with the following command:
furyctl vendor -H
Apply your Kustomize project that uses Ingress module packages as bases with:
kustomize build <your-project-path/ingress> | kubectl apply -f - --server-side
âšī¸ INFO you may need to apply twice or thrice because a new Validating webhook is added with this release and it needs some time to come up.
Disaster Recovery module upgradeâ
To upgrade the Disaster Recovery module to the new version, update the version on the Furyfile.yml
file to the new version:
versions:
...
dr: v1.11.0
...
Then, download the new modules with furyctl
with the following command:
furyctl vendor -H
Delete the old velero-restic
DaemonSet, since it has been replaced with a new DaemonSet node-agent
if it's in use with:
kubectl delete ds velero-restic -n kube-system
kubectl delete job minio-setup -n kube-system
Apply your Kustomize project that uses Disaster Recovery module packages as bases with:
kustomize build <your-project-path/dr> | kubectl apply -f - --server-side
Check that all velero's pods are up and running, you may want to manually trigger a backup to test that everything is working as expected. For example:
# create a backup
velero backup create --from-schedule manifests test-upgrade -n kube-system
# ... wait a moment
# check that Phase is completed
velero get backup -n kube-system test-upgrade
# you may want to see some details
velero backup describe test-upgrade -n kube-system
đĄ TIP you can use a port-forward minio'UI and login to check that the backups are there.
OPA module upgradeâ
To upgrade the OPA module to the new version, update the version on the Furyfile.yml
file to the new version:
versions:
...
opa: v1.8.0
...
Then, download the new modules with furyctl
with the following command:
furyctl vendor -H
Apply your Kustomize project that uses OPA module packages as bases with:
kustomize build <your-project-path/opa> | kubectl apply -f - --server-side
You can try to deploy a pod that is not compliant with the rules deployed in the cluster and also check in Gatekeeper Policy Manager for new violations of the constraints.
âšī¸ INFO seeing errors like
http: TLS handshake error from 172.16.0.3:42672: EOF
in Gatekeeper Controller logs is normal. The error is considered harmless. See Gatekeeper's issue #2142 for reference.
Auth module upgradeâ
To upgrade the Auth module to the new version, update the version on the Furyfile.yml
file to the new version:
If you were using these components, adjust your Kustomize project to use the new auth
module as a base:
versions:
...
auth: v0.0.3
...
Then, download the new modules with furyctl
with the following command:
furyctl vendor -H
To upgrade this module from v0.0.2
to v0.0.3
, you need to download this new version and do the following changes before applying the kustomize
project:
- Edit your policy configuration file to use
routes
instead ofpolicy
, for example, from:
policy:
# from and to should be set to the prometheus ingress
- from: https://prometheus.example.com
to: https://prometheus.example.com
allowed_idp_claims:
groups:
# ldap groups configured in dex
- group1
- group2
to:
routes:
- from: https://prometheus.example.com
to: https://prometheus.monitoring.svc # notice the internal service. See (2.) below.
policy:
- allow:
or:
- claim/groups: group1
- claim/groups: group2
# - email:
# is: someone@sighup.io
- Forward auth mode has been deprecated by Pomerium in v0.21.0 (the one included in this release), you will need to switch to proxy auth.
If you were using forward auth with annotations in your ingresses, you will need to adjust them.
For example, if you had an ingress for Grafana in the monitoring
namespace with the following annotations:
nginx.ingress.kubernetes.io/auth-url: "https://pomerium.example.com/verify?uri=$scheme://$host$request_uri"
nginx.ingress.kubernetes.io/auth-signin: "https://pomerium.example.com/?uri=$scheme://$host$request_uri"
You will have to:
2.1. Create a new ingress in the pomerium
namespace with the hostname for Grafana using the pomerium
service and the http
as backend configuration.
2.2. In the policy definition file, make sure that the from
field matches the hostname and that the to
field points to grafana's service (the same one than the ingress in the monitoring namespace).
2.3. Delete the Ingress in the monitoring
namespace.
2.4 Repeat for the other ingresses.
Apply your Kustomize project that uses Auth module packages as bases with:
kustomize build <your-project-path/auth> | kubectl apply -f - --server-side
đ CONGRATULATIONS you have now successfully updated all the core modules to KFD 1.23.4