Add AppArmor Support #8299 #18858

nnzv · 2024-05-12T01:47:25Z

Implemented AppArmor support for Minikube on both aarch64 and x86_64 architectures. Testing pending, relying on documentation for guidance. Excluded unnecessary extras:

BR2_PACKAGE_APPARMOR_PROFILES: Community profiles deemed non-essential for Minikube users.

linux-foundation-easycla · 2024-05-12T01:47:29Z

The committers listed above are authorized under a signed CLA.

✅ login: nnzv / name: Enzo Venturi (c66cc13, e64a71b, e109ed0, 494ee1d, e0c9cc8, 0a9728b, a6103c8)
✅ login: minikube-bot (16527f5, 3841b3d)

k8s-ci-robot · 2024-05-12T01:47:30Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: nnzv
Once this PR has been reviewed and has the lgtm label, please assign prezha for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2024-05-12T01:47:34Z

Welcome @nnzv!

It looks like this is your first PR to kubernetes/minikube 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/minikube has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2024-05-12T01:47:34Z

Hi @nnzv. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

minikube-bot · 2024-05-12T01:48:01Z

Can one of the admins verify this patch?

nnzv · 2024-05-12T02:38:22Z

I'm checking if the kernel image from Buildroot supports AppArmor. Following the docs, I tried linux-menuconfig in the Makefile, but it says there's nothing to be done. Am I overlooking any essential steps? Also, for an overview of the kernel modifications required, here's a summary of the necessary options:

General setup --->
    -*- Auditing support

Security options  --->
    -*- Enable the securityfs filesystem
    -*- Socket and Networking Security Hooks
    [*] Enable different security models
    [*] AppArmor support
          [*]   Enable introspection of sha1 hashes for loaded profiles
          [*]      Enable policy hash introspection by default
    [ ] Build AppArmor with debug code
          First legacy 'major LSM' to be initialized (AppArmor)  --->
          "yama,apparmor" Ordered List of enabled LSMs

nnzv · 2024-05-12T03:35:05Z

CC: @tstromberg

nnzv · 2024-05-13T05:00:33Z

I've gone through the entire image building process to test out the new environment, following the documentation closely. It instructs to use the target out/minikube-amd64.iso, but it failed near the end. I fixed it by switching "amd64" to "x86_64". If this change gets merged into the master branch, we might need to update the documentation. Check the logs, especially after I switched to the "x86_64" target. The original log file is massive, around 56MB of plain text.

...
make[1]: Leaving directory '/mnt/out/buildroot'
# x86_64 ISO is still BIOS rather than EFI because of AppArmor issues for KVM, and Gen 2 issues for Hyper-V
if [ "x86_64" = "aarch64" ]; then \
                mv /mnt/out/buildroot/output-aarch64/images/boot.iso /mnt/out/minikube-arm64.iso; \
        else \
                mv /mnt/out/buildroot/output-x86_64/images/rootfs.iso9660 /mnt/out/minikube-amd64.iso; \
        fi;
Please enter a valid architecture. Choices are x86_64 and aarch64.
make: *** [Makefile:959: deploy/iso/minikube-iso/board/minikube/amd64/rootfs-overlay/usr/bin/auto-pause] Error 1
make: *** [Makefile:333: out/minikube-amd64.iso] Error 2
...

Seems like the only thing left is to tweak the kernel, you know, enabling those configurations I mentioned earlier. Once that's done, the feature should be ready, although we still need to do some testing regarding the use of AppArmor at the Kubernetes level. I'll roughly leave the final commands I used and some evidence. Note that the last "cat" command results in N, it should be Y.

minikube start --driver=kvm2 --cni calico --kubernetes-version=1.29.0 --feature-gates=AppArmor=true --iso-url=file://$(pwd)/out/minikube-amd64.iso
minikube ssh

medyagh · 2024-05-14T21:19:21Z

@nnzv do you mind sharing what use cases this would help with ? are there any issues that users are already asking for this ?

nnzv · 2024-05-14T23:17:22Z

Do you mind sharing what use cases this would help with ? are there any issues that users are already asking for this ?

User @acoulon99 requested this feature some time ago via an issue (#8299). For detailed information on its use cases, you can refer to the Kubernetes documentation at https://kubernetes.io/docs/tutorials/security/apparmor, particularly the example of a profile that blocks all file write operations on nodes. It's surprising that Minikube hasn't incorporated this feature, considering that Kubernetes has supported AppArmor since version 1.14. Implementing it would be a logical next step.

medyagh · 2024-05-16T17:16:27Z

Do you mind sharing what use cases this would help with ? are there any issues that users are already asking for this ?

User @acoulon99 requested this feature some time ago via an issue (#8299). For detailed information on its use cases, you can refer to the Kubernetes documentation at https://kubernetes.io/docs/tutorials/security/apparmor, particularly the example of a profile that blocks all file write operations on nodes. It's surprising that Minikube hasn't incorporated this feature, considering that Kubernetes has supported AppArmor since version 1.14. Implementing it would be a logical next step.

thank you for sharing that thats a valid reason to add the support, in the past each time apparomor gets enabled it breaks a lot of other things... I curious how it would affect addons or other normal use cases

medyagh · 2024-05-16T17:16:51Z

ok-to-build-iso

medyagh · 2024-05-16T17:19:18Z

Do you mind sharing what use cases this would help with ? are there any issues that users are already asking for this ?

User @acoulon99 requested this feature some time ago via an issue (#8299). For detailed information on its use cases, you can refer to the Kubernetes documentation at https://kubernetes.io/docs/tutorials/security/apparmor, particularly the example of a profile that blocks all file write operations on nodes. It's surprising that Minikube hasn't incorporated this feature, considering that Kubernetes has supported AppArmor since version 1.14. Implementing it would be a logical next step.

if we I think we should have a flag to like --apparomor so if users face issues they could disable it.
I am gonna build the ISO on the PR just to see initial tests results

nnzv · 2024-05-16T18:37:19Z

If we I think we should have a flag to like --apparomor so if users face issues they could disable it. I am gonna build the ISO on the PR just to see initial tests results

You're welcome. Indeed, it does seem like a good idea. I wasn't aware that it was implemented long ago but disabled due to breaking changes. Regarding the new flag, I'm not sure how to dynamically ignore things by buildroot, specifically referring to the iso/minikube-iso/configs/* files depending on the architecture.

nnzv · 2024-05-16T18:39:17Z

Additionally, keep in mind that this is only partially implemented. The remaining task involves configuring the kernel. You can find details in one of my comments below. @medyagh

medyagh · 2024-05-17T18:40:47Z

yes I think there was issues with github action running with apparmor (for KIC - docker/podman drivers)
#7658

@nnzv the ISO build failed btw but not related to this PR (yet) the build failure is due to the changes in SHA by containernetworking/plugins#1038 that they unexpectedly built the binary and overwrote the previous ones

for enable disabling it I am thinking if there should be a linux command to turn it on or off after the ISO is built. so if users start minikube --apparmor=true it would be enabled in the kernel module. that way we would not have a lot of new headaches for breaking people's existing workflows

medyagh · 2024-05-20T17:33:04Z

@nnzv do you mind pulling upstream just merged #18896

medyagh · 2024-05-20T17:33:10Z

/ok-to-build-iso

nnzv · 2024-05-20T20:52:44Z

For enable disabling it I am thinking if there should be a linux command to turn it on or off after the ISO is built. so if users start minikube --apparmor=true it would be enabled in the kernel module. that way we would not have a lot of new headaches for breaking people's existing workflows.

Yeah, sounds like a modprobe. So, the AppArmor config needs to be <M> instead of <*>. I didn't check Minikube's code, how it handles flags, etc. But I guess it'll look something like this. Apologies for the simplistic code.

...
if (*WantAppArmorSupport) {
    sshtunnel("sudo modprobe apparmor")
    // Additional logic can go here for further validation
}
...

minikube-bot · 2024-05-20T23:42:39Z

Hi @nnzv, we have updated your PR with the reference to newly built ISO. Pull the changes locally if you want to test with them or update your PR further.

medyagh · 2024-05-22T18:23:33Z

/ok-to-test

medyagh · 2024-05-23T17:13:58Z

deploy/iso/minikube-iso/configs/minikube_aarch64_defconfig

@@ -3,6 +3,7 @@ BR2_aarch64=y

 # Toolchain
 BR2_TOOLCHAIN_BUILDROOT_WCHAR=y
+BR2_TOOLCHAIN_BUILDROOT_CXX=y


was enabling c++ libraries needed ?

In menuconfig, under Target Packages -> Security, you'll find that you can't enable AppArmor until you meet some prerequisites, like enabling C++.

medyagh · 2024-05-23T17:24:24Z

@nnzv the ISO has been built succesfully, I think could you please test it manually to see if it is enabled ?
a quick check I did seems like it is not enabled

$ mk ssh
                         _             _            
            _         _ ( )           ( )           
  ___ ___  (_)  ___  (_)| |/')  _   _ | |_      __  
/' _ ` _ `\| |/' _ `\| || , <  ( ) ( )| '_`\  /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )(  ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)

$ 
$ lsmod | grep apparmor
$ ps aux | grep apparmor
docker      3430  0.0  0.0   3112   408 pts/0    S+   17:34   0:00 grep apparmor
$ sudo journalctl -u apparmor
-- No entries --

and also how about we explore setting somethig like

BR2_TARGET_GENERIC_INITSCRIPT_APPARMOR=n

to prevent AppArmor from starting automatically at boot and then if the user wants it they can enable it

nnzv · 2024-05-23T19:47:21Z

and also how about we explore setting somethig like

BR2_TARGET_GENERIC_INITSCRIPT_APPARMOR=n

to prevent AppArmor from starting automatically at boot and then if the user wants it they can enable it

Is this config unique to Buildroot? Haven't found it. You can try apparmor_parser, but I haven't set up AppArmor in the kernel #18858 (comment). Let me commit that change. Sorry for the delay.

medyagh · 2024-05-24T20:35:11Z

@nnzv I didnt see any new change in your commit, if apparmor is added you should be able to verify it locally
(try running minikube start --iso-url) and provide local file path as the url to test it

medyagh · 2024-05-24T20:35:20Z

ok-to-build-iso

Apparmor configured for aarch64 and x86_64 platforms, requiring activation via kernel parameters (e.g., via grub.cfg). Debug settings enabled for future apparmor issue resolution.

Added apparmor utilities and extra utilities to configurations for both aarch64 and x86_64 platforms. Apparmor utilities such as aa-status are helpful for exploring this LSM, while additional utilities require Perl and other dependencies, hopefully already enabled.

nnzv

After researching, I found that apparmor can't be activated as a kernel module. So, with this setup, the kernel won't use apparmor LSM by default. To enable it dynamically, we may need to tweak kernel parameters in the grub.cfg. So, after setting up a node, we'd SSH in, adjust the grub.cfg, or generate it with grub-mkconfig after editing /etc/default/grub.

minikube ssh
# ...
vi /etc/default/grub
# ...
grep CMDLINE_LINUX /etc/default/grub
# GRUB_CMDLINE_LINUX="... apparmor=1 security=apparmor ..."
grub-mkconfig -o /boot/grub/grub.cfg
# ...

If for some reason Minikube doesn't handle the modification of the grub configuration, we can manually edit /boot/grub/grub.cfg or its equivalent on the node to add the required kernel parameters. Afterward, a reboot is required for the kernel to recognize the new LSM. Of course, this explanation is simply to outline the process; ideally, automation could be achieved using a boolean flag variable like --apparmor, as you mentioned.

nnzv · 2024-05-24T21:59:53Z

@nnzv I didnt see any new change in your commit, if apparmor is added you should be able to verify it locally (try running minikube start --iso-url) and provide local file path as the url to test it.

No worries. It's committed now. Since I don't have my ThinkPad (the one I used for development) with me, I'll wait for Minikube's CI to finish building the ISO. Once that's done, I'll test the POC I mentioned earlier, specifically about enabling apparmor with kernel parameters. Enabling it by default (adding CONFIG_DEFAULT_SECURITY_APPARMOR) in the kernel configuration could pose a problem. Take a look at this quote from a GitHub blog post: https://github.blog/2023-07-05-introduction-to-selinux/

Please note that some Linux distributions have different MAC systems enabled, and running multiple exclusive LSMs at the same time will cause issues. For example, Ubuntu has AppArmor enabled by default, therefore enabling SELinux while AppArmor is enabled will prevent you from booting properly. Please check include/linux/lsm_hooks.h for the LSM_FLAG_EXCLUSIVE to see if your LSM is exclusive.

minikube-bot · 2024-05-25T02:33:32Z

Hi @nnzv, we have updated your PR with the reference to newly built ISO. Pull the changes locally if you want to test with them or update your PR further.

minikube-pr-bot · 2024-05-25T03:29:12Z

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 18858) |
+----------------+----------+---------------------+
| minikube start | 47.9s    | 50.3s               |
| enable ingress | 25.3s    | 24.8s               |
+----------------+----------+---------------------+

Times for minikube ingress: 27.0s 27.5s 23.5s 24.0s 24.5s
Times for minikube (PR 18858) ingress: 24.9s 23.9s 27.0s 23.9s 24.4s

Times for minikube start: 49.4s 46.3s 46.1s 47.5s 50.4s
Times for minikube (PR 18858) start: 47.5s 49.6s 52.4s 51.3s 50.8s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 18858) |
+----------------+----------+---------------------+
| minikube start | 21.6s    | 22.6s               |
| enable ingress | 21.5s    | 21.7s               |
+----------------+----------+---------------------+

Times for minikube start: 21.1s 21.3s 21.9s 22.2s 21.5s
Times for minikube (PR 18858) start: 24.0s 20.9s 23.5s 23.6s 20.8s

Times for minikube ingress: 21.3s 21.8s 21.8s 21.8s 20.8s
Times for minikube (PR 18858) ingress: 22.3s 21.3s 21.8s 21.8s 21.3s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 18858) |
+----------------+----------+---------------------+
| minikube start | 21.3s    | 21.2s               |
| enable ingress | 35.1s    | 31.8s               |
+----------------+----------+---------------------+

Times for minikube start: 20.3s 20.3s 23.0s 22.4s 20.4s
Times for minikube (PR 18858) start: 20.0s 22.7s 19.9s 22.9s 20.3s

Times for minikube ingress: 31.8s 47.8s 32.3s 31.3s 32.3s
Times for minikube (PR 18858) ingress: 32.3s 31.3s 31.8s 32.3s 31.3s

nnzv · 2024-05-25T04:56:07Z

Hey @medyagh, could we give "ok-to-build-iso" another shot? I've checked the ISO created by the bot, but it seems my changes weren't taken into account (494ee1d and e109ed0). By the way, do you happen to know how we can tweak kernel parameters? I'm aware that each architecture in deploy/iso/minikube-iso/board has its own grub.cfg, but I couldn't find anything when I ran find / -name "grub.cfg" -type f from the root after SSHing into the minikube node.

minikube-pr-bot · 2024-05-25T17:25:32Z

These are the flake rates of all failed tests.

Environment	Failed Tests	Flake Rate (%)
Hyper-V_Windows	TestStartStop/group/no-preload/serial/DeployApp (gopogh)	n/a
KVM_Linux_crio	TestFunctional/serial/ComponentHealth (gopogh)	0.65 (chart)
Docker_Linux_containerd_arm64	TestAddons/parallel/InspektorGadget (gopogh)	2.55 (chart)
Docker_Linux_containerd_arm64	TestAddons/StoppedEnableDisable (gopogh)	2.55 (chart)
Docker_Linux_containerd_arm64	TestAddons/parallel/Volcano (gopogh)	3.85 (chart)
Hyper-V_Windows	TestCertExpiration (gopogh)	20.00 (chart)

To see the flake rates of all tests by environment, click here.

k8s-ci-robot requested review from afbjorklund and prezha May 12, 2024 01:47

k8s-ci-robot added the cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. label May 12, 2024

k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels May 12, 2024

nnzv force-pushed the patch-apparmor branch from 9f9526a to f0e57e7 Compare May 12, 2024 01:49

nnzv added 3 commits May 11, 2024 21:18

add support for apparmor

e0c9cc8

Add Support For Apparmor (x86_64) kubernetes#8299

a6103c8

Add Support For Apparmor (aarch64)

c66cc13

nnzv force-pushed the patch-apparmor branch from f0e57e7 to c66cc13 Compare May 12, 2024 02:20

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels May 12, 2024

Merge branch 'kubernetes:master' into patch-apparmor

e64a71b

Updating ISO to v1.33.1-1716226929-18858

16527f5

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels May 22, 2024

This comment has been minimized.

Sign in to view

medyagh reviewed May 23, 2024

View reviewed changes

Merge branch 'kubernetes:master' into patch-apparmor

0a9728b

This comment has been minimized.

Sign in to view

nnzv added 2 commits May 24, 2024 15:44

iso: add apparmor kernel config

494ee1d

Apparmor configured for aarch64 and x86_64 platforms, requiring activation via kernel parameters (e.g., via grub.cfg). Debug settings enabled for future apparmor issue resolution.

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels May 24, 2024

nnzv commented May 24, 2024

View reviewed changes

This comment has been minimized.

Sign in to view

Updating ISO to v1.33.1-1716583330-18858

3841b3d

This comment has been minimized.

Sign in to view

Add AppArmor Support #8299 #18858

Are you sure you want to change the base?

Add AppArmor Support #8299 #18858

Conversation

nnzv commented May 12, 2024 • edited

linux-foundation-easycla bot commented May 12, 2024 • edited

k8s-ci-robot commented May 12, 2024

k8s-ci-robot commented May 12, 2024

k8s-ci-robot commented May 12, 2024

minikube-bot commented May 12, 2024

nnzv commented May 12, 2024 • edited

nnzv commented May 12, 2024

nnzv commented May 13, 2024 • edited

medyagh commented May 14, 2024

nnzv commented May 14, 2024

medyagh commented May 16, 2024

medyagh commented May 16, 2024

medyagh commented May 16, 2024 • edited

nnzv commented May 16, 2024

nnzv commented May 16, 2024 • edited

medyagh commented May 17, 2024

medyagh commented May 20, 2024

medyagh commented May 20, 2024

nnzv commented May 20, 2024

minikube-bot commented May 20, 2024

medyagh commented May 22, 2024

This comment has been minimized.

This comment has been minimized.

medyagh May 23, 2024

Choose a reason for hiding this comment

nnzv May 23, 2024

Choose a reason for hiding this comment

medyagh commented May 23, 2024 • edited

nnzv commented May 23, 2024 • edited

This comment has been minimized.

This comment has been minimized.

medyagh commented May 24, 2024

medyagh commented May 24, 2024

nnzv left a comment • edited

Choose a reason for hiding this comment

nnzv commented May 24, 2024 • edited

This comment has been minimized.

minikube-bot commented May 25, 2024

minikube-pr-bot commented May 25, 2024

nnzv commented May 25, 2024

This comment has been minimized.

minikube-pr-bot commented May 25, 2024

nnzv commented May 12, 2024 •

edited

linux-foundation-easycla bot commented May 12, 2024 •

edited

nnzv commented May 12, 2024 •

edited

nnzv commented May 13, 2024 •

edited

medyagh commented May 16, 2024 •

edited

nnzv commented May 16, 2024 •

edited

medyagh commented May 23, 2024 •

edited

nnzv commented May 23, 2024 •

edited

nnzv left a comment •

edited

nnzv commented May 24, 2024 •

edited