Guest Agent for Cloud Hypervisor

[weltling]

TL;DR Guest agents are deployed into a provisioned VM to execute OS specific tasks like managing the systems, hardware or users, retrieving system info, etc. There are guest agent implementations existing for QEMU, VMware or in a cloud like Azure. There is currently no guest agent for Cloud Hypervisor and this thread is about discussing a possible guest agent approach.

Introduction

Guest Agent is a help service running in a guest VM to facilitate communication between host and guest. The communication takes place through a predetermined to exchange information and execute commands on the guest. This mechanism allows for advanced guest management including pervasive system configuration like fine tuning devices, editing users, creating filesystem snapshots, etc.

Specific to already existing guest agent implementations is the integration with VM orchestration tools. For example, the QEMU guest agent is integrated with libvirt and virsh is the corresponding tool on host to perform the communication. Given there is currently no upstreamed libvirt integration for Cloud Hypervisor, still keeping it in mind and having it as a good example seems a good flavor. Having the outright libvirt/QEMU implementation as a mental picture, I’d like to discuss a more scoped approach for the Cloud Hypervisor guest agent.

Why?

Guest Agent is nowadays a standard feature in many OSS and commercial virtualization stacks. Along with similar efforts already going like for kata containers integration, the libvirt integration efforts already started in the Cloud Hypervisor specific fork, there is no reason to let neglect an opportunity for targeting an improvement in the guest VM management as in the form of guest agent.

Considerations

Communication Transport

There are several transports thinkable. VSOCK is already available on kernels >= 5.5, so that might be a choice for an easy start. QEMU seems to use a dedicated virtio-serial device that is projected into the guest under /devl/vport…. As it’s virtio based, the performance should be good enough, however Cloud Hypervisor might need some extra implementation to provide the same.

Bidirectional Channel

While in most case the question is to control the guest, it is thinkable to also do the opposite. Connections can be initiated from the guest to host to send messages concerning important events taking place in the guest. Like, for example, the guest might notify it’s going to reboot in 3 minutes so the host can take an action on that. I currently don’t see this kind of handling in the existing implementations for libvirt.

Host Tooling

In first place, it would depend on the transport choices, but in general there would seem to be a dedicated host client needed to perform the communication. Like what in the case of existing QEMU implementation, virsh is what comes as the host client tool.

Libvirt/QEMU Compatibility

It seems that a guest agent should be in first place developed to meet the nedes in Cloud Hypervisor. It is of course possible to implement a redundant compatibility layer to provide the compatibility with other host tooling solutions.

Choices to be made

Option	Choices
Programming language	Rust (or Python, or C++, …)
Communication data format	JSON (XML, any other)?
Communication protocol	To be defined, using Open API to formalize?
Runtime daemon supervision	systemd, windows service?
Communication transport	Virtio-serial, vsock (the easiest to be chosen)
Managing code	Integrate into cloud-hypervisor.git like ch-remote, or maybe a separate repo?

Refs

• https://wiki.qemu.org/Features/GuestAgent
• https://wiki.libvirt.org/Qemu_guest_agent.html
• https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/virtualization_deployment_and_administration_guide/chap-qemu_guest_agent

[cryptoknight]

Seems like the most straightforward approach would just be to use/support the QEMU guest agent. It could be extended to support vsock if desired, and it looks like there's already a QAPI crate for managing the protocol.

Also there is an upstream libvirt driver for CH, but it has bit-rotted and probably doesn't work for recent versions.

[weltling]

@cryptoknight, thanks for the hint on the QAPI implementation. Reusing already implemented crates is a very appealing idea. Just for the docs here, it is https://github.com/arcnmx/qapi-rs .

While using the QEMU agent as is is doable, the point in this thread is actually to suggest a CH own guest agent. There are obvious advantages there, too. CH specific tasks can be integrated, the life cycle wouldn't need to depend on QEMU, choices can be made that are more suitable for Cloud Hypervisor. That's why I was also talking about the CH own guest agent in first place, and then about the QEMU compatibility as a secondary option.

With regard to the libvirt integration, another part that probably should be targeted is the MSHV as a domain, too.

Thanks

[blakegao]

I have some experience with this feature, for your information only.

• virtio-vsock is undoubtedly a better choice than virtio-serial, but there are newer version requirements for kernel, and virtio driver of windows guest does not support vsock, so this is why qemu-guest-agent does not enable vsock by default.
• Initiating a connection from the guest-agent is a good start, assuming that guest-agent will also periodically report health status or heartbeat, so that the management task data sent by host will not be discarded or the buffer will not be filled.
• Another significant disadvantage of serial (chardev) is that it does not have stream semantics, and it needs special treatment when implementing the application layer protocol. A simple solution is to use flag byte(e.g. 0xFF, 01111110) as the start and end delimiter, guest-agent can choose to discard or retransmit when incomplete data is identified.

[weltling]

Thank you for these points! Indeed it looks like for Windows guest the only option today is virtio-serial. I see also some driver code present here, but it's not a part of the official driver ISO (yet?).

It looks like after all, the serial option would be still the most portable nowadays. A vsock one could be a good quick starter to focus on the the base agent structure and prototype implementation while using Linux as guest. The QEMU guest agent also supports multiple transports, so seems supporting multiple transports right by design wolud be a suitable approach here, too.

Thanks

[xietuo]

Kuasar is a recently released project that provides native support for cloud-hypervisor. The project includes a vmm-task component as a daemon process within the guest, serving a similar role to the Guest Agent. Communication between the vmm-task and the host is using vsock.

[weltling]

It's a good example indeed, thanks for bringing this up. It looks also like besides the transport implementation, there are parts to learn from like communication protocol, components used and cases covered. It seems, that Kuasar would rather focus on a container scenarios, so the implementation seems specifically geared. In any case, it is a good source of learning that should not be missed.

Thanks!

[weltling]

An update here. In the scope of an intern project this summer, there's a guest agent prototype implementation based on QMP, uses VSOCK and written in Python. There are several commands implemented including provisioning a user and setting up network. OFC there are still some open questions and we'd plan to continue on exploring this field. Below is the mentioned guest agent prototype, any comments are appreciated:

https://github.com/kasanchez519/ch-guest-agent/

/cc @kasanchez519

Thanks