GDB support on ARM64 KVM

[liuw]

The support for x86_64 was merged some time ago. There is a Debuggable trait. It is only implemented for x86_64.

My first thought on this is there is no functional translate_gva call on ARM64. I'm not sure if it should be implemented in kernel or in CH. Implementing a page walker is error-prone. Doing it in kernel benefits not just Cloud Hypervisor but other projects too. On the other hand, implementing it in kernel means the host has to get a new kernel or backport the feature. That's not a good idea in conservative production environments.

The rest are just reading and writing registers and memory which seem to be rather straightforward.

CC @yangxile @MrXinWang @michael2012z

[yangxile]

CC @jongwu

[michael2012z]

Another gap is in gdbstub crate, where the support of AArch64 is still missing. But it seems a small gap, we only need to add the definition of core registers and implement the Arch trait.

[michael2012z]

@liuw Is anybody working on this issue now? Recently I began to look into this topic. If nobody is on this yet, I will continue to try.

[liuw]

@liuw Is anybody working on this issue now? Recently I began to look into this topic. If nobody is on this yet, I will continue to try.

No one is looking into this yet. The team is occupied by another project. Thanks for looking into it.

[michael2012z]

Here comes the list of gaps to fill (also a plan) to support GDB on Arm64:

• AArch64 support in gdbstub_arch: Add the core registers definition and implement gdbstub::arch::Arch
• kvm-ioctls bugfix: The target_arch name for AArch64 need to be corrected for function set_guest_debug
• Break Points: On AArch64 only software break points are supported. KVM_GUESTDBG_USE_HW_BP flag for API KVM_SET_GUEST_DEBUG is not available on ARM.
• GVA->GPA translation: API KVM_TRANSLATE is not available on AArch64. This is the biggest gap to fill. I prefer to walk through guest page tables to do the translation "manually". Going to try it. Will focus on armv8-a case only.
• Some refactoring works in hypervisor crate: X86 uses KVM_GET/SET_REGS for that, but they are not for AArch64. Instead we use KVM_GET/SET_ONE_REG to handle each register one by one. We can hide the difference in hypervisor crate and make unified interfaces for "setting/getting core registers" to vmm.

[russell-islam]

Here comes the list of gaps to fill (also a plan) to support GDB on Arm64:

* AArch64 support in `gdbstub_arch`: Add the core registers definition and implement `gdbstub::arch::Arch`

* `kvm-ioctls` bugfix: The `target_arch` name for AArch64 need to be corrected for function `set_guest_debug`

* Break Points: On AArch64 only software break points are supported. `KVM_GUESTDBG_USE_HW_BP` flag for API `KVM_SET_GUEST_DEBUG` is not available on ARM.

* GVA->GPA translation: API `KVM_TRANSLATE` is not available on AArch64. This is the biggest gap to fill. I prefer to walk through guest page tables to do the translation "manually". Going to try it. Will focus on armv8-a case only.

* Some refactoring works in `hypervisor` crate: X86 uses `KVM_GET/SET_REGS` for that, but they are not for AArch64. Instead we use `KVM_GET/SET_ONE_REG` to handle each register one by one. We can hide the difference in `hypervisor` crate and make unified interfaces for "setting/getting core registers" to `vmm`.

Seems like a good plan. Initially we could focus minimal implementation and we could improve gradually.

[michael2012z]

Thanks for reminding, I almost forgot to update the progress on this issue:
(I am working on prototype in my fork for this: https://github.com/michael2012z/cloud-hypervisor/tree/aarch64-gdb)

• [Prototype ready] AArch64 support in gdbstub_arch: Add the core registers definition and implement gdbstub::arch::Arch
• [Upstreamed by @MrXinWang ] kvm-ioctls bugfix: The target_arch name for AArch64 need to be corrected for function set_guest_debug
• [Prototype ready] Break Points: On AArch64 only software break points are supported. KVM_GUESTDBG_USE_HW_BP flag for API KVM_SET_GUEST_DEBUG is not available on ARM.
• [Now I am on the coding] GVA->GPA translation: API KVM_TRANSLATE is not available on AArch64. This is the biggest gap to fill. I prefer to walk through guest page tables to do the translation "manually". Going to try it. Will focus on armv8-a case only.
• [Done, going to upstream this part at first soon] Some refactoring works in hypervisor crate: X86 uses KVM_GET/SET_REGS for that, but they are not for AArch64. Instead we use KVM_GET/SET_ONE_REG to handle each register one by one. We can hide the difference in hypervisor crate and make unified interfaces for "setting/getting core registers" to vmm.

[keiichiw]

Just FYI, @ptosi is adding ARM64 GDB support to crosvm at https://crrev.com/c/3785467/ with a gdbstub_arch PR.
Since some code for CH's x64 GDB support (#3575) came from crosvm, I thought it might be somehow helpful to CH folks. Please take a look and give some comments if you're interested.

[liuw]

Just FYI, @ptosi is adding ARM64 GDB support to crosvm at https://crrev.com/c/3785467/ with a gdbstub_arch PR. Since some code for CH's x64 GDB support (#3575) came from crosvm, I thought it might be somehow helpful to CH folks. Please take a look and give some comments if you're interested.

Thanks for the heads-up. Are you looking for review on the crosvm changes also? Or are you looking for review on only the gdbstub_arch crate?

[keiichiw]

Are you looking for review on the crosvm changes also? Or are you looking for review on only the gdbstub_arch crate?

I thought it'd be nice if CH folks could take a look at gdbstub_arch to make sure that the change would work well with the CH too. And, I just realized that @ptosi is talking about the same thing at #4355 (comment) :)

[rbradford]

@michael2012z Has some work at https://github.com/michael2012z/cloud-hypervisor/tree/aarch64-gdb but is waiting for aarch64 gdbstub_arch support to land!

[MarkGass]

Would this explain why I am seeing various crashes using gdb's Next and Step commands with an Ubuntu ARM64 VM on Azure?

Breakpoint 1, main () at hello.c:4
4 printf("Hello, world!\n");
(gdb) n

Program received signal SIGSEGV, Segmentation fault.
0x0000fffff7ea98cc in ?? () from /lib/aarch64-linux-gnu/libc.so.6
(gdb) info stack
#0 0x0000fffff7ea98cc in ?? () from /lib/aarch64-linux-gnu/libc.so.6
#1 0x0000fffff7e7ae90 in puts () from /lib/aarch64-linux-gnu/libc.so.6
#2 0x0000aaaaaaaa0768 in main () at hello.c:4

I realize that my program could be crashing. But I am running a very simple Hello world type program that runs fine:

• without gdb on the ARM Azure VM
• with gdb on an Azure VM
• with gdb on my nVidia Jetson (physical ARM64 CPU)

If so, do you all have any idea what Microsoft (or I) would have to do to make it work on Azure? I am trying to teach a course involving C and ARM64 assembler.

Or is there some work-around (setting a gdb option, etc.) that can still provide simple debugging without triggering the problem?

Thanks,
--Mark Gass
(mkgass@meredith.edu)

[michael2012z]

Hi, Mark

You were using gdb to debug an application program in an ARM64 VM, right? That is a different topic with this issue. This issue was tracking the support of debugging the VM kernel from the host side (the debugging happens on the host, rather than in the VM), which is a different scenario from yours.

Just to confirm, were you using Cloud Hypervisor to manage the VM on Azure? What's the version?

[MarkGass]

Michael,

You were using gdb to debug an application program in an ARM64 VM, right?

Correct.

were you using Cloud Hypervisor to manage the VM on Azure?
I am just a regular user of ARM64 Azure VMs. So I don't even know if they use Cloud Hypervisor. And if they do, I don't know the version.

I do know that the Azure ARM Neoverse-N1 offering was still in limited availability when I requested access. So maybe this is something they have not worked out yet. I did make a post on an Azure Virtual machine support forum as well.

Thanks for the insights. I was unsure as to the role of the Cloud Hypervisor project in my scenario.