SWITCH | SWITCHdrive | SWITCHengines |

Nested Virtualization (KVM)

We are using Switch Engines to run our Continuous Integration (CI) server and as part of that we wanted to run some tests in a VM. The VM is necessary because we want to test our installation procedure and to ensure reproducibility, we can’t do that on the Switch Engine Machine directly. However, we noticed that running VMs was really slow, we are running into all kind of timeouts because the VM takes ages to boot. The VM is just a small NixOS image, so the VM itself shouldn’t be the problem.

If I am not mistaken, Switch Engines runs on Openstack? So this means if we run a VM inside Switch Engines Machines, we are trying to do nested virtualization. Some googling revealed that hardware support for nested virtualization is disabled by default, which means our nested VMs probably fallback to full emulation explaining the low speed. Relevant OpenStack documentation: https://docs.openstack.org/developer/devstack/guides/devstack-with-nested-kvm.html .

Long story short, can you tell us if nested (kvm) virtualization is enabled on Switch Engines? Or would it be possible to enable it?

// Philipp

Dear Philipp,

good timing! As you have already found out, nested virtualization is normally disabled on SWITCHengines for slightly complex technical and operational reasons. But we know that it can be very useful in some situations, so we have done some work to make it possible again. We have done successful tests in our staging environments but haven’t deployed this in production yet. Would you like to be an alpha tester?

For the timing of our deployment to production, [UPDATED:] I hope we can do this next Tuesday. When this is deployed, it will be possible to get instances that support (nested) virtualization, but it will not (reliably) happen by default. Please contact us so that we can make sure you get the feature on instances that need it.

Hi sleinen,

That’s great news :slight_smile:

I am happy to be an alpha tester. I am quite busy this week, but I will come back to you next week.

We have deployed a configuration change today that enables nested virtualization for new instances when they are started on a certain subset of our compute nodes. And I live-migrated what I suspect is the right instance to one of these nodes. You may want to try to stop (“Shut Off”) your instance and then start it again. When it comes up, you should find that it now has vmx in /proc/cpuinfo, and that KVM should work on it.

I’m trying to think of a less “ad hoc” way for users to select that feature, but all the possibilities I can think of have other drawbacks. So for now I think the best way is “if you need nested virtualization on a given instance, talk to us”.

We changed the configuration of SWITCHengines a few days ago. The result is that in all new instances, “recursive” hardware virtualization should now be available. If not, please let us know!

If you have an existing VM and want to activate recursive virtualization, you can shutdown (virtually power down) and restart it. A way to test whether it is available is to grep vmx /proc/cpuinfo (on Linux).

In case you are curious, the main reason why we changed the configuration was the Spectre/Meltdown security problem: The workarounds have significant performance overhead, and the new configuration allows instances to use some native CPU features that reduce that overhead somewhat. Basically we now expose all features of the underlying hardware to your instances. The “vmx” feature is one of those. One drawback of the new configuration is that we can no longer live-migrate instances freely between machines for maintenance etc. We’ll still try to use live-migration to make maintenance transparent to users, we just have to do it more carefully.