Nested Hypervisors in Xenserver 7

[John Fullbright]

Hi All,

 

I installed Xenserver 7 yesterday.  Today I think I want to play with nested hypervisors.  I was looking on xen.org, and it seems I need to add two parameters and change 1.  I have some prior experience making templates, and this looks doable. http://wiki.xen.org/wiki/Nested_Virtualization_in_Xen

 

 

Has anyone actually tried it yet?  If so, would you care to share your experiences?

 

TIA

 

J

 

[Tobias Kreidl]

It's doable, known to work, but is of course unsupported. Have not played with (yet) myself, but may. Be cautious about trying to run newer version of XenServer under of older ones as not all functionality may be recognized in terms of how in particular connectivity is handled up the chain.

-=Tobias

[Willem Boterenbrood]

Keep in mind that when running a hypervisor under another hypervisor CPU virtualization extensions are not available for the 2nd hypervisor. So you can only run a 2nd hypervisor that is able to work without those virtualization extensions. Xen for instance can do this but it losses certain capabilities that will limit its functionality and lower its speed.

 

There are also some hypervisors that can emulate those CPU virtualization extensions for VM's. So these will provide full fucntionality to the 2nd hypervisor though with reduced performance.

 

So running a hypervisor under another hypervisor is never recommended for production use, there are just too many downsides. For testing it might be fine. (I sometimes run XenServer under Vmware player/Virtualbox on my Windows desktop, to find out if a certain driver is available, or how to make yum install work under XenServer 7.0) For real testing scenarios I keep some old servers and storage devices though.

[John Fullbright]

Hi Willem,

 

"Keep in mind that when running a hypervisor under another hypervisor CPU virtualization extensions are not available for the 2nd hypervisor."

 

This is what has changed, and what I want to experiment with.  It's based on hardware (EPT and VT-d) and has been in "tech preview" since xen 4.4.  Recently, it is possible to do the same with Hyper-v in Windows Server 2016 TP5. https://msdn.microsoft.com/en-us/virtualization/hyperv_on_windows/user_guide/nesting  I have tried this on Windows Server 2016 TP5 and it does work.

 

In either case, you expose the virtualization extensions to the guest.  For the Xen hypervisor, prior testing is documented here http://wiki.xen.org/wiki/Nested_Virtualization_in_Xen

 

In both cases, it's not for production and is considered experimental. I wanted to try on Xenserver 7 since it uses the 4.6 hypervisor.   In Xen, the configuration looks like:

    hap=1    nestedhvm=1    cpuid = ['0x1:ecx=0xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx']

Well, I know where to mask the CPU bits in a xenserver template, and am fairly familiar with manipulating them or building them.  I had to make for FreeBSD last year, and it wasn't really that hard.  hap= and nestedhvm= look like QEMU parameters that get passed through, so I just need to figure out where in the template to add them.

 

J

[John Fullbright]

I think I figured it out.  At least I can see vmx in flags of cpuinfo in my Ubuntu guest. 

 

You enable it in the vm with xe vm-param-set uuid=<uuid> platform...

 

I had to dig through the source code to figure out which platform setting, but it appears to be working.  I'll try some various guest hypervisors tomorrow and see which ones work/which ones don't.

 

J

[John Fullbright]

So,

 

It does work.

 

L0 = Fujistu Primergy Rx200 S7 - Xenserver 7

L1 = Xenserver 7

L2 = Ubuntu Xenial 16.04

 

I used the Centos 7 template with the xenserver 7 ISO to create the L1.  Adjust the boot LUN size to whatever you want.  Uncheck start the VM.  Then, from the console, find the uuid of the vm and:

 

xe vm-param-set uuid=<your vm uuid> platform:exp-nested-hvm=true

 

The start the VM and load xenserver 7 in the L1.  I added the L1 to my pool.  I was then able to access all the shared SRs and create an ubuntu xenial L2.

 

I tried just ubuntu with Xen 4.6 as an L1, and that worked.  I tried ubuntu with KVM in the L1, and that appeared to work as well.  I then tried Windows Server 2016 TP5 in the L1.  Windows loads, you can see the hypervisor, vmx, and ept are there, you can install the hyper-v role, but when you go to launch the L2 hyper-v errors.  It's event ID 46.  MSR index 48d has a value that is not within the accepted capabilities range for Hyper-v essentially. 

 

coreinfo says

 

Intel® Xeon® CPU E5-2620 0 @ 2.00GHz

Intel64 Family 6 Model 45 Stepping 7, GenuineIntel

Microcode signature: 00000710

HYPERVISOR      *       Hypervisor is present

VMX             *       Supports Intel hardware-assisted virtualization

EPT             *       Supports Intel extended page tables (SLAT)

PS C:\>

 

Probably something wrong with the way the CPUid bits are masked.  In any event, I have not tried vmware as an L1 yet.

J

[Shakil Shaikh]

Hi there,

 

I'm trying this out, but with an Ubuntu install instead.

 

Can a PV Ubuntu guest "nest HVM" or would I have to install Ubuntu as an HVM guest?

 

Or put another way: when you installed XS7 as L1, was that PV? Using the CentOS template perhaps implies that?

[John Fullbright]

The Centos 7 template is HVM with paravirtualized drivers.

[Shakil Shaikh]

I see, so if I wanted to run say Ubuntu with KVM at L1, I should stick with an HVM template (I'm using a windows 10 one), and not upgrade the kernel to a PV enabled one?

[marc snijders1709159882]

on my home-lab with Server 2016 DC within hyper-v eventually Xenserver 7.5 installed on vm with legacy networkcard and vm configured as nested vm via PS: Set-VMProcessor -VMName "vmname" -ExposeVirtualizationExtensions $true

 

 

[Shane Sorensen1709160797]

I was getting an error when I attempted to install the Hyper-V role through the GUI on a 2016 server. Found an article that said running the command version of this would work. Low and behold, it did work for me. Give it a shot from admin powershell console.

Enable-WindowsOptionalFeature –Online -FeatureName Microsoft-Hyper-V –All -NoRestart