Really slow when importing OS Layer or creating app layers

[Terry Rebstein]

Hi, I'm finding at a new App Layer 4.4 deployment that importing OS layers takes 1.5 hrs to import a 100GB OS layer that is probably ~50GB thin provisioned. This doesn't seem normal. Also when creating an App Layer to packaging it takes about 1hr 10 minutes to create the app layer. I'm not sure where the bottleneck is as this is on an enterprise vSphere deployment. It's causing the whole project to drag out and is not ideal.

 

Any guidance on identifying the cause of slowness is appreciated. I've monitored network performance and its about 4000-7000 KB/s and IO latency on the SAN is around 5-15ms on average.

 

I have specified a 200GB image cache in the vsphere connector as a heads up.

 

Thanks

[Dennis Austin]

Hi Terry,

 

That does sound strange.  If you haven't already I would dig into the ELM logs; they're pretty extensive.  

 

Is everything running on premise or do you have cloud components?

 

Our old friend WireShark may give you some clues.  If running on premise is the ELM maybe choking out on CPU or memory?

 

Sorry, don't have any brilliance to share on this.

 

d

[Rob Zylowski1709158051]

Hi Terry,

 

What type of storage are you using.  The speed is pretty much directly proportional to both network and storage throughput.  For instance if you are using Spinning Disk and a 1 GB network those times are probably in line if you are using vSphere as your Hypervisor (vSphere goes though extra steps to convert and expand the disks) .

 

The 56 Mb/s network speed is obviously pretty slow.  The question is, is it slow due to a network issue or the speed of the SAN.

 

Do you know the path between your hypervisor and your appliance?  Are they on the same network or could there be some unknown bottleneck between them?

 

To give you an idea in my lab i have older servers like r710 and r610 with1 GB networking and i use single flash disks for storage some local and some NAS.  It probably takes about 20-30 minutes to import an OS.  It takes about 12 minutes to create a packaging machine from cache but about 40 minutes if the cache needs to be initialized first.

 

Some customers i have worked with that have 10GB networks with good flash arrays can spin up a packing machine from cache in 2-3 minutes.  So its very dependent on infrastructure.

 

BTW remember the appliance is where we store the layers so its important for the appliance itself to be on fast storage.

 

 

Rob

[Terry Rebstein]

Thanks for the responses guys. 

 

I am offsite today but will probe the logs and get back to you.

 

The storage is a 3Par with a shared pool of disks including some SSD. I don't have more details at this time, but it's not likely something that should struggle. I've put the appliance on the same host as the OS layer VM to be imported. Does traffic route through vcenter or direct to the host? It's a 10GB NIC on the hosts itself with minimal competing traffic. The hosts are DL460G9's with plenty of processing power and memory, I've not seen any bottleneck. There are no cloud components, everything is local within the datacenter.

 

The closest thing to a bottleneck seems to be network and disk. Can you detail the exact data flow path for when the ELM imports an OS layer or creates a packaging machine? Also it doesn't appear that the cache is making much difference, how can I validate the cache is in fact being created/leveraged? Any more info you can provide will help me try and resolve this.

 

Thanks again

[Rob Zylowski1709158051]

When we import the OS its via OVF export from vCenter.  After we download it we clone it into a layer on the ELM appliance.  Creating a packaging machine is totally different.  We store the layers as vhd files in the appliance.  So the first task to create a packaging machine is to create the boot disk where we merge the os layer and any pre-requisite layers into a separate vhd,we then convert the vhd to a sparse format vmdk and upload it to the datastore defined in the connector, we then have to expand it to a thick disk because ESX doesn't understand the sparse format used.  Its important because of that last step to make sure when your OS layer is not created too large for whats in it because it will be expanded to that size.

 

Then we upload the blank layer disk which is uusally 10G and it also goes through that process.

 

What caching does is store the OS disks (with any pre-reqs etc) in teh datastore so we can just reuse them if nothing has changed so it skips much of that process.

 

But after an upgrade all the cache disks have to be recreated so the first time you create a layer after upgrading it takes alot longer.  You also have to recreate the cache disks if you update you OS layer.

 

You can actually see the cache disks on the datastore defined in the connector.

[Terry Rebstein]

Moved to an all SSD datastore and it has significantly improved performance. Creating a new version of a platform layer (10GB) took about 30 min instead of 90min.

 

Thanks for all assistance

[Trond Eirik Haringvarstein]

Hi Terry, after reading this you inspired me to do some test. 1 minute to App Layer :-)

https://twitter.com/xenappblog/status/902873846952452098

 

A blog post will be out next week, testing iSCSI, SSD and NVMe. Jarian Gibson will later do 10 vs 40GB (I only have 1GB)

 

I would like to know/understand why your OS Thin Provisioned VHD is 50GB?

Is it Windows 2012 R2 with tons of Windows Updates that haven't been cleaned up?

 

My image is Windows 2016 with latest 08/2017 CU with all C++ libraries and it only takes 11.9GB on disk..

 

By understanding this, we can probably educate the community on the best practices to Speed Up CELM.

 

Thanks

Eric

[Terry Rebstein]

It's server 2016 and I've not gone through purging data or updates. Any reference docs or tips are appreciated! I also think 50GB is pretty high but haven;t gone looking into why yet.

[Terry Rebstein]

Checked again and its actually closer to ~25 GB

[Sandy Williams]

Same issue here.. using latest version.  My ELM is on the same XenServer host as my cache location (local disk).  Took me about 75 minutes to copy down the cache for my OS during a layer creation.  When looking at NIC performance in XenServer it shows an average of about 30mbps