Target Devices "sometimes" boot in a blackscreen

[Tobias Schörgenhofer]

Hello Everyone,

 

we have a very strange situation with one Customer.

 

The Customer has 6 Citrix Servers, which running with PVS 7.6 (it was 6.1 before).

 

Everytime in the morning, when the servers restarts.....0-4 of the 6 servers hanging in a black screen!

 

Its totaly random which of the 6 servers fail to reboot! After i reboot the failing servers manualy, they boot just fine.

 

Some Facts:

The master Server war original on XenServer (Convert to VMware and then Imaging Wizard)
DHCP Server is in the same VLAN

vSphere 5.5.0

VMXNET 3 Network Adapter

We have 3 other Customers with PVS Servers on the same Hypervisior...none of them has this problem.

 

we already tried the following:

New PVS Server from 6.1 to 7.6 including Target Device Update

DHCP Server changing to a central DHCP Server

Reschedule the Reboot Cycles

Attaching the Boot Image Directly

 

 

Has anyone maybe a new approach? :/

 

 

greetings Tobi from the cold Switzerland :)

[Björn Schläfli]

issue solved with update from esxi 5.5 patch 1 to 6.0 express patch 6.

[James Pethigal]

Having the exact same issue.

Migrated from PVS 6.1 & XenServer to 5.5 vSphere 5.5 & PVS 7.6

Had to use E1000 NIC because of BSOD with VMXNET 3 NIC

DHCP is in the same isolated PVS vlan on a single server.  (disabled split scope)

 

Boot iso with Interrupt Safe Mode checked

 

About 1/3 of the reboots hang with a black screen.   Usually will reboot with a couple power cycles

 

Regards

[Avi elmaliah1709155232]

when i had a static ip configured in the master image server and i would later designate that as standard, i would get black screen in boot up, start up your gold image and make sure it's configured to use DHCP,

 

if that's the case you would see that if all target devices are turned off, you can boot the first target device, all subsequent target will receive black screen and sometimes blue screen when they try to boot.

[James Pethigal]

DHCP on master image & target devices.

Did clear the registry entries for DhcpDefaultGateway, DhcpDomain, DhcpIPAddress, DhcpNameServer, DhcpServer, DhcpSubnetMask, and DhcpSubnetMaskOpt  on the target device.  That seemed to improve boots somewhat.

 

Also made sure no phantom hardware or drivers hanging around

[Tobias Schörgenhofer]

Having the exact same issue.

Migrated from PVS 6.1 & XenServer to 5.5 vSphere 5.5 & PVS 7.6

Had to use E1000 NIC because of BSOD with VMXNET 3 NIC

...

Regards

 

Arghs okey. How was the performance afterwards?

 

DHCP on master image & target devices.

Did clear the registry entries for DhcpDefaultGateway, DhcpDomain, DhcpIPAddress, DhcpNameServer, DhcpServer, DhcpSubnetMask, and DhcpSubnetMaskOpt  on the target device.  That seemed to improve boots somewhat.

 

Also made sure no phantom hardware or drivers hanging around

 

i have heard that, that was a problem with pvs 6.1. I tried this but no change....

[Avi elmaliah1709155232]

how do you boot? via ISO, TFTP, PXE?

how many NICs do the VMs have?

 

a new approach will be to create a VM on VMWARE and check, maybe?

 

 

 

also, make sure you check this pdf for anything related to ESX, like what the VM hardware version, whether you need intermediate buffering enabled and whether you ISCSI KB from MSFT in your configuration.

 

http://citrix.edocspdf.com/media/output/en.provisioning-7.pvs-provisioning-7.pdf

[Tobias Schörgenhofer]

how do you boot? via ISO, TFTP, PXE?

 

how many NICs do the VMs have?

 

a new approach will be to create a VM on VMWARE and check, maybe?

 

ehm i guess you mean tftp + pxe right?

 

The VMs get their IP and TFTP Server via DHCP Server.

We tried with ISO Images...same behavior.

 

They have only 1 NIC.

 

Yeah but then i will have to install everything from scratch, and i dont want to do that.

[Rasmus Kindberg]

Perhaps related to below?

 

http://support.citrix.com/article/CTX139498

 

 

Yeah but then i will have to install everything from scratch, and i dont want to do that.

 

You could reverse image (see http://pvsguy.com/tag/reverse-imaging/) to perform NIC changes.

 

I would suggest you try removing the CD ROM device from the target VMs through vSphere. If that does not help, try changing to "paravirtual vmware" (or whatever it is called) on the SCSI controller on the VM to see if that has an effect.

[Tobias Schörgenhofer]

Perhaps related to below?

 

http://support.citrix.com/article/CTX139498

 

i guess that wont helpt.

Its not that the machines have a slow boot time (in fact its fast, around 30sec IF they boot).

They Problem is that some of the 6 VMs sometime just hang in a black screen if i check in the morning.

When i press "reset" at the vSphere Client, they boot normaly.

[Avi elmaliah1709155232]

also, make sure you check this pdf for anything related to ESX, like what the VM hardware version, whether you need intermediate buffering enabled and whether you ISCSI KB from MSFT in your configuration.

 

http://citrix.edocspdf.com/media/output/en.provisioning-7.pvs-provisioning-7.pdf

 

also, check maybe the times they are rebooting in the night, maybe it's too many and then you will need to give more time between reboots or change the PVS server advanced settings.

[James Pethigal]

Arghs okey. How was the performance afterwards?

 

 

i have heard that, that was a problem with pvs 6.1. I tried this but no change....

Performance wasn't bad.  Obviously...  it's not going to be the same as using VMXNET 3, but the customer is using a nasty Oracle app the cpu bounds the processor before the NIC becomes an issue.

 

Going to open a Citrix Support case on Monday when I am back.   Let you know if we get any resolution.

[James Pethigal]

Think I have our issue finally resolved. 

(Citrix support recommended using a boot iso with static IP, but not really feasible because we are spinning up 50 + servers.)

 

Current rev:

 

Target:

VMware version 10

VMXNET 3 NIC

VMware paravirtual SCSI drivers. (replaced the LSI)

Disable IPV6

Disable large send offload

Clear the registry entries for DhcpDefaultGateway, DhcpDomain, DhcpIPAddress, DhcpNameServer, DhcpServer, DhcpSubnetMask, and DhcpSubnetMaskOpt

 

Server:

Disable IPV6

Disable large send offload

 

Boot ISO/tftp & DHCP

 

Took a few days to resolve the BSOD with VMXNET 3 NICs.  

 

 

Stay thirsty, my friends

 

...  James

[Björn Schläfli]

Hi Tobias,

 

Long time ago but same issue here with PVS 7.9. no issues with 7.7. Did you ever solved your black screen issue?

[Joe W]

On 4/2/2015 at 5:23 PM, James Pethigal said:

Think I have our issue finally resolved. 

(Citrix support recommended using a boot iso with static IP, but not really feasible because we are spinning up 50 + servers.)

 

Current rev:

 

Target:

VMware version 10

VMXNET 3 NIC

VMware paravirtual SCSI drivers. (replaced the LSI)

Disable IPV6

Disable large send offload

Clear the registry entries for DhcpDefaultGateway, DhcpDomain, DhcpIPAddress, DhcpNameServer, DhcpServer, DhcpSubnetMask, and DhcpSubnetMaskOpt

 

Server:

Disable IPV6

Disable large send offload

 

Boot ISO/tftp & DHCP

 

Took a few days to resolve the BSOD with VMXNET 3 NICs.  

 

 

Stay thirsty, my friends

 

...  James

 

 

Was this fix permanent? Running into similiar issue: PVS 7.18, esxi 6.5 on Lenovo SR630s.