Servers randomly crashing

[Jens Strohschnitter1709152492]

We have randomly crashes of our provisioned RDS running Server 2016 (1607).

 

Environment:

 

Citrix XenDesktop 7.15 CU3

Server 2016 Build 1607 (24 vCPU, 90GB RAM, ~30-35 sessions per RDS)

Hyper-V 2016 (S2D) as Hypervisor

Citrix Provisioning Services 7.15 CU3

Citrix Workspace Environment 18.08 with enabled memory and cpu management

 

What we've done:

 

Eventlogs do not show any useable entry 

CDF Tracing on PVS does not show any entry on reboot-times

New installation of Server 2016 from scratch mit last MS Feb CU 

Only an entry in Hyper-V Clusterlog that showes, that Hyper-V has hard rebootet the VM:

(sorry only in german available)

 

""

Fehler in der Clusterressource "XXXXXXXXXXXX" des Typs "Virtual Machine" in der Clusterrolle "XXXXXXXX Resources".

Abhängig von den Fehlerrichtlinien für die Ressource und die Rolle wird vom Clusterdienst möglicherweise versucht, die Ressource auf diesem Knoten online zu schalten oder die Gruppe auf einen anderen Knoten des Clusters zu verschieben und die Ressource dann neu zu starten. Prüfen Sie den Ressourcen- und Gruppenzustand mit dem Failovercluster-Manager oder mit dem Windows PowerShell-Cmdlet "Get-ClusterResource".

""

 

Users reports that before crashing of the server, the session freezes (no keyboard input or mouse move) and then tries to reconnect without result.

 

We do not have any error-message in any log that can help us to find the issue.

 

The issue appears only on servers that have more than 30 sessions connected. Other servers with less then 10 sessions with same image are not effected.

 

Does someone report same issue with Server 2016 ???

[Jens Strohschnitter1709152492]

It seemed that the issue comes up with a to low entered pagefile and a to high RAM-Cache.

 

I've set pagefile to 24GB fix and RAM-Cache on PVS to 16GB.

 

Since change, no RDS rebooted.

[Jens Strohschnitter1709152492]

Now I've seen the issue on my own session:

 

While working, the session freezes, no mouse move, no keyboard input.

The screen goes blurred.

After round about 1 or 2 minutes the screen goes black and the sessions disconnects.

 

Looking in the Hyper-V logs, the VM was rebooted by Hyper-V.

 

Still no further entries in eventlogs of the VM and in CDF-Trace logs of Citrix Provisioning Servers.

 

I've found an article that describes that Hyper-V looks at every VM if the OS is reachable. If not, the VM will be restarted by Hyper-V
and logs it with the error (sorry still in german):

 

""

Fehler in der Clusterressource "XXXXXXXXXXXX" des Typs "Virtual Machine" in der Clusterrolle "XXXXXXXX Resources".

Abhängig von den Fehlerrichtlinien für die Ressource und die Rolle wird vom Clusterdienst möglicherweise versucht, die Ressource auf diesem Knoten online zu schalten oder die Gruppe auf einen anderen Knoten des Clusters zu verschieben und die Ressource dann neu zu starten. Prüfen Sie den Ressourcen- und Gruppenzustand mit dem Failovercluster-Manager oder mit dem Windows PowerShell-Cmdlet "Get-ClusterResource".

""

This article describes the result of Hyper-V:

https://blogs.msdn.microsoft.com/clustering/2013/01/24/understanding-how-failover-clustering-recovers-from-unresponsive-resources/

 

This happens only on VMs that has more than 20 sessions in peak logged on. And only when users logged off and near 10 sessions are still connected.

 

And to make it complicated, not at all. Only radomly and not every day :-(

 

On Citrix RDS with the same PVS-Image but in another DeliveryGroup with 10 sessions in peak, the issue has not happened at all.

 

Very strange ... what can I do further on ???

 

 

 

[Erno Alanen1709160612]

Hi

 

What was your pagefile size in the beginning when you had problems?

 

It seems that the pagefile usage is at the highest when users are disconnected and Wem dumps their apps memory in to the pagefile. When user logs out completely the pagefile usage decreases.

 

This is definitely something that is different in RDS compared to VDI.

 

Erno

[Jens Strohschnitter1709152492]

@Erno Alanen1709160612

Thank you fpr your reply.

The pagefile was set from 16GB up to 24GB. After setting it to fixed 24GB, the issue has gone.