high cpu Citrix Wem Norskale service

[Wim Tooren1709161660]

I did a search but didn't find any problems like this. Doe anyone know what the problem could be. Our WEM broker servers running constant between 80 and 100 % cpu,  Also our users complaining about long login times and setting not loading correctly.

 

We are using Citrix Wem 1909.  We have 2 brokers (4 vcpu wit 16 gb ram), both are load balanced through a netscaler. The database is located on a sql cluster. In the logging of the wem clients we see a lot of events "Error while retreiving metrics for session ''"

[James Kindon]

it's not documented but it's been fixed in 1912

https://docs.citrix.com/en-us/workspace-environment-management/current-release/fixed-issues.html

 

The hack in 1909 and previous versions is to disable the VIP running on port 8286 (leave 8285 up). If you don't do this, WEM will never process from the Cache even if you set "use cache even if online" as it can talk to the brokers as required - mark the VIP down and suddenly it believes it's offline, but can still sync over the 8285 port

 

Trust me, its been there for a long time but no one believed until recently - it's fixed now though thankfully

[Alain Assaf]

Presumably this was not the case with prior versions. Did you update your AV to exclude the renamed WEM services on your VDA's. Also what does the WEM logs show during login? Is user experience of long login times reflected in Director (or similar) monitoring? Are the infrastructure servers constantly pegged on CPU or only during login storms?

 

Thanks,

Alain

[Wim Tooren1709161660]

Before we didn't use  Citrix WEM for our citrix environment. We did us another workspace management tool. We did include the wem services in our AV. When i look in the logs i don't see any errors or warnings. The Director doesn't show the slow logins. Between the hours when the users are in (office hours)  The infrastructure servers constantly have high cpu.

 

thanks,

Wim

[James Kindon]

if your brokers are getting hit, then your clients will suffer. You can force WEM into operating from cache to bypass the pain for a while (need to be on the latest release for that to work properly without a hack on the netscaler)

 

I would log a ticket with Citrix on your broker issues and get them to trace

[Wim Tooren1709161660]

Thanks, i will log a ticket with citrix.

 

You are talking about a hack on the netscaler? Is this documented anywhere? 

[Nils Johansson]

On 1/28/2020 at 1:20 PM, Wim Tooren1709161660 said:

Thanks, i will log a ticket with citrix.

 

You are talking about a hack on the netscaler? Is this documented anywhere? 

@Wim Tooren1709161660 Have you gotten any answer from Citrixsupport?

We have the same problem, Norskale service eating up all CPU.

[Wim Tooren1709161660]

We did 2 actions, we rebuild the performance counters on the wem brokers and we increase the size of the transaction log files of the wem database

[Derek Black]

We have 4 WEM servers load balanced and ironically we only noticed the issue after upgrading from 1906 to 1912 Cu1.

The documentation is pretty vague regarding the number of clients in regards to the impact of the broker service:

 

• When the number of connected agents exceeds a certain threshold (for example, 800), Norskale Broker Service.exe might consume a significant amount of CPU resources. For example, its CPU usage can increase to 70%. [WEM-7773]
• When the number of connected agents exceeds a certain threshold, Norskale Broker Service.exe might consume a significant amount of CPU resources. [WEM-7886, CVADHELP-14272]

 

What is the actually threshold, or even rough threshold?

We have ~300 agents so using their example of 800 agents it would seem to me like we should be well beneath the threshold.  (Again we have 4 WEM servers that are load balanced).

 

Disabling the VIP fixes the issue for us, we immediately see the CPU of all 4 WEM servers drop when the VIP is disabled.  Conversely the Broker service immediately consumes an excessive amount of CPU when enabling the VIP. 

 

Our agents are not configured to 'Use Cache Even When Online" but have 'Offline' enabled.

 

Review of the SQL transaction log looks to have sufficient space.

 

Curious if rebuilding the perf counters and increasing the size of the transaction log files was the resolution?

 

Is anyone still experiencing this issue with 1912?

 

Thank You!