WMIPRVSE high CPU utilization

[Scott Knights]

I have a Xenapp environment, running 2012R2 servers on ESXi 6 hosts. VDA version 7.16. The servers are provisioned using MCS. CPM is used to manage profiles. Sophos antivirus is installed with Citrix and Microsoft recommended exclusions.

 

The WMIPRVSE service running under the NETWORK SERVICE user context will periodically use constant excessive CPU. Running procmon shows that it is constantly touching the file tzres.dll and its associated mui.  It also tries and fails to read the non existent registry key HKEY_USERS\S-1-5-20\Software\Citrix\SessionSfr\0. I can eliminate this by creating the key.

 

 

The WMI-Activity Trace log shows constant Event 12s from CIMWIN32 under this PIDs HostID, executing this query:

 

ProviderInfo for GroupOperationId = 64558; Operation = Provider::ExecQuery - CIMWin32 : select __RELPATH, __RELPATH, __DERIVATION, Name, SessionId from Win32_Process; HostID = xxxxx; ProviderName = CIMWin32; ProviderGuid = {d63a5850-8f16-11cf-9f47-00aa00bf345c}; Path = %systemroot%\system32\wbem\cimwin32.dll

 

 

Running this query manually gives extensive output. The WMI repository validates OK.

 

There do not appear to be any corresponding event 11s to show the ClientProcessID.

 

The issue still occurs with the VMTools, Citrix Broker, telemetry, CPM and End User Experience monitoring services stopped and disabled. I have excluded tzres.dll and the WBEM folders from on access AV scanning.

 

If I kill the wmiprvse process, it immediately starts again using high CPU. The only thing I have found which stops the excessive CPU usage is to rename the cimwin32.dll file. Once this is done and the wmiprvse is ended it will start without the excessive CPU usage. However, I don't know what negative effects this may have. Procmon shows it is still trying to repeatedly access cimwin32.dll.

 

Anybody have any idea on where to look next?

 

[vikash]

Hi @Scott Knights,

 

Good Day :)

 

I do recollect 7.12 environments facing similar CPU hike issues with WMI provide. Not sure if this would work but could you try and reset the WMI counter (referring to https://support.citrix.com/article/CTX223754), you could achieve that by following the below steps:

 

1. Click Start, click Run, type Wbemtest, and then click OK.
2. In Windows Management Instrumentation Tester, click Connect.

3. In the Namespace box, type root\cimv2, and the click Connect.

4. 4Click Enum Classes.

5. In the Enter superclass name box, type Win32_Perf, click Recursive and then click OK.

6. In Query Results, you will not see results for the counters that are not transferred to WMI.

   For example, the counter object for Remote Process Explorer is Win32_PerfFormattedData_PerfProc_Process. If this counter does not exist, you will not see the following line in Query Results - Win32_PerfFormattedData_PerfProc_Process.

To Manually reset the WMI counters:

1. Click Start , click Run, type cmd, and then click OK.
2. Stop the Windows Management Instrumentation service or at the command prompt, type net stop winmgmt, and then press ENTER.
3. At the command prompt, type winmgmt /resyncperf, and then press ENTER.
4. At the command prompt, type wmiadap.exe /f, and then press ENTER.
5. Start the Windows Management Instrumentation service or at the command prompt, type net start winmgmt, and then press ENTER.
6. Type exit, and then press ENTER to close the command prompt.
7. After resetting the WMI counters, retest WMI.

You could also try restarting the Windows  Management Instrumentation Service and check if that works.

* Strong recommendation is that you run the antivirus scan for the all systems, sometimes virus attacks may also indicate a WMIPRVSE high CPU utilization.

 

Let me know if this help.

 

Regards,

Vikas Hiremath

[Riley Foster]

I've been working with Citrix and Microsoft for a good two weeks now and I *hope* this is solved as of this morning.

 

After some log collecting Citrix support suggested that the issue is with 3rd party software and to open a case with Microsoft. I opened a case with Microsoft (and paid the $500 fee). After some logging and monitoring via their WMI tools they pointed out some issues with long-running WMI queries originating from Citrix Profile Management. Microsoft captured more logs yesterday and says someone from the debug team might reach-out. We used a tool called WMISpy and in the logs the long-running queries look like:

 

[09/12 09:38:20.0488 AM] [Session: 28  Command Line: "C:\Program Files\Citrix\Virtual Desktop Agent\upmEvent" wait] [33360.1796] [ExecQuery: SELECT * FROM Win32_Process where Name = 'explorer.exe' AND SessionId = 28]

 

Citrix support phoned me yesterday and asked how the troubleshooting was going with Microsoft. I mentioned the suspicion of UPM causing the issues and right away the tech told me that there was a "known issue" with UPM causing high wmiprvse.exe CPU usage. They had a private fix which they sent to me (dated 9/6/2018). I installed it last night and so far WMI CPU usage has been negligible. 

 

Riley

[Balaji Muthukrishnan]

Hi @Vikas Hiremath

 

You are spot on, I had similar issue in my environment with XD 7.17 / VDA 7.16 (Dedicated Machines), were almost 100 + machines affected by this CPU spike issue due to WmiPrvSE.exe process

 

-WMI Activity Log (Error event 5858) points to brokerAgent.exe and PICA

-One more observation was Citrix Director was not able to pull VDA metrics like ICA Latenct, RTT and Process Memory Utilization

- Machine usage Graph with metrics like Memory, CPU, and IOPS was not generated, Data collection error

-Additionally stopped Citrix Profile management service to remove any anomalies as i am not using it.

 

Followed your steps and the steps outlined in https://support.citrix.com/article/CTX223754, bingo...issue got resolved.

 

Thanks...It was really helpful.

 

 

-- Run with Elevated privilege

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Microsoft Windows [Version 10.0.15063]
(c) 2017 Microsoft Corporation. All rights reserved.

C:\WINDOWS\system32>lodctr /R

Error: Unable to rebuild performance counter setting from system backup store, error code is 2
C:\WINDOWS\system32>cd\

C:\>cd Windows\SysWOW64

C:\Windows\SysWOW64>lodctr /R

Info: Successfully rebuilt performance counter setting from system backup store
C:\Windows\SysWOW64>cd\

C:\>cd Windows\System32
-re-run again to reset counters
C:\Windows\System32>lodctr /R

Info: Successfully rebuilt performance counter setting from system backup store
C:\Windows\System32>WINMGMT.EXE /RESYNCPERF

C:\Windows\System32>wmiadap.exe /f

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

or 

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Microsoft Windows [Version 10.0.15063]
(c) 2017 Microsoft Corporation. All rights reserved.

C:\WINDOWS\system32>services.msc

- Select Windows Management Instrumentation service
- Select Pause
- Run the below commands

C:\WINDOWS\system32>winmgmt /resyncperf

C:\WINDOWS\system32> wmiadap.exe /f

- Now Resume the Windows Management Instrumentation service
- Now run the below commands

C:\WINDOWS\system32>lodctr /R

Error: Unable to rebuild performance counter setting from system backup store, error code is 2

-- If it fails to reset run the command again after running the below command. If it resets successfully, no need to run the command again

C:\WINDOWS\system32>cd\

C:\>cd Windows\SysWOW64

C:\Windows\SysWOW64>lodctr /R

Info: Successfully rebuilt performance counter setting from system backup store
C:\Windows\SysWOW64>cd\

C:\>cd Windows\System32
-re-run again to reset counters
C:\Windows\System32>lodctr /R

Info: Successfully rebuilt performance counter setting from system backup store
C:\Windows\System32>WINMGMT.EXE /RESYNCPERF

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 

[Riley Foster]

It doesn't look like this is a numbered LC yet. It's just replacing two .exe files and editing the registry. 

 

The "folder" that the files were in was named "7.17" - so take that as you will. I don't know if this fix is specific just to 7.17 or others.

 

I can give you my case # - hopefully that will help. 

 

Case#: 78304839

 

Riley

[Riley Foster]

I'm running into this *exact* issue right now. I've tried tracing, logs, etc. and I can see the files/processes involved, but I can't determine why it's happening.

 

Restarting the WMI service solved it for a while - I believe the issue compounds itself with each user logon. I have also reset the WMI repository and the result is the same. It calms down for a while then slowly builds to 40-50% CPU usage.

 

This is cutting our capacity in half..

[Dennis Parker]

I don't have an answer, but I have some more information that may help.  I do believe this is a Microsoft issue though based on some findings.

 

I can make the issue happen: 

Pause WMI service

Stop Desktop Broker Service and our monitoring tool/agent (Check_MK)

All WMIPRVSE instances should be stopped at this point.

Resume WMI service

 

At this point, unless you have something else that uses WMI Performance Data Counters, you should have minimal WMIPRVSE processes.

 

Starting either of these services (in our environment) will kick up a WMIPRVSE.exe with High CPU and looking into the details of the threads, can see that it is hitting the Performance Data Helper libraries.

Letting either of these run, every ~17 to 18 minutes, the process will again kick up the CPU process. 

 

Watching the High CPU process with ProcMon, can see it running through the Performance Counter registry keys, several times. Suspect that it is rebuilding some sort of cache.

 

Reading this thread on an unrelated product, they see the same sort of thing: https://github.com/martinlindhe/wmi_exporter/issues/89 

The issue on our VDAs gets more pronounced as there are more users logged in, presumably because it has to rebuild a larger cache of performance counters.

 

We do plan on opening a ticket with Microsoft, armed with this information, and hopefully can come up with something that can be fixed/changed. I am slightly worried that it is "working as designed".

 

Also: Note that in the March Microsoft Patches, there was an update to WMI that appears to have made this more pronounced as well.

 

Edited August 28, 2018 by Dennis Parker
Add info

[Dennis Parker]

Can you please tell me the version of UPM this patch applies to and the patch ID we can request from Citrix?

[Trevor Trujillo]

Hi Riley, 

After trouble shooting this with Citrix, they have requested we reach out to Microsoft as well. Before I reach out to Microsoft for the same issue, do you have a ticket number for Microsoft to reference? 

[Riley Foster]

19 hours ago, Trevor Trujillo said:

Hi Riley, 

After trouble shooting this with Citrix, they have requested we reach out to Microsoft as well. Before I reach out to Microsoft for the same issue, do you have a ticket number for Microsoft to reference? 

 

I'm not sure I understand. Are you asking for our Microsoft case number? Because the issue, in our case, was not a Microsoft problem - it is a Citrix issue with UPM. The Citrix case# is above and I would perhaps start there if you're having the same symptoms.

 

Riley

[Alexandre Smialoski]

Hello all,

 

Fixed Issues (Profile Management 1808):
The CPU usage of the WMIPrvSE.exe process might be high because of the UPMEvent.exe process. [LC9257]

https://docs.citrix.com/en-us/profile-management/current-release/whats-new/fixed-issues.html

 

I will try this today.

 

Regards,

[Trevor Trujillo]

@Alexandre Smialoski any update on your previous commented fix?

[Alexandre Smialoski]

@Trevor Trujillo

 

Yes, the issue was solved with VDA 1808.2 (with Profile Management 7.19.0). I have updated all my VDAs to 1808.2 and the problem don´t occur anymore.

 

Thanks,

Alexandre Smialoski

[Riley Foster]

So, we're trying to get Citrix to reimburse us for the Microsoft case we had to open. Basically, Citrix did very little to troubleshoot the issue and their stance was that wmiprvse.exe is a Microsoft component. According to Citrix it's usually "3rd party" software that causes the problem. My argument with them is that after opening the case with Microsoft and *immediately* after mentioning that Microsoft determined UPM was the cause the tech told me about the private fix. There was no way he didn't already know about it.

 

Either the tech lacked the skill to find the issue or they wanted Microsoft to do the heavy lifting to diagnose. Given the immediate recommendation I'm leaning towards the latter.

 

Now that it's been integrated into the latest update, I Googled the fix number (LC9257) and came across this: https://echelog.com/logs/browse/citrix/1521673200. Search for LC9257 and then follow that IRC conversation.

 

This has been ongoing since MARCH. The individual there (Minnebo) went through the exact same process. They were passed off to Microsoft and had to pay for the support case. They're also trying to get Ctirix to reimburse them.

 

Unbelievable..

[Trevor Trujillo]

5 minutes ago, Riley Foster said:

So, we're trying to get Citrix to reimburse us for the Microsoft case we had to open. Basically, Citrix did very little to troubleshoot the issue and their stance was that wmiprvse.exe is a Microsoft component. According to Citrix it's usually "3rd party" software that causes the problem. My argument with them is that after opening the case with Microsoft and *immediately* after mentioning that Microsoft determined UPM was the cause the tech told me about the private fix. There was no way he didn't already know about it.

 

Either the tech lacked the skill to find the issue or they wanted Microsoft to do the heavy lifting to diagnose. Given the immediate recommendation I'm leaning towards the latter.

 

Now that it's been integrated into the latest update, I Googled the fix number (LC9257) and came across this: https://echelog.com/logs/browse/citrix/1521673200. Search for LC9257 and then follow that IRC conversation.

 

This has been ongoing since MARCH. The individual there (Minnebo) went through the exact same process. They were passed off to Microsoft and had to pay for the support case. They're also trying to get Ctirix to reimburse them.

 

Unbelievable..

We are being pushed to open a support ticket with Microsoft as well. Citrix support assisted in pulling WMI logs and stated they can not review them.  Microsoft has to pin point the issue in the logs before Citrix will do anything. 

[Balaji Muthukrishnan]

One other reason WmiPrVSe.exe Process consuming high CPU in my case with Storefront and Delivery Controller was due to Microsoft Management Agent installed on these servers. After reviewing the WMI-Activity log, it shows HealthService.exe and MonitoringHOst.exe process utilizing  it. 

 

After removing Microsoft Monitoring Agent issue seems resolved and tons of error (EventID 5858) in the WMI - Activity log also gone.

[Anthony Steffens1709158608]

Hello,

     We are seeing the issues on our VDA Servers running 7.1912 update 3...constant EventID 5858 from ClientProcessId=5156 which appears to be vmtools.exe running on Server 2016 v1607 (OS Build 14393.4104).  We keep seeing the WmiPrvSE.exe max'ing out CPU and we assume crashing...and the WMI-Activity logs are non-stop hammered with these:

 

Log Name:      Microsoft-Windows-WMI-Activity/Operational
Source:        Microsoft-Windows-WMI-Activity
Date:          6/23/2021 1:53:11 PM
Event ID:      5858
Task Category: None
Level:         Error
Keywords:      
User:          SYSTEM
Computer:      dc-vdb-01.xxxxx.xxx
Description:
Id = {00000000-0000-0000-0000-000000000000}; ClientMachine = DC-VDB-01; User = NT AUTHORITY\SYSTEM; ClientProcessId = 5156; Component = Unknown; Operation = Start IWbemServices::ExecQuery - ROOT\CIMV2 : Select CommandLine from Win32_Process Where ProcessID = '26916'; ResultCode = 0x80041032; PossibleCause = Unknown
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-WMI-Activity" Guid="{1418EF04-B0B4-4623-BF7E-D74AB47BBDAA}" />
    <EventID>5858</EventID>
    <Version>0</Version>
    <Level>2</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x4000000000000000</Keywords>
    <TimeCreated SystemTime="2021-06-23T17:53:11.509111300Z" />
    <EventRecordID>1534598</EventRecordID>
    <Correlation />
    <Execution ProcessID="2688" ThreadID="23836" />
    <Channel>Microsoft-Windows-WMI-Activity/Operational</Channel>
    <Computer>dc-vdb-01.parda1us.wl.com</Computer>
    <Security UserID="S-1-5-18" />
  </System>
  <UserData>
    <Operation_ClientFailure xmlns="http://manifests.microsoft.com/win/2006/windows/WMI">
      <Id>{00000000-0000-0000-0000-000000000000}</Id>
      <ClientMachine>DC-VDB-01</ClientMachine>
      <User>NT AUTHORITY\SYSTEM</User>
      <ClientProcessId>5156</ClientProcessId>
      <Component>Unknown</Component>
      <Operation>Start IWbemServices::ExecQuery - ROOT\CIMV2 : Select CommandLine from Win32_Process Where ProcessID = '26916'</Operation>
      <ResultCode>0x80041032</ResultCode>
      <PossibleCause>Unknown</PossibleCause>
    </Operation_ClientFailure>
  </UserData>
</Event>

 

...I assume this what others are seeing above...but trying the "lodctr /R despite being successful, in terms of both successful rebuild and resync...I am still seeing the same issue.

https://support.citrix.com/article/CTX223754

 

We are using a "golden image" via MCS...so I was going to look into this as well - https://support.citrix.com/article/CTX238677

 

...I really appreciate any assistance.  Thanks again.

[Joe W]

All of our high WMI cpu instances are always coupled with the Windows spooler service (both on Windows 2008 R2/XA7.6 & Windows 2016/XA7.15).

 

 

I have yet to find a cause... or a solution.