Jump to content
Welcome to our new Citrix community!
  • 6

Just Grey Screen instead of Published Desktop


Rene Balz1709162432

Question

Hi,

 

over the last few weeks, a weird Citrix issue started to occur on several completely independent Citrix Session Hosts (from different AD domains). Some details about the environments:

 

  • Server 2019 based Hyper-V architecture
  • Virtual Citrix Session Host (Published Desktop) with fully patched Windows Server 2019 and VDA 7 2012

 

The issue is as follows:

 

  • In the event log of the Session Host, an CtxUvi error with ID 1005 is being logged: "The Citrix Universal DLL Injection Driver has encountered an unexpected error."
  • Right after that, an error 1003 (also from CtxUvi) is being logged:  "The Citrix Universal DLL Injection Driver has detected an integrity error during process creation. The Citrix Universal DLL Injection Driver has been disabled."
  • From now on, all users who freshly log into the desktop just receive a blank grey screen as soon as Citrix Workspace is creating the session window.
  • After a server reboot, everything works normal again.

 

Some observations / details:

 

  • The issue is occuring about once every week (per Server) but there is no temporal pattern to be found.
  • The CtxUvi crash is always being caused by a user who's logging off, so the issue always occurs either before lunchbreak or in the evening (~5 pm). But it's never the same user who's causing it.
  • Apart from the grey screen, the session actually works. It is, as an example, possible to blindly create a text file on the desktop (with keyboard shortcuts), and the file actually gets created.
  • Like I said, several customers are affected.

 

Does someone have any ideas how to fix this or tips for further diagnosis?

 

Best regards

 

 

Link to comment
  • Answers 68
  • Created
  • Last Reply

Top Posters For This Question

Recommended Posts

  • 1

Hi All, 

I'm not alone on this hooray!

I've been going around in circles trying to work out what's going on here and have the exact same issues as you with the CtxUvi events (how I found this post).

First I thought it was the version we were using which was Citrix Virtual Apps and Desktop 7 1906.  Haven't wanted to disrupt anything with upgrades during pandemic and last week realised its out of support so upgraded to 2012 and still the same issue happening. 

 

We think this started occuring about 6 weeks ago but can't be 100% sure and if it was then that coincides with our monthly windows updates server patching.  But again can't be 100% on that. 

 

I have 30 Servers and it always starts happening after lunch for my users on random servers but not all of them.   Looking at the event logs today on one server this ties in as the first instance of the CtxUvi issue which was at 12:55:10 then every 5 mins there after until I reboot the server.   So can't be a GPO applying as that happening all the time (24hrs we have users in with night shifts etc).  I cant see any obvious correlation with other events on the server but its like a needle in a haystack 

 

We don't use Trend AV but have Sophos with Behaviour Monitoring.  Nothing showing in the AV logs  at normal logging level so I'm turning on verbose logging on a number of the servers and will see what that shows up.  

 

I will update anything else I find also but has anyone logged this with Citrix yet?

 

Regards

  • Like 1
Link to comment
  • 1
5 hours ago, Dennis Reimer1709161851 said:

Can you tell more details. Why should this Adobe Reader process cause the CTXUvi/Greyscreen Issue? I´m asking because I have several customers with this issue and haven´t found a final solution yet. 

 

Sure! 

 

Last year, when I created this discussion, every CTXUvi crash I've analyzed was triggered by a user logging off - no exceptions. Thanks to the post from Graeme Dunkley (page 4 of this thread), we got the problems under control by adding some processes to the reg value "UviProcessExcludes".

 

This year, however, there never was a logoff event to be found at the same time as the CTXUvi event 1005. So I tried to find out what could be triggering the CTXUvi crash this time. I've enabled the logging of process creation and termination on several VDAs from different customers. When the next greyscreen issues occured, I located the process log entries whose timestamps were (temporally) close to the 1005 event. On multiple occasions, I saw that a process called RdrCEF.exe has been launched within a few milliseconds of the CTXUvi crash.

 

So, I've added this process to the "UviProcessExcludes" value on all affected VDAs, and there hasn't been any greyscreen problem for the last 3 weeks.

 

Best regards

  • Like 1
Link to comment
  • 1

Hi All, 

 

So we are getting this issue yet again.  Last time over a year ago and sorted with some Sophos AV exclusions and sorted. 

However I can't seem to work out what's causing the issue this time.  I've enabled verbose Sophos logging which really impacts our Citrix servers (we do a shared desktop) so I get users complain on slowness.  If i then enabled the windows audit policy of success and failures on Audit process tracking it grinds the server to unusable.  So I'm between a rock and a hard place, need to enable logging to work out what causes it, but can't enable logging as the system is unusable.  

 

We don't have Adobe installed anywhere like the issues Christian Radatz found.  At the moment I've setup a scheduled task to watch for the CtxUvi 1003 & 1005 error in the event log and send me an email so I know its occurred.  Next I need to get the server in question to be put in to maintenance mode also as part of that script just to stop the helpdesk tickets coming in.  But this is just a temp solution until we have a fix.

 

With the Sophos verbose logging I couldn't see anything out the ordinary around the time of the issue. 

 

So my question is can anyone offer a solution that doesn't impact the server to the extent it does while trying to work this out.  Christian Radatz, how did you do the Process Tracking?

 

Thanks in advance!

 

  • Like 1
Link to comment
  • 0

I have experienced a similar issue. RDP works fine and it is isolated to ICA sessions i assume?

 

We had GPOs that applied registry changes related to graphics which caused this. I spent some hours localizing the GPO, unlinked and problem solved. For me it was registry keys changes under:

 

"EnableWPFHook" - SOFTWARE\Citrix\CtxHook\AppIni_Dlls\Multiple Monitor Hook

"EnableWPFHook" - SOFTWARE\Wow6432Node\Citrix\CtxHook\AppInit_Dlls\Multiple Monitor Hook

"Open GL" - SOFTWARE\Citrix\CtxHook\AppIni_Dlls\Graphics Helper

"Open GL" - SOFTWARE\Wow6432Node\Citrix\CtxHook\AppInit_Dlls\Graphics Helper

"CUDA" -  SOFTWARE\Citrix\CtxHook\AppIni_Dlls\Graphics Helper

"CUDA" -  SOFTWARE\Wow6432Node\Citrix\CtxHook\AppInit_Dlls\Graphics Helper

 

Basically, these GPOs aided us with VDA 7.15, when upgraded to 1912 these registry settings made all ICA session connections gray. I would say the best option you have to move forward would be to unlink all GPOs in a test environment, link one at a time and then see which ones break the ICA sessions. 

 

Good luck!

Link to comment
  • 0

Hi James, 

 

Could you elaborate?

I also saw your answer in: https://discussions.citrix.com/topic/412405-citrix-1912-cu2-server-2012r2-grey-screen-on-login/

 

We have at least 3 customers on server 2019 with TrendMicro that are having this issue (the frequency of grey screens have slowed down from 1 server/day to 1/week for now)

Is there a fix or more info about TrendMicro being the issue?

Should we add new exclusions to the list?

 

The issue seems to be the same as mentioned by Rene, The CtxUvi crashes and will not work for new sessions after this.

Existing sessions are not impacted.

 

The issue seemed to be stable for the past week after we modified the system variable order on the Golden Image but today the issue occurred again.

 

Thanks

 

Link to comment
  • 0

Hi folks,

 

thank you for your replies! "Relieved" to see that others are affected as well. Indeed, TM is in the mix. WFBS 10 SP1 Patch 2274 to be precise. Servers with OfficeScan / ApexOne are affected as well. Adding all relevant TM processes to "HKLM\SYSTEM\CurrentControlSet\Services\CtxUvi\UviProcessExcludes" didn't help at all, so I've now added the following paths to the Behavior Monitoring exclusion list of Trend Micro:

 

  • C:\Program Files (x86)\Citrix\HDX\bin\*.dll
  • C:\Program Files\Citrix\HDX\bin\*.dll
  • C:\Program Files (x86)\Citrix\HDX\bin\*.exe
  • C:\Program Files\Citrix\HDX\bin\*.exe

I'll keep you posted.

 

@Gus Johansson: There are no policies writing to these locations. Verified with GPResult and CtxCseUtil.

Link to comment
  • 0

Hi again,

 

my 4 wildcard exclusions did not help; the issue is still occurring.

 

@James Kindon

 

From all relevant exclusion suggestions (from both Trend Micro and Citrix), these are the only paths that actually exist on the affected Citrix Session Host (I've tested all paths based on the attached file with PowerShell's Test-Path):

  • C:\Logs\CDF
  • C:\Program Files (x86)\Citrix\HDX\bin\WebSocketService.exe
  • C:\Program Files\Citrix\User Profile Manager\UserProfileManager.exe
  • C:\Program Files\Citrix\Virtual Desktop Agent\BrokerAgent.exe
  • C:\Windows\system32\csrss.exe
  • C:\Windows\system32\smss.exe
  • C:\Windows\System32\spoolsv.exe
  • C:\Windows\System32\winlogon.exe

 

So I've now excluded these paths as well as any .exe in "C:\Program Files\Citrix" and "C:\Program Files (x86)\Citrix" and will report back.

 

ExclusionSuggestions.txt

Link to comment
  • 0

The grey screen problem has just reoccurred. To be honest, I highly doubt that this issue is "fixable" with TM exclusions. Don't know how to move forward...

 

These are the Citrix programs excluded from Behavior Monitoring and added to Trusted Programs:

  • C:\Program Files (x86)\Citrix\Online Plugin\wfcrun32.exe

  • C:\Program Files (x86)\Citrix\Online Plugin\Browser\CtxWebBrowser.exe

  • C:\Program Files (x86)\Citrix\Online Plugin\Receiver\Receiver.exe

  • C:\Program Files (x86)\Citrix\UWACacheService\UWACacheService.exe

  • C:\Program Files (x86)\Citrix\HDX\bin\MultimediaRedirector.exe

  • C:\Program Files (x86)\Citrix\HDX\bin\cmstart.exe

  • C:\Program Files (x86)\Citrix\HDX\bin\CtxKlMapHost.exe

  • C:\Program Files (x86)\Citrix\HDX\bin\CtxSession.exe

  • C:\Program Files (x86)\Citrix\HDX\bin\IcaSt.exe

  • C:\Program Files (x86)\Citrix\HDX\bin\iexplore.exe

  • C:\Program Files (x86)\Citrix\HDX\bin\VdaRedirector.exe

  • C:\Program Files (x86)\Citrix\HDX\bin\WfShell.exe

  • C:\Program Files (x86)\Citrix\HDX\bin\EncSvc.exe

  • C:\Program Files (x86)\Citrix\HDX\bin\SemsService.exe

  • C:\Program Files (x86)\Citrix\HDX\bin\CtxMtHost.exe

  • C:\Program Files (x86)\Citrix\HDX\bin\RevSeamLauncher.exe

  • C:\Program Files (x86)\Citrix\HDX\bin\IcaImeUtil.exe

  • C:\Program Files (x86)\Citrix\HDX\bin\TouchOptimizedDesktop.exe

  • C:\Program Files (x86)\Citrix\HDX\bin\UWALauncher.exe

  • C:\Program Files (x86)\Citrix\HDX\bin\WebSocketService.exe

  • C:\Program Files (x86)\Citrix\HDX\bin\WebSocketAgent.exe

  • C:\Program Files (x86)\Citrix\HDX\bin\MediaPlayer.exe

  • C:\Program Files\Citrix\GroupPolicy\CseEngine.exe

  • C:\Program Files\Citrix\CdfCaptureService\CdfCaptureService.exe

  • C:\Program Files\Citrix\Virtual Desktop Agent\DirectorComServer.exe

  • C:\Program Files\Citrix\HDX\bin\CpSvc.exe

  • C:\Program Files\Citrix\IcaConfigTool\CreateAnonymousUsersApp.exe

  • C:\Program Files\Citrix\IcaConfigTool\IcaConfigConsole.exe

  • C:\Program Files\Citrix\HDX\bin\CtxGfx.exe

  • C:\Program Files\Citrix\HDX\bin\GfxStatusIndicator.exe

  • C:\Program Files\Citrix\HDX\bin\CtxRdr.exe

  • C:\Program Files\Citrix\HDX\bin\icak2meng.exe

  • C:\Program Files\Citrix\HDX\bin\CtxKlMapHost64.exe

  • C:\Program Files\Citrix\HDX\bin\ImaAdvanceSrv64.exe

  • C:\Program Files\Citrix\HDX\bin\SCService64.exe

  • C:\Program Files\Citrix\HDX\bin\CtxInjectMobileDesktopHook64.exe

  • C:\Program Files\Citrix\HDX\bin\CtxLocalUserSrv.exe

  • C:\Program Files\Citrix\HDX\bin\CtxSvcHost.exe

  • C:\Program Files\Citrix\HDX\bin\CtxPreLaunch.exe

  • C:\Program Files\Citrix\HDX\bin\ss3admin.exe

  • C:\Program Files\Citrix\User Profile Manager\ISessionMetrics.exe

  • C:\Program Files\Citrix\User Profile Manager\CEIP\UpmCeipSender.exe

  • C:\Program Files\Citrix\User Profile Manager\UserProfileManager.exe

  • C:\Program Files\Citrix\User Profile Manager\UpmSearchHelper.exe

  • C:\Program Files\Citrix\User Profile Manager\UpmEvent.exe

  • C:\Program Files\Citrix\User Profile Manager\UpmUserMsg.exe

  • C:\Program Files\Citrix\HDX\bin\HdxWebProxy.exe

  • C:\Program Files\Citrix\ExceptionHandler\CtxExceptionHandler64.exe

  • C:\Program Files\Citrix\ExceptionHandler\CtxExceptionHandler.exe

  • C:\Program Files\Citrix\CDF\Service\CdfSvc.exe

  • C:\Program Files\Citrix\XenDesktopVdaSetup\ConfigurationApp.exe

  • C:\Program Files\Citrix\XenDesktopVdaSetup\VerifyVdaMsiInstallStatus.exe

  • C:\Program Files\Citrix\XenDesktopVdaSetup\XenDesktopVdaSetup.exe

  • C:\Program Files\Citrix\XenDesktopVdaSetup\AnalyticsUploader.exe

  • C:\Program Files\Citrix\XenDesktopVdaSetup\CitrixMSILogAnalyzer.exe

  • C:\Program Files\Citrix\Virtual Desktop Agent\StartMenuScan.exe

  • C:\Program Files\Citrix\Virtual Desktop Agent\Agent Configuration\AgentConfig.exe

  • C:\Program Files\Citrix\ImageAnalysis\Citrix.Cam.ImageAnalysis.Console.exe

  • C:\Program Files\Citrix\Virtual Desktop Agent\Delivery Agent Tests\DeliveryAgentTests.exe

  • C:\Program Files\Citrix\Virtual Desktop Agent\BrokerAgent.exe

  • C:\Windows\system32\csrss.exe

  • C:\Windows\system32\smss.exe

  • C:\Windows\System32\spoolsv.exe

  • C:\Windows\System32\winlogon.exe

Link to comment
  • 0
On 3/19/2021 at 3:42 PM, Rene Balz1709162432 said:

The grey screen problem has just reoccurred. To be honest, I highly doubt that this issue is "fixable" with TM exclusions. Don't know how to move forward...

Did you get anywhere further with resolving this issue Rene? We have been affected on the morning of the 23rd and again this morning on the 30th. Oddly not on any other days. Our VDAs reboot daily and once rebooted they are fine again. Only around 3-4 VDAs affected on either given day... Very strange issue. We are seeing the same CtxUvi errors and we also use TM.

 

Thanks

Link to comment
  • 0
18 hours ago, Rene Balz1709162432 said:

Hi Chris,

 

not yet, unfortunately. But I disabled Behavior Monitoring today and will keep you updated. 

 

Thank you. We have not disabled anything yet. Have you had it on any particular days, seems odd that we have had it on Tuesday 23rd and then again on Tuesday 30th and not on any other days...

 

Thanks

Link to comment
  • 0

Hi!

 

I think a sample size of 2 is a bit small to draw conclusions.. Anyway, these are all occurrences in March (from just one customer):

  • 01.03.2021    Server 1
  • 02.03.2021    Server 3
  • 03.03.2021    Server 4
  • 08.03.2021    Server 1
  • 10.03.2021    Server 2
  • 12.03.2021    Server 1
  • 12.03.2021    Server 2
  • 18.03.2021    Server 3
  • 18.03.2021    Server 1
  • 19.03.2021    Server 2
  • 22.03.2021    Server 4
  • 29.03.2021    Server 3
  • 29.03.2021    Server 4
  • 31.03.2021    Server 2

I've created a scheduled task which shoots an e-mail alert when the 1003 event from CtxUvi is being logged. Most of these occurrences haven't been noticed by the customer, since we ASAP enable maintenance mode when we're getting an alert. As mentioned in my original post, the issue is always being caused by a user logging off. The timestamps of the 1003 error are always matching a logoff timestamp down to the second.

 

 

 

Link to comment
  • 0
On 4/1/2021 at 3:11 PM, Rene Balz1709162432 said:

Hi!

 

I think a sample size of 2 is a bit small to draw conclusions.. Anyway, these are all occurrences in March (from just one customer):

  • 01.03.2021    Server 1
  • 02.03.2021    Server 3
  • 03.03.2021    Server 4
  • 08.03.2021    Server 1
  • 10.03.2021    Server 2
  • 12.03.2021    Server 1
  • 12.03.2021    Server 2
  • 18.03.2021    Server 3
  • 18.03.2021    Server 1
  • 19.03.2021    Server 2
  • 22.03.2021    Server 4
  • 29.03.2021    Server 3
  • 29.03.2021    Server 4
  • 31.03.2021    Server 2

I've created a scheduled task which shoots an e-mail alert when the 1003 event from CtxUvi is being logged. Most of these occurrences haven't been noticed by the customer, since we ASAP enable maintenance mode when we're getting an alert. As mentioned in my original post, the issue is always being caused by a user logging off. The timestamps of the 1003 error are always matching a logoff timestamp down to the second.

 

 

 

 

I would agree, but as it had only happened twice, and both instances were on a Tuesday, seemed odd. Anyway, it happened on Thursday as well, so that's that!

 

Have you seen anymore instances since the 31st when you disabled the behavioral monitoring?

 

We have not yet been able to tie this down to a log off event, which is different to your issue at the moment. We are waiting to see if we have another instance so we can have more data to investigate...

 

Thanks!

Link to comment
  • 0

Hi Chris! No issues since I've disabled Behavior Monitoring. But I doubt that the servers have been heavily used since (due to the easter holidays). 

 

Just to make sure we're on the same page regarding the logoff timestamps, here's my way to analyze this (I use a script for this, but that's irrelevant):

  • Open the security log of an affected VDA server (with Event Viewer)
  • Make sure that the oldest (security) log entry is older than the most recent CtxUvi error
  • If not, consider increasing the file size limit of the security log and wait for the next incident
  • Filter the security log by event ID 4647
  • Keep the timestamp of the most recent CtxUvi error in mind and look for a logoff event with a matching timestamp (let's say down to the minute)

Have we been doing exactly the same?

Link to comment
  • 0
9 hours ago, Rene Balz1709162432 said:

Hi Chris! No issues since I've disabled Behavior Monitoring. But I doubt that the servers have been heavily used since (due to the easter holidays). 

 

Just to make sure we're on the same page regarding the logoff timestamps, here's my way to analyze this (I use a script for this, but that's irrelevant):

  • Open the security log of an affected VDA server (with Event Viewer)
  • Make sure that the oldest (security) log entry is older than the most recent CtxUvi error
  • If not, consider increasing the file size limit of the security log and wait for the next incident
  • Filter the security log by event ID 4647
  • Keep the timestamp of the most recent CtxUvi error in mind and look for a logoff event with a matching timestamp (let's say down to the minute)

Have we been doing exactly the same?

 

Hi

 

Yes, we are doing the same, but we can't seem to find a log off event matching the error event...

 

We disabled BM on Thursday (while waiting for another occurrence of the issue), but we had a recurrence of the error yesterday, after it had been disabled, so 90% sure it isn't BM...

 

We found an app long event ID 1530 entry, with the same timestamp as the CtxUvi error, saying that certain registry handles were locked by TMBMSRC.exe (Trend). Research shows that exe is used by:

Behavior Monitoring

Device Control

Certified Safe Software Service

Client Self-Protection (stops the Trend services being messed with)

 

We are now going to test each of them to see if we can identify which component is causing the issue...

 

We have however now also identified that in at least two cases we have a log on 4648 event for an internal application, at the exact same second as the CtxUvi error... That application does edit the registry! It is a custom application, which is run automatically when all users login, but it will only 'login' and edit the registry when the users enter some info and clicks OK. This might happen straight after the user has logged in, or they might ignore it and do it 10, 20 mins later, or not at all. But almost all of our users would be clicking OK at some point shortly after logging in, so it is odd that not all of our users/VDAs are being affected every day...

 

We are waiting to confirm which Trend component is likely the cause, then we will look to add an exception in for that logon app.

 

Will update with outcome...

Link to comment
  • 0
24 minutes ago, Rene Balz1709162432 said:

Hi! Since we've disabled BM 16 days ago, the issue hasn't been occurring at all. Zero incidents! I just re-enabled BM and updated WFBS to patch 2309. Will report back.

 

When I posted the other day I said it was disabled and we had another occurrence, it was supposed to be disabled, but somehow it had become enabled again.

 

In any event, we excluded this application from the BM and since doing so it has been a week and no more occurrences! 

 

24 minutes ago, Rene Balz said:

@Chris Gundry: I don't think we've got the same issue...

 

Perhaps not, but certainly similar...

 

24 minutes ago, Rene Balz said:

Does it change values under the CTXHook key?

 

No, which is the strange part of all of this. But we have confirmed it is this application which is lining up with the error in our case. Perhaps Trend is changing something in that location as a result, we are not sure.

Link to comment
  • 0
On 4/16/2021 at 4:49 PM, Rene Balz1709162432 said:

Hi! Since we've disabled BM 16 days ago, the issue hasn't been occurring at all. Zero incidents! I just re-enabled BM and updated WFBS to patch 2309. Will report back.

 

@Chris Gundry: I don't think we've got the same issue...

 

 

Does it change values under the CTXHook key?

 

I have a customer with the same issue. We've had all TM parts disabled from the start, except the real time scan (with all the exclusions) and predictive machine learning. Might just kill that one as well and see where it goes.

Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×
×
  • Create New...