Disconnected ghost sessions with 4 processes

[Phil Harle]

I've noticed some of our 2012 R2 session host servers display odd behaviour when some user log off. It hangs onto a 'ghost' session in task manager where the username isn't visible - the screengrab attached explains this better.

Right-clicking on a user to sign-off doesn't do anything. Any ideas?

 

Phil

[David Tesdall]

From command line run:

qwinsta

 

From here you will see a line with a blank username but the status will be "disc". Short for disconnected. Next to "disc" will be a number like "91"

 

 run "query process /ID:yourid” no quotes and with the number from the first line

 

query process /ID:91

 

taskkill does not always work for me but going back into task manager and finding the pid and right clicking on it and selecting end task has always worked for me.

 

The user should now be kicked out and the user can try again.

 

This is still happening to me on WIndows server 2016 with Citrix 7.15.CU2

[Casey Cockfield]

This occurs when the system can not finish logging someone out. If it kills enough processes where only the system processes are running, and nothing is tied to your account, it will show as a ghost. If there is still something running under your users name, such as in the picture, Outlook.exe, it will keep your name. 

A lot of times I've seen AV software hang up a session from fully logging off. McAfee is the worst. Make sure your IA team is using best practice for your servers. 

Another note, between the tickets I've opened with Citrix and Microsoft, this tends to be a Microsoft issue most of the time. No silver bullet though for this one, just have to troubleshoot and understand what you are dealing with etc. 

[Balaji Muthukrishnan]

In my case, all the affected sessions are showing as disconnected in studio with "Current user" as Blank. if I check the session state by command line using get-brokersession cmdlet...I can see -LogonInProgress parameter is set to "True" for all the affected sessions., whereas for the other disconnected sessions -LogonInProgress parameter is set to "False".

But I am seeing similar event id of 10 and 12 with event source as "RPM". When I login to HSD to check the running process for the user session, i can see only very few process (Runtime process,User mode font drive host, citrix graphics, windows logon Application and Windows Logon User Interface Host).

 

Tried killing the LogonUI process for the user session (Windows Logon User Interface Host), and session gracefully logoff.  From the scenario, so far, when a user tries to login for some reason session is broken, after certain timeout, studio marks the session as disconnected, but controller thinks, session is still in logon phase, probably thats the reason -LogonInProgress parameter is still set to "True".  So far, I am manually killing the process LogonUI to fix the issue.

 

I am correlating to KBs for the issue, which I am yet to try...

 

https://support.citrix.com/article/CTX232490   - After Installing VDA 7.15 on Windows 2016, Unable to Reconnect to Existing Session or Start a New Session

https://support.citrix.com/article/CTX286701  -  Users getting disconnected while working in the published applications

 

 

I have applied the fix suggested in  (https://support.citrix.com/article/CTX232490    - After Installing VDA 7.15 on Windows 2016, Unable to Reconnect to Existing Session or Start a New Session ) CTX KB in the OS layerbut it didnt fix the issue. I am still seeing ghost disconnected sessions

 

 

after spending a lot of time in collecting memory dump and cdf montior traces. support came up with few findings on this issue...

 

1. could be due to user trying to do multiple clicks on the receiver for web SF Application/Desktop shortcuts, causing one of the earlier launched session to fail and goes in to disconnect/Ghost session. from the logs saw, user tried multiple clicks on the SF side causing the first and subsequent launches to fail finally the last click succeeded - extend multiple click timeout using https://support.citrix.com/article/CTX236169

 

2. As usual AV or Security tools causing, revisit on the Exclusions (In My case  Trend, Tanium)

 

am still not seeing a way out on this issue...

[Balaji Muthukrishnan]

Finally after a long struggle, I was able to fix the issue with multiple changes...here are the registry tweaks...

 

1. AutoEndTasks set 1 for force closing Apps like Notepad, word, etc...which are not saved. causing signoff/shutdown/restart issues (HKEY_USERS\.DEFAULT\Control Panel\Desktop" -Type String -Name "AutoEndTasks" -Value 1)

 

2.Adding a set of Process list (system/Application/AV including winlogon.exe,csrss.exe,dwm and LogonUI.exe, etc...) to LogoffCheckSysModules key. After trying many many combinations, finally graceful logoff worked by adding csrss.exe in the list (HKLM:\SYSTEM\CurrentControlSet\Control\Citrix\wfshell\TWI\LogoffCheckSysModules)

 

3.Adding same set of process ((system/Application/AV including winlogon.exe,csrss.exe,dwm and LogonUI.exe, etc...)) in the Sysprocs key (HKLM:\SYSTEM\CurrentControlSet\Control\Terminal Server\Sysprocs)

 

4.Previously i had the these 2 entries added as well, 

HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\Reconnect - FastReconnect=0

HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\Reconnect  - DisableGPCalculation=1

 

Hope it helps someone...Thanks...

[Rob Stephens]

Restart the Citrix Profile Management Service. 

[Phil Harle]

We don't use Citrix UPM, instead we use vanilla Windows roaming profiles.

The Citrix Profile Management Service is stopped on the session hosts. I wonder if starting the service across all hosts, even though it shouldn't be used, would assist?

[Phil Harle]

Just to add some further information to this in the hope that it might heed some more clues.

 

It seems that most (99%, but not all!) of these ghost sessions disconnect after a while. I assume this is due to our session timeout configuration in group policy RDS settings, that logs off all inactive disconnected sessions after 1 hour of inactivity.

 

What this doesn't explain is why some disconnected sessions show the username, and also the list of running applications (attached), whereas these ghost sessions don't list the username nor any 'real' applications.

 

First thought is that there was a process running that I should be adding to the HKLM\SYSTEM\CurrentControlSet\Control\Citrix\wfshell\TWI\LogoffCheckSysModules key, however this doesn't explain why the username is showing as blank for the session, and out of the four processes running I think they're all required and shouldn't be added to the LogoffCheckSysModule key.

 

Still head-scratching...

[Phil Harle]

This problem got worse over the last 24 hours and the session host servers were hanging onto loads of 'ghost' sessions, never timing out and disconnecting, and were therefore preventing users from reconnected or starting new sessions.

 

It transpired that stopping the windows search service cleared all the ghost sessions within a few seconds. However as the service automatically starts up again the issue would reoccur.

 

The search service was installed to fix this issue with Outlook 2013 (also on Server 2012 R2) http://social.technet.microsoft.com/Forums/office/en-US/b8e8277c-a2f9-426a-bb3d-3f45bd0a47fc/installation-runs-everytime-we-start-outlook-2013-application-in-terminal-services?forum=officesetupdeploy

 

However, it seems that you can also fix the same issue by having the search service installed but disabled. I've disabled the service across all of our session hosts and am monitoring today.

[Muhammad Quaseem Noor Siddiqi]

Hi Phil, I know I am posting in a very old post. But wanted to check that did the issue resolved completely as we are also facing the same issue.

[Zaheer Patel1709156747]

Is this issue still happening... Any fix yet ? 

[Michael Brown1709162373]

This solution for me was adding a DWORD32 entry for DisableLogonUISuppression here

 

HKLM\Software\Citrix\Citrix Virtual Desktop Agent and setting it to 1 in the registry.

 

This forces the user to enter credentials each time they try to access the app, but at least the app launches now.

[Mike Geubel1709154567]

Hi Balaji, do you mean the antivirus process with system/Application/AV?

 

[Joshua Millen]

I am adding this because I am sure it will help someone from banging their heads against a wall.

 

Server 2012 r2
VDA 7.15 cu3

Storefront 7.15 cu3 (then upgraded to 1912 CU5)

DDCs 7.15 cu3

 

Did start seeing the "disconnected" with "Application not running" sessions in Studio with no username associated with the session. I could track down the IP/Client info.

In windows task manager on the server, we had the "disconnected" with the number of processes running for that failed session.

 

Turns out, the sessions were not "disconnected", they just failed to log in entirely. I had users, when launching apps to certain delivery groups that would get stuck at "welcome" and then disappear without launching the app.

 

My eureka moment...

I added one of the impacted users to my citrix admin group to give them admin on the VDA.

I launched into vmware on their machine and had them log DIRECTLY into the windows console of the Server 2012 R2 VDA. And it just sat at the "Welcome" and spinning. Never ended. I could kill the "winlogon.exe" process for that instance to kill the login process and get back to the credential prompt. But WINDOWS could not process the login for that particular user.

 

I could see in event viewer that the "explicate credentials" logon was "successful". But it was not going beyond that. I did not dive deep into logs after that and event viewer was useless. Also, since the logon process was not being handed off to UPM there were no useful citrix logs. Which is also why the app never launched and you get the "disconnected" sessions... windows is still trying to log the user session in while Citrix has timed out and abandoned it.

 

And it followed the user, not any particular device. I even deleted the VDA AD computer objects and let PVS recreate them. No dice.

So, I figured something must be fudged with the user object. Some arbitrary BS SID or Kerberos setting or attribute that did not get changes/updated for whatever reason.

Figured I would have to delete the AD user object and recreate it. Thankfully I did not have to.

 

Reset the user's password.....

And holy crap, they could now launch all their apps.

 

This outcome was duplicated over every user that had this issue. (only 5 so far)

 

__________________________

The only error event recorded from Citrix was on the DDC. When Citrix abandoned the session after logon timeout was reached.

Citrix-XenDesktop-BrokerMonitor

EventID 45 

BrokeringUpn USERNAME@ORG 

ClientName TESTINGPC01

FailureReason 7