Citrix UPM stops cleaning out profiles after a couple of locked files

[Christopher Andrews]

Our environment:

• Windows 2008 R2 64 bit
• XenApp 6.5 RU4
• PVS 6.1
• UPM 5.2.1.50.20
• 120 second delay set on deleting logged off profiles

There are acutally two issues, one there are lots of locked folders and registries from random profiles.  Two the UPM stops trying to remove old profiles if it cannot remove the oldest profile.

 

The locked files and folders:

• The SYSTEM process is holding users NTUSER.DAT (registry) files open.
• The C:\Users\userid\AppData\Local\Microsoft\Credentials and C:\Users\userid\AppData\Roaming\Microsoft\Credentials folders occasiotnally have ACLs that prevent Local Admins from viewing or change permissions, or changing ownership of the folder.

The more annoying part of this is that after five attempts to delete a profile, UPM gives up and stops trying to delete any more logged off profiles.

 

2015-07-06;11:12:44.297;ERROR;;;;5356;DeleteLocalProfile: Deleting 'C:\Users\userid01\ntuser.dat' from the local profile failed. Error 0x20: The process cannot access the file because it is being used by another process.
2015-07-06;11:12:44.297;ERROR;;;;5356;DeleteLocalProfile: Could not delete local profile: <C:\Users\userid01>: The process cannot access the file because it is being used by another process.
2015-07-06;11:13:44.299;INFORMATION;;;;12480;PeriodicMaintenance: A thread has terminated. Closing its handle.
2015-07-06;11:13:44.299;ERROR;;;;10864;DeleteLocalProfile: Deleting 'C:\Users\userid01\ntuser.dat' from the local profile failed. Error 0x20: The process cannot access the file because it is being used by another process.
2015-07-06;11:13:44.299;ERROR;;;;10864;DeleteLocalProfile: Could not delete local profile: <C:\Users\userid01>: The process cannot access the file because it is being used by another process.
2015-07-06;11:14:44.300;INFORMATION;;;;12480;PeriodicMaintenance: A thread has terminated. Closing its handle.
2015-07-06;11:14:44.300;ERROR;;;;11892;DeleteLocalProfile: Deleting 'C:\Users\userid01\ntuser.dat' from the local profile failed. Error 0x20: The process cannot access the file because it is being used by another process.
2015-07-06;11:14:44.300;ERROR;;;;11892;DeleteLocalProfile: Could not delete local profile: <C:\Users\userid01>: The process cannot access the file because it is being used by another process.
2015-07-06;11:15:44.302;INFORMATION;;;;12480;PeriodicMaintenance: A thread has terminated. Closing its handle.
2015-07-06;11:15:44.302;ERROR;;;;8408;DeleteLocalProfile: Deleting 'C:\Users\userid01\ntuser.dat' from the local profile failed. Error 0x20: The process cannot access the file because it is being used by another process.
2015-07-06;11:15:44.302;ERROR;;;;8408;DeleteLocalProfile: Could not delete local profile: <C:\Users\userid01>: The process cannot access the file because it is being used by another process.
2015-07-06;11:16:44.304;INFORMATION;;;;12480;PeriodicMaintenance: A thread has terminated. Closing its handle.
2015-07-06;11:16:44.304;ERROR;;;;9856;DeleteLocalProfile: Deleting 'C:\Users\userid01\ntuser.dat' from the local profile failed. Error 0x20: The process cannot access the file because it is being used by another process.
2015-07-06;11:16:44.304;ERROR;;;;9856;DeleteLocalProfile: Could not delete local profile: <C:\Users\userid01>: The process cannot access the file because it is being used by another process.
2015-07-06;11:17:44.305;INFORMATION;;;;12480;PeriodicMaintenance: A thread has terminated. Closing its handle.

 

If you restart the UPM process it will try again and stop functioning after five more attempts.  There are no messages or errors added to the Windows event logs. I can reboot the servers back to the golden image, but that is a gross fix and a waste of people's times.

 

Any one have any ideas?  Any way Citrix will fix this bug without me having to burn one of our support tickets?

 

 

[Miroslav Blasko]

what files or folders left on the server...

 

what about this:

http://discussions.citrix.com/topic/355546-cannot-delete-acrord32-sbx-folder/page-2

[Steven Arrington1709155611]

We are having this exact same issue.  We opened a ticket with Citrix and all they have recommended was to turn on additional logging.....Not the most helpful at the moment but hopefully our logs reveal something.

[Steve H]

Hello - did you ever resolve this - very similar issue here. NTUser.dat's held open by the System process. I can go in and unload the hive just fine, UPM removes it shortly after.

[Wenfeng Yi]

Hi all

It  seems that UPM can't delete the local profile when logoff, you could  try to enable the UPM policy "delay before delete local cache", it may help.

As a workaround, you could also disable the UPM policy "delete local cache profile on logoff".

[Ken Z]

Wenfeng

 

Christopher mentions that he's already got this set to 120 seconds...

 

I'm assuming that everyone's got AntiVirus software installed? have they included UPM in the AV exclusions list?

 

regards

 

Ken Z

[don shears]

We are having the same issue, delay on deleting profile is set to 180 seconds but we still have a very small number of users having the issue. renaming their profile fixes it for a couple of days but the issue re-occurs for the same users.

[Joe Robinson]

I've had this problem from time to time, and I noticed that the spooler was locking the folders.  I have no idea why spooler would have something in the credentials subfolder of the user profile open, but it did.  I could then restart the spooler and clean up the stale profile.

 

If you fire up process explorer, you can search for a handle to that folder and it will help you identify the process that has the profile open.  Then you can (hopefully) restart that process and get your profiles cleaned up.

I know..the real solution is why these folders are getting locked, but I've been fighting this problem since UPM 3.x.  I've had cases open, and I know others have as well. This will at least give you a way to fix a machine if you can't restart it.

[don shears]

same issue here, I ran process monitor and seen that one of our effected users on logoff there was still a handle on their cookies session. I can also see that it is failing to release the NTuser.dat file also. We have had a support all open with Citrix for over a month now and still no where nearer in fixing the issue. we have followed all the best practices. We only a have small amount of users with the issue on medium size farm and it seems to be the same 5 people constantly effected.

 

I can see a profile now that is dated back to the 20th Jan and I unable to remove the inactive profile, when I go advanced options and select user profiles to try to remove the profile cleanly the delete option is greyed out. I tried using delprof tool to delete inactive profiles but fails to delete the profile. We have a GPO that deletes cached profiles older than 3 days but this seems to be failing to work even though it has the gpo successfully applied.

[CarlStalhood]

You can use procexp.exe to search for the process that has the file or registry key opened. Usually a security program.

[don shears]

I have tried using process explorer and found that some of users during logoff their profile still had a handle on their cookie sessions so we added this to the list of exclusions and worked for about week and the same set of users have all reported the same problem. everytime they log back on they have to reconfigure their profile. Looking at one of their accounts now that should have been deleted on logoff and delprof won't delete the profile and the user is not even connected at the moment. I have tried to delete the profile through system properties, user profiles but the delete option is greyed out.

[CarlStalhood]

Find a file in c:\users\%username% that isn't being deleted. Or find the user's registry under HKEY_USERS.

 

In procexp.exe, click the binoculars icon and search for the file or registry key. This should tell you which process has a handle to the file or registry key.

[David Tesdall1709161072]

The year is 2020 and and I am running Windows Server 2019 and running into this same issue. System (PID 4) is holding the ntuser.dat file open. Anyone every get this fixed?

[Beat Buser1709156910]

On 4/7/2020 at 7:23 PM, David Tesdall1709161072 said:

The year is 2020 and and I am running Windows Server 2019 and running into this same issue. System (PID 4) is holding the ntuser.dat file open. Anyone every get this fixed?

 

Same here. Win2019, LTSR 1912 CU1, System with PID 4 is locking/blocking the ntuser.dat and some other files from deleting by the UPM.

[Peter Mommerency]

On 4/7/2020 at 7:23 PM, David Tesdall1709161072 said:

The year is 2020 and and I am running Windows Server 2019 and running into this same issue. System (PID 4) is holding the ntuser.dat file open. Anyone every get this fixed?

The year is 2021 and I'm seeing this on a W2K12 server with LTSR 1912 CU1. Although in my case it's not ntuser.dat, but the WinX directories and sometimes Google Chrome directories. 
Anyone know if this is fixed in CU2?

[Vincent Lewin1709156903]

The year is 2021, March to be exact and I am having this issue on a windows server 2019 1809 running LTSR 1912 CU2.

My folder is 3D Objects pluse WinX folders.

The process that is locking the 3D Objects folder is UserProfileManager.exe!

I also have no roaming desktop on 2019 pluse other issues.

 

I would like to start with this duplicate profile .000/.001/.002 problems first.

Does anyone have a fix?

 

[Bradd Schlosser]

I am in the same boat on VDA 2106 with Server 2016 VDAs (w UPM hot fix) in UPM enabled. 60 second delay deleting locally cached profiles. UPM is locking various files kind of at random with users, and subsequently....local profiles stick. For what it's worth we migrated away from 7.15 LTSR and were on CU5 previously. Did not experiencing this issue there. Nightly reboots correct the problem, and I have combed through all of my policies to make sure things should not be in conflict. I know this is reviving the oldest thread ever, but it is what it is. Carl if you have any insight as to what could be causing this, it would be much appreciated. 

[Quang Hong Huynh1709162616]

Same problem here with Server 2019 and 2103 VDA.

In my case, only "LOG.old" are being held by UPM process, 60 s delay is configured, AV is disabled (for testing), and "LOG.old" is excluded from sync.

 

These LOG.old files are in MS Teams, MS Edge and Chrome subfolders.

 

 

Edit:

After restarting UPM service the files are freed to delete but this is not a satisfying "solution"...

[Bradd Schlosser]

I did see in the patch notes for Profile Management 2109 that they acknowledged issues in 2106 (maybe past issues as well). I wish they were more transparent with this stuff. I have not upgraded at this point, but will put it in planning.

 

The fixes mentioned that might correlate to us specifically (though not even remotely detailed) are as follows:

 

"Logging on to the same VDA again might cause a duplicate profile. [CVADHELP-17676]"

"The desktop might become unusable in the following cases:

When you log off and log back on in a short interval [CVADHELP-17941]"

 

https://docs.citrix.com/en-us/profile-management/current-release/whats-new/fixed-issues.html

 

Actually semi-hilarious how broad those fix descriptions are, for probably issues that are extremely granular but detrimental in use.