Citrix FAS and Event ID 107

[John Taussig]

Our Citrix FAS implementation is working fine for initial authentication.  All users are able to access their virtual desktops with no problems or errors on any of the components.  The problem that we're having occurs 10 hours after the initial login.

 

If a user logs in and then disconnects the session, then the VDA crashes (and reboots) exactly 10 hours after the initial login.  We know that this is caused by our recent FAS implementation because the crash is preceded by a large number of Event ID 107 logged in the application event log of the VDA (event details below).  No errors are found on the FAS, StoreFront, DDC, or CA servers; the errors are only found on the VDA.  We do not use session limits or power management on our virtual desktops -- our users have always been able to maintain a session for days at a time.

 

Our Citrix components are all on the 1808 version.

 

Log Name:      Application
Source:        Citrix.Authentication.IdentityAssertion
Date:          1/23/2019 4:00:47 AM
Event ID:      107
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      xd10g-x86-0199.asco.local
Description:
[S107] HdxCredentialSelector::PerformCertificateHash() Failed: [Error: Access Denied 
Server stack trace: 
   at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc)
   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]: 
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at Citrix.Authentication.UserCredentialServices.ILogonCsp.SignHash(String cookie, String containerName, Int32 keyNumber, Int32 hashId, Byte[] hashToBeSigned)
   at Citrix.Authentication.IdentityAssertion.HdxCredentialSelector.<>c__DisplayClass14.<PerformCertificateHash>b__13()]
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Citrix.Authentication.IdentityAssertion" />
    <EventID Qualifiers="0">107</EventID>
    <Level>2</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2019-01-23T09:00:47.953931400Z" />
    <EventRecordID>25287</EventRecordID>
    <Channel>Application</Channel>
    <Computer>xd10g-x86-0199.asco.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>[S107] HdxCredentialSelector::PerformCertificateHash() Failed: [Error: Access Denied 
Server stack trace: 
   at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc&amp; rpc)
   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]: 
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData&amp; msgData, Int32 type)
   at Citrix.Authentication.UserCredentialServices.ILogonCsp.SignHash(String cookie, String containerName, Int32 keyNumber, Int32 hashId, Byte[] hashToBeSigned)
   at Citrix.Authentication.IdentityAssertion.HdxCredentialSelector.&lt;&gt;c__DisplayClass14.&lt;PerformCertificateHash&gt;b__13()]</Data>
  </EventData>
</Event>

 

 

[Jesse Mortimore]

We are seeing the exact same situation, except that instead of the machine crashing it breaks all mapped drives and the users cannot save any documents they are working on and they cannot open anything from the mapped drive! Have you found a solution to this as of yet? We have found that if the simply disconnect and reconnect it will reconnect the mapped drives and start the 10 hour timer over again until the next day. We are seeing the same thing in regards to the event viewer only showing issues on the VDA and not the SF, DDC, FAS servers

[John Taussig]

Thanks for responding, jmortimore_NC.  I was eventually able to determine that the problem was a kerberos ticketing problem.  Because user authentication uses a certificate instead of a password, the kerberos renewal process wasn't completing properly unless a disconnect/reconnect was performed before ticket expiry.

 

The reason our VDAs were crashing completely is because we use profile disks (FSLogix) and the logged in users were losing access to the file share where the profile disks are stored.  I configured FSLogix to use machine account authentication instead of user account authentication to fix that specific problem.

 

The "klist" and the "klist purge" commands were very helpful in troubleshooting the problem.  I found that my kerberos tickets had a "Renew Time" of 10 hours when the default is 7 days.

 

It seems like a session reconnect event is fundamentally different than an initial login because all subsequent tickets are renewed successfully after a reconnect, whereas they still don't seem to renew correctly if they expire before a session reconnect takes place.  I'm not sure why this is the case.....it's something that I'm still looking at.

 

 

[Jesse Mortimore]

JTaussig145, have you been able to resolve this or get any further information? Where did you find the renewal time and were you able to change that time frame?

[John Taussig]

It's resolved for us.  Run the klist command inside of a command prompt on your VDA.  Look at the "Renew Time" value on cached ticket #0.  The default setting for this value is 7 days, not 10 hours (ours was originally stuck at 10 hours).  The relevant GPO setting is Computer Configuration....Windows Settings....Security Settings......Account Policies....Kerberos Policy.....Maximum lifetime for user ticket renewal.  Note that this GPO setting must be applied to the entire domain.

 

[Jesse Mortimore]

So I am looking at that and my end time is exactly 10 hours after my start time. My Renew time is in fact 7 days. Is that what you see as well?

 

[Jessie Keisling]

We just fought an issue related to this for about a week and a half. We were seeing very similar symptoms, but triple verified our Kerberos settings were correct.

The root cause for us was that we were taking advantage of the "Temporary Group Membership" PAM feature that was introduced in Server 2016. This essentially sets a TTL flag on the user's account for a group membership add. See URL below:

https://docs.microsoft.com/en-us/archive/blogs/askds/previewing-server-2016-tp4-temporary-group-memberships

When you use this feature, the TGT lifetime is set to the shortest TTL that the user currently has. Anything less than 1 week (if your renew time is one week) sets the renew time to match the end time of the Kerberos ticket. Any TTL less than 10 hours, and the ticket life is shortened to whatever the shortest TTL is.

Unfortunately FAS wasn't developed around this and only presents the logon Smartcard (PIN) for 5 minutes after logon. When the Kerberos ticket expires and it's reached the end of the "Renew Time", default behavior is to use cached credentials to request a kerberos ticket (with a smartcard, this is a PIN). When this happens, the logon smartcard isn't available, it isn't able to renew and you see the symptoms mentioned above. 

[Andrew Gresbach1709152664]

curious if anyone else found anything more on this issue.  we are in the exact situation where we have that policy set to 7 days but renew time shows 10hrs.   Our DC's are still on 2012 so not sure if the temporary group membership feature will apply to our situation.  I am going to open a ticket on it but thought i'd check in here as well

[Matt Grojean1709162436]

Hi everyone, reviving this thread as I am experiencing this issue but the renew time is indeed 7 days and expiration is 10 hours as expected. But I am still getting FAS error 107 on the VDAs. Not sure what else could be missing right now. Still digging. 

 

[Torsten Noslashrgaard Thomsen]

On 11/15/2020 at 9:08 PM, Matt Grojean1709162436 said:

Hi everyone, reviving this thread as I am experiencing this issue but the renew time is indeed 7 days and expiration is 10 hours as expected. But I am still getting FAS error 107 on the VDAs. Not sure what else could be missing right now. Still digging. 

 

Hi. Probably this one: https://www.bleepingcomputer.com/news/microsoft/windows-kerberos-authentication-breaks-due-to-security-updates/

/ Torsten

[Matt Grojean1709162436]

6 minutes ago, Torsten Noslashrgaard Thomsen said:

Hi. Probably this one: https://www.bleepingcomputer.com/news/microsoft/windows-kerberos-authentication-breaks-due-to-security-updates/

/ Torsten

 
Indeed it was. I had to roll back a DC patch prior to MS fixing this in a later update around Thursday I believe. 

[Joshua Collins1709156758]

This KB doesn't appear to be on our DCs but it is on our Storefronts and the install time does line up with when we started having the issue.

 

Do you think it being on the Storefronts could cause the issue?

[Matt Grojean1709162436]

5 minutes ago, Joshua Collins1709156758 said:

This KB doesn't appear to be on our DCs but it is on our Storefronts and the install time does line up with when we started having the issue.

 

Do you think it being on the Storefronts could cause the issue?

Look for KB4586793 or KB4587793 on the DCs. I don't expect the storefront to have the issue but I'm using Citrix Cloud so my infrastructure is only Cloud Connectors, FAS servers and VDAs. No StoreFront and no DCs of course. 

[Datacom Vic Citrix]

We were are also having the same issue.




The following patch is responsible.

CAUSE:
https://support.microsoft.com/en-au/help/4586830/windows-10-update-kb4586830

 

FIX:

https://support.microsoft.com/en-au/help/4594441/windows-10-update-kb4594441


Hope this helps.

[Joshua Collins1709156758]

On 11/24/2020 at 9:59 AM, Matt Grojean1709162436 said:

Look for KB4586793 or KB4587793 on the DCs. I don't expect the storefront to have the issue but I'm using Citrix Cloud so my infrastructure is only Cloud Connectors, FAS servers and VDAs. No StoreFront and no DCs of course. 

 

We did eventually find the bad KB on the a couple of the DCs and this resolved all issues.

[Matt Grojean1709162436]

1 hour ago, Joshua Collins1709156758 said:

 

We did eventually find the bad KB on the a couple of the DCs and this resolved all issues.

 Be careful with the patch they said resolves the issue. I too tried to apply the updates followed by the resolution patch and continued to receive the 107 errors again and had to roll back. 

[Datacom Vic Citrix]

8 hours ago, Matt Grojean1709162436 said:

 Be careful with the patch they said resolves the issue. I too tried to apply the updates followed by the resolution patch and continued to receive the 107 errors again and had to roll back. 

You are still receiving errors until the user sessions have renewed Kerberos certificates. All users must log off and back on again. Until they have logged off and back on again the errors will persist with the expired Kerberos sessions.

[Joe Robinson]

I'm battling this problem as well.  I've applied the kb4594442, which is the correct OOB Update for the version of Server 2019 I'm running, but a day later (after log on/log off) users are still receiving the error.  December patches are available now (or soon), so hopefully there is something in the notes to fix that.

[Matt Grojean1709162436]

Yeah, I had to roll back and forget the OOB patch. Brought me to halt.

[Bob Ballard]

We are facing this as well. Two questions, are any others that have this issue experiencing issues with logging in at all? We are getting many users who are getting "cannot complete your request" errors when disconnecting their session or logging in for the first time in the morning. Second, does anyone know if the current December patch resolves this without needing to role back any of the other patches?

[Matt Grojean1709162436]

That sounds sort of unrelated but with bad Kerberos behavior, a lot can fall apart. 

 

As for the December patches, I have heard they included a fix but the update must be applied to both the domain controllers and the VDAs to work. VDAs must have it first or at the same time as the DCs. Not second. 

[Bob Ballard]

Also do the DC' specifically have to have KB4586793 and KB4587793 on them to be the cause of this Kerberos or is the cumulative updates from November (KB4586830) also the cause of these errors?