Occasional Protocol Driver Error - EDT Enabled - 7.15 CU1

[Dennis Parker]

Can't find any related posts that sound like what we have going on....

 

Occasionally, a server will be unable to make new EDT/UDP connections and get a "Protocol Driver Error" until the server is restarted. I have attempted to restart almost every service on the systems having the issue and none seem to correct the issue. Since the RDP service is tied to the UDP port, I believe this is where the issue lies. The only way to get the server to accept connections again is to restart.

 

In the event logs, I always see some sort of update, usually Adobe Flash Player Update Service, but occasionally Windows Module Installer that seem to correlate to the start of the issue for the server.

 

Environment:

Mix of 2008R2, 2012R2 and 2016 VDA servers - have not seen the error on the 2016 servers yet, 2008R2 is vastly more affected, but there have been a few 2012R2 instances, including one yesterday. I have about 200 VDA systems. The VDAs are all running 7.15 CU1. The issue did not start when we enabled EDT/UDP but rather about 2 months after.

Core component servers (which shouldn't seem to matter for this) are all 7.15CU1 releases running on 2012R2 servers.

NetScaler 11.1.57.19

 

The issue only affects one VDA at a time and a restart clears the issue.

 

A Process Monitor trace shows the UDP connection from the Netscaler reaching the SVCHOST (termserv) process, but the process does not respond.

WireShark trace also shows the UDP packet making it to the server in question, of course.

 

Any ideas what might be breaking the EDT connection? Do I need to disable EDT for now? Obviously, upgrading to 7.16/7.17 is not an option.

[Leon Poortvliet]

We were having this issue as well and for us it started with the Cumulative Windows updates released for March. We're running only 2008R2 VDA's, so I can only speak for those servers. Installing CU2 fixed the protocol driver error message... but now we experience seemingly random disconnects. Following your post and this one: https://discussions.citrix.com/topic/394895-receiver-connection-interrupted-adaptive-transport/?do=findComment&comment=2008178 i'm disabling EDT to see if that solves the problem for now.

[Dennis Parker]

Thanks for chiming in!  I hadn't seen the other thread. I will take a look through it.

 

7.15 CU 2 did not fix the issue for me. It did make it much less common on though. I am still in the process of rolling out to the VDAs, but I have provided trace information to Citrix support from a CU 2 system.   The issue did also start with the March updates as well as near as I can narrow it down, so glad for some confirmation there.

 

I have disabled EDT on a few systems. Yesterday one of the EDT disabled clients was experiencing random disconnects so I will pay a bit more attention to that today. Those servers did not get the CU 2 update either though.

 

My support tickets for the issue are:

77058606 for the VDA

77077252 for the NetScaler (support forced me to open a separate NetScaler ticket for the issue)

 

I'll keep this updated as I know more. Citrix says it has the highest priority assigned to the ticket at this time, but they are still reviewing logs. If you have a ticket open, I would like to get the number so I can maybe ask them to compare them.

 

Thanks again for the post.

[Leon Poortvliet]

Thanks for the quick reply.

 

Running a few hours with EDT disabled and the clients that experienced the issue, have not complained since. In fact, one user perceived it as being faster for this session. I have not yet opened a ticket, but I will as soon as I find some time to provide traces.

[Dennis Parker]

We were forced to disable EDT across the environment as well. Citrix escalation engineer is still attempting to work on our case, but has had very little luck recreating our issue and of course the logs show nothing really of any use. Just that, yup, there is a connection problem, which we already know about.

 

Not sure what else to do at this point. Not that I wish the issue on anybody else, but I do hope someone else can provide more information to Citrix and they can find some problem and resolution with EDT.  Been a frustrating issue. 

 

The NetScaler ticket was closed.

 

Anybody else with this issue? Anybody seeing fallback to TCP with receiver 4.11 until server restart?

[Carlo Piva1709156772]

The company I work for has been having these issues for the last few weeks. I thought it was something with the network configuration because we were alerted that engineers at the datacenter had to replace a power module to our rack, and thought maybe something was touched - but nothing appeared obvious.

Today users are reporting the Citrix Protocol Driver error.

We had upgraded our Netscalers back in February or March to what was current at the time.

Windows updates are installed weekly.

The issue has progressed from being a minor annoyance only affecting one or two servers for a short period (they appeared to self correct after a few minutes), to all of the servers randomly for extended periods of time.

 

If the only solution is to disable EDT than I'll do that tonight.

 

 

[Dennis Parker]

Haven't been brave enough to try enabling again on 7.15.4000 with one exception that is running on a low latency connection with very few disconnects and Windows Server 2016. It has never to my knowledge happened against any of our 2016 servers. The issue was more pronounced against high latency connections that also experienced many disconnected sessions. All of our users connect over the Internet from various places around the world. I wouldn't expect a power module to introduce this issue, it just seems to happen. 

I did see where sometimes, fairly rarely, that it would self-correct too. Most servers would just hang and require a restart. 

Wish I had better news on this.

What version of VDA are you on by chance? 

[Carlo Piva1709156772]

On 27/08/2019 at 3:09 PM, Dennis Parker said:

Haven't been brave enough to try enabling again on 7.15.4000 with one exception that is running on a low latency connection with very few disconnects and Windows Server 2016. It has never to my knowledge happened against any of our 2016 servers. The issue was more pronounced against high latency connections that also experienced many disconnected sessions. All of our users connect over the Internet from various places around the world. I wouldn't expect a power module to introduce this issue, it just seems to happen. 

I did see where sometimes, fairly rarely, that it would self-correct too. Most servers would just hang and require a restart. 

Wish I had better news on this.

What version of VDA are you on by chance? 

 

We have a mixed back of 7.15.CU1 through 7.15CU4.

Most have been upgraded to 7.15CU4.

Right now we have been troubleshooting a mysterious time sync issue that may be tied to the errors we saw, and have been grasping at straws. I created another post about the time sync problems. 

My manager was away last week so did not get the OK to disable EDT.

 

 

 

[Dennis Parker]

Thanks for that information. I think I saw the TIme Sync post as well. Quite an interesting read.

 

Also, interesting to tie these two together. We also have time sync changes (long story I won't go into here) that I wouldn't classify as "issues" though so I didn't ever even think they could be related and I do have any events that I could correlate to the time changes and the EDT issues. Mine are from servers having various methods of getting time and very occasionally they get very slight time adjustments from different sources and then get changed back, usually pretty quickly. (The differences are usually in micro seconds.) Might be an incentive to be allowed to fix the time sync configuration. 

[Carlo Piva1709156772]

We were getting the drops between the VDA's and the Delivery Controllers, and some weirdness happening with the Netscaler (possibly unrelated). It just sort of came out of the blue back in July.  With no logs correlating to the issues, and nothing showing up in MAAS, it became a literal needle in a haystack occurrence.  Citrix was called a few times but they were not able to assist. They believed the issue was network related as the communication would drop and re-establish itself, like a flapping circuit, but there were no network logs to that extent, traces came back clean, and the network adapters never seemed to indicate a drop in connection.  So when I saw the time change on servers last week (by the time we we were getting to the VDA's the time had already sync'd back and was fine), the bells started going off. I was able to confirm that if the time goes out on the VDA, then it will unregister from the Delivery Controller.  The problem was then trying to track down why the time kept changing.   Someone already suggested the fix found on this article  https://discussions.citrix.com/topic/399684-xenserver-tools-timesync-disable/ and have implemented it on a few servers so will see if it fixes the problem. I have also talked with Citrix support, and I am now working further with the Citrix XenServer support team to identify the problem, but it sounds like they were aware of a similar issue but believed it to be resolved in an update that was pushed out back in July (which is when these problems all started for us).