Jump to content


Photo

XA 7.13 upgrade, Delivery Controller problems

Started by Art Frank , 07 March 2017 - 05:29 PM
11 replies to this topic

Best Answer Art Frank , 17 March 2017 - 06:34 PM

Fixed it. Short answer: install Microsoft Visual C++ 2015 Redistributable Update 3, x64.

 

The long description:

Searching the text "specified module could not be found HRESULT: 0X8007007E" seemed to point the finger at some type of DLL dependency that might be missing. So I downloaded Dependency Walker and ran it against the CredentialSecurity.dll file. There were quite a few missing dependencies, so I ran it on the working XenApp 7.8 system, and there were only 3. So I focused on the ones that were missing but different on 7.13. They all seemed to be stemming from VCRUNTIME140.DLL, which upon further investigation is a Visual C++ 2015 dll. Not only was this already installed on the system (version 14.0.24212) but Citrix's 7.13 documentation doesn't say ANYTHING about that version ANYWHERE, and instead specifies, for the delivery controller, Visual C++ 2008 SP1. I was tempted to reinstall 2008 SP1, but that VCRUNTIME140 is definitely 2015, so I downloaded the latest 2015 installer, installed it (version 14.0.24215) and all of those dependencies were resolved. Restarted the desktop service on the VDA, and it connected no problem. The site now seems to work as expected.

 

Citrix, if you're reading this, please: (1) update your documentation to reflect to correct requirements for the 7.13 Delivery Controller, and (2) update your "component installer" which is supposed to deploy software prerequisites automatically but in this case is failing. It would also be really cool if the installer had noticed that my licensing was set to XenApp and kept it that way instead of changing it to XenDesktop, but that's a little quibble.

Art Frank Members

Art Frank
  • 26 posts

Posted 07 March 2017 - 05:29 PM

I have a XenApp 7.8 site. Yesterday I upgraded the app servers using the 7.13 XenApp/XenDesktop ISO. This went without a problem. Then I upgraded the Studio/Storefront server, and while that appeared to have gone well, the app servers could no longer connect to the site. They appeared in the console as "Unregistered". Obviously, because of this, I couldn't launch any apps at all. I believe this happened after upgrading the databases, but I didn't check connectivity after installing 7.13 but before upgrading the database and site.
 
The event viewer on the app servers showed event 1002 "The Citrix Desktop Service cannot connect to the delivery controller 'http://myurl:80/Citrix/CdsController/IRegistrar' (IP Address 'XXX.XXX.XXX.XXX')
 
After struggling for a bit, I rolled back the Studio/Storefront server to a pre-upgrade snapshot, restored the databases from backup, rebooted, and everything started working fine. But that server is still running 7.8 components.
 
A few questions, before I try this again:
  • Is the leap from 7.8 to 7.13 supported? I know that sometime between 7.8 and 7.13 Citrix consolidated XenApp and XenDesktop. Do I need to upgrade my XenApp site to the latest XenApp version before hopping to XenApp/XenDesktop?
  • Is there a way to test the delivery controller URL referenced in the event? Now (working) the URL loads a blank page. After 7.13 (not working) exhibited the same behavior.
  • Is there anything I could have done ahead of time to head off this problem? In Citrix Studio, I tested the site, all tests came back good. Tried testing the Delivery Group, but "No applicable tests could be found." Now that I'm aware of the Citrix Health Assistant (during this failed upgrade I was struggling with XDPing instead) I've run the VDA connectivity tests from the app servers and they're all good. Anything else? 
Thanks for any advice you may have.

 



Automatisering Pantein Members

Automatisering Pantein
  • 5 posts

Posted 08 March 2017 - 10:13 AM

7.12 is a much more stable release imho. I have also upgraded to 7.13 and have a few issues, one of which I have posted here. It might be worth your while to upgrade to 7.12 instead of 7.13. Everything worked perfectly with 7.12 in my environment.



Julian Mooren Members

Julian Mooren
  • 78 posts

Posted 08 March 2017 - 01:26 PM

I also had problems updating from XD 7.8 --> 7.13 (Upgrade Loop)

Here I documented how I could fix it: https://citrixguyblog.wordpress.com/2017/03/08/xendesktop-studio-gets-stuck-in-mandatory-upgrade-loop/

 

Can you provide us detailed log and error messages?  What was the status of the broker machines (Get-BrokerController)?

Sometimes you need to set the product edition in Studio again (after upgrade). Otherwise no connections will be established.



Art Frank Members

Art Frank
  • 26 posts

Posted 09 March 2017 - 12:07 AM

7.12 is a much more stable release imho. I have also upgraded to 7.13 and have a few issues, one of which I have posted here. It might be worth your while to upgrade to 7.12 instead of 7.13. Everything worked perfectly with 7.12 in my environment.

 

Thanks for the suggestion. I may do that for the delivery controller. The VDAs seemed to upgrade okay, and Studio shows them as running "Agent Version 7.13.0.22", so I left them as they are.



Art Frank Members

Art Frank
  • 26 posts

Posted 09 March 2017 - 12:25 AM



I also had problems updating from XD 7.8 --> 7.13 (Upgrade Loop)

Here I documented how I could fix it: https://citrixguyblog.wordpress.com/2017/03/08/xendesktop-studio-gets-stuck-in-mandatory-upgrade-loop/

 

Can you provide us detailed log and error messages?  What was the status of the broker machines (Get-BrokerController)?

Sometimes you need to set the product edition in Studio again (after upgrade). Otherwise no connections will be established.

 

That's some interesting information I will keep on hand when I try upgrading the delivery controller again. But for me, the site database upgraded (apparently) successfully, this wasn't in a loop. And after the site upgrade, the VDAs all stopped talking to the controller. They appeared as "Unregistered." Looking at the VDAs, the Citrix Desktop Service was spitting out warnings like these:

 

Event 1002: The Citrix Desktop Service cannot connect to the delivery controller 'http://myurl:80/Citrix/CdsController/IRegistrar' (IP Address 'XXX.XXX.XXX.XXX')

Error Details: Exception 'Error occurred when attempting to connect to endpoint at address http://myurl:80/Citrix/CdsController/IRegistrar, binding WsHttpBindingIRegistrarEndpoint and contract Citrix.Cds.Protocol.Controller.IRegistrar: System.ServiceModel.ServerTooBusyException: The HTTP service located at http://XXX.XXX.XXX.XXX/Citrix/CdsController/IRegistrar is unavailable.  This could be because the service is too busy or because no endpoint was found listening at the specified address. Please ensure that the address is correct and try accessing the service again later. ---> System.Net.WebException: The remote server returned an error: (503) Server Unavailable.

 
Event 1002: The Citrix Desktop Service cannot connect blah blah blah...

Error Details:  Exception 'Internal error' of type 'System.ServiceModel.FaultException`1[Citrix.Cds.Protocol.Cbp.Fault]'..

 

Event 1002: The Citrix Desktop Service cannot connect blah blah blah...

Error Details:  Exception 'The server was unable to process the request due to an internal error.  For more information about the error, either turn on IncludeExceptionDetailInFaults (either from ServiceBehaviorAttribute or from the <serviceDebug> configuration behavior) on the server in order to send the exception information back to the client, or turn on tracing as per the Microsoft .NET Framework SDK documentation and inspect the server trace logs.' of type 'System.ServiceModel.FaultException'..

 

Event 1002: The Citrix Desktop Service cannot connect blah blah blah...

Error Details:  Exception 'Error occurred when attempting to connect to endpoint at address http://myurl:80/Citrix/CdsController/IRegistrar, binding WsHttpBindingIRegistrarEndpoint and contract Citrix.Cds.Protocol.Controller.IRegistrar: System.ServiceModel.EndpointNotFoundException: There was no endpoint listening at http://XXX.XXX.XXX.XXX/Citrix/CdsController/IRegistrar that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond XXX.XXX.XXX.XXX:80

 

So many ways of saying the same thing--the server isn't responding to the connection request. Yes, the URL is correct. Yes, the IP address is correct. And after rolling the server back to the previous version, all of the VDAs connected no problem without any changes.

 

I did not run Get-BrokerController while troubleshooting. I will keep that in mind next time.



Julian Mooren Members

Julian Mooren
  • 78 posts

Posted 09 March 2017 - 08:11 AM

Maybe there was a problem with the port binding after the upgrade. Can be fixxed with: "BrokerService.exe /VDAPort 80". After this the broker service will be restarted and should listen on port 80 (Its just an assumption).

 

 If you have some free time try this:

- Setup fresh XDC and VDA (7.8)

- Add the XDC to a restored copy of the XD 7.8 database

- Upgrade VDA to 7.13

- Upgrade XDC to 7.13

 

Will the error occur again? Then you can troubleshoot without touching the production.



Art Frank Members

Art Frank
  • 26 posts

Posted 09 March 2017 - 04:55 PM

Maybe there was a problem with the port binding after the upgrade. Can be fixxed with: "BrokerService.exe /VDAPort 80". After this the broker service will be restarted and should listen on port 80 (Its just an assumption).

 

 If you have some free time try this:

- Setup fresh XDC and VDA (7.8)

- Add the XDC to a restored copy of the XD 7.8 database

- Upgrade VDA to 7.13

- Upgrade XDC to 7.13

 

Will the error occur again? Then you can troubleshoot without touching the production.

 

Exactly. We haven't ever set up a test environment, and my previous upgrades went without incident, but clearly running through the upgrade on a test platform is a good idea. But the VDAs are already on 7.13 and seem to be operating well, so I didn't roll those back.



Art Frank Members

Art Frank
  • 26 posts

Posted 16 March 2017 - 09:29 PM

Okay, I finally got a test environment working, using clones of my delivery controller/storefront and one of my XenApp/VDA servers. Upgraded the Delivery Controller to 7.13. Recall, the VDA is already at 7.13 and seems to be happy enough.

 

Did the site tests ahead of time. All came back good. Upgraded the delivery controller/storefront server. Before upgrading the databases, I tested the site, and it wasn't working. "Cannot connect to delivery controller" error. Upgraded the databases -- same error. Tried the Citrix Health Assistant from the VDA -- check succeeded, all green check marks. Tried the "brokerservice.exe /vdaport 80" command -- no change.

 

Get-BrokerController did not reveal anything interesting as far as I could tell. However, it showed that the licensing got messed up (LicensingGraceState=Active, LicensingGracePeriodReasons={OutOfBox}) because the upgrade changed my license type to XenDesktop (for which I have no license). Changing it back to XenApp resolved the licensing issue. If there's something else I should be looking for here, I don't know what it is.

 

Tried upgrading the Machine Catalog, that didn't make a difference. Also tried upgrading the Delivery Group, that didn't help either. Removed the server from the Machine Catalog, re-added it, same error.

 

The error is now pretty consistent (last time there were a variety of error details):

Event 1002: The Citrix Desktop Service cannot connect to the delivery controller 'http://myurl:80/Citrix/CdsController/IRegistrar' (IP Address 'XXX.XXX.XXX.XXX')

Error Details:  Exception 'Internal error' of type 'System.ServiceModel.FaultException`1

[Citrix.Cds.Protocol.Cbp.Fault]'..

 

Anything else I should look at before rolling back? Next time around I'm going to try upgrading to 7.12 instead, see how that works out for me.



Art Frank Members

Art Frank
  • 26 posts

Posted 16 March 2017 - 10:21 PM

I figured out how to turn on logging for the Broker Service. Here's where I think the error is:

BrokerComponent.IRegistrar.Register(LONG SID): Exception System.DllNotFoundException: Unable to load DLL 'CredentialSecurity.dll': The specified module could not be found. (Exception from HRESULT: 0X8007007E)

 

(I'm sorry, I cannot copy/paste the surrounding text from the log file. The test box is really isolated.)

 

I found a copy of this file in C:\Program Files\Citrix\Broker\Service. I tried to just regsvr32 the file, but I got the error:

The module "C:\Program Files\Citrix\Broker\Service\CredentialSecurity.dll" failed to load. Make sure the binary is stored at the specified path or debug it to check for problems with the binary or dependent .DLL files. The specified module could not be found.

 

And I can see that this error happens every time my VDA server tries to register.



Art Frank Members
  • #10

Art Frank
  • 26 posts

Posted 17 March 2017 - 06:34 PM

Fixed it. Short answer: install Microsoft Visual C++ 2015 Redistributable Update 3, x64.

 

The long description:

Searching the text "specified module could not be found HRESULT: 0X8007007E" seemed to point the finger at some type of DLL dependency that might be missing. So I downloaded Dependency Walker and ran it against the CredentialSecurity.dll file. There were quite a few missing dependencies, so I ran it on the working XenApp 7.8 system, and there were only 3. So I focused on the ones that were missing but different on 7.13. They all seemed to be stemming from VCRUNTIME140.DLL, which upon further investigation is a Visual C++ 2015 dll. Not only was this already installed on the system (version 14.0.24212) but Citrix's 7.13 documentation doesn't say ANYTHING about that version ANYWHERE, and instead specifies, for the delivery controller, Visual C++ 2008 SP1. I was tempted to reinstall 2008 SP1, but that VCRUNTIME140 is definitely 2015, so I downloaded the latest 2015 installer, installed it (version 14.0.24215) and all of those dependencies were resolved. Restarted the desktop service on the VDA, and it connected no problem. The site now seems to work as expected.

 

Citrix, if you're reading this, please: (1) update your documentation to reflect to correct requirements for the 7.13 Delivery Controller, and (2) update your "component installer" which is supposed to deploy software prerequisites automatically but in this case is failing. It would also be really cool if the installer had noticed that my licensing was set to XenApp and kept it that way instead of changing it to XenDesktop, but that's a little quibble.


Best Answer

Art Frank Members
  • #11

Art Frank
  • 26 posts

Posted 17 March 2017 - 08:11 PM

Fixed it. Short answer: install Microsoft Visual C++ 2015 Redistributable Update 3, x64.

 

The long description:

... (2) update your "component installer" which is supposed to deploy software prerequisites automatically but in this case is failing.

 

Point of clarification, now that I'm doing this again in test: it looks like the 7.13 installation ISO includes the Visual C++ 2015 Redistributable, but it's version 14.0.24212. Version 14.0.24215 is required. It also looks like the installer stomps on the existing C++ 2015 install, so I had to repair the 14.0.24215 after installing XenDesktop 7.13. Moral of the story, install 14.0.24215 AFTER installing 7.13.


George Fridman Members
  • #12

George Fridman
  • 13 posts

Posted 08 May 2017 - 12:57 PM

I also had problems updating from XD 7.8 --> 7.13 (Upgrade Loop)

Here I documented how I could fix it: https://citrixguyblog.wordpress.com/2017/03/08/xendesktop-studio-gets-stuck-in-mandatory-upgrade-loop/

 

Can you provide us detailed log and error messages?  What was the status of the broker machines (Get-BrokerController)?

Sometimes you need to set the product edition in Studio again (after upgrade). Otherwise no connections will be established.

 

my upgrade went fine