Jump to content


Photo

Netscaler 11 62.10 - Outlook Anywhere Broken?

Started by Patrick Sudderth , 22 August 2015 - 05:22 PM
59 replies to this topic

Teemu Keski-Valkama Members
  • #21

Teemu Keski-Valkama
  • 13 posts

Posted 08 September 2015 - 10:11 AM

Our case seems to be dead in the water since 28th of August, anyone else had more luck with responses from support?



Rohan Blake Members
  • #22

Rohan Blake
  • 9 posts

Posted 08 September 2015 - 10:52 AM

Nope, zero useful response so far.. attempting to push it further up the food chain.. needs to get to the Netscaler development team as it appears to a bug with their 11.0 code base..



Adam Sieting Members
  • #23

Adam Sieting
  • 4 posts

Posted 08 September 2015 - 01:12 PM

We're having the same issue. Just set it the SSL_BRIDGE for a work around and that seems to work fine for now. Citrix hasn't given us any information either except it's a "bug". 

 

It also appears to have broken our View environment for clients using the fat client. Going to attempt SSL_BRIDGE to fix this one too otherwise we have to revert code back. 

 

Sad, this is the first time I've had such major issues with a NS upgrade. 



Teemu Keski-Valkama Members
  • #24

Teemu Keski-Valkama
  • 13 posts

Posted 09 September 2015 - 12:31 PM

We were asked to fiddle with some settings (SSL Profile, basic to strict to basic) and checkboxes (nondescript, tried enabling and disabling TLS 1.2 as that was listed here before). No help.

 

Alas, the old "have you tried turning it off and on again" -solution did not help this time, at least when it came to settings.

 

Still hoping on getting escalated to be noticed by the people that are responsible for recoding the SSL stack, obviously they broke something.



Henrik - Members
  • #25

Henrik -
  • 69 posts

Posted 15 September 2015 - 11:36 AM

Any updates since the 9th of sept?

This is kind of urgent.



Jörn Reitwießner Members
  • #26

Jörn Reitwießner
  • 76 posts

Posted 18 September 2015 - 11:21 AM

Spoken with a Citrix-engineer at a Citrix-event last Tuesday. He told me that this issue is on high-priority at Citrix-support and it will be fixed at the next release.



Teemu Keski-Valkama Members
  • #27

Teemu Keski-Valkama
  • 13 posts

Posted 25 September 2015 - 09:57 AM

Our case was escalated today, might be they are collecting data and hopefully finding a way to fix this in future release.



Teemu Keski-Valkama Members
  • #28

Teemu Keski-Valkama
  • 13 posts

Posted 27 September 2015 - 05:43 PM

Update and a more functional workaround!

Edit: This solution tested with NS11 62.10.nc build

 

 

Been debuggin with Citrix after our case got escalated. We had both Outlook Anywhere and M-Files client access via RPC-over-HTTP giving us trouble, it was the M-Files traffic that eventually lead us to what we now suspect is the root of the problem.

 

Currently it seems that this issue is caused with having AppFlow enabled, especially the global Appflow rule enabled.

 

We were able to return to using SSL Offloading for out Exchange and M-Files traffic after completely disabling the AppFlow feature on the NetScaler (from console "disable ns fea appflow"). 

 

For us this solution is perfectly fine as our AppFlow was only really used when testing Splunk, currently we are not using it but will most likely return to testing at a later time, the initial results were promising. In any case, for us the AppFlow feature is currently not essential. 

Support did mention that they suspect that the AppFlow global policy is the root of this issue, so theoretically disabling the global policy and enabling individual policies for XA/XD should work. But we have not tested this, for us the whole feature was disabled.



Scott Osborne Members
  • #29

Scott Osborne
  • 137 posts

Posted 30 September 2015 - 09:32 PM

Also seeing very similar where just like the other link, for StoreFront (even 3.0.1) had to now uncheck TLS 1.2 even though 1.0 and 1.1 were enabled as well. That worked for StoreFront. Our Secure XML\STA was broke as well and will test doing something with internal services for TLS 1.2.

 

In addition to get more specific, we were fine with SHA1 certs on SF and Controllers and only when going SHA2, we could break it at will. So most likely the built in ciphers related to TLS 1.2 SHA 256 seemed to be a problem..

 

 



Rohan Blake Members
  • #30

Rohan Blake
  • 9 posts

Posted 30 September 2015 - 09:55 PM

Thats a different problem Scott, this specific OA issue is not a TLS problem.

Packet captures over the Outlook Anywhere http-rpc flows show the Netscaler resetting the TCP connection much further on down the traffic flow.. after the TLS negotiation etc has been successfully completed. 

Netscaler interrupts the OA traffic flow by incorrectly sends a RST to the client before one of the response's from the Exchange RPCProxy.

 

Teemu our devices are running NS11.0-55.23.nc, and appflow has never been enabled but we still experience the issue. Are you running the newer NS11.0-62.10 version?



Teemu Keski-Valkama Members
  • #31

Teemu Keski-Valkama
  • 13 posts

Posted 01 October 2015 - 05:27 AM

Rohan: Edited the solution message to indicate that it is tested with the (currently) latest NS 11 62.10.nc build.

 

From our conversation with support:

Latest few data sets provided allowed to compare /match Mfiles traffic with Outlook - and even more!
So far I have seen HTTP200 (with large content-length) not being passed from VIP to Client, but some one of the Mfiles traces show those to be passed to client after very long time (as if it took long time to 'process').

Disabling Appflow removed this slowdown for us and restored both OA and M-Files Client functionality.



Rohan Blake Members
  • #32

Rohan Blake
  • 9 posts

Posted 01 October 2015 - 08:10 AM

Yeah OA specifies a "content-length" header of 1GB.. not sure what Netscaler is doing though which is taking more than 30secs to perform/process (pre-allocating a buffer for it maybe.. but surely it shouldnt take that long??)..  30secs is generally the TCP timeout and results in the RST been eventually sent from the Netscaler to cleanly close the connection.

 

I'd originally thought perhaps it was just discarding/ blocking the flow because it didnt "like" the overly large content-length header (some WAF's alert on it), but serialisation or processing delay exceeding the default TCP timeout value perhaps makes more sense (race condition) as I've on rare occasions during testing seen it successfully complete the MAPI MailStore 6001 RPC ping query. Perhaps 1% of the time..

 

OA http-rpc is also very dependant on http-keepalives, without them it will generally fail with a similar error/problem.

 

Will try the NS 11 62.10.nc build with appflow disabled and see if that resolves the issue for us as well.



Rohan Blake Members
  • #33

Rohan Blake
  • 9 posts

Posted 01 October 2015 - 10:19 AM

TCP timeout may have been the wrong term. The 30sec query/response time limitation may actually be built in at the application layer, it just happens to result in the TCP connection been brought down after the timeout has been exceeded.



Andrzej Starmach Citrix Employees
  • #34

Andrzej Starmach
  • 137 posts

Posted 04 October 2015 - 12:53 PM

Rohan - I have looked at your config

Please disable "HTMLInjection" feature - that should help for now.

 

Andrzej



Rohan Blake Members
  • #35

Rohan Blake
  • 9 posts

Posted 04 October 2015 - 01:17 PM

Indeed it does.. :-) Muchas gracias Andrezej.

 

Now.. do we have any timeline for when Citrix will be releasing patch/update to correct the bug?



Andrzej Starmach Citrix Employees
  • #36

Andrzej Starmach
  • 137 posts

Posted 04 October 2015 - 01:27 PM

We are still debugging this in-house to find exact root cause. Once we have a fix added to development version of NS code I will share ETA on version with permanent fix.

-Andrzej



Espen Fjeldstad Members
  • #37

Espen Fjeldstad
  • 7 posts

Posted 05 October 2015 - 08:37 AM

We are still debugging this in-house to find exact root cause. Once we have a fix added to development version of NS code I will share ETA on version with permanent fix.

-Andrzej

 

Hi.

We just recently upgraded to Netscaler 11.0 62.10 and now we experience a lot of issues with Outlook Anywhere.

Do you have any updates on this issue?

 

The appflow feature has been disabled, as well as TLS1.2 towards the service group, vserver, etc.



Andrzej Starmach Citrix Employees
  • #38

Andrzej Starmach
  • 137 posts

Posted 05 October 2015 - 08:51 AM

Espen - What type of issue *exactly* you are experiencing with OA ? is it *only* OA or OWA as well ?
Let's record the problem in NS trace and upload to your case ID with show tech support file



datru Members
  • #39

David Truttmann
  • 32 posts

Posted 09 October 2015 - 06:26 AM

Is the issue fixed in 11.0-63.16?



Andrzej Starmach Citrix Employees
  • #40

Andrzej Starmach
  • 137 posts

Posted 09 October 2015 - 10:23 AM

David-we do have a root cause and currently working on a fix.
That said 63.x MR of NS11 does not have this fix.