Citrix ADC core dumps when HTTP Request without/empty Hostname hits 80-443 Redirect Policy

[Jens Dellner]

Hey guys,
we have some problems with our Citrix ADC with firmware 13.0.36.27.

 

We use the Responder Policy Method for HTTP-HTTPS Redirect ( https://support.citrix.com/article/CTX120664) since many years and never had problems with it. Since a few weeks - maybe since the last firmware update - our ADCs are core dumping when a Request without Hostname hits the Virtual Server for HTTP-HTTPS redirect. In this moment a failover to the secondary appliance happens. If a client makes directly a new request without hostname, the new primary appliance will core dump too and this could be a circle of death. One core dump needs ~2GB of disk space and after 3-5 core dumps the disk is full and old log files get deleted to free up space.

 

I can reproduce this issue in our environment. I use the Firefox Extension "HTTP Resource Test", configure the HTTP Header "Host" without value and open http://<IP</AnyDirectory. The Primary ADC makes a core dump, fails over to the secondary appliance and since the Extension fires the request again, the new primary ADC makes a core dump too.

 

HTTP-HTTPS Redirect like in any official Citrix HowTo:
add responder action act_rsp_80-443-Redirect redirect "\"https://\" + HTTP.REQ.HOSTNAME.HTTP_URL_SAFE + HTTP.REQ.URL.PATH_AND_QUERY.HTTP_URL_SAFE" -responseStatusCode 302
add responder policy pol_rsp_80-443-Redirect "HTTP.REQ.IS_VALID" act_rsp_80-443-Redirect
bind lb vserver vs_Name_80-443-Redirect -policyName pol_rsp_80-443-Redirect -priority 100 -gotoPriorityExpression END -type REQUEST

 

I opened a citrix case but citrix says it is related to our environment. Citrix is fix focused on the HTTP Reqeust "http:///ECP" because it was the logged issue in the logfiles. /ECP is related to Microsoft Exchange Services but we don´t have this directory. It was not accessed by one of our stuff, it was any kind of internet robot who was scanning IPs for an open Exchange Control Panel in the Internet and didn´t send a Hostname. The problem is not the directory, it could be any directory. Every time when a request without Hostname hits a Responder VServer and doesn´t send a Hostname the problem occures. So I think propably more than our environment could be affected.

 

The request hits the Responder Policy, because the Policy Expression "HTTP.REQ.IS_VALID" is evaluated to true. HTTP1.0 does not need a Hostname specified in the request. So the global setting "Drop Invalid HTTP Requests" and the nshttp_default_profile is ignored. If a HTTP1.1 request without Hostname is made, it should be dropped. I thought. But the Response in the Firefox Extension shows that the Request was made with HTTP1.1. So why was it not dropped?!? And why do we have this problem since a few weeks and not since the beginning?

 

Citrix recommended to change the Expression to:
add responder policy pol_rsp_80-443-Redirect "HTTP.REQ.IS_VALID && HTTP.REQ.HEADER(\"Host\").EXISTS" act_rsp_80-443-Redirect

 

I think this is no fix and even no workaround. If the request does not send a hostname but your backend has the called directory and is accessable via http (due a mistake or something), there would be no redirect to HTTPS and the content is accessed with http.

 

I changed the suggested Expression, but it was also possible to core dump the ADCs. There is difference between sending no Hostname or sending an empty Host Header. And the Firefox Extension sends an empty Host Header...and the ADCs makes a core dump.

 

Okay. Revert back to "HTTP.REQ.IS_VALID" because it is one of the recommended ways of Citrix and worked all the time. I created a new Responder Policy which drops all Requests without specified Hostname oder empty Host Header:

add responder policy pol_rsp_DROP_No_Hostname "HTTP.REQ.HEADER(\"Host\").EXISTS.NOT || HTTP.REQ.HOSTNAME.EQ(\"\")" DROP
bind lb vserver vs_Name_80-443-Redirect -policyName pol_rsp_DROP_No_Hostname -priority 99 -gotoPriorityExpression END -type REQUEST
bind lb vserver vs_Name_80-443-Redirect -policyName pol_rsp_80-443-Redirect -priority 100 -gotoPriorityExpression END -type REQUEST

 

I was not able to core dump the ADCs with the Firefox Extension again. But one day later the ACDs had a new dump. I am not sure what really happened but I think the Policy with Expression 99 was some kind of ignored. After i changed the Redirect Expression from "HTTP.REQ.IS_VALID" to "HTTP.REQ.IS_VALID && HTTP.REQ.HEADER(\"Host\").EXISTS" back, we do not have any problem. But why was this necessary?

 

Citrix suggested to me to use static Redirects like "https://Domain/Directory" for every webservice. But thats no solution. I escaladed the case to another engineer because the problem looks to me like a bug. Maybe I'm doing the Engineer wrong but it looks like a bug to me. It always worked, it is a recommended redirect method and now i can reproduce a core dump, because the Responder Engine doesn´t drop Requests without Hostname (which results in https:///Directory) by itself?

 

It was frightening to see that our appliances and webservices could be shutdown easier than with a DDos. I needed only 2 Requests per Minute.

 

What do you think about it? Am i on the wrong track? Or do you know the problem and can reproduce it too?

 

Best regards,
Jens

[Jens Dellner]

After further Investigation Citrix identified this issue and will fix it in the upcoming releases.

 

Tentative release dates:
13.0.39.x - Mid of September
12.1.53.x - End of July
12.0.62.x - End of August

 

Problem Description:

3196 - A appliance might crash if you use responder action of redirect action type.

 

Workaround:

Use Responder Action "Respond with" instead of "Redirect"

 

All Information provided by Citrix Engineering.

 

Best regards,

Jens