Netscaler external DNS service DOWN, internal DNS service UP

[Alex Learoyd1709154956]

We are using Netscaler 12.1 51.16 and having a problem getting the external DNS services to appear UP on our Pre-production device. 

The name server and associated localdns VIP are working so we can run successful dig commands against external sites using internal DNS however external DNS services are down, so when running dig commands using external services this therefore fails which is causing a blockers for some GSLB project work

 

Creating a new 8.8.8.8 service on port 53 appears down

 

Our security/firewall team have confirmed ports are open.  The whole SNIP/ADNS subnet is open on port 53.  This leads out to the internet on the external firewall

As per our production device, we have created services with a standard DNS monitor.

Netscaler routing has been checked and all looks fine.
 

Are there any other tests we can carry out or areas to check which would cause external services created on port 53 to be down?

[Siddhartha Sarmah]

try taking a nstrace then (on wireshark) filter it using "ip.addr==8.8.8.8" - do you see any traffic ?

[Alex Learoyd1709154956]

Thanks ^

I have carried out some troubleshooting nstcpdump, dig and trace using GUI filter expression trace 'CONNECTION.DSTIP.EQ(8.8.8.8) -link ENABLED'
(whilst running Dig to google.com)

cd /usr/bin
./dig @8.8.8.8 www.google.com

Prod

root@'NETSCALER_PROD'# nstcpdump.sh host 8.8.8.8 and dst port 53
reading from file -, link-type EN10MB (Ethernet)
11:28:07.116059 IP *SNIP Address*.42076 > 8.8.8.8.53: 65264+ [1au] A? . (28)
11:28:12.119529 IP *SNIP Address*.22242 > 8.8.8.8.53: 64126+ [1au] A? . (28)
11:28:17.120575 IP *SNIP Address*.15718 > 8.8.8.8.53: 20224+ [1au] A? . (28)
11:28:22.125078 IP *SNIP Address*.28340 > 8.8.8.8.53: 49264+ [1au] A? . (28)
11:28:27.127599 IP *SNIP Address*.17559 > 8.8.8.8.53: 28813+ [1au] A? . (28)

Dig resolves google.com

; <<>> DiG 9.10.6 <<>> @8.8.8.8 www.google.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 8883
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;www.google.com.                        IN      A

;; ANSWER SECTION:
www.google.com.         198     IN      A       216.58.213.4

;; Query time: 6 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Wed Jun 05 11:37:16 BST 2019
;; MSG SIZE  rcvd: 59

-------------------------------------------------------------------------------------
PreProd
 

root@'NETSCALER_PREPROD'# nstcpdump.sh host 8.8.8.8 and dst port 53
reading from file -, link-type EN10MB (Ethernet)
11:29:47.402279 IP *SNIP Address*.50465 > 8.8.8.8.53: 19805+ [1au] A? . (28)
11:30:19.418682 IP *SNIP Address*.36353 > 8.8.8.8.53: 43860+ [1au] A? . (28)
11:30:24.421371 IP *SNIP Address*.44370 > 8.8.8.8.53: 39147+ [1au] A? . (28)
11:30:29.426578 IP *SNIP Address*.56285 > 8.8.8.8.53: 21659+ [1au] A? . (28)
11:31:01.443476 IP *SNIP Address*.62276 > 8.8.8.8.53: 32952+ [1au] A? . (28)
 

Dig does NOT resolve google.com

; <<>> DiG 9.10.6 <<>> @8.8.8.8 www.google.com
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached

NS trace captures both show traffic, inspecting the packet from SNIP to 8888 'Standard query A <Root> OPT' then Domain Name System (query) 

Navigating down to Additional records, <root>: type OPT, then Z: 0x0000 on PROD working it shows 'DO bit: Accepts DNSSEC security RRs' on PREPROD not working this shows 'DO bit: Cannot handle DNSSEC security RRs'

Does anyone know what this means? 'DO bit: Cannot handle DNSSEC security RRs'

 

 

edit - I have also carried out a trace, this time with the destination as the SNIP address when running another 8.8.8.8 dig command

On the Production Netscaler filtering 'udp.port ==53' there is a quite a lot of traffic as expected including the 8.8.8.8 traffic, when using '(udp.port ==53) && (ip.src == 8.8.8.8)'
When I look for tcp port 53 '(tcp.port ==53)' there is no traffic.

On the pre-production Netscaler 'udp.port ==53' there is no traffic.  However when filtering on 'tcp.port ==53' I see traffic, but only from the Pre-production SNIP to the Pre-Production SNIP.  TCP out-of-order, TCP re-transmissions and resets. 

Could the Pre-production device be failing to use UDP port 53, instead trying to use only TCP port 53?  It looks to me like UDP 53 is blocked, but this is not the case, 53 TCP and UPD are confirmed open from the Preprod SNIP

[Siddhartha Sarmah]

If the service is not coming up that means the monitor itself is failing. So the type of traffic will depend on what monitor you have configured. 

[Alex Learoyd1709154956]

On 6/7/2019 at 1:55 AM, Siddhartha Sarmah said:

If the service is not coming up that means the monitor itself is failing. So the type of traffic will depend on what monitor you have configured. 

 

Thanks, though this is sort of a given.... We use the same monitor as on the production working Netscaler, just a standard 'DNS' monitor.

Prod this shows 'Success - DNS response with status code 0'

Pre-prod this shows 'Failure - probe timed out'

Global DNS parameter settings are the same on both devices and we have tried custom monitors with a few different basic and advanced parameters.

 

Any specific custom DNS monitors are welcome

[Siddhartha Sarmah]

The standard DNS monitor should work just fine (it won't use TCP),  the issue seems to be that SNIP isn't communicating with 8.8.8.8

 

The trace snippets you posted earlier are all one way, there is no dns response seen. The DO bit is not relevant here as 8.8.8.8 should respond to dns queries with or without the DO bit set. (Simply test DIG from any working setup with and without the +dnssec option)

 

I'd suggest you take a simulataneous trace on the ADC and on your perimeter firewall and see if the DNS requests are seen on the firewall and if the requests were allowed and natted correctly.

[Alex Learoyd1709154956]

20 hours ago, Siddhartha Sarmah said:

The standard DNS monitor should work just fine (it won't use TCP),  the issue seems to be that SNIP isn't communicating with 8.8.8.8

 

The trace snippets you posted earlier are all one way, there is no dns response seen. The DO bit is not relevant here as 8.8.8.8 should respond to dns queries with or without the DO bit set. (Simply test DIG from any working setup with and without the +dnssec option)

 

I'd suggest you take a simulataneous trace on the ADC and on your perimeter firewall and see if the DNS requests are seen on the firewall and if the requests were allowed and natted correctly.

Thanks, ill try this when I eventually get perimeter firewall access

Correct, no DNS response seen, as per above.  Prod I can see from source 8.8.8.8 to SNIP query responses from www.google.com.  Preprod this does not exist...
 

edit - I have also carried out a trace, this time with the destination as the SNIP address when running another 8.8.8.8 dig command

On the Production Netscaler filtering 'udp.port ==53' there is a quite a lot of traffic as expected including the 8.8.8.8 traffic, when using '(udp.port ==53) && (ip.src == 8.8.8.8)'
When I look for tcp port 53 '(tcp.port ==53)' there is no traffic.

On the pre-production Netscaler 'udp.port ==53' there is no traffic.  However when filtering on 'tcp.port ==53' I see traffic, but only from the Pre-production SNIP to the Pre-Production SNIP.  TCP out-of-order, TCP re-transmissions and resets. 

Could the Pre-production device be failing to use UDP port 53, instead trying to use only TCP port 53?  It looks to me like UDP 53 is blocked, but this is not the case, 53 TCP and UPD are confirmed open from the Preprod SNIP

[Alex Learoyd1709154956]

This is now resolved.  It turned out to be traffic from our ISP being incorrectly routed to the external firewall. 

The public ADNS IP was changed to a new IP which identified this fault.  Another team is looking why the ISP is blocking dns/incorrectly routing traffic

Thanks for the help everyone