VPX HA in AWS not working

[Cindee Mcmillen]

I have seen a number of posts but I continue to have issues with an HA config in AWS.

There are repeated errors in the /var/log/ns.log like the below.

(there are more error but all seem to related to an attempt to connect to AWS API and retrieve information)

 

awsconfig: AWSCONFIG ERROR : Request to AWS API sever failed check error files /flash/nsconfig/.AWS//error.xml-1-14

 awsconfig: AWSCONFIG ERROR: HTTP RespCode:401,  Unauthorized! Incorrect Secret and/or Access Keys.

 

 

I have double and triple checked to enter keys in the user_data for the instance and have triple checked the IAM users permissions.

I am guessing that maybe the format I am using is wrong but it is pretty simple.

ACCESS_KEY=XXXXXXXXXXXXXXXXXXX
SECRET_KEY=XXXXXXXXXXXXXXXXXXXXXXXX

The keys get saved successfully in /nsconfig/.AWS/nws_details

 

I have successfully verified that the keys work by running an python script from the VPX  and  retrieving the same information that I see in

the log file.  e.g. Action=DescribeInstances&Filter.1.Name=private-ip-address&Filter.1.......

 

 

Maybe something has changed from some previous version that worked ?

The AMI version that I am using from the Marketplace is:

Citrix NetScaler and CloudBridge Connector 12.0-60.10-32-daf08ece-57d1-4c0a-826a-b8d9449e3930-ami-01f2e6510c37e7967.4

(ami-0e58c36d11321550f)

 

Any tips or experiences greatly appreciated..

 

[Arvind Kandula]

Hi,

Sorry for the inconvenience caused. We have published a note for Citrix ADC VPX listings.

**At this time, High Availability deployments may not work as expected. We are working on the issues and will provide a resolution soon**

 

Recently AWS announced and immediately changed the size of STS session tokens, which Citrix ADC (formerly NetScaler) relies upon during the High Availability process. More information on the session token change can be found at the link below:

https://aws.amazon.com/blogs/security/setting-permissions-to-enable-accounts-for-upcoming-aws-regions/

This has affected the High Availability process of Citrix ADC VPX.

 

Citrix team is working with AWS team to inform the Citrix ADC users in AWS marketplace.

Citrix team is also working with AWS team in releasing an updated AMI, we can keep you posted. For further information please feel free to mail aws@citrix.com.

[John Moody1709158020]

Any progress on this post from April?

 

Having issues.   Support says it is an AWS issue, and AWS says it is a citrix issue.

 

 

and my HA doesn't work!

[Eli James]

Still no reponses and no suggestions from April ?

______________________________

VPN Sai Mannat AnyDesk

[Arvind Kandula]

Hi, we apologize for the inconvenience caused. We have updated the AWS marketplace immediately after the issue was resolved and we have released the versions of Citrix ADC supporting the EC2 changes and we did not see any issues from our customers from then. Can you please kindly share the support case number, ADC version number that you are using to my mail id, arvind.kandula@citrix.com. we will look into it and resolve the issue.

[Farhan Ali1709152717]

To clarify few things for others

In Latest Netscaler builds 13.0, 12.1.51.20 above , 12.0.61.x above  there is no need for

ACCESS_KEY=XXXXXXXXXXXXXXXXXXX
SECRET_KEY=XXXXXXXXXXXXXXXXXXXXXXXX

 

We only need IAM Role with proper permissions. In older builds there was some issue with the STS token size which is fixed on all the builds released after May 2019. So if your build is after May 2019 you should not face this issue.

 

Some Key points.

1. Access key and secret key are not needed any more. only IAM role is need with proper permissions. Only in 11.0 release accesskey, secret key are required.

2. If the HA is in same availability zone then Interface movement will happen and different IAM permissions are required and if HA is in different Availability zone then Elastic ip movement will happen and different IAM permission are required. Refer to Citrix docs for the IAM permissions required

3. HA failover need Netscaler to make some calls to AWS Rest API server which is reachable over internet. So Make sure your default route is pointing properly to internet.

4. DNS resolution is happening properly

5. HA sync and Heartbeat ports are open in the AWS Security group

6. Under VPC there is a route to reach internet

7. go to CLI and then go to shell and type "cat /var/log/ns.log | grep aws -i". and check what errors you are noticing to identify the issue.

 

 

These are the basic key points required for troubleshooting

 

[Elavendhan Barathi1709162772]

Hi,

 

I do have the same issue, covered all the key points mentioned above.

 

But, upon inspecting cloud trail logs, ADC is making api call "Describe routes" but nothing like "delete routes" or "change routes" which I guess is expected from ADC after the "describe routes" call .

 

The failover of the ADC is no issue but the route table is not getting changed after it's failed over, no events in ha-daemon.log or ns.log.

 

Any response would be highly appreciated.