NetScaler Always On VPN before Windows Logon failing on first hurdle

[Ken Z]

Hi Everyone

 

well, another day another question....

 

I've followed whatever guides i could find on configuring "Always On before Windows Logon" for the NetScaler. Namely...

 

https://docs.citrix.com/en-us/citrix-gateway/current-release/vpn-user-config/always-on-vpn-before-windows-logon/configure-always-on-vpn-before-windows-logon.html

https://docs.citrix.com/en-us/tech-zone/learn/tech-insights/citrix-gateway-alwayson.html

 

I've rebuilt my test NetScaler multiple times, and used both 13.0.67.39 and 13.0.71.55 versions, and i can't even get the machine tunnel working :-(

Getting the error at the windows login screen "Citrix Gateway Plugin is not connected due to an error. Please contact your IT administrator"

 

At this point the only thing that both documents skim over is the Device Certificate. What format does this need to take? is this just a computer certificate with application policies set to Client Authentication? or something more? been pilling my hair out on this one for a couple of days... in my test environment i'm using my internal Enterprise CA to deploy computer certificates via autoenroll to my test desktops, and i've installed the root CA for it onto the NetScaler in all the places it needs to go.

 

Regards

 

ken Z

[Rhonda Rowland1709152125]

The device certificate is certificate issued to the machine by a trusted CA (like a Domain CA).

You will then need to enable the vpn vserver for client cert authentication.

1) Be sure the vpn vserver has a server cert bound and issued by a trusted CA that the client will trust. This should be issued to the FQDN that gateway fqdn resolves to the VIP with.

2) Then on the vpn vserver also bind a root cert for the CA authority used to issue the client cert to the endpoint device.  This will allow the gateway to trust the client certs presented by the client device. 

3) Be sure the "client device" has a root cert installed to trust this certificate. And then use AD certauthority to issue a machine cert (x.509 cert) to the client device you are testing with.

 

Configure the Gateway for client cert authentication:

4) Create a client cert policy/action.    (For usual client cert authentication, this would normally have Two Factor Authe:ON and require an ldap password as well. For this scenario, you would likely leave the 2FA setting OFF in the client cert policy.)  Bind this to the authentication vserver as mentioned in the setup docs you were looking at, which will handle aaa on behalf of the vpn vserver.

5)  Update the authentication vserver AND the vpn vserver ssl parameters (or use an ssl profile and bind to both).  Set Client Cert Authentication in the ssl parameter (or profile) to Client Certification: Optional (I believe)

 

Device Cert info:  https://docs.citrix.com/en-us/citrix-gateway/current-release/install-citrix-gateway/certificate-management-on-citrix-gateway/use-device-certificates-for-authentication.html

 

What you might want to do first as a test case, is just get a regular vpn configured with client cert authentication working and then transition to the always on tunnel.  So you can see it work on its own before adding the always on tunnel to the testing.

 

See if this helps fill in some gaps for you.

 

[Ken Z]

Rhonda

 

thanks for your swift response...

 

I've got a Domain CA running, and the root certificate is deployed to all Windows Devices (no intermediate cert in my test environment), plus the NetScaler (both VPX NetScalers I was using for testing at different times)

A 1 year Wildcard Server cert was generated (with appropriate SAN entries) for the AO VPN vServer and installed. Connecting to the vServer using Chrome from the test device worked without any errors, so i normally assume that all certificate chains are good.

I've also installed the same root certificate into the vServer in the appropriate places as explained in points 4 and 6 of this link https://docs.citrix.com/en-us/citrix-gateway/current-release/vpn-user-config/always-on-vpn-before-windows-logon/configure-always-on-vpn-before-windows-logon.html

 

As far as the device Certificate goes, I'm not too au fait with device certificates, but I started the Domain CA's Certificate Authority console > Certificate Templates > Manage. Duplicated the Computer template and edited it (name, 1 year validity, Security (autoenroll for domain computers), max key size = 2048, etc), saved it, then issued the certificate template, and rebooted my test windows 10 laptop. A certificate appeared in the laptops' local computer certificate store under "Personal" with client Authentication & Server Authentication. I assume this is good enough but will read up on X.509 certificates to confirm.

 

After that, I installed the corresponding Citrix Gateway Agent and did the registry keys on the test laptop (AlwaysOn=1, AlwaysOnService=2, AlwaysOnURL=https://<URL>, DisableIconHide=1) and rebooted.

After that the machine tunnel failed and here I am (I did a lot more testing, rebuilding, Google searching etc) but that's the gist (obviously the Authentication Policy, Policy Actions, etc were also completed as per the documentation I mentioned at the top).

 

So, my next port of call is the article you posted - https://docs.citrix.com/en-us/citrix-gateway/current-release/install-citrix-gateway/certificate-management-on-citrix-gateway/use-device-certificates-for-authentication.html to see if I've done anything wrong with the device certificate, then the last point you've mentioned about getting Client Cert authentication working on it's own.

 

Thanks and regards

 

Ken Z

 

[Gareth Beattie1709158154]

Check your ciphers that are applied to the Gateway VServer. Try setting it to the Default set of ciphers. We had ours set to a specific group of ciphers and was getting the same error as you, once the DEFAULT set was added it started working.

[Ken Z]

On 2/24/2021 at 12:35 AM, Gareth Beattie1709158154 said:

Check your ciphers that are applied to the Gateway VServer. Try setting it to the Default set of ciphers. We had ours set to a specific group of ciphers and was getting the same error as you, once the DEFAULT set was added it started working.

Gareth

 

sorry for the delay in answering - was dragged onto another project.

 

The SSL Ciphers are set to Default, so unfortunately that's not the issue. If only it was that simple :-)

 

Regards

 

Ken Z

[Rhonda Rowland1709152125]

Can you summarize your cert bindings on the vpn vserver (and cs vserver) if relevant.  

Are you also using a AAA authentication vserver?  (As authentication/ssl cert requirements need to be same between vpn vserver/authe vserver and cs vserver)

Also confirm if you are using SSL Parameters or an SSL profile and whether the client cert settings are set (and how) consistently across all three vservers.

 

Again, I would try just a simple gatweay cert authentication to see if that works to see what the underlying issue is and then go back to the always on vpn troubleshooting.  I'll see if I can find another client cert reference before end of day.

[Daniel Bazan]

facing same issue, hpowever in my case it works 6 out 10 times, reboot and it may work then next reboot error and so forth, it is random and I wish Citrix support would have a fix for this, I have not found one tech that knows this inside out, most of them never even heard of AON, always the same story, lets get a trace going and then you dont hear fro them for days. frustrating as hell

 

if anyone has any other ideas please share, I am ready to go find the CEO and tell him to get more seasoned support techs.

[Ken Z]

Guys

 

as I've never got this working properly, I'm going back to first principles in my test lab and building a NetScaler from scratch to see where i'm going ewrong.

 

Can anyone recommend a good step-by-step guide (not necessarily a Citrix article) on how to set this up using using either NetScaler 12.1 or 13.0 (13.0 preferred)?

 

Thanks.

 

Ken Z

[Ken Z]

On 3/2/2021 at 4:47 PM, Rhonda Rowland1709152125 said:

Can you summarize your cert bindings on the vpn vserver (and cs vserver) if relevant.  

Are you also using a AAA authentication vserver?  (As authentication/ssl cert requirements need to be same between vpn vserver/authe vserver and cs vserver)

Also confirm if you are using SSL Parameters or an SSL profile and whether the client cert settings are set (and how) consistently across all three vservers.

 

Again, I would try just a simple gatweay cert authentication to see if that works to see what the underlying issue is and then go back to the always on vpn troubleshooting.  I'll see if I can find another client cert reference before end of day.

Rhonda

 

very sorry for the delay. This was put on the back burner for the last few months, but now I have a desperate requirement to get this working.

Answers to your points are...

 

1) Upgraded the NetScaler to 13.0.82.45 and confirmed that my NFR license was still valid (VPX 3000 Platinum)

1) From my internal Enterprise CA, I've created a valid wildcard server certificate (with SAN addresses) and applied it to a dedicated vServer just for testing Always On.

2) I also installed the root CA into the NetScaler

3) Added the Server Certificate into the "Server Certificate" section of the AO vServer

4) Added the root certificate into the "CA Certificate" section of the AO vServer

5) Edited the Basic settings of the vServer, added the root CA to the "CA for Device Certificate" section  (I also have 'DTLS' ticked... assume it's not relevant?)

6) Set the SSL Parameters so that SSLv3, TLSv1, TLSv11 and TLSv12 are all ticked (bad practise, but this is a test environment)

7) Only Profile set is HTTP which is set to nshttp_default_profile

8) SSL Ciphers is set to DEFAULT

9) No Session Policies configured (just want to get the machine tunnel up at this point)

10) Created an Authentication Policy with just a machine tunnel policy configured as per official docs. This is using AAA.

11) Using a computer certificate deployed from the same Enterprise CA as the server certificate used on the NetScaler onto the test client device.

12) Can connect to the AO vServer from the test client device via Chrome with no SSL errors - Chrome reports the Certificate as valid

13) The client device (x64 Windows 10 20H2) is accessing the NetScaler VIP via a NAT rule on my Smoothwall firewall in my test lab. The NetScaler is a single-leg device on the internal network, the client device is in a second network separated by the Smoothwall firewall. Everything has access to the internet via a second firewall (an obsolete Netscreen NS204, but does the job)

14) the SSL Parameters and SSL Ciphers are set identically for both the AO and Auth vServers

 

Rhonda, your question about using the same certificate (currently using a wildcard one) between the AO vServer, Auth vServer and CS vServer... I'm not using a CS vServer for this configuration. No where in the docs (https://docs.citrix.com/en-us/citrix-gateway/current-release/vpn-user-config/always-on-vpn-before-windows-logon/configure-always-on-vpn-before-windows-logon.html) does it mention the requirement for a CS vServer. I assume it's optional? Also, I followed the above article, and the Auth vServer config is show in steps 10 - 23 and nowhere in this section does it mention the requirement for adding an SSL certificate. For the hell of it, I checked the state of the Auth vServer (Security > AAA - Application Traffic > Virtual Servers - don't know why I never did before - doh) and it was marked as "DOWN". I added the same wildcard and root CA to this vServer and it was then shown as UP. I will now test this this afternoon to see if the issue was that simple...

 

As an aside, looking at the logs in the Secure Gateway Client on the client device (specifically nssslvpn.txt) , I was getting this every time 

 

sendGARequestThread | 165 | Failed at HttpSendRequest Error code: 12029 

 

Error code 12029 appears to be a generic Microsoft error code meaning "a socket connection failed because encrypted communication could not be established" indicating (to me) an SSL certificate issue, which is why I've been questioning my client (computer) certificate..

 

Thanks to everyone for all the suggestions....

 

Regards

 

Ken Z

 

[Rhonda Rowland1709152125]

Hey, Ken.

Just saw this. One thought - because a lot of people look at the most recent posting and not the most recent activity, feel free to repost these details in a NEW thread (with a link back to this one if useful) to get as many eyes on it to see if someone can help you.    Its fine to post again in this context  as you probably want more than just me commenting :)

 

First, if this is a citrix error 12029 seems to be related to epa processing. Do you have any epa scans?  https://support.citrix.com/article/CTX203650

 

This is still a little hard for me to follow.

1) You are using both the vpn vserver (in always on vpn mode) with a AAA vserver for advanced authentication (and no contentswitching vserver).  Yes, cs vservers are optional but often used.  Will not affect you if you don't have one (and simplifies some of the config troubleshooting at the moment). But regardless certain cert bindings and ssl cert settings must be configured on the vpn vserver AND the associated aaa vserver if both are in use.  (Valid/trusted server cert; a valid root cert to trust the client cert on the device, and the right ssl parameters for ssl cert processing.)

2) You are trying to get client cert working and while you got further, you are still having issues.  Is this cert issued to the machine account or the user doing the login?

 

Generic thoughts:
1) When enabling client cert access on the vpn vserver, the vpn vserver AND the aaa vserver (which actually handles authentication) must be set to support cert authentication as well. (CS vserver too, if in use).  

2) Client Cert requirements, usually involve the following: (regular user flow; not necessarily always on vpn). I'm describing a traditional vpn vserver without always on, because its easy to setup and you can focus on just the cert behavior before troubleshooting always on behavior too.

• Issue "user" cert from the Domain CA to the endpoint device.
• Bind server cert on the vpn vserver and authentication vserver to confirm the identify of the vservers signed by a CA trusted by the client device. (Distribute root cert to endpoint, if needed for Domain signed cert or if using a public signed cert no additional requirement)
• Bind CA Root (or intermediate cert with proper cert chain) to the vpn vserver and authentication vserver as a CA Cert so they can trust the issuer of the client cert on the endpoint device.
• Enable vpn vserver/aaa vserver (both) to accept client cert:mandatory (or optional) under the ssl parameters (or use an ssl profile and bind to both vservers). 
• Then create the appropraite client cert policy (2FA:ON; userPrincipalName or other account format) and then the ldap policy on the AAA vserver.  Steps vary slightly if using advanced vs. classic policies 
• Ensure vpn vserver is integrated with AAA for authentication.

This for a regular user-based authentication and not necessarily the always on vpn config.

If you can get this basic test up and running with some slight modifications to the session policy, then you can confirm that the client connection to the gateway is working, and the aaa and vpn vserver are doing a proper client cert and LDAP validation.

 

While the always on vpn will be slightly different, you will still need 1) the ssl parameters on the vpn vserver/aaa vserver to require the client cert parameter to be on, 2) a root cert for the issuer of the client cert to be bound to both vpn and aaa vserver.   3) You still need to configure a client cert policy followed by a ldap policy for the user flow.

First section on authentication scenarios:  https://docs.citrix.com/en-us/citrix-gateway/current-release/vpn-user-config/alwayson.html

 

 

The point is if you can get the client cert to process in the plain vpn scenario, then you should have all the requirements for always on vpn and just need to fix any always on vpn issues.  Or at least we would know cert processing is good and it is just an always on vpn issue.