Unable to login with Default Credentials with NEW Installation

[Pearson VUE ATS]

Hello.  We've recently stood up instances of the Citrix Virtual Netscaler, NSVPX-ESX-12.1-60.19_nc_64 residing in Vsphere.  I am able to set the NSIP, Subnet, and Gateway.  I can see via SSH and via a browser.  However; I am unable to login with the default nsroot or nsrecover.  Doesn't matter if I'm at the console, browser or SSH.

From the console, I see the error "Internal Authentication Error, Exiting".  Then it brings me to a bash prompt.  These are brand new.. so how do I get logged in or reset this?  Is there another user?  This is happening to 7 instances....   I only see nsroot and nsrecover as documented options.

 

I've also tried the article, https://docs.citrix.com/en-us/citrix-adc/current-release/system/authentication-and-authorization-for-system-user/how-to-reset-nsroot-administrator-password.html to reset the password.  

 

So, just a question, getting stuck here on trying to mount flash.  I ran the LS command and see a flash directory.  I ran df but don't see flash mounted, just;

Filesystem 1K-blocks Used         Avail   Capacity  Mounted on
/dev/md0   422318     395666     18206    96%         /
devfs                   1                1            0    100%       /dev

So tried mount/dev/md0 /flash and mount/dev/devfs /flash but it doesn't like either.  Guessing I am not following this.

What is the ad0s1a supposed to reference?

Run the following command to display the mounted partitions: df If the flash partition is not listed, you must mount it manually.

Run the following command to mount the flash drive:
mount/dev/ad0s1a /flash

 

Side note - I don't have full control in VSphere - this is handled by Corporate and they setup the image and confirmed they didn't set a password.

 

Thanks in advance for your expertise.

[Rhonda Rowland1709152125]

First,

Are you sure you are running the correct command:

mount/dev/ad0s1a /flash

needs a space (which is incorrect in the article referenced):

mount  /dev/ad0s1a   /flash

 

ad01s1a is the default flash card device name.

However, on some models or vpx instances the device name *might* be different:

The following models use the /dev/ad0s1a device name for the flash disk:
7000 9010 9950 10010 12000 MPX-15000 MPX-17000

The following models use the /dev/ ad4s1a device name for the flash drive:
MPX-5500 MPX-7500 MPX-9500 MPX-9700 MPX-10500 MPX-12500 MPX-13500

 

Edit:  Try this first for the vm (if you have a running vpx, you can also login and go to shell and do: df -h to confirm. But if not, this is likely the on you are looking for.

For VMWare, it might be /dev/da0s1a/

 

 

Found in the comments from this post by JG Spiers:  https://www.jgspiers.com/reset-netscaler-nsroot-password/

 

 

 

 

Edited April 3, 2021 by Rhonda Rowland
Added notes.

[Pearson VUE ATS]

Hello.  That is a step in the right direction.  Thanks the flash is located here, da0s1a.

 

However; following the article, step by step, https://www.jgspiers.com/reset-netscaler-nsroot-password/, I still cannot login with nsroot.

 

I seem to be able to login via nsrecover via the Console or SSH but not through the Browser.

 

I seem to have limited options however and I did notice some errors when logging in.  Even after I login with nsrecover, I get the following <local10.err> ns login:  Error Connecting to PEO: Connection Failed.  Then will bring me to a bash prompt but unable to show nsconfig for instance.

 

I am able to run df and see the drives however;

 

FileSystem                     1K-blocks                                        Used                        Avail                    Capacity           Mounted on

/dev/Md0                           422318                                   396298                        17574                        96%                  /

devfs                                                1                                                  1                                  0                       100%                   /dev

procfs                                              4                                                 4                                 0                        100%                   /proc

/dev/da0s1a                     1623950                                  146046                    1347988                         10%                   /flash

/dev/da0s1e                   14519676                                 5831250                  7526852                         44%                 /var          

 

 

Thanks for your help.

 

 

[Pearson VUE ATS]

I noticed if I do an LS I don't see /nsconfig but see it under /flash.  So do see contents under ls nsconfig as long as I changed the directory to /flash.  So maybe something more is missing.

 

[Rhonda Rowland1709152125]

/nsconfig is a symbolic link to /flash/nsconfig.  This is the full path; /nsconfig is a shortcut.

 

During the safeboot mode the path isn't present and you have to mount the device per the article.

 

To reset the password, you must be on a console connection and not remote ssh - for phsyical systems this means physical access via serial cable (or other console-based access means); for VMs it means console connection provided by hypervisor.

The safeboot bypasses the  authenitcaiton requirement so you can "reset" the password in the node.

If this build in a newer than Oct 2020, then you need the serial number to login or the default credential for vpx may be blank and need to be set for first time use.

You may want to rest both the nsroot and the nsrecover account.

 

 

Can you confirm that via console you successfully initiated the Ctrl+C command and the loader config step, got the hearts in the prompt (as you may need to continuously hit ctrl+c until the interrupt occurs) from there you do the boot -s command to proceed with the rest process.

 

 

Edited April 6, 2021 by Rhonda Rowland

[Pearson VUE ATS]

I've been connecting via VSphere at the console layer, I did get the Ctrl-C and follow the steps but will try again.

 

Thanks

Jennifer

 

[Rhonda Rowland1709152125]

I'll repeat myself this evening on my test environment and see if I can see any problems or alternate articles.

 

[Pearson VUE ATS]

I ran through again, my prompts mach the screen shots in jgspiers.com article, using da0s1a

 

<local0.crit>  ns nsnetsvc:  See errors such as ns nsnetsvc: boot_netscaler: ns_start returned an error

<local0.crit>  ns nsnetsvc:  BOOT FAILURE:  One or more packet engines failed to start

<local0.crit>  ns [25] Pitboss:  Netscaler boot has failed; another reboot won't help

<local0.crit>  ns [25] Pitboss: Use nsrecover to login, Review build, config and logs for causes,

<kern.info> ns kernell:  ns kernel pid 1279 (nsnetsvc), uid 0 : exited on signal 6 (core dumped).

Startup Failed - writing dmesg to the log file.

 

Then it brings me to the login prompt.

 

If I try nsroot, get nsnet_connect:  No such file or directory

Login incorrect

 

Then I can login as nsrecover and get Error connecting to PE0:  Connection failed.

 

Seems something didn't load with the initial image when corporate mounted via VMWare..

 

Thanks in advance.

[Pearson VUE ATS]

Attaching some screen shots.

 

 

[Pearson VUE ATS]

Thinking through this and looking at your comments again, I did see references to the serial number in some documentation but this is a VM and not a physical device so not share if this is relevant.  If it is, where would you get the serial number on a vm?

 

Thanks

 

 

[Rhonda Rowland1709152125]

1) for a vpx, the first time connection without a password set is <blank> and therefore you have to set password on first connect. Or it defaults to nsroot and requires change.

 

The above issue looks like a kernel load issue and that is a different problem.

So your issue might not be the password reset it might be something else; Does this system boot normally (you just can't login)?  And was this a post upgrade issue or a change in hypervisors when this started.  That sounds like kernel recovery and not just a password reset but support might be needed for better options.

 

 

NOTE: This are really old articles on kernel recovery AND without knowing for sure that is what is going on I would hesitate to go down this path without checking with support to make sure the problem isn't bigger than just the password reset. Procedures for kernel recovery AND password resets are similar in that you do a safe mode boot AND you mount flash but there are some differences.  And at this point don't want you to lose config that you are trying to preserve.

https://support.citrix.com/article/CTX202541

http://terenceluk.blogspot.com/2016/03/citrix-netscaler-vpx-appliance-fails-to.html

https://support.citrix.com/article/CTX121992

[Pearson VUE ATS]

Thanks for your input its appreciated.  These are the first we are running on VMWare as our others were physicals.  I am happy to open a support ticket, just like to see if it's an easy fix I can do first and this is happening to 7 brand new instances, so my guess is something was not configured correctly on the VMWare/ESX side or something else.

[Paul Blitz]

Silly question: how many CPS and how much RAM has been allocated to this VM?

[Pearson VUE ATS]

4x2 sockets = 8 total vCPUs, 12 GB RAM

[Pearson VUE ATS]

Thanks for your assistance on this topic.  I did open a ticket with Citrix and they indicated there is a known bug in version 12.1-60.19 that were causing the kernel errors amongst the others and suggested that the version 12.1-61.18/19 is a better fit.  And actually seems we needed a different version all together since we use FIPS so we are going to blow the VM's away and install using version 12.1-55.210 to accommodate our FIPS requirement and allow us to backup/restore so to speak our configuration from our physical Netscaler MPX9700's.

 

Thanks