Jump to content


Photo

Suddenly Disconnect to SSH Server in Management -mashine

Started by Matt Matsen , 10 February 2017 - 01:02 PM
15 replies to this topic

Matt Matsen Members

Matt Matsen
  • 21 posts

Posted 10 February 2017 - 01:02 PM

Hi,

 

i have problems with the connection to my Xenserver: SSH and XenCenter, so i can enter the rootuser and ping also works fine. then the next thing is a connection close, but the password i definitly correct.

 

Maybe something goes wrong when i had done a: "sudo rm -rf /*" on a VM Mashine Linux system (My fault i wanted to delete /backups_temp/ not the root).

But why is there also a problem on the management vm? i thought the VM's are isolated from the Managemnt VM??

 

Now i have this trouble, can maybe plaese someone help me here?

 

PS: The VM's are still working. for my luck at a friday afternoon ;)



Ivan Esterajher Citrix Employees

Ivan Esterajher
  • 16 posts

Posted 10 February 2017 - 01:31 PM

Hello Matt,

 

can you access the console of that host and try logging in from there? That way we can rule out any networking issues.

If you can't log from the console, you can try changing the password by booting to the single user mode as described in this article. This will however require downtime on the VMs that are currently running on that host.

 

When did this start happening? Have you changed anything in your environment besides doing "rm" in that one VM?

 

Doing an rm -rf / on a VM doesn't affect the dom0 (management VM) so I don't think you would have these problems because of that.



Matt Matsen Members

Matt Matsen
  • 21 posts

Posted 10 February 2017 - 02:02 PM

Hey, thanks i will try to access the console next time, so it is not in the same house.

 

- The ping already works fine, its just the ssh and the xencenter which make troubles.

 

- The connection timeout started 3-6 sec after the 'rm rf', so this make me thinking about it

 

- i have done just the command "sudo ethtool -e eth0 raw off" on the dom0 host to read out the EEProm from network device, so there weren't any write procedures etc.

The only write procedure was on one VM with "xe vif-param-set uuid={uuid of vif} other-config:ethtool-tx="off""



Ivan Esterajher Citrix Employees

Ivan Esterajher
  • 16 posts

Posted 10 February 2017 - 02:22 PM

Just to understand this correctly:

 

You are able to SSH to the machine? It accepts root and password?

Is the connection closing soon after you logged in over SSH? 

Maybe there is a problem with the remote connection from where you have XenCenter installed to the actual XenServer.

 

It would be interesting if you can access /var/log/kern.log and see if there are any more info there.



Matt Matsen Members

Matt Matsen
  • 21 posts

Posted 10 February 2017 - 02:31 PM

no/yes i can't access ssh..

so im connecting to the IP of the dom0 and get the Login Promt: there i can type in the user: root , then the password and now i get the "connection close" error.

 

and the first time, right after i put in the "rm -rf" on the VM  i was in my xencenter console and got just kicked by the server, without return back into



Matt Matsen Members

Matt Matsen
  • 21 posts

Posted 10 February 2017 - 02:44 PM

the ssh error is:

 

root@192.16.11.2's password: (typing in the right pass)
Connection to 192.16.11.2 closed by remote host.
Connection to 192.16.11.2 closed.

 

My first thought is here there was something deleted, i dont know why...

But if it this the case, is there a way to recover and rescue the system dom0 ?



Ivan Esterajher Citrix Employees

Ivan Esterajher
  • 16 posts

Posted 10 February 2017 - 02:49 PM

This seems like a network / firewall issue. If you're not currently at the location and don't have access to a remote console I don't think there is much you can do.

 

Did you maybe change the .ssh/authorized_keys file?

 

If you can get access to the machine, paste parts of the daemon and kern logs which you will find interesting so we can troubleshoot more.

 

You can also try ssh -vv root@192.16.11.2 which will show more verbose messages about connecting through SSH.



Matt Matsen Members

Matt Matsen
  • 21 posts

Posted 10 February 2017 - 03:23 PM

ok, on monday i will try to get access to the console on xenserver:

 

another thuing is: on the http://192.16.11.2 ive got also a Not Found:

 

Not Found

The requested URL / was not found on this server.

Xapi Server

 

 

 

 

-- Im at this point not sure if there was maybe my fault and i have deleted the files on the wrong terminal console accidently in my working rush  :\

 

 

here are the ssh debug logs:

 

serv:~# ssh -vv root@192.16.11.2
OpenSSH_6.7p1 Debian-5+deb8u3, OpenSSL 1.0.1t  3 May 2016
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to 192.16.11.2 [192.16.11.2] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.7p1 Debian-5+deb8u3
debug1: Remote protocol version 2.0, remote software version OpenSSH_4.3
debug1: match: OpenSSH_4.3 pat OpenSSH_4* compat 0x00000000
debug2: fd 3 setting O_NONBLOCK
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1,diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-dss-cert-v00@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-md5-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-md5-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: setup hmac-sha1
debug1: kex: server->client aes128-ctr hmac-sha1 none
debug2: mac_setup: setup hmac-sha1
debug1: kex: client->server aes128-ctr hmac-sha1 none
debug2: bits set: 992/2048
debug1: sending SSH2_MSG_KEXDH_INIT
debug1: expecting SSH2_MSG_KEXDH_REPLY
debug1: Server host key: RSA 82:33:3f:6b:8c:08:72:3b:eb:e5:d8:da:4a:f8:af:21
debug1: Host '192.16.11.2' is known and matches the RSA host key.
debug1: Found key in /root/.ssh/known_hosts:6
debug2: bits set: 974/2048
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /root/.ssh/id_rsa ((nil)),
debug2: key: /root/.ssh/id_dsa ((nil)),
debug2: key: /root/.ssh/id_ecdsa ((nil)),
debug2: key: /root/.ssh/id_ed25519 ((nil)),
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Trying private key: /root/.ssh/id_rsa
debug1: Trying private key: /root/.ssh/id_dsa
debug1: Trying private key: /root/.ssh/id_ecdsa
debug1: Trying private key: /root/.ssh/id_ed25519
debug2: we did not send a packet, disable method
debug1: Next authentication method: password
root@192.16.11.2's password:
debug2: we sent a password packet, wait for reply
Connection closed by 192.16.11.2
 

 



Matt Matsen Members

Matt Matsen
  • 21 posts

Posted 13 February 2017 - 10:30 AM

Hello again,

 

now i was able to take a look at the command managemnt window, and there is no pissibility to access the console or something else:

 

when i've trying to log-in i get the error: "Critical error - immediate abort','26'" with the correct password!

 

So I'm sure there is something not ok, and I Think i had deleted something on the managemnt side with my 'rm -rf /*' command (i thing also, the active, accessed files weren't deleted): such a distster :(

 

So im a little bit scared about to restart the Xenserver Mashine. (VM's still running)

 

Is there maybe a way to rescue the mashine with Install CD or something?



Ivan Esterajher Citrix Employees
  • #10

Ivan Esterajher
  • 16 posts

Posted 13 February 2017 - 10:44 AM

Take a look at this blog post

Could be down to a full filesystem but could be some other problems too. If you are sure you did the rm rf on the VM, than you're fine, but if there's a possibility of you doing it on dom0 then there could be problems.



Matt Matsen Members
  • #11

Matt Matsen
  • 21 posts

Posted 13 February 2017 - 02:30 PM

ok, thanks first, so i will test the reboot at wednesday, im still hoping it's maybe a funny coincidence and there is just a problem with the full rootpath or something.

 

But: If its not the case, there is no pissibility to rescue the system and i will lost all my vm data? :\



Matt Matsen Members
  • #12

Matt Matsen
  • 21 posts

Posted 15 February 2017 - 12:25 PM

Hey, it seems i have a problem ... and worst case triggered.

 

Please i need some help here :( It would be horror if the Storage repository with all the VM's would be lost... so there aren't any backups (shame on me)

 

please help thnaks very much in adavance :(



Ivan Esterajher Citrix Employees
  • #13

Ivan Esterajher
  • 16 posts

Posted 16 February 2017 - 01:58 PM

Hello Matt,

 

What happened?

You can boot the server with a Clonezilla or a similar tool and create a disk image just to be sure, before proceeding to troubleshoot.

 

Is it booting back up? Any messages on the console?



Boby kunjukunju John Citrix Employees
  • #14

Boby kunjukunju John
  • 5 posts

Posted 16 February 2017 - 04:10 PM

.



Matt Matsen Members
  • #15

Matt Matsen
  • 21 posts

Posted 16 February 2017 - 05:12 PM

yes, but i cannot access the console and im also using a Raid0 for my two raid controllers, so i think everything is lost and i have to reinstall the full server :(



Ivan Esterajher Citrix Employees
  • #16

Ivan Esterajher
  • 16 posts

Posted 17 February 2017 - 08:36 AM

Did you try serial access?

https://support.citrix.com/article/CTX121442