Node stucks in Phoenix (no boot back to Hypervisor)

If a node stucks in Phoenix you can get it back to the HV, but first check that there are no tasks runing from phoenix: On a CVM check this:

ecli task.list include_completed=false
host_upgrade_status
firmware_upgrade_status

If there is nothing running, you can do this in Phoenix:

python /phoenix/reboot_to_host.py

Then the host will boot back to HV, you will have to remove the CVM from maintenance mode:

cluster status (to see, if everything in the Cluster is ok)

ncli host list (to get UUID from the host)

ncli host edit id=<insert uuid here> enable-maintenance-mode='false'

Check the metadataring:

nodetool -h localhost ring

If the Host is AHV, you have to leave Maintenance Mode for AHV, too:

acli host.list
acli host.exit_maintenance_mode <hostname>

or in newer Versions use the Nutanix provided script:

also first check upgrade status etc.

then on a CVM:

python /home/nutanix/cluster/bin/lcm/lcm_node_recovery.py <IP of affected CVM/phoenix>

This works with ESXi and AHV, maybe you have to provide the vCenter and Credentials.

The script should also end the foundation service, but this don't work.

So after bringing back the node you have to check:

allssh 'genesis status|grep foundation'

If foundation is running on one node

allssh genesis stop foundation

Then the Node should be back and running in your cluster.