The process for upgrading the NSX-T managers in an environment is a automated process that works through three managers and finishes the moment all the NSX-T managers are upgraded to the new desired version. Recently I was upgrading a NSX-T datacenter environment from version 3.1.0.0.017107177 to version 3.1.1.0.0.17483065 in my lab environment. The Edge nodes and Transport Nodes had already been upgraded successfully. While we where in the middle of upgrading the the NSX-T manager upgrade got interrupted and the NSX-T managers rebooted when the upgrade was not yet finished.

After all the nodes where back up again I was not able to logon to the Management environment, the designated Virtual IP (VIP) appeared to be down.  When I connected to the first NSX-T Manager machine I was presented with a message indicated that the upgrade had not fully completed. When I executed the following command at the prompt Get upgrade progress-status I was presented with the following output:

The output shows that all the upgrade steps where completed successfully. When I connected to the second NSX-T manager machine I got the same output.

I then connected to the third NSX-T Manager, this one was not completed and caused the other NSX-T managers to remain in the upgrading status and the Management VIP to remain unavailable.

I first executed the command to see the available upgrade packages on the NSX-T Manager machine.  get upgrade-bundle playbooks To resume the NSX-T Manager upgrade I executed the following command:  resume upgrade-bundle VMware-NSX-appliance-3.1.1.0.0.17483186 playbook

The upgrade process resumed and completed successfully in a manner of minutes, after which the environment became functional again and Management VIP became accessible again.