Categoriearchief: NSX-T

Fixing an interrupted NSX-T Manager upgrade

nsxtThe process for upgrading the NSX-T managers in an environment is a automated process that works through three managers and finishes the moment all the NSX-T managers are upgraded to the new desired version. Recently I was upgrading a NSX-T datacenter environment from version 3.1.0.0.017107177 to version 3.1.1.0.0.17483065 in my lab environment. The Edge nodes and Transport Nodes had already been upgraded successfully. While we where in the middle of upgrading the the NSX-T manager upgrade got interrupted and the NSX-T managers rebooted when the upgrade was not yet finished.

After all the nodes where back up again I was not able to logon to the Management environment, the designated Virtual IP (VIP) appeared to be down.  When I connected to the first NSX-T Manager machine I was presented with a message indicated that the upgrade had not fully completed. When I executed the following command at the prompt Get upgrade progress-status I was presented with the following output:

2021-03-27 13_46_50-SSDC-Man-PEC - TeamViewer

The output shows that all the upgrade steps where completed successfully. When I connected to the second NSX-T manager machine I got the same output.

I then connected to the third NSX-T Manager, this one was not completed and caused the other NSX-T managers to remain in the upgrading status and the Management VIP to remain unavailable.

2021-03-27 13_45_22-SSDC-Man-PEC - TeamViewer

I first executed the command to see the available upgrade packages on the NSX-T Manager machine.  get upgrade-bundle playbooks To resume the NSX-T Manager upgrade I executed the following command:  resume upgrade-bundle VMware-NSX-appliance-3.1.1.0.0.17483186 playbook

The upgrade process resumed and completed successfully in a manner of minutes, after which the environment became functional again and Management VIP became accessible again.

image

Advanced Cross vCenter vMotion

vmware_vSphere7_graphicVMware released vSphere version 7.0 U1c – 17327586 in December 2020. Next to the cool new features that is included in this version (This blog is al about one of those cool features) another very important reason to download and install this version of vSphere is that it closes a major security issue with previous versions. You can find more info on this here.

New features in this version of vSphere include the following:

  • Physical NIC statistics
  • Advanced Cross vCenter vMotion
  • Parallel remediation on host in clusters that you manage with vSphere Lifecycle Manager baselines
  • Third-party plug-ins to manage services on the vSAN Data Persistence platform

The VMware release notes have the following to say about this new feature:

With vCenter Server 7.0 Update 1c, in the vSphere Client, you can use the Advanced Cross vCenter vMotion feature to manage the bulk migration of workloads across vCenter Server systems in different vCenter Single Sign-On domains. Advanced Cross vCenter vMotion does not depend on vCenter Enhanced Linked Mode or Hybrid Linked Mode and works for both on-premise and cloud environments. Advanced Cross vCenter vMotion facilitates your migration from VMware Cloud Foundation 3 to VMware Cloud Foundation 4, which includes vSphere with Tanzu Kubernetes Grid, and delivers a unified platform for both VMs and containers, allowing operators to provision Kubernetes clusters from vCenter Server. The feature also allows smooth transition to the latest version of vCenter Server by simplifying workload migration from any vCenter Server instance of 6.x or later.

In this blog we will describe the process of importing VMs form a 6.7 vCenter to the updated 7.0.1 vCenter, making use of the cross vCenter technology. To prepare the environment for cross vCenter vMotion the vMotion network has to be configured with a gateway.

image

At the receiving side we tried to VMKping the sending host over the vMotion VMKernel port. When this failed we added a route to any foreign network across the gateway. When we retried the VMKping it was successful.

On the sending side we also configured the vMotion network with a gateway entry.

image

To start the process of performing a cross vCenter vMotion we right click  on the cluster or ESXi host.

image

Click on Import VMs

image

Select source vCenter

image

Select the VMs you want to move.

image

Select the host to transfer the compute to.

image

Select the destination storage.

image

Select networks.

image

Select vMotion priority.

imageReady to complete, click Finish.

The 7.0.1 environment also makes use of NSX-T network virtualization. Why is this important to mention? If you want to perform a roll back you can’t move a VM that is connected to a NSX-T managed portgroup to a none NSX-T managed portgroup. To remediate this issue you should create a none NSX-T portgroup with the same vLAN and add the VM you want to rollback to that portgroup.

Upgrade NSX-T Edge Nodes

image-1VMware NSX-T delivers virtual networking in a software defined datacenter. In this article we are going to take a look at a VMware NSX-T environment that is ready for upgrading. In this blog we will upgrade the seven NSX-T Edge nodes. Let’s first take a look at what is the function of Edge nodes within the NSX-T architecture. An NSX Edge nodes are service appliances that run centralized network services that cannot be distributed to the hypervisors. An NSX Edge node can belong to one overlay transport zone and multiple vLan transport zones.

Today we are performing an upgrade for the Edge Nodes of a NSX-T environment. We are upgrading 7 Edge Nodes from version 3.1.0.0.017107177 to version 3.1.1.0.0.17483065. Before the upgrade we first preform a pre check of the environment, to make sure it is ready for the upgrade.

2021-03-15 18_47_45-SSDC-Man-PEC - TeamViewer

The above image shows that during the pre check there where 6 NSX-T Edge nodes with issues in the environment that could prevent a successful upgrade. Before we go any further we are going to investigate what those issues are.

2021-03-15 18_53_29-SSDC-Man-PEC - TeamViewer

By clicking on one of the affected NSX-T Edge nodes we can see that this node had two issues.

2021-03-15 18_53_55-SSDC-Man-PEC - TeamViewer2021-03-15 18_54_17-SSDC-Man-PEC - TeamViewer

When we click on the blue two with the exclamation mark next to it we can drill further down to identify the current issue. The two alarms indicate that the password expiration is approaching for both the admin and root account.

2021-03-15 18_59_55-SSDC-Man-PEC - TeamViewer

To remediate this issue we will change the password for the Admin and Root account. To accomplish this task we connect to the NSX-T Edge node as root via SSH and execute the following commands:

  • /etc/init.d/nsx-edge-api-server stop
  • passwd admin
  • passwd root
  • touch /var/vmware nsx/reset_cluster_credentials
  • /etc/init.d/nsx-edge-api-server start

2021-03-15 19_06_13-SSDC-Man-PEC - TeamViewer
The Edge-TN-07 is now without errors, we proceed by checking the other NSX-T Edge nodes and preform the same actions on those nodes.

2021-03-15 19_21_02-SSDC-Man-PEC - TeamViewer

The other NSX-T Edge nodes are now also without errors.

2021-03-15 21_35_40-TraXall – Toegang tot de car configurator_ Robin PLOMP - Message (HTML)

In the upgrade window we select the Edge Node cluster and we start the upgrade.

2021-03-15 22_48_41-SSDC-Man-PEC - TeamViewer

Grab a drink (coffee) and wait for the progress bar to fill up to 100%

2021-03-15 22_50_04-SSDC-Man-PEC - TeamViewer

In the upgrade overview window we can now see that the seven NSX-T Edge nodes are now upgraded.