Nested OpenStack VMs in vcenter cannot ping external network problem Resolution

Problem Description:

Recently built the vcenter environment, and using the VMS created by Vcenter to build a set of OpenStack environment, in verifying the external network functions of OpenStack, found that the message ping is not through the outside network, packet capture found in the DVS at the Vcenter lost, This is a very strange thing, after careful troubleshooting, now vcenter unexpectedly aware of the message of the Mac to the VM is not managed by the VMS sent by the message directly ignored.

First:

The explanations are as follows:

1) esx-b016 is a host with VMware ESX installed, managed and controlled by vcenter, I use vcenter to create a virtual distributed switch (DVS01), using the eth2 of the esx-b016 host as the upstream port of this switch (some may ask, Why use Eth2 as the upstream port, this because eth0 and Eth1 were occupied by others:); esx-b016 's eth2 Nic is connected to the SWITCH1 port of the external physical switch g1 and then connected to the physical gateway router G3 via the Rrouter1 port of the switch The physical gateway router has an IP of 162.3.110.1.

2) using Vcenter to create a VM on the ESX-B016 host (virtual machine name OPENSTACK_VM), used to install the OpenStack environment, the VM's network card eth0 connected to the DVS01 port group DVPort1;

3) in the OpenStack environment, I created a tenant network (NET1:192.168.0.0/24), a virtual machine (USER_VM), a router (R), associated Net1 to router, and created a network (ext1:162.3.0.0/ 16) as an external network

4) The Ext external network created is VLAN type, Vlanid is set to DVPort1, the port group is VLAN trunk, allow 2-4094 through, while G1 is set to trunk port, G2 is set to access port, only VLAN 1000 is allowed to pass.

When I was ready, I went into the USER_VM virtual machine to ping the 162.3.110.1 operation, theoretically it should be able to ping the gateway, but it is strange how not to pass.

Positioning process

All right, just a weapon to grab a bag. Tcpdump, first look at the transmission route: USER_VM, Br-int->r, Snat, Br-int, Br-eth0-eth0 Uplink (eth2), g1–> G3, Router1,

1) The first step I perform ping 162.3.110.1 in USER_VM

2) First determine whether the ping message is sent out, the eth0 on the USER_VM virtual machine to grab packets, found to be able to catch the ICMP request to 162.3.110.1 packet, but no response;

3) Grasp the packet on R, but also can catch the ICMP request message to 162.3.110.1, but no response;

4) in the eth0, the same can catch the ICMP request message to 162.3.110.1, but no response, indicating that the message has been sent from the OPENSTACK_VM virtual machine, into the dvs01 distributed switch;

5) distributed switch DVS01 through the upstream port to the physical switch switch1 G1 port, I execute displaymac-address on switch1 | include GE0/0/1 command, monitoring all messages passing through the G1 port, Did not find the source MAC address for the snat of the external network port MAC any message (here is why the source Mac outside the network port, do not understand the classmate can think carefully), this indicates that the message was not sent to Switch1 in time, is it after the dvs01 disappeared out of thin air?

Workaround:

After checking the data, found that the original was dvs01 lost, this is due to the configuration of the port group, the OPENSTACK_VM corresponding port group configuration can be modified as follows to resolve the problem:

The root cause is that the snat above the outer gateway we created with OpenStack is not recognized by DVS01, and if the pseudo-transmission is set to reject, ESX will compare the message Mac that is being transmitted with all of the available Macs and find that inconsistent messages will be discarded. What is valid here? It is certain that ESX's own assigned MAC address is valid, and OpenStack allocates a MAC address that ESX does not perceive and is therefore considered invalid.

Additional Information:

The following is a description of the promiscuous mode and pseudo-transfer in the VMware official documentation:

Promiscuous mode (promiscuous):

Promiscuous mode controls whether the virtual machine can view the unicast traffic for other nodes on the ESX host. By default, this option is set to [Reject (Deny)], which means that the virtual network adapter cannot run in promiscuous mode. In promiscuous mode, the virtual network adapter does not need to perform any receive filtering, so the guest operating system can receive all the traffic observed on the line. Although promiscuous mode can effectively track network activity, this mode of operation is extremely insecure because, regardless of whether some packets are only received by a particular network adapter, all adapters in promiscuous mode can access such packets. This means that the administrator or Root user in the virtual machine can view traffic that is transferred to other clients or to the host operating system.

Although the most common promiscuous mode should be turned off, the virtual switch can also be configured to run in promiscuous mode if the network intrusion detection software or packet port scanner is running.

Forged transmits (pseudo-transfer):

Spurious transmissions will affect outbound traffic. By default, this option is set to accept, which means that the ESX host does not compare the source MAC address to a valid MAC address. If you set this option to [Reject (Deny)],esx The primary opportunity compares the source MAC address that the operating system is transmitting to a valid MAC address of its adapter to see if they match. If the address does not match, ESX discards the packet. The guest operating system does not detect that its virtual network adapter cannot send packets using the emulated MAC address. The ESX host intercepts the packet before it is transmitted using an analog address, so the guest operating system may assume that the packet has been discarded.

Copyright NOTICE: This article for Bo Master original article, without Bo Master permission not reproduced.

Nested OpenStack VMs in vcenter cannot ping external network problem Resolution