Case study of service interruption caused by Metro cutover and cutover (1)

This is an example of a failure caused by ios software Bug on an mpls vpn network. I have sorted it out for your reference and discussion. The occurrence and resolution of faults are contingent, it is not important. We only hope that through discussion, we can improve our ability to analyze and solve problems.
I. Fault Cause
A city A man's backbone two 7206R1 and R2) Uplink core network, downlink city domain, mutual backup, run MPLS, the figure is not painted, huh, huh. Upgrade the city region one day to adjust and expand the man. (The metro structure of City B in the following section is different and the scale is larger. However, in this example, we can also understand it based on this structure, so we will not describe it in detail)
Upgrade process:
1. Upgrade 7206ios
2. Business cutover
3. Perform the redundancy test, HSRP test, full-network test, and service test, and perform the last user connectivity test. OK, let's go!
4. observation phase.
The next day, a user reported that the mpls vpn network was disconnected. Note: The cutover process does not modify the user configuration !)
Ii. Fault
The user reports that the mpls vpn service from cutover City A to another city B is interrupted.
A user's business in City A is connected to the PE router in the form of mpls vpn through the CE device, that is, the upgraded Backbone Router Cisco VXR 7206). Similarly, A user's business in City B is connected to the corresponding PE router of the provincial backbone network through the CE device in the form of MPLS VPN.
A fault occurs when A user in City A can PING the local PE router in City A, and the local network connection of A user in City A is normal, however, you cannot PING a user business sub-interface of the PE router of the Remote City B.
In the same way, A user in City B can PING the PE router in City B. The local network connection of A user in City B is normal, but cannot PING the PE router in City.
The most strange thing is that, on the PE router connected to A user's business in City A, there are also five other VPN users, and the network services of these five VPN users are completely normal. The network service settings are the same as those of a user's business. Compare the commands between all VPN services.
For A user's VPN service, the VPN between City A and City B and the IPV4 route are completely normal!
Iii. troubleshooting Process

1) fault isolation
First, open the logical port Loopback1 on the Cisco 7206VXR on the backbone PE router of A user VPN in City A, and route Loopback1 into A user VPN, at this time, a user in City B can receive the Loopback address route through the VPN, but still cannot PING the Loopback address. Therefore, it can be proved that the VPN interruption of a user is not caused by the new man device, and the problem occurs in the MPLS interaction process between provincial backbone routers. In this case, we will focus on Troubleshooting the provincial backbone network.

2) command processing process
The main processing work is as follows:
1. Run the show ip vrf command on the backbone router r2 of City A to check the mpls vpn configuration and RD name. All are correct. Run the show ip vrf detail vpn9: XXX. VPN and show ip vrf interface commands to check the route attributes of a user's vpn and its interface status and ip address. All are correct. Run the show ip bgp neighbors/show ip bgp vpnv4 all/show ip bgp vpnv4 vrf vpn9: XXX. VPN command to check the BGP relationship between the two pes of City A and City B. All are correct.
2. Run the show ip route vrf vpn9: XXX. VPN command on the backbone router R2. You can see that the vpn route is correct, which is A directly connected route city A local user) and A bgp Route City B user ). Run the ping command to ping vrf vpn9: XXX. VPN 210.5.2.130. Test the connectivity between local PE of City A and local CE of City. The test result is successful. Run the ping command to ping vrf vpn9: XXX. vpn x. Test the connectivity between A local PE and BPE. The test result is ping failure.
3. telnet to the PE router of City B and run the show ip route vrf vpn9: XXX. VPN command. You can see that the vpn route is correct, which is A user in the directly connected routing City B) and A user in the bgp routing City ). Run the ping command to ping vrf vpn9: XXX. VPN 210.5.0.222. Test the connectivity between the local PE of city band the local CE of city B. The test result is successful. Run the ping command to ping vrf vpn9: XXX. VPN 210.5.2.129 .. Test the connectivity between City B PE and city A PE. The test result is ping failure.
Because the routes at both ends of the vpn are correct, and no changes are made during the cutover process. The cutover test is also normal. After the discussion, we come to a preliminary conclusion that the problem is not caused by cutover. It may be caused by other reasons. The direction of the analysis should be unrelated to the cutover, focusing on the direction of mpls vpn.
4. run the clear ip route vrf vpn9: XXX. VPN *, run the show ip route vrf vpn9: XXX. VPN, you can see that the vpn route has changed, and a user in bgp Route City B has disappeared. Run the show ip route vrf vpn9: XXX. VPN can be seen that the vpn route has changed, except for A user in the directly connected route City A), A user in the bgp Route City B) appears again. Run the show ip route vrf vpn9: XXX. VPN command on the PE router of City B. You can see that the vpn route is correct, which is A user in the directly connected routing City B) and A user in the bgp routing City ). Ping vrf vpn9: XXX. vpn x. X on the backbone router R2. Test the connectivity between city BPE and city APE. The test result is ping failure.
5. run the clear ip route vrf vpn9: XXX. VPN *, run the show ip route vrf vpn9: XXX. VPN, you can see that the vpn route has changed, and A user in bgp Route City A has disappeared. Run the show ip route vrf vpn9: XXX. VPN can be seen that the vpn route has changed, except for A user in the directly connected routing City B), A user in the bgp routing City A) is re-displayed. Run the show ip route vrf vpn9: XXX. VPN command on the backbone router R2. You can see that the vpn route is correct, which is A user in the directly connected routing City B) and A user in the bgp routing City ). Ping vrf vpn9: XXX. vpn x. X on the backbone router R2. Test the connectivity between City B PE and city A PE. The test result is ping failure.
6. run the command shutdown on A user's sub-port on the backbone router R2. run the command show ip route vrf vpn9: XXX. VPN, you can see that the vpn route has changed, and A local user in the directly connected route City A has disappeared. Run the show ip route vrf vpn9: XXX command with no shutdown. VPN can be seen that the vpn route has changed, and the directly connected route City A is A local user) is re-displayed, and A bgp Route City B user still exists. Run the show ip route vrf vpn9: XXX. VPN command on the PE router of City B. You can see that the vpn route is correct, which is A user in the directly connected routing City B) and A user in the newly learned bgp routing City ). Ping vrf vpn9: XXX. vpn x. X on the backbone router R2. Test the connectivity between city BPE and city APE. The test result is ping failure.