Multi-function PCIe switch nine: Problems needing attention in single NT system

Multifunction PCIE Switch IX: Issues needing attention in a single NT system

1. Difference between single NT and dual NT system

NTB is typically used on dual-or multi-control systems to achieve cross-node data transfer. On a system composed of multiple nodes, there is usually a NTB chip on each node to achieve the system-wide address space partitioning and routing. As the name implies, single NT refers to the use of a PCIe non-transparent bridge to achieve inter-node data transfer scheme, usually used between two nodes. Compared with the multi-NT system, this method is relatively simple, low cost and short development period, which can shorten the verification cycle of the key functions and performance of the whole system.

2. Implementation of single NT system

The implementation of single NT needs to combine the design requirements of the whole system, according to the specific server's motherboard and various settings, determine the physical NTB need to install on which machine, and then consider and check the following settings:

A. Check the BIOS for NTB related enable/disable switch, if any, open the corresponding switch;

B. On the node where the physical NTB is required, the EEPROM can enable the corresponding NTB, while on the other nodes, the EEPROM is DISABLENTB;

C. According to the design requirements of the system, set the number and size of address conversion window.

3. Features of PCIe devices in a single NT system

In a single NT system, because the NTB of the Linkport side is actually disable, the NTB corresponding bar space seen on the linkport side is actually mapped over the Virtualport side and then set by the Linkport side operating system when it initializes the PCIe. Therefore, after virtualport end multiplicity, the corresponding bar register of Linkport NTB will change, which will result in linkport not being able to read and write data to virtual port. In addition, if the Linkport node is power-on, the virtualport side is not power-up,

Then Linkport can not see the NTB device when initializing the PCIe bus, resulting in the linkport side of the NTBPCEI device will not exist, the driver is not loaded.

4. How to evade the problems encountered in single NT system

In view of the above problem, understand the root cause of the problem, it is not difficult to think of the following solution:

1. Make sure that the Ntbpcie device on the Virtualport side has been probed and initialized when the Virtualport is power-on prior to Linkport and the Linkport is guaranteed to initialize the PCIe;

2. If the Linkport node is able to use NTB normally, then the Virtualport node restarts, then Linkport will need to restart or revert to the previous Ntbpcie settings, and after the Virtualport succeeds, the NTB can be used.

This article from "Storage Chef" blog, reproduced please contact the author!

Multi-function PCIe switch nine: Problems needing attention in single NT system