NTB EEPROM setup and data transfer across nodes

Ntbeeprom settings and data transfer across nodes

In addition to monitoring the heartbeat of the system state, the dual-control dual-live system also requires a channel capable of transmitting data across nodes. The PCIe non-transparent bridge (NTB) is favored by many people because of its standard-based PCIE specification, low software dependency, fast speed, and easy configuration. The principle of cross-node data transfer based on PCIENTB is simple, as indicated:

650) this.width=650; "src=" http://s2.51cto.com/wyfs02/M01/7D/38/wKioL1bisMvSjHKnAADVgqof8MQ879.jpg "title=" ntb_5_ 1.jpg "alt=" Wkiol1bismvsjhknaadvgqof8mq879.jpg "/>

Compared to the normal PCIE/PCI device configuration space Register, NTB more BAR2/3,BAR4/BAR5 address conversion register, the local node sent over the hit BAR2/BAR3 or bar4/ The address high of the BAR5 TLP is converted to the value in the corresponding address translation register and forwarded to the other node.

In my recent NTB debugging process, the discovery data can be written from the Linkport side to the virtualport side of the specified memory address, and virtualport this side can read back from the specified address to the Linkport end of the data written back But a similar program exchange executes on the virtual port and the Linkport end, it is found that the Virtualport end can write the data to the Linkport side, and the data read back from the Linkport side is consistent with the previously written data. However, the Linkport side cannot read back the expected data from the specified address.

In order to analyze this problem, the author first uses GDB to trace the code, did not find the problem of the program logic, and then the two ends of the program unified into a file, with different input parameters to control whether to do remote reading or writing, re-test, or can see the same phenomenon. is baffled by the time, suddenly think of the principle of address conversion, the mind of the process of address conversion:

First of all, to determine whether the address of the TLP received by the NT hit its corresponding bar, in the actual example I used, the actual use of the bar3/bar2 composed of a 64bit base address. After checking, the CPU is determined to access the data to this address;

Next, check whether the value in the address translation register is the desired value. The correct value should be the bus address of the node where the memory is to be accessed across the node, and in the Intelx86 architecture, the bus address is the physical address of the memory. Check the value of this register is also no problem, the following value is: 0x0d6d80007800003

[[Email protected]bin]# lspci-xxxxx-s 06:00.0 | grep "C30"

c30:00 (0b) B7 (6d) at XX

[[Email protected]bin]# lspci-xxxxx-s 06:00.0 | grep "C40"

c40:0d 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

Since there is no problem with the base site register and the address translation register, where is the problem? The value of the RequestID is also set, and there is no problem. What the hell is the problem?

When I think about it, I remember the bit length of the high position of the address translation register is also set by the Setup register, and the Setup register is up to the first not 1 bit as mask from high to low, and the back bit is all 0, With the value in the address translation register, the address is converted to a high position.

Based on the above conversion method, it is found that on the Virtualport side, the BAR3/BAR2 setup register is 0xfffffffe00000000, so the converted address is:

0xfffffffe00000000& 0x0d6d80007800003 =0x0d6d800000000000. The actual address window I use is only 2 m, so I expect the post-conversion address to be High: 0x0d6d80007800000. It is no wonder why the total sense of data can be written to each other node, but at the other side the expected address is always read without expected data. Therefore, the setup register must be changed to: 0xffffffffffe000000. Use the following command to reset the value of the register, as set out above:

[Email protected]bin]# setpci-s 06:00.0 d8.l=0xffffffff

The test is followed by a complete pass. Continue to analyze why the Setup register is having a problem, originally its value was initialized by the NTB EEPROM, and a problem occurred in the middle of the EEPROM firmware I maintained, causing F to become E.

The lesson this time is to be sure to deeply understand the principle of NTB data transmission, only the process of address conversion and related registers in the chest, in order to root planing the end.

This article from "Storage Chef" blog, reproduced please contact the author!

NTB EEPROM setup and data transfer across nodes