Application of multithreading technology in Real-time Data Collection and Analysis

Application of multithreading technology in Real-time Data Collection and Analysis

[Date: 2008-7-23]

Source: China Power Grid Author: He Jia, College of Automation Engineering, University of electronic science and technology, Li Wang houjun

[Font:Large Medium Small]

　

Introduction A digital interface device is a dedicated communication device that converts a standard 422 serial bus to a custom serial bus. The digital interface test system outputs 422 serial data and control signals (YCK and YZM) of the custom serial bus to the digital interface device according to the working principle of the digital interface device, it also collects, stores, analyzes, and processes the output serial data (YDATA) to detect the devices under test. The digital interface testing system provides eight test channels. The parameters of the 422 serial bus and the custom serial bus for each channel can be set by the tester through the application software. In order to facilitate data analysis, the data format agreed on in the software is: AA xx 01 23 45 67 89 AB CD EF 01 23... 10. The frame header is 0xAA, the end of the frame is 0x10, and xx is the sending counter value. 1 is added for each sending. The application software uses algorithms to analyze and process the data received by the custom Serial Bus in real time, such as the number of bytes received and the total number of frames received, total number of errors in bytes ...... The test results are displayed dynamically on the test interface. The tester can determine whether the device to be tested works properly based on the real-time test results. Once the test data error rate is found to be too high, you can immediately stop the test after power failure to prevent the tested device from burning out. System Implementation Scheme Figure 1 shows the principle of the digital interface testing system. The test device is connected to the operation control computer through the USB2.0 bus. Each test device provides eight data transmission and receiving units. The data transmission unit is used to output 422 asynchronous serial data, the maximum baud rate is 614.4 kbps. The frame length can be adjusted according to the test requirements. The maximum cycle of data transmission per frame is 5 ms; the data receiving unit is used to generate control signals YZM and YCK of the custom serial bus, and read back data from YDATA. YCK and YZM can reach up to 1.6384MHz and 25.6 kHz respectively. Due to the large amount of data transmitted during the test and the need to perform real-time analysis on the collected data, real-time data collection and analysis are processed in multiple threads. In the application software, the main interface thread is mainly responsible for completing man-machine interface operations. At the same time, the two threads of data collection and data analysis are opened to synchronize and coordinate the work. To ensure the continuity and integrity of the data collection and analysis process, the two threads use the high-speed memory buffer and memory ing file to implement the first and second-level buffering of high-speed data streams. The level-1 buffer zone caches the data uploaded by the USB bus. It is used to split the received USB data packets according to each channel. Level-2 buffer stores the output data separately according to the channel number, it is used by data analysis threads for data analysis and processing. In the whole solution, data packet upload, storage, data decomposition, data analysis, and other operations are completed in the memory, which is fast and accurate. In addition, a new thread synchronization method is used, it not only ensures high-speed data throughput of data collection threads, fast response and real-time analysis of data analysis threads, but also ensures high performance and lowest system overhead of the entire solution. Data Cache processing 2. Data collection To save the data returned by USB, a 512 K high-speed memory buffer is built in the memory. the space of 12 K is divided into 24 equal parts in units of bytes (the size of a USB packet. In multi-thread synchronization, semaphores are commonly used to control the number of threads accessing a shared resource. In combination with the idea of producers and consumers in the operating system, the semaphores can be extended to complete thread synchronization. When the data collection thread is operating, the 12 K high-speed buffer uses the read/write semaphores as the status indication to Control Data Reading and output. The number of write semaphores is initialized to 24 (indicating that there are 24 data ranges to be written), and the number of read semaphores is initialized to 0 (indicating that there are 0 spaces and data needs to be read ), the data collection thread is equivalent to the producer, and the data analysis thread is equivalent to the consumer. 3. Each packet of data is returned by the USB interface. First, the number of written semaphores is determined. If the number is zero, the thread blocks and waits. If the number is not zero, the data is written. The USB data packet is saved to the current producer pointer (pWriteIndex) in the last 512 byte buffer of the address, After completion, the producer pointer is incremented by 512, the write semaphore is subtracted by 1, and the read semaphore is incremented by 1. This is the production process. When the thread switches to the data analysis thread and starts consumption (decomposition and processing of collected data packets), it first determines the number of read semaphores, which is zero (there is no consumable data), and the thread blocks the waiting, if the value is not zero, the pReadIndex is used to control the reading of a packet of data. After a packet of data is read, the consumer pointer is increased by 512, the read semaphore is reduced by 1, and the write semaphore is increased by 1. The above analysis shows that the coordination of the two semaphores in the buffer zone of the entire 12 K not only achieves synchronization of data collection and data analysis threads, the Interlock mechanism ensures that the producer pointer and consumer pointer do not point to the same memory area, which significantly improves the reliability of the entire system. Data Analysis Because the eight channels of the digital interface testing device can work simultaneously, in order to distinguish the data in the data packets uploaded by the USB bus from the data in which the data is mapped and meet the real-time data transmission requirements, each channel sends the test data to the internal buffer of the USB interface in 64 bytes. The first byte is used to identify the data of the next 63 bytes of the channel ...... . When the internal buffer of the USB interface is filled with 512 bytes, It is uploaded to the high-speed buffer in the computer memory through the USB bus. Therefore, in data analysis, We need to extract the test data returned by each channel from the USB packet, and then compare and analyze the data with the standard format. It can be seen that during a long test, the data analysis thread processes a large amount of data and the task is heavy. To ensure the real-time performance and data analysis accuracy of the entire system, the data received by the test is stored in different memory ing files according to the channel number. After the data analysis thread successfully reads a packet of data under the control of the semaphore and consumer pointer, it extracts data from each channel in the package based on the channel ID and writes the data to the corresponding memory ing file, call the data processing function to analyze the error code of each channel. The error code analysis result can be displayed by calling the main interface thread. Data decomposition process 4 is shown in. To ensure test efficiency and data processing accuracy, pWriteCounter is used to control the movement of data pointers written to memory ing files and determine the number of received data bytes, as a reference for reading the memory ing file pointer during data processing and a condition for starting data analysis. During data processing, data is not analyzed immediately after data is written into the memory ing file, but the number of received data bytes is determined based on pWriteCounter, the error rate analysis on the frame data is not started until the received data is longer than the standard frame length. This not only avoids the analysis of one frame data multiple times, but also ensures the correctness of the data analysis, it also reduces the time for data analysis threads to exclusively occupy the CPU. According to the final running results, this method effectively solves the problem of data loss during real-time data collection. After the data is decomposed and written into the memory ing file of the corresponding channel, you also need to compare and analyze the test data and standard data received by each channel in real time, and display the data in the form of a bit error rate, the tester can monitor whether the digital interface device works properly based on the real-time error rate. The data analysis process is shown in step 5. In actual use, the digital interface device may have many faults. The data collected and received is large and the data errors are different. Therefore, it is not possible to misjudge or miss any situations, through repeated tests, the following methods of data processing in fixed formats are obtained: ① Scan the frame header 0xAA one byte in the memory ing file of each channel. Once the frame header 0xAA is detected, enter step ②. ② First, determine whether the end position of the frame data is 0x10, and whether the first data at the end of the frame is the same as the corresponding standard data (the number before the end of the standard data frame, if both conditions are met, the data of this frame is a standard frame (the data length of this frame is equal to the length of the sent standard frame). If not, perform step 4. ③ After judging this frame as a standard frame, compare and judge from the second position of the frame data and the standard data to the end of the frame one by one (skip 0th frame header data and 1st counter data ), if they are not equal, an error is recorded. Each time an error byte is found, the number of error bytes is increased by 1. Go to Step 6. ④ Check whether 0xAA or 0xAA exists in the range from the frame header to the frame length data. First, use step ② To determine whether 0xAA is the frame header of the next frame data, if the next frame data frame header is used, the first byte of the frame header is recorded at the end of the frame. This indicates that the data of the frame data is lost. Otherwise, the data is incorrect and non-standard frame processing is called. ⑤ From the second position of the frame data and the standard data to the determined data within the frame length range for one-to-one comparison and judgment (skipping 0th frame header data and 1st counter data ), if they are not equal, an error is recorded, and a byte is regarded as an error. ⑥ Check whether the counter data in the frame is continuous with the counter data in the front and back frames. If the counter data is continuous, no frame error occurs; otherwise, frames are dropped, determine the Frame Length Based on the front and back counter data and convert it to the number of corresponding error bytes. Connect two tested devices to verify the performance of the entire system, even if each channel chooses to work at a maximum baud rate of 614.4 kbps and a maximum cycle of 5 ms, it can still ensure high real-time data collection speed, stable data quality, and low error rate. Conclusion Real-time testing and Result Display 6 are shown. This technology has been successfully applied to the detection and Maintenance System of a digital interface device and achieved good results. A large number of tests have proved that this multi-thread, memory ing file, and two-level buffer method works well in high-speed real-time data collection and analysis. The read/write semaphores established based on the idea of producers and consumers effectively synchronize the collection and analysis threads. The size of the memory ing file is 100 MB before the test starts, during a longer period of test, you can dynamically apply for new memory space, which not only guarantees the real-time requirements of the system, but also effectively saves system memory resources.