Performance testing indicators

Performance testing indicators

Original article: Http://www-900.ibm.com/cn/products/servers/pseries/tech/tpcc.shtml TPC-C As a non-profit organization, the transaction processing Performance Board (TPC) is responsible for defining transaction processing and database performance benchmarks such as TPC-C, TPC-H and TPC-W benchmarking, provide objective performance data based on these benchmark test items. The TPC benchmark test adopts an extremely strict operating environment and must be conducted under the supervision of an independent audit institution. The Committee includes most major database product vendors and server hardware systems vendors. Related enterprises participate in TPC Benchmark Testing in order to obtain objective performance verification in the prescribed operating environment, and develop more robust and scalable software products and hardware devices through the technologies used during the application testing. TPC-C is an industry standard benchmark to measure the performance and scalability of online transaction processing (OLTP) systems. This benchmark project will test a wide range of database functions, including queries, updates, and queuing-type small batch transactions. Many IT professionals see TPC-C as an effective indicator of the performance of an "authentic" OLTP system. The TPC-C benchmark is designed for a simulated order entry and sales environment to measure the per-minute business transaction (tpmc) throughput. It is particularly worth mentioning that it will specifically measure the system to execute four other transaction types at the same time (such as payment, order status update, delivery and securities-level changes) the number of new order transactions generated per minute. The independent audit body will be responsible for notarization of the benchmark test results, and TPC will produce a comprehensive and thorough test report. This test report can be obtained from the TPC web site (http://www.tpc.org. Tpmc defines the throughput of the TPC-C, measured by the average number of transactions processed per minute during a valid TPC-C configuration, at least 12 minutes of operation. 1. Overview of TPC-C specifications TPC-C is dedicated to the online transaction processing system (OLTP system), we generally call this kind of system business processing system. The TPC-C Test Specification simulates a complex and representative OLTP application environment: assuming there is a large commodity wholesaler, it has several commodity libraries distributed in different regions; each Warehouse is responsible for supplying 10 sales points; each sales point provides services to 3000 customers; each customer has 10 products on average per order; about 1% of the products in all orders do not have inventory in the warehouse directly to which they belong, and must be supplied by warehouses in other regions. The system processes the following transactions: New-order: the customer enters a new order transaction; Payment: updates the customer's account balance to reflect the payment status; Delivery: Delivery (batch transaction simulation ); Order-status: query the status of the customer's recent transactions; Stock-level: queries the inventory status of the warehouse to facilitate timely replenishment. For the first four types of transactions, the response time must be less than 5 seconds; for inventory status query transactions, the response time must be less than 20 seconds. Logical Structure diagram: Flowchart: 2. Evaluation Indicators After two years of development, the TPC-C testing specification was released in July 1992. Almost all vendors that provide software and hardware platforms in the OLTP market have released the corresponding TPC-C test results. With the continuous development of computer technology, these test results are also refreshing. The TPC-C test results mainly have two indicators: ● Throughput (tpmc) According to the definition of TPC, the traffic indicator describes how many new-order transactions can be processed per minute while the system executes the four types of transactions: payment, order-status, delivery, and stock-level. The response time for all transactions must meet the requirements of the TPC-C test specification. The larger the traffic indicator value, the better! ● Price/performance ratio (price/tpmc) That is, the ratio of the system price (quoted in the United States) to the traffic indicator. The less cost-effective the better! 3. Result Publishing The TPC-C test results for each vendor are published in two forms per the TPC organization: the executive summary and the full disclosure report ). The main test indicators, test environment, and complete system configuration and quotation are described in the test result summary. In addition to the above content, the detailed test report, the configuration and testing process of the entire test environment are also described in detail. P690 tpmc test value: 76,389,839.00 $/Tpmc: 831.00 US $6,349,223.0 Number of CPUs: 32 Database: IBM DB2 UDB 8.1 Operating System: AIx 5l v5.2 Middleware: Tuxedo 8.0 Tested on: 2003.6.30 P690 TPC-C Test Configuration: 1. Background: 1 x eserver pseries 690 with 32x1.7 GHz power4 + processors with 128 MB L3 cache per mcm (total of four mcms), 512 GB memory 2. Front-end: 30 x eserver pseries 630 model 6e4 each with 4x1.0 GHz power4 CPUs with 32 MB L3 cache, 16 GB memory Specweb: Specweb96: Benchmark Test in specweb96ProgramThe maximum number of HTTP operations per second, with no significant degradation in response time. Specweb99: number of accesses. The number of concurrent accesses supported by the network server with predefined workload. The specweb99 detection device simulates a customer's slow Internet connection and sends an HTTP workload request to the network server. Specweb99 test the Web server running status Specweb99 is a web server benchmark developed by the standard performance evaluation organization (SPEC. It measures the maximum number of concurrent connections to Web servers that meet the specific throughput and customer request response rate requirements. The total baud rate of concurrent connections is within the range of 320 Kbps to kbps. Specweb99 runs on a machine called the master client, which contains a configuration file that allows users to load specific load requests. The master client must also handle the transmission coordination problem between the client and the server or the system (SUT) in the test. The client generates an independent HTTP request stream through many sub-processes/threads and simulates enough load to send to SUT. Figure 2 shows the hierarchy of the Client/Server. Figure: Typical specweb99 lab environment In this test, the client sends request data to the server in the test. The test specification requires that the connection between the client and the server cannot use a TCP protocol with a segment size greater than 1460 bits. Therefore, each client reads a response of 1460 bits or fewer data blocks. Two types of load are used in the test: Static Load. Static Load has four types of files. The minimum file size increases by kb, the second file type increases by 1 kb, and the last two types of files increase by 10 KB and kb. Each directory contains a total of 36 files for each type of 9 files. The file types of the target request are widely used in various types. 9 files in each category are redistributed. The final target file is mixed as follows: 35% of request files are smaller than 1 KB 50% of request files are smaller than 10 KB 14% of request files are smaller than 100 kb, but greater than or equal to 10 KB 1% of request files are smaller than 1000 kb, but greater than or equal to 100 KB Dynamic Load. Dynamic Load is based on advertising and user registration. There are four types of request content used in specweb99: Standard Dynamic fetch operation, dynamic random fetch operation, dynamic send operation, and dynamic fetch operation of customer graphics interface. The standard dynamic fetch operation and the dynamic fetch operation of the customer's graphic interface demonstrate the simple advertisement rotation feature of the Web server. The dynamic fetch operation with AD rotation tracks users and users, so the ads can be customized in different ways. Finally, a user is registered on the corresponding website for dynamic publishing. P690 specweb99 test value: 21,000 Web server: Zeus 4.0 Operating System: AIx 5l v5.1 (64-bit) Number of CPUs: 16 Tested on: 2001-10-1 Test Configuration: 16x1.3 GHz power-4 Processors w/1440kb unified on chip L2 cache, 192 GB memory, 32x32 IBM Gigabit Ethernet-sx pci controllers, 32 x Gigabit Ethernet network (1 Gigabit/sec), 96 X clients (4x375 MHz POWER3-II, RS/6000 44p-270), requested connections = 21000, Max fileset size = 67319.6 MB P650 specweb99 test value: 12,400 Web server: Zeus 4.1r3 Operating System: AIx 5l v5.2 (64-bit) Number of CPUs: 8 Tested on: 2002-10-1 Test Configuration: 8x1.45 GHz power4 + processors w/1.5 MB (I + d) Unified on chip L2 cache, 32 MB unified off chip/SCM L3 cache, 64 GB memory, 8 x Gigabit Ethernet-SX PCI-X controllers, 8 x Gigabit Ethernet network (1 Gigabit/sec), 48 X clients (6x668 MHz RS64-IV, pseries 620 model 6f1 ), requested connections = 12400, Max fileset size = 39801.28 MB P630 specweb99 test value: 6,895 Web server: Zeus 4.2r1 Operating System: AIx 5l v5.2 (64-bit) Number of CPUs: 4 Tested on: 2003-2-1 Test Configuration: 4x1450 MHz power4 + processors w/1536kb (I + d) Unified on chip L2 cache, 8 Mb unified (off chip)/SCM L3 cache, 32 GB memory, 4 x Gigabit Ethernet-SX PCI-X controllers, 4 x Gigabit Ethernet networks (1 Gigabit/sec), 24 x clients (4x375 MHz POWER3-II, pseries 640 model b80 ), requested connections = 6900, Max fileset size = 22199.12 MB Notesbench: Notesbench is a driver used to test various Lotus Notes. The purpose is to execute commands in the Custom workload tutorial to simulate client operations. Notesbench tests "test mail only" and "test mail and Database ". All published IBM results are "test mail workload only ". P680 notesbench test value: 150,197 User: 108,000 Average Response Time: 0.584 seconds Domino server version: 5.06a Operating System: AIx 4.3.3 Number of CPUs: 4 Tested on: 2001.11.20 Test Configuration: IBM eserver pseries 680 (24 * rs64 IV/600 MHz; 96 gb ram, 30 partitions)