How to implement benchmark standard test (one)---problem and tpc-c standard

Standard | questions
How to implement benchmark standard test

The raise of---problem and tpc-c standard

Last Updated:monday, 2004-10-18 16:36 eygle




1. The question was raised

No matter what kind of system you implement, you may have considered a series of questions like this:

What kind of equipment should I buy?
How is my system performing?
How many users can my system withstand?
How much concurrency can my system endure?
When will the performance problem occur?
When Will I upgrade?

Obviously, the answer to these questions is not so simple, more people are based on experience to give an empirical value to evaluate the system.
But in the big system design process, experience shows is not enough, you have to come up with enough evidence to illustrate your architecture, your system must theoretically be able to meet the needs of users.

Here we have seen that one of the most important elements has been introduced, and that is: User needs.
This is in line with economic law, there is demand for development, demand to promote everything. So there is the standard test (benchmark) We are going to discuss today.

In recent time, has been involved in a large system of project design, planning and testing, now put some of the experience to write for your reference.

2.TPC Organization

For the benchmark test, the most famous is TPC (Transaction processing performance Council, transaction Processing Performance Committee) organization. You can
The official site www.tpc.org for more detailed information. TPC was founded in 1988 by a group of original vendors that provide hardware and software systems or related solutions in the computer field.
, members from the beginning of the 8 companies developed to the current 50, IBM, NCR, HP, Oracle, Microsoft and other international well-known companies are its members. Headquartered in the United States,
and open to the world, but so far, TPC organization of the vast majority of members are the United States, Japan, Western Europe, large computer software and hardware companies.

As a non-profit organization, the Transaction Processing Performance Committee (TPC) is responsible for defining transaction processing and database performance benchmarks such as TPC-C, Tpc-h, and Tpc-w Benchmarking
Test and release the objective energy data based on these benchmark projects. The TPC benchmark is highly operational and must be supervised by an independent audit body. Committee
Members include most of the major database product vendors and server hardware system vendors.

TPC does not give the code of the benchmark program, but only the standard specification (Standard specification) of the benchmark program.
Note that here we propose one of the most important elements in the benchmark test: the standard specification.
You might hear a lot of people say to you, tpc-c values have no real meaning, and no test can simulate the load in a real production environment.
This general statement naturally has its own reason, but it is one-sided, or that this kind of argument actually misinterpreted the TPC-C and other standard test intention.
We say that the most important thing in the benchmark test is the standard specification, that is, in fact, tools and other factors are not important, as long as everyone is using the same standard specification, the same tool for system
Test, the results of the test will be more meaningful.

From this point of view, the benchmark test has actually become a stage where manufacturers show their technical prowess. Any manufacturer or other tester may, according to the standards promulgated by TPC,
Optimally construct their own systems, using the best platform and the most efficient applications. So you might listen to Oracle attack Db2,oracle said: IBM's TPC Test uses
are Oracle databases, not their own DB2. In fact, this is also one-sided, tpc-c test OLTP performance, and we know that DB2 in the Data warehouse is the real
Powerful.


In order to ensure the objectivity of the test results, the tester (usually the manufacturer) must submit a complete set of reports to TPC (Fulldisclosure report), including the detailed configuration of the system under test,
Class price and total price including five years maintenance cost. The report must be verified by the TPC authorized Auditor (TPC itself does not audit). There are now only a few auditors in the world, all in the United States.

TPC has launched four sets of benchmark procedures known as TPC-A, Tpc-b, Tpc-c and tpc-d. Where A and B are obsolete and are no longer used. Tpc-c is an online transaction processing (OLTP)
Benchmark program, TPC-D is the benchmark program for decision Support (Decisionsupport). TPC later tried to launch the Tpc-e,tpc-s, and eventually was abandoned.

3.tpc-c

Our focus is on the tpc-c test.

TPC-C is an industry standard benchmark project designed to measure the performance and scalability of online transaction processing (OLTP) systems. This benchmark project will include queries, updates, and queue-type
A wide range of database functions, including small volume transactions, are tested. Many IT professionals see TPC-C as a valid indicator of the performance of the "real" OLTP system.

The TPC-C benchmark tests business Transaction (TPMC) throughput per minute for a simulated order entry and sales environment. It is particularly noteworthy that it will specialize in measuring the system at the same time executing its
The number of new order transactions generated per minute for each of the four transaction types (such as payment, order status update, delivery, and security-level changes). The independent audit body will be responsible for the benchmark test results
Notarization, at the same time, TPC will be out according to a comprehensive and thorough test report. This test report can be obtained from the TPC Web site (http://www.tpc.org).

TPMC definition: tpc-c throughput, measured by the average number of transactions processed per minute during a valid tpc-c configuration, at least 12 minutes to run

3. 1TPC-C Specification Overview

Tpc-c is specifically targeted at online transaction processing systems (OLTP systems), which are generally referred to as business processing systems.

The TPC-C Test specification simulates a more complex and representative OLTP application environment:
Suppose there is a large commodity wholesaler, which has a number of commodity banks distributed in different regions, each of which is responsible for supplying 10 points of sale, and each point of sale serves 3,000 customers;
Each customer has an average of 10 products per order, and about 1% of all orders have no stock in their immediate warehouse and need to be supplied by warehouses in other regions.

The system needs to deal with the following types of transactions:

New-order: The customer enters a new order transaction;

Payment: Update customer account balance to reflect its payment status;

Delivery: Delivery (analog batch transaction);

Order-status: Inquire the status of the customer's recent transaction;

Stock-level: Check the warehouse inventory status in order to be able to replenish goods in time.

For the first four types of transactions, the response time is less than 5 seconds, and for the Inventory Status inquiry transaction, the response time is within 20 seconds.

Logical structure diagram:





Flow chart:



3.2. Evaluation indicators

TPC-C Test specification After two years of development, published in July 1992. Almost all vendors offering hardware and software platforms in the OLTP market have released corresponding TPC-C test results, with the calculation
The continuous development of machine technology, these test results are constantly refreshing.

Tpc-c test results are mainly two indicators:

Flow index (throughput, abbreviation TPMC)

According to TPC, the flow index describes how many of the four transactions that the system performs in the payment, Order-status, Delivery, Stock-level, per minute
New-order Trading. The response time for all transactions must meet the requirements of the TPC-C test specification.

The greater the flow indicator value, the better!

Price/Performance (Price/performance, short PRICE/TPMC)

That is, the ratio of the price of the test system to the flow index.

The less cost-effective the better!

For this section, detailed instructions can be found in the official documents of TPC-C, and you can obtain the official documents of TPC at the following links:

Http://www.tpc.org/tpcc/spec/tpcc_current.pdf

On the IBM website, you can also find part of the explanation:

Http://www-900.ibm.com/cn/products/servers/pseries/tech/tpcc.shtml

The current version of Tpc-c is 5.2.







The author of this article:
Eygle,oracle technology concern, from China's largest Oracle technology forum Itpub.
Www.eygle.com is the author's personal site. You can contact the author by Guoqiang.Gai@gmail.com. Welcome technical discussions and exchange of links.



Original source:

Http://www.eygle.com/special/How.To.implement.Benchmark.Test.01.htm