gpdb Parallel Load Test

Test file information

10G Dec 12 14:10 A111G Dec 12 14:32 A210G Dec 12 14:10 B111G Dec 12 14:35 B2

Gpfdist Scenario a single server

drop table if exists host_1;drop EXTERNAL TABLE if exists exttable_ext_1_host;drop table if exists host_1_err;create table host_1 (like sourcetable) distributed randomly;CREATE EXTERNAL TABLE exttable_ext_1_host (like sourcetable) LOCATION (‘gpfdist://10.2.22.81:9999/A*‘) FORMAT ‘text‘ (delimiter as ‘,‘ null as ‘‘ escape ‘OFF‘) ENCODING ‘UTF8‘ LOG ERRORS INTO host_1_err SEGMENT REJECT LIMIT 100 PERCENT;insert into host_1 select * from exttable_ext_1_host;

Scenario 22 Server

drop table if exists host_2;drop EXTERNAL TABLE if exists exttable_ext_2_host;drop table if exists host_2_err;create table host_2 (like sourcetable) distributed randomly;CREATE EXTERNAL TABLE exttable_ext_2_host (like sourcetable) LOCATION (‘gpfdist://10.2.22.81:9999/B1‘,‘gpfdist://10.2.22.82:9999/B2‘) FORMAT ‘text‘ (delimiter as ‘,‘ null as ‘‘ escape ‘OFF‘) ENCODING ‘UTF8‘ LOG ERRORS INTO host_2_err SEGMENT REJECT LIMIT 100 PERCENT;insert into host_2 select * from exttable_ext_2_host;

Gpload Plan III

Gpload Single Table

---VERSION: 1.0.0.1DATABASE: gpdbUSER: gpadminHOST: 10.4.2.4PORT: 5432GPLOAD:  INPUT:    - SOURCE:        LOCAL_HOSTNAME:          - 10.2.22.81        PORT: 9999        FILE:          - /data/ptest/A*    - FORMAT: text    - DELIMITER: ‘,‘    - ESCAPE: ‘OFF‘    - NULL_AS: ‘‘    - ENCODING: UTF8    - ERROR_LIMIT: 10000    - ERROR_TABLE: host_1_err  OUTPUT:    - TABLE: host_1    - MODE: insert

Programme IV

Gpload two units

Test results

In order to prevent the cache interference test, after many tests the results are as follows, we can find that scenario two is obviously better than scheme one

. File Info S File | File Size | Storage Size | Number of inserted Records | Number of exception Records------| --------------------------------------------Scheme One | 21GB | 25GB |49826141 |2199 Solution II | 21GB | 25GB | 52108083| 1867

. Load time-consuming

It is unclear why the difference between the 1th read (test sequence one, scheme one/two) and the following is so large that it may be related to GPFS

Scenario Four of the B-machine file read failed, and the entire test process three/four feeling is not very stable (hang), in view of the scenario three-contrast program does not have much advantage, and by observing program Four can be found that a machine loading time has reached 22.24s, the comparison scheme II may not have an advantage, so did not complete the test

Test Order	programme I	Scenario Two	Programme III	Scenario Four
Test Order One	440403.263 ms	204201.096 ms	36.41 seconds	22.24 seconds+b?
Test Order Two	35854.612 ms	26303.240 ms	Programme III	Programme IV
Test Order Three	42007.990 ms	25593.730 ms	Programme III	Programme IV
Test Order Four	43795.502 ms	25706.479 ms	Programme III	Programme IV
Test Order Five	36576.681 ms	26405.977 ms	Programme III	Programme IV

-eof-

gpdb Parallel Load Test