Hive table creation does not use the lzo storage format, but the data is in the lzo format.

Today, when I sent an email to the microblogs data platform, they failed to run hql, but the logs on the gateway did not show the cause. I checked it for help, finally, the cause of the problem is found. The following is the analysis process:

1. Failed hql:

INSERT OVERWRITE TABLE brand_ad_user_with_interact_score_3 select a.uid, a.brand, a.friend, CASE b.weight WHEN NULL THEN '0.000000' ELSE b.weight ENDfrom brand_ad_2hop_3 aLEFT OUTER join ods_bas_user_interact_score_to_thin_3 bon (a.uid = b.fid and a.friend = b.tid);

The hql is very simple, that is, two tables are associated and then output to another table. It is a common join operation without group by operations, so there is no data skew problem of map.

2. View job logs

Check the status and log information of the job on the 50030 page. The job is killed, and the map task is killed before it is completed, the running time of the kill map task exceeds 10 hours, as shown in:

According to the analysis in 1, the job does not store data skew in hql. Why does a single map run for more than 10 hours, view the counter information of the kill map task, as follows:

It turns out that a single map task reads 10 Gb of data from HDFS. No, it's hard to say that the data files to be processed are not sharded. A single map task processes a single large file, with such speculation, I checked the files under the two table directories in hql. The following are all files in the lzo format, but no index is created.

In the brand_ad_2hop_3 table, a single file has reached 10 Gb +, which is probably the cause: files in the lzo format are not indexed and data files cannot be sharded. As a result, during running, A single file can only be processed by one map task. If a single file is large, the map task will be processed for a long time.

After checking the table creation Statement of brand_ad_2hop_3, the storage format is text.

Since the cause of the problem is found, the following is the solution:

(1) Create an index for the lzo File

(2) Use the lzo storage format when creating a table

Hive table creation does not use the lzo storage format, but the data is in the lzo format.