Small files automatically merged and output based on Hive Optimization

1. First set the small file standard in the hive-site.xml. lt; propertygt; lt; namegt; hive. merge. smallfiles. avgsizelt; namegt; lt; va

1. First set the small file standard in the hive-site.xml. lt; propertygt; lt; namegt; hive. merge. smallfiles. avgsizelt;/namegt; lt; va

Homepage → Database Technology

Background:

Read News

Small files automatically merged and output based on Hive Optimization

[Date:] Source: Linux community Author: houzhizhen [Font:]

1. First set the small file standard in the hive-site.xml.

Hive. merge. smallfiles. avgsize
536870912
When the average output file size of a job is less than this number, Hive will start an additional map-reduce job to merge the output files into bigger files. this is only done for map-only jobs if hive. merge. mapfiles is true, and for map-reduce jobs if hive. merge. mapredfiles is true.

2. Only map mapreduce output and merge small files.

Hive. merge. mapfiles
True
Merge small files at the end of a map-only job

3. Output mapreduce with reduce and merge small files.

Hive. merge. mapredfiles
True
Merge small files at the end of a map-reduce job

Hive programming guide PDF (Chinese Version)

Hadoop cluster-based Hive Installation

Differences between Hive internal tables and external tables

Hadoop + Hive + Map + reduce cluster installation and deployment

Install in Hive local standalone Mode

WordCount word statistics for Hive Learning

Hive operating architecture and configuration and deployment

Hive details: click here
Hive: click here

This article permanently updates the link address:

0

Oracle 11gR2 uses RMAN Duplicate to copy a database

Asm disk Group loading ORA-15183 error example

Related information: Hive optimized Hive to merge small files

Hive optimization tips-How to Write HQL (07:16:54)

Hive optimized tips (10:23:00)

Image Information

View All comments in this article (0)

Comments

Latest Information

Popular this week

Introduction to Linux community-advertisement service-website map-help Info-contact us
The articles published on this site (LinuxIDC) do not mean that they agree with the statement or description. They only provide more information and do not constitute any suggestions.


Copyright©2006-2015 Linux community All rights reserved Shanghai ICP filing no. 15008072-1