Use parquet (CDH4.3) in hive)

After hadoop version cdh4.3 uses impala to create a parquet table, an error occurs during query. [Impala: 21000] SELECT * FROMfoo; Query: SELECT * FROMfooERROR: AnalysisException: FailedTOLOADmetadataFORTABLE: DEFAULT. fooCAUSEDBY: TableLoadingException: FailedTOLOADmet

After hadoop version cdh4.3 uses impala to create a parquet table, an error occurs during query. [Impala: 21000] SELECT * FROM foo; Query: SELECT * FROM fooERROR: AnalysisException: Failed to load metadata for table: DEFAULT. fooCAUSED BY: TableLoadingException: Failed to load met

Hadoop version cdh4.3

An error occurs when you use impala to create a parquet table.

[impala:21000] > SELECT * FROM foo;Query: SELECT * FROM fooERROR: AnalysisException: Failed TO LOAD metadata FOR TABLE: DEFAULT.fooCAUSED BY: TableLoadingException: Failed TO LOAD metadata FOR TABLE: fooCAUSED BY: MetaException: org.apache.hadoop.hive.serde2.SerDeException SerDe parquet.hive.serde.ParquetHiveSerDe does NOT exist

The reason is that hive does not have these lib. Download them and put them in the/opt/cloudera/parcels/CDH/lib/hive/lib directory (I deployed it using cloudera manager). Create a script to download it.

#!/bin/sh#parquet-pig parquet-scrooge parquet-test-hadoop2 parquet-thrift parquet-avro parquet-cascading ?for f in parquet-column parquet-common parquet-encoding parquet-generator parquet-hadoop parquet-hive docurl -O http://repo1.maven.org/maven2/com/twitter/${f}/1.2.4/${f}-1.2.4.jar#curl -O http://oss.sonatype.org/service/local/repositories/releases/content/com/twitter/${f}/1.2.4/${f}-1.2.4.jardonecurl -O http://repo1.maven.org/maven2/com/twitter/parquet-format/1.0.0/parquet-format-1.0.0.jar

And copy them in.

cp parquet-*  /opt/cloudera/parcels/CDH/lib/hive/lib

You may need to restart metastore and refresh metastore in impala.

INVALIDATE METADATA;

Modify the parquet table in impala

create table test2 (name STRING) STORED AS PARQUETFILE;

Insert data

insert into test2 select * from test;

Reference

Https://issues.cloudera.org/browse/IMPALA-574

Original article address: Use parquet (CDH4.3) in hive. Thank you for sharing it with the original author.