Hive Insert Overwrite directory error:
2017-06-07t16:56:11,766 ERROR [main]: Exec. Task (:())-Failed with exception unable to move source hdfs://testhadoop/tmp/t1/.hive-staging_hive_2017-06-07_16-56-05 _916_7465107102137731594-1/-ext-10000 to Destination/tmp/t1 org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move source hdfs://testhadoop/tmp/t1/.hive-staging_hive_2017-06-07_16-56-05_916_7465107102137731594-1/- ext-10000 to Destination/tmp/t1 at Org.apache.hadoop.hive.ql.exec.MoveTask.moveFile (movetask.java:103) at Org.apache.hadoop.hive.ql.exec.MoveTask.execute (movetask.java:254) at Org.apache.hadoop.hive.ql.exec.Task.execut ETask (task.java:197) at Org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential (taskrunner.java:100) at or G.apache.hadoop.hive.ql.driver.launchtask (driver.java:1858) at Org.apache.hadoop.hive.ql.Driver.execute ( driver.java:1562) at Org.apache.hadoop.hive.ql.Driver.runInternal (driver.java:1313) at Org.apache.hadoop.h Ive.ql.Driver.Run (driver.java:1084) at Org.apache.hadoop.hive.ql.Driver.run (driver.java:1072) at org.apache.hadoop.hive. Cli. Clidriver.processlocalcmd (clidriver.java:232) at ORG.APACHE.HADOOP.HIVE.CLI.CLIDRIVER.PROCESSCMD (Clidriver.java : 183) at Org.apache.hadoop.hive.cli.CliDriver.processLine (clidriver.java:399) at org.apache.hadoop.hive.cl
I.clidriver.executedriver (clidriver.java:776) at Org.apache.hadoop.hive.cli.CliDriver.run (clidriver.java:714) At Org.apache.hadoop.hive.cli.CliDriver.main (clidriver.java:641) at Sun.reflect.NativeMethodAccessorImpl.invok E0 (Native Method) at Sun.reflect.NativeMethodAccessorImpl.invoke (nativemethodaccessorimpl.java:62) at Sun. Reflect. Delegatingmethodaccessorimpl.invoke (delegatingmethodaccessorimpl.java:43) at Java.lang.reflect.Method.invoke ( method.java:497) at Org.apache.hadoop.util.RunJar.run (runjar.java:221) at org.apache.hadoop.util.RunJar.ma
In (runjar.java:136)
caused by:org.apache.hadoop.hive.ql.metadata.HiveException:java.io.IOException:rename for SRC path:hdfs:// Testhadoop/tmp/t1/.hive-staging_hive_2017-06-07_16-56-05_916_7465107102137731594-1/-ext-10000/000000_0 to Dest Path:/tmp/t1/000000_0 returned false at Org.apache.hadoop.hive.ql.metadata.Hive.moveFile (hive.java:2942) A T Org.apache.hadoop.hive.ql.exec.MoveTask.moveFileInDfs (movetask.java:118) at Org.apache.hadoop.hive.ql.exec.Move Task.movefile (movetask.java:96) ... caused by:java.io.IOException:rename for SRC path:hdfs://testhadoop/tmp/t1/.hive-staging_hive_2017-06-07_ 16-56-05_916_7465107102137731594-1/-ext-10000/000000_0 to Dest Path:/tmp/t1/000000_0 returned false at Org.apache. Hadoop.hive.ql.metadata.hive$3.call (hive.java:2922) at Org.apache.hadoop.hive.ql.metadata.hive$3.call (Hive.java : 2911) at Java.util.concurrent.FutureTask.run (futuretask.java:266) at JAVA.UTIL.CONCURRENT.THREADPOOLEXECU Tor.runworkeR (threadpoolexecutor.java:1142) at Java.util.concurrent.threadpoolexecutor$worker.run (ThreadPoolExecutor.java
: 617) at Java.lang.Thread.run (thread.java:745)
Reason:
Hive2.1.0bug, upgrade to hive2.1.1
The reason for the error is insert overwrite directory, if the original directory already has data, MV will error. See HIVE-13997, Address https://issues.apache.org/jira/browse/HIVE-13997
Bug Fix idea: Delete the target directory data before the MV result data to the target directory. This policy differs from the hive1.2.1 version.
Hit patch,https://issues.apache.org/jira/browse/hive-13997 download hive-13997.2.patch, then put path to HIVE-2.1.0-SRC, execute PATCH-P1 < Hive-13997.2.patch; Then recompile the QL module to get Hive-exec-2.1.0.jar