使用sqoop1.4.4從oracle匯入資料到hive中錯誤記錄及解決方案

標籤：des   style   blog   http   color   java   使用   os   

  在使用命令導資料過程中，出現如下錯誤

sqoop import --hive-import --connect jdbc:oracle:thin:@192.168.29.16:1521/testdb --username NAME --passord PASS --verbose -m 1 --table T_USERINFO

錯誤1：File does not exist: hdfs://opt/sqoop-1.4.4/lib/commons-io-1.4.jar

FileNotFoundException: File does not exist: hdfs://opt/sqoop-1.4.4/lib/commons-io-1.4.jar
      at org.apache ... ...
      at org.apache ... ...

原因分析: 

感謝 Daniel Koverman‘s answer http://stackoverflow.com/questions/19375784/sqoop-jar-files-not-found

It is common for Hadoop services to look for jars in HDFS because all nodes in the cluster can access files in HDFS. This is important if the MapReduce job being kicked off by the Hadoop service, in this case Sqoop, has a dependence on those jars. Remember, the Mappers are running on a DataNode, not the NameNode even though you are (probably) running the Sqoop command from the NameNode. Putting the jars on HDFS is not the only possible solution to this problem, but it is a sensible one.Now we can deal with the actual error. At least one, but probably all, of your Mappers are unable to find a jar they need. That means that either the jar does not exist or the user trying to access them does not have the required permissions. First check if the file exists by running hadoop fs -ls home/SqoopUser/sqoop-1.4.3-cdh4.4.0/sqoop-1.4.3-cdh4.4.0.jar by a user with superuser privileges on the cluster. If it does not exist, put it there with hadoop fs -put {jarLocationOn/NameNode/fileSystem/sqoop-1.4.3-cdh4.4.0.jar} /home/SqoopUser/sqoop-1.4.3-cdh4.4.0/sqoop-1.4.3-cdh4.4.0.jar. 

解決方案：

 將提示中涉及的jar檔案put到hdfs檔案系統中的相同位置，如果檔案系統中沒有對應的目錄，則需要建立相應目錄，在我的錯誤提示中，由於hdfs://master:8020/中缺少了 /opt/sqoop-1.4.4/lib/檔案夾中的各種jar,所以我的做法是把此處整個/opt/sqoop-1.4.4/lib檔案夾put到hdfs://master:8020/中

<!--查看以下檔案系統中的檔案目錄，這是遞迴查詢，如果檔案很多 建議不要家-R參數，而是逐層查看-->hadoop fs -ls -R /<!--建立相同的目錄結構-->hadoop fs -mkdir /opthadoop fs -mkdir /opt/sqoop-1.4.4<!--將本地的/opt/sqoop-1.4.4/lib 拷貝到hdfs中的/opt/sqoop-1.4.4目錄中-->hadoop fs -put /opt/sqoop-1.4.4/lib /opt/sqoop-1.4.4/<!--查看一下結果,確認拷貝成功-->hadoop fs -ls -R /opt/sqoop-1.4.4

錯誤2 ：java.lang.ClassNotFoundException: Class U_BASICINFO not found

對於要匯入到hive中的表，錯誤提示說找不到對應的.class和.jar檔案

java.lang.Exception: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class U_BASICINFO not found    at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class U_BASICINFO not found    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)    at org.apache.sqoop.mapreduce.db.DBConfiguration.getInputClass(DBConfiguration.java:394)    at .....

原因分析：暫時不知道

解決方案：

感謝 user236575’s answer： http://stackoverflow.com/questions/21599785/sqoop-not-able-to-import-table/21626010#21626010

預設sqoop在執行匯入table過程中會產生對應的table的java檔案和編譯產生的.class和.jar檔案，.java檔案儲存在sqoop/bin目錄下，而class 和 jar檔案則儲存在/tmp/sqoop-hduser/compile/ 下相應的檔案夾中。

我的解決方式是找到要匯入表的class和jar檔案，然後將他們拷貝到sqoop/bin目錄下面和hdfs檔案系統中的/user/USERNAM/ 目錄下面(後期測試後，只要將.class和.jar拷貝到sqoop/bin目錄下就可以成功import)。

<!--拷貝到sqoop/bin目錄下-->cp /tmp/sqoop-root/compile/某個臨時檔案夾包含需要的class和jar檔案/*  /opt/sqoop-1.4.4/bin/<!--put到hdfs中的/user/USERNAME/檔案夾下-->hadoop fs -put /tmp/sqoop-root/compile/某個臨時檔案夾包含需要的class和jar檔案/* /user/root/

錯誤3 org.apache.hadoop.mapred.file already exists exception:output directory hdfs://user/root/... ...

解決方案：

在執行過一次匯入資料表命令後，當再次執行時，可能會出現這種錯誤，這是只要進入到hdfs中將對應的檔案或者檔案夾刪除即可。

hadoop fs -rm /user/USERNAME/*