解決Hadoop0.21.0 HDFS/MapReduce編譯錯誤(ivy-resolve-common does not find hadoop-commo)

轉自：http://junsky.iteye.com/blog/1070647

解決Hadoop0.21.0 HDFS/MapReduce編譯錯誤(ivy-resolve-common does not find hadoop-commo)

     最近老大安排，要跟蹤HDFS ConcatenateFile的前提條件；於是嘗試用ant/ivy編譯了一下hdfs，提示ivy找不到hadoop-common組件；於是又編譯了一下mapreduce，提示ivy找不到hadoop-comm/hadoop-hdfs；編譯hadoop-common，提示編譯成功；在編譯mapreduce錯誤如下：

 

ivy-resolve-common:

[ivy:resolve] 

[ivy:resolve] :: problems summary ::

[ivy:resolve] :::: WARNINGS

[ivy:resolve] 
module not found: org.apache.hadoop#hadoop-common;0.21.0

[ivy:resolve] 
==== apache-snapshot: tried

[ivy:resolve] 
  https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-common/0.21.0/hadoop-common-0.21.0.pom

[ivy:resolve] 
  -- artifact org.apache.hadoop#hadoop-common;0.21.0!hadoop-common.jar:

[ivy:resolve] 
  https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-common/0.21.0/hadoop-common-0.21.0.jar

[ivy:resolve] 
==== maven2: tried

[ivy:resolve] 
  http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-common/0.21.0/hadoop-common-0.21.0.pom

[ivy:resolve] 
  -- artifact org.apache.hadoop#hadoop-common;0.21.0!hadoop-common.jar:

[ivy:resolve] 
  http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-common/0.21.0/hadoop-common-0.21.0.jar

[ivy:resolve] 
module not found: org.apache.hadoop#hadoop-common-test;0.21.0

[ivy:resolve] 
==== apache-snapshot: tried

[ivy:resolve] 
  https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-common-test/0.21.0/hadoop-common-test-0.21.0.pom

[ivy:resolve] 
  -- artifact org.apache.hadoop#hadoop-common-test;0.21.0!hadoop-common-test.jar:

[ivy:resolve] 
  https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-common-test/0.21.0/hadoop-common-test-0.21.0.jar

[ivy:resolve] 
==== maven2: tried

[ivy:resolve] 
  http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-common-test/0.21.0/hadoop-common-test-0.21.0.pom

[ivy:resolve] 
  -- artifact org.apache.hadoop#hadoop-common-test;0.21.0!hadoop-common-test.jar:

[ivy:resolve] 
  http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-common-test/0.21.0/hadoop-common-test-0.21.0.jar

[ivy:resolve] 
module not found: org.apache.hadoop#hadoop-hdfs;0.21.0

[ivy:resolve] 
==== apache-snapshot: tried

[ivy:resolve] 
  https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-hdfs/0.21.0/hadoop-hdfs-0.21.0.pom

[ivy:resolve] 
  -- artifact org.apache.hadoop#hadoop-hdfs;0.21.0!hadoop-hdfs.jar:

[ivy:resolve] 
  https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-hdfs/0.21.0/hadoop-hdfs-0.21.0.jar

[ivy:resolve] 
==== maven2: tried

[ivy:resolve] 
  http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-hdfs/0.21.0/hadoop-hdfs-0.21.0.pom

[ivy:resolve] 
  -- artifact org.apache.hadoop#hadoop-hdfs;0.21.0!hadoop-hdfs.jar:

[ivy:resolve] 
  http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-hdfs/0.21.0/hadoop-hdfs-0.21.0.jar

[ivy:resolve] 
::::::::::::::::::::::::::::::::::::::::::::::

[ivy:resolve] 
::          UNRESOLVED DEPENDENCIES         ::

[ivy:resolve] 
::::::::::::::::::::::::::::::::::::::::::::::

[ivy:resolve] 
:: org.apache.hadoop#hadoop-common;0.21.0: not found

[ivy:resolve] 
:: org.apache.hadoop#hadoop-common-test;0.21.0: not found

[ivy:resolve] 
:: org.apache.hadoop#hadoop-hdfs;0.21.0: not found

[ivy:resolve] 
::::::::::::::::::::::::::::::::::::::::::::::

[ivy:resolve] 

[ivy:resolve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS

 

BUILD FAILED

/opt/hadoop-0.21.0/mapred/build.xml:1861: impossible to resolve dependencies:

resolve failed - see output for details

 

 

通過Google，發現原來這是Apache-Hadoop的BUG:

https://issues.apache.org/jira/browse/HDFS-1519(hdfs ivy配置bug)

 

主要原因是ivy配置問題：

hdfs的ivy在配置關聯類別庫hadoop-common時候位置指定錯誤，具體修改如下：

 

hdfs/ivy.xml

<dependency org="org.apache.hadoop" name="hadoop-common" 

-----
rev="${hadoop-common.version}" 

+++ rev="${hadoop-common.version}-SNAPSHOT"

conf="common->default"/>

hdfs/src/contrib/hdfsproxy/ivy.xml

       <dependency org="org.apache.hadoop"

         name="hadoop-common"

----       rev="${hadoop-common.version}"

+++      rev="${hadoop-common.version}-SNAPSHOT"

         conf="common->default"/>

       <dependency org="org.apache.hadoop"

         name="hadoop-common-test"

---        rev="${hadoop-common.version}"

+++     rev="${hadoop-common.version}-SNAPSHOT"

         conf="common->default"/>

hdfs/src/contrib/thriftfs/ivy.xml

       <dependency org="org.apache.hadoop"

             name="hadoop-common"

---        rev="${hadoop-common.version}"

+++     rev="${hadoop-common.version}-SNAPSHOT"

         conf="common->default"/>

 

 

 

mapreduce的ivy在配置關聯類別庫hadoop-common/hadoop-dfs時候位置指定錯誤，修改方法與hdfs類似：

1.修改mapreduce project裡的ivy.xml(多個)

    在eclipse裡search file(ivy.xml):

    (1)rev="${hadoop-common.version}"  replace rev="${hadoop-common.version}-SNAPSHOT"

    (2)rev="${hadoop-hdfs.version}"  replace rev="${hadoop-hdfs.version}-SNAPSHOT"

2.修改buid.xml

          <unzip

         -- --      src="${common.ivy.lib.dir}/hadoop-hdfs-${hadoop-hdfs.version}.jar"

         +++     src="${common.ivy.lib.dir}/hadoop-hdfs-${hadoop-hdfs.version}-SNAPSHOT.jar"

         dest="${build.dir}">