SBT announces assembly Resolve jar Package conflicts deduplicate:different file contents found in the following

One, problem definition

The recent use of the SBT Battle Assembly fails when the package, at what time, a jar package conflict/file conflict occurs, two identical classes from different jar packages classpath inner conflict.

For more information: I have a self4j jar, Hadoop-common-hdfs jar package. The Hadoop-common-hdfs.jar contains the SELF4J jar package, which causes the conflict.

Such exceptions are usually caused by packaging irregularities and packaging negligence.

(Individuals feel that the right packaging strategy is to simply package their core functions.) Do not package dependencies together. But sometimes they have to be played together for convenience or sometimes. Be aware that the above problems may occur)

the Exception log is as follows :

[Trace] Stack Trace Suppressed:run Last *:assembly for the full output. [ERROR] (*:assembly) deduplicate:different file contents found in the Following:[error] C:\Users\shengli.victor\.ivy2\cache\ Org.slf4j\slf4j-api\jars\slf4j-api-1.7.7.jar:org/slf4j/imarkerfactory.class[error] C:\Users\shengli.victor\. Ivy2\cache\com.xxx.xx.hdfsfile\hdfscommon\jars\hdfscommon-1.1.jar:org/slf4j/imarkerfactory.class[error] Total Time:4 s, completed 2014-11-20 19:07:33

The anomaly is very obvious, from 2 different jar packages self4j, Hdfscommon-1.1.jar. In the Org/slf4j/imarkerfactory.class this kind of conflict.

For example, with:

Hdfscommon-1.1/jar

watermark/2/text/ahr0cdovl2jsb2cuy3nkbi5uzxqvb29wc29vbq==/font/5a6l5l2t/fontsize/400/fill/i0jbqkfcma==/ Dissolve/70/gravity/southeast "/>

Slf4j-api-1.7.2.jar

Second, the way to solve

There are two scenarios for resolving jar package conflicts:

1. Remove one of the jars, or say, in the packaging. Do not place the 2 same class in the classpath within the same jar package, that is, the exclude jar.

2. Merge conflicts

1. Excluding JARs and files% "provided"

Exclude one in the same jar, due to repeated. Ability to use "provided" keyword.

For example, Spark is a container class, and we need the spark core jar to write the spark application. However, the real package commits to the cluster and does not need to be put into the jar package.

This is where we use the% "provided" keyword to exclude it.

Librarydependencies ++= Seq (  "Org.apache.spark" percent "Spark-core"% "0.8.0-incubating"% "provided",  " Org.apache.hadoop "%" hadoop-client "%" 2.0.0-cdh4.4.0 "%" provided ")

Maven defines "provided" as:

This was much likecompile, but indicates you expect the JDK or a container to provide the dependency at runtime. For example, when building a Web application for the Java Enterprise Edition, you would set the dependency on the Servlet API and related Java EE APIs to scopeprovidedBecause the Web container provides those classes. This scope was only available on the compilation and Test Classpath, and was not transitive.

2. Merge Strategy

Assume that there are multiple identical files or jars under a relative path. At this point we are able to use the merge strategy.

The assemblymergestrategy is defined in the BUILD.SBT .

For example, the following:

Mergestrategy in Assembly <<= (Mergestrategy in assembly) {(old) + =  {case    PathList ("org", "slf4j", XS @ _*)         = Mergestrategy.first case    PathList (PS @ _*) if Ps.last endsWith "Axiom.xml" and Mergestrategy.filterd Istinctlines Case    PathList (PS @ _*) if Ps.last endsWith "log$logger.class" = Mergestrategy.first    case PathList (PS @ _*) if Ps.last endsWith "iloggerfactory.class" = Mergestrategy.first case    x = = Old (x)  }}

Workaround: combine the org, slf4j all classes and files under this. The strategy used is: in the classpath. 2 Select 1, choose the first self4j in the classpath order.

Multiple formats are supported here. such as Ps.lat EndsWith "Axiom.xml". Axiom.xml is the end of the file, all adopt the Filterdistinctlines strategy, that is, merging two files, to the repeated part.

Through the above changes, finally overcame the problem of slf4j conflict, that is, deduplicate:different file contents found in the following problem.

Again SBT Assembly:

[Warn] Merging ' Meta-inf\index. LIST ' with strategy ' discard ' [warn] merging ' meta-inf\manifest. MF ' with strategy ' discard ' [warn] merging ' meta-inf\maven\log4j\log4j\pom.properties ' with strategy ' first ' [Warn] Merging ' Meta-inf\maven\log4j\log4j\pom.xml ' with strategy ' first ' [warn] merging ' Meta-inf\maven\org.slf4j\slf4j-api \pom.properties ' with strategy ' first ' [warn] merging ' meta-inf\maven\org.slf4j\slf4j-api\pom.xml ' with strategy ' first ' [Warn] merging ' meta-inf\maven\org.slf4j\slf4j-log4j12\pom.properties ' with strategy ' first ' [warn] merging ' META-INF \maven\org.slf4j\slf4j-log4j12\pom.xml ' with strategy ' first ' [warn] merging ' com\esotericsoftware\minlog\log$ Logger.class ' with strategy ' first ' [warn] merging ' com\esotericsoftware\minlog\log.class ' with strategy ' first ' [Warn] Merging ' Org\apache\log4j\helpers\loglog.class ' with strategy ' first ' [warn] merging ' org\slf4j\iloggerfactory.class ' With strategy ' first ' [warn] merging ' org\slf4j\imarkerfactory.class ' with strategy ' first ' [Warn]Merging ' Org\slf4j\logger.class ' with strategy ' first ' [warn] merging ' org\slf4j\loggerfactory.class ' with strategy ' First ' [Warn] merging ' org\slf4j\mdc.class ' with strategy ' first ' [warn] merging ' org\slf4j\marker.class ' with strategy ' First ' [Warn] merging ' org\slf4j\markerfactory.class ' with strategy ' first ' [warn] merging ' org\slf4j\helpers\ Basicmdcadapter.class ' with strategy ' first ' [warn] merging ' org\slf4j\helpers\basicmarker.class ' with strategy ' first ' [Warn] Merging ' Org\slf4j\helpers\basicmarkerfactory.class ' with strategy ' first ' [warn] merging ' org\slf4j\helpers\ Formattingtuple.class ' with strategy ' first ' [warn] merging ' org\slf4j\helpers\markerignoringbase.class ' with strategy ' First ' [warn] merging ' org\slf4j\helpers\messageformatter.class ' with strategy ' first ' [warn] merging ' org\slf4j\ Helpers\noplogger.class ' with strategy ' first ' [warn] merging ' org\slf4j\helpers\noploggerfactory.class ' with strategy ' First ' [warn] merging ' org\slf4j\helpers\nopmdcadapter.class ' with strategy' First ' [warn] merging ' org\slf4j\helpers\namedloggerbase.class ' with strategy ' first ' [warn] merging ' org\slf4j\ Helpers\substituteloggerfactory.class ' with strategy ' first ' [warn] merging ' org\slf4j\helpers\util.class ' with Strategy ' first ' [warn] merging ' org\slf4j\impl\log4jloggeradapter.class ' with strategy ' first ' [warn] merging ' org\ Slf4j\impl\log4jloggerfactory.class ' with strategy ' first ' [warn] merging ' org\slf4j\impl\log4jmdcadapter.class ' with Strategy ' first ' [warn] merging ' org\slf4j\impl\staticloggerbinder.class ' with strategy ' first ' [warn] merging ' org\ Slf4j\impl\staticmdcbinder.class ' with strategy ' first ' [warn] merging ' org\slf4j\impl\staticmarkerbinder.class ' with Strategy ' first ' [warn] merging ' org\slf4j\spi\locationawarelogger.class ' with strategy ' first ' [warn] merging ' org\ Slf4j\spi\loggerfactorybinder.class ' with strategy ' first ' [warn] merging ' org\slf4j\spi\mdcadapter.class ' with Strategy ' first ' [warn] merging ' org\slf4j\spi\markerfactorybinder.class ' with strategy ' firSt ' [warn] merging ' rootdoc.txt ' with strategy ' concat ' [warn] strategy ' concat ' is applied to a file[info] strategy ' dedup Licate ' is applied to 373 files (Run the task at debug level to see details) [Warn] strategy ' discard ' is applied to 2 fi Les[warn] Strategy ' first ' was applied to + files

[INFO] Done packaging. [Success] Total time:84 s, completed 2014-11-20 19:04:52

There are a number of merge strategies:

Able to participate in the examiner's SBT assembly document: Https://github.com/sbt/sbt-assembly

http://stackoverflow.com/questions/19606243/resolving-dependencies-in-creating-jar-through-sbt-assembly

• MergeStrategy.deduplicateIs the default described above
• MergeStrategy.firstPicks the first of the matching files in classpath order
• MergeStrategy.lastPicks the last one
• MergeStrategy.singleOrErrorBails out with a error message on conflict
• MergeStrategy.concatSimply concatenates all matching files and includes the result
• MergeStrategy.filterDistinctLinesAlso concatenates, but leaves out duplicates along the
• MergeStrategy.renameRenames the files originating from Jar files
• MergeStrategy.discardSimply discards matching files

Many other formulations, example:

Assemblymergestrategy in Assembly:={ Case PathList( "javax" , "servlet" Xs@_*)= Mergestrategy. First Case PathList(PS@_*)ifPs.last EndsWith ". html"  = Mergestrategy. First Case  "application.conf"                             = Mergestrategy. concat Case  "unwanted.txt"                                 = Mergestrategy. Discard CaseX=    Val Oldstrategy =(Assemblymergestrategy in assembly). Value Oldstrategy (x)}

Final Qucik Hack: Suppose none of the above works. There is also the best one, forcing the default of all merges, not the last resort, do not use:

Mergestrategy in Assembly <<= (Mergestrategy in assembly) {Mergestrategy = {case entry = {   val strateg y = mergestrategy (entry)   if (strategy = = mergestrategy.deduplicate) Mergestrategy.first   Else Strategy}}}

Iii. Summary

Don't panic when you encounter similar problems. Look at the meaning of the description of the log in detail.

The exception reported a conflict of content redundancy. The root cause of the problem is the fact that there are exactly the same 2 classes in the Classpath in the now-seen path.

Find out why. Resolve the party hair. Two methods of eliminating conflicts. has been the removal method and the other is the merger method.

Relative to Maven and Gradle. SBT's conflict resolution approach is still closer to the bottom. If you remember correctly, MAVEN and Gradle can resolve conflicts on their own initiative.

This article only solves this problem to propose the solution way and the thought. Detailed configurations also need to be studied.

--eof--

Original articles, reproduced please specify: http://blog.csdn.net/oopsoom/article/details/41318599

SBT announces assembly Resolve jar Package conflicts deduplicate:different file contents found in the following