SBT release assembly resolves jar package conflict issues deduplicate:different file contents found in the following

I. Definition OF the problem

When I recently encountered a problem with SBT in the assembly package, there was a jar conflict/file conflict problem with the packages, and two identical classes from different jar packages generated conflicts within the classpath.

Specifically: I have a self4j jar, and a hadoop-common-hdfs jar package, where Hadoop-common-hdfs.jar contains self4j this jar package, causing the conflict.

Such exceptions are generally caused by packaging irregularities and packaging negligence.

(Personally, the correct packaging strategy is: only packaging their own core functions, not to pack dependencies together, but sometimes for convenience or sometimes have to play together, be aware of the problem may occur)

The Exception Log is as follows :

[Trace] Stack Trace Suppressed:run Last *:assembly for the full output. [ERROR] (*:assembly) deduplicate:different file contents found in the Following:[error] C:\Users\shengli.victor\.ivy2\cache\ Org.slf4j\slf4j-api\jars\slf4j-api-1.7.7.jar:org/slf4j/imarkerfactory.class[error] C:\Users\shengli.victor\. Ivy2\cache\com.xxx.xx.hdfsfile\hdfscommon\jars\hdfscommon-1.1.jar:org/slf4j/imarkerfactory.class[error] Total Time:4 s, completed 2014-11-20 19:07:33

The exception is obvious, from 2 different jar packages self4j, Hdfscommon-1.1.jar, in Org/slf4j/imarkerfactory.class this class conflicts.

Such as:

Hdfscommon-1.1/jar

Slf4j-api-1.7.2.jar

Second, the solution

There are two scenarios for resolving jar package conflicts:

1. Remove one of the jars, or, in the packaging, do not place 2 identical classes in the Classpath within the same jar package, that is, the exclude jar.

2. Merge conflicts

1. Excluding JARs and files% "provided"

Exclude one in the same jar, because you can use the "provided" keyword for duplicates.

For example, Spark is a container class, and we need the spark core jar to write the spark application. But the real package commits to the cluster execution, it does not need to be in the jar package.

This is where we use the% "provided" keyword to exclude it.

Librarydependencies ++= Seq (  "Org.apache.spark" percent "Spark-core"% "0.8.0-incubating"% "provided",  " Org.apache.hadoop "%" hadoop-client "%" 2.0.0-cdh4.4.0 "%" provided ")

Maven defines "provided" as:

This was much likecompile, but indicates you expect the JDK or a container to provide the dependency at runtime. For example, when building a Web application for the Java Enterprise Edition, you would set the dependency on the Servlet API and related Java EE APIs to scopeprovidedBecause the Web container provides those classes. This scope was only available on the compilation and Test Classpath, and was not transitive.

2. Merge Strategy

If there are multiple identical files or jars under the relative path, then we can use the merge policy.

The assemblymergestrategy is defined in the BUILD.SBT .

As follows:

Mergestrategy in Assembly <<= (Mergestrategy in assembly) {(old) + =  {case    PathList ("org", "slf4j", XS @ _*)         = Mergestrategy.first case    PathList (PS @ _*) if Ps.last endsWith "Axiom.xml" and Mergestrategy.filterd Istinctlines Case    PathList (PS @ _*) if Ps.last endsWith "log$logger.class" = Mergestrategy.first    case PathList (PS @ _*) if Ps.last endsWith "iloggerfactory.class" = Mergestrategy.first case    x = = Old (x)  }}

solution: the Org, slf4j all classes and files under this, all do merge, adopt the Strategy is: in Classpath, 2 Choose 1, choose is Classpath order in the first self4j.

This supports multiple formats, such as Ps.lat EndsWith "Axiom.xml", which is the end of the Axiom.xml file, all using the Filterdistinctlines strategy, that is, merging two files to remove the duplicated parts.

Through the above modification, finally solved the problem of slf4j conflict, that is, deduplicate:different file contents found in the following problem.

Again SBT Assembly:

[Warn] Merging ' Meta-inf\index. LIST ' with strategy ' discard ' [warn] merging ' meta-inf\manifest. MF ' with strategy ' discard ' [warn] merging ' meta-inf\maven\log4j\log4j\pom.properties ' with strategy ' first ' [Warn] Merging ' Meta-inf\maven\log4j\log4j\pom.xml ' with strategy ' first ' [warn] merging ' Meta-inf\maven\org.slf4j\slf4j-api \pom.properties ' with strategy ' first ' [warn] merging ' meta-inf\maven\org.slf4j\slf4j-api\pom.xml ' with strategy ' first ' [Warn] merging ' meta-inf\maven\org.slf4j\slf4j-log4j12\pom.properties ' with strategy ' first ' [warn] merging ' META-INF \maven\org.slf4j\slf4j-log4j12\pom.xml ' with strategy ' first ' [warn] merging ' com\esotericsoftware\minlog\log$ Logger.class ' with strategy ' first ' [warn] merging ' com\esotericsoftware\minlog\log.class ' with strategy ' first ' [Warn] Merging ' Org\apache\log4j\helpers\loglog.class ' with strategy ' first ' [warn] merging ' org\slf4j\iloggerfactory.class ' With strategy ' first ' [warn] merging ' org\slf4j\imarkerfactory.class ' with strategy ' first ' [Warn]Merging ' Org\slf4j\logger.class ' with strategy ' first ' [warn] merging ' org\slf4j\loggerfactory.class ' with strategy ' First ' [Warn] merging ' org\slf4j\mdc.class ' with strategy ' first ' [warn] merging ' org\slf4j\marker.class ' with strategy ' First ' [Warn] merging ' org\slf4j\markerfactory.class ' with strategy ' first ' [warn] merging ' org\slf4j\helpers\ Basicmdcadapter.class ' with strategy ' first ' [warn] merging ' org\slf4j\helpers\basicmarker.class ' with strategy ' first ' [Warn] Merging ' Org\slf4j\helpers\basicmarkerfactory.class ' with strategy ' first ' [warn] merging ' org\slf4j\helpers\ Formattingtuple.class ' with strategy ' first ' [warn] merging ' org\slf4j\helpers\markerignoringbase.class ' with strategy ' First ' [warn] merging ' org\slf4j\helpers\messageformatter.class ' with strategy ' first ' [warn] merging ' org\slf4j\ Helpers\noplogger.class ' with strategy ' first ' [warn] merging ' org\slf4j\helpers\noploggerfactory.class ' with strategy ' First ' [warn] merging ' org\slf4j\helpers\nopmdcadapter.class ' with strategy' First ' [warn] merging ' org\slf4j\helpers\namedloggerbase.class ' with strategy ' first ' [warn] merging ' org\slf4j\ Helpers\substituteloggerfactory.class ' with strategy ' first ' [warn] merging ' org\slf4j\helpers\util.class ' with Strategy ' first ' [warn] merging ' org\slf4j\impl\log4jloggeradapter.class ' with strategy ' first ' [warn] merging ' org\ Slf4j\impl\log4jloggerfactory.class ' with strategy ' first ' [warn] merging ' org\slf4j\impl\log4jmdcadapter.class ' with Strategy ' first ' [warn] merging ' org\slf4j\impl\staticloggerbinder.class ' with strategy ' first ' [warn] merging ' org\ Slf4j\impl\staticmdcbinder.class ' with strategy ' first ' [warn] merging ' org\slf4j\impl\staticmarkerbinder.class ' with Strategy ' first ' [warn] merging ' org\slf4j\spi\locationawarelogger.class ' with strategy ' first ' [warn] merging ' org\ Slf4j\spi\loggerfactorybinder.class ' with strategy ' first ' [warn] merging ' org\slf4j\spi\mdcadapter.class ' with Strategy ' first ' [warn] merging ' org\slf4j\spi\markerfactorybinder.class ' with strategy ' firSt ' [warn] merging ' rootdoc.txt ' with strategy ' concat ' [warn] strategy ' concat ' is applied to a file[info] strategy ' dedup Licate ' is applied to 373 files (Run the task at debug level to see details) [Warn] strategy ' discard ' is applied to 2 fi Les[warn] Strategy ' first ' was applied to + files

[INFO] Done packaging. [Success] Total time:84 s, completed 2014-11-20 19:04:52

There are a number of merge strategies:

You can refer to the official SBT Assembly documentation: Https://github.com/sbt/sbt-assembly

• MergeStrategy.deduplicateIs the default described above
• MergeStrategy.firstPicks the first of the matching files in classpath order
• MergeStrategy.lastPicks the last one
• MergeStrategy.singleOrErrorBails out with a error message on conflict
• MergeStrategy.concatSimply concatenates all matching files and includes the result
• MergeStrategy.filterDistinctLinesAlso concatenates, but leaves out duplicates along the
• MergeStrategy.renameRenames the files originating from Jar files
• MergeStrategy.discardSimply discards matching files

More of the wording, example:

Assemblymergestrategy in Assembly:={ Case PathList( "javax" , "servlet" Xs@_*)= Mergestrategy. First Case PathList(PS@_*)ifPs.last EndsWith ". html"  = Mergestrategy. First Case  "application.conf"                             = Mergestrategy. concat Case  "unwanted.txt"                                 = Mergestrategy. Discard CaseX=    Val Oldstrategy =(Assemblymergestrategy in assembly). Value Oldstrategy (x)}

Iii. Summary

Do not panic about similar problems, look closely at the meaning of the log description.

Exception report the conflict of content redundancy, look at the path, found that there are exactly the same 2 classes within the Classpath, this is the root cause of the problem.

Find out the reason, solve the party, eliminate the conflict two methods, has been the removal method, the other is the merger method.

Conflict resolution versus Maven and GRADLE,SBT is still relatively close to the bottom. If you remember correctly, both MAVEN and Gradle can resolve conflicts automatically.

This article only proposes the solution and the idea for this question, the concrete configuration also needs to continue to study.

--eof--

Original articles, reproduced please specify from: http://blog.csdn.net/oopsoom/article/details/41318599

SBT release assembly resolves jar package conflict issues deduplicate:different file contents found in the following