Spark2.3-run Exception NoSuchMethodError:io.netty.buffer.PooledByteBufAllocator.metric () __.net

Spark2.3-run exception NoSuchMethodError:io.netty.buffer.PooledByteBufAllocator.metric () I. Description of the problem

Multiple frameworks were introduced in a project HBase 1.4.1 Kafka 1.1.0 Spark 2.3.0

When you run the Spark sample program with the local model, you receive the following error:

Exception in thread "main" java.lang.NoSuchMethodError:io.netty.buffer.PooledByteBufAllocator.metric () lio/netty/
    Buffer/pooledbytebufallocatormetric; At Org.apache.spark.network.util.NettyMemoryMetrics.registerMetrics (nettymemorymetrics.java:80) at Org.apache.spark.network.util.nettymemorymetrics.<init> (nettymemorymetrics.java:76) at Org.apache.spark.network.client.transportclientfactory.<init> (transportclientfactory.java:109) at Org.apache.spark.network.TransportContext.createClientFactory (transportcontext.java:99) at Org.apache.spark.rpc.netty.nettyrpcenv.<init> (nettyrpcenv.scala:71) at Org.apache.spark.rpc.netty.NettyRpcEnvFactory.create (nettyrpcenv.scala:461) at org.apache.spark.rpc.rpcenv$. Create (rpcenv.scala:57) at Org.apache.spark.sparkenv$.create (sparkenv.scala:249) at Org.apache.spark.sparkenv$.cre Atedriverenv (sparkenv.scala:175) at ORG.APACHE.SPARK.SPARKCONTEXT.CREATESPARKENV (sparkcontext.scala:256) at Org.ap Ache.sPark. Sparkcontext.<init> (sparkcontext.scala:423) at Org.apache.spark.sparkcontext$.getorcreate (  sparkcontext.scala:2486) at org.apache.spark.sql.sparksession$builder$ $anonfun $7.apply (sparksession.scala:930) at org.apache.spark.sql.sparksession$builder$ $anonfun $7.apply (sparksession.scala:921) at Scala. Option.getorelse (option.scala:121) at Org.apache.spark.sql.sparksession$builder.getorcreate (SparkSession.scala : 921) at Spark.ml.JavaAFTSurvivalRegressionExample.main (javaaftsurvivalregressionexample.java:50) at SUN.REFLECT.N Ativemethodaccessorimpl.invoke0 (Native method) at Sun.reflect.NativeMethodAccessorImpl.invoke ( nativemethodaccessorimpl.java:62) at Sun.reflect.DelegatingMethodAccessorImpl.invoke ( delegatingmethodaccessorimpl.java:43) at Java.lang.reflect.Method.invoke (method.java:498) at COM.INTELLIJ.RT.EXECU Tion.application.AppMain.main (appmain.java:147)

Second, the problem analysis

In the search, according to: http://www.aboutyun.com/thread-24429-1-1.html to the problem, may be dependent on the existence of multiple versions of the Netty cause the program to appear Nosuchmethoderror exception.

Next, use the command mvn dependency:tree >> log/dependency.log to analyze the existing jar pack dependencies in the program and find it in the search log file: org.apache.hbase: The hbase-client:jar:1.4.1:compile contains a io.netty:netty-all:jar:4.1.8.final:compile jar package org.apache.spark:spark-core_ The 2.11:jar:2.3.0:compile contains a io.netty:netty:jar:3.9.9.final:compile

Initial positioning is the conflict caused by these two jar packages.

My first option was to use the <exclusions></exclusions> tag to remove all the lower versions of the Netty in the Spark-core, without the fruit.

It was later found that a similar problem was found in StackOverflow: https://stackoverflow.com/questions/49137397/ Spark-2-3-0-netty-version-issue-nosuchmethod-io-netty-buffer-pooledbytebufalloc?utm_medium=organic&utm_ Source=google_rich_qa&utm_campaign=google_rich_qa

In this, the main emphasis on the issue of MAVEN's delivery dependencies, https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html.

And the workaround in the comments is to use <dependencyManagement></dependencyManagement> to manage multiple versions of the project, adding dependencies as follows:

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupid>io.netty</ groupid>
            <artifactId>netty-all</artifactId>
            <version>4.1.18.Final</version>
        </dependency>
    </dependencies>
</dependencyManagement>

Note: The version here is determined by the version of Netty in the log file generated by MVN. iii. Results of Operation

After adding this configuration, the program can function correctly:

Iv. Full Reliance

<?xml version= "1.0" encoding= "UTF-8"?> <project xmlns= "http://maven.apache.org/POM/4.0.0" xmlns:xsi= "HTT" P://www.w3.org/2001/xmlschema-instance "xsi:schemalocation=" http://maven.apache.org/POM/4.0.0 Http://maven.apach E.org/xsd/maven-4.0.0.xsd "> <modelVersion>4.0.0</modelVersion> <properties> <java .version>1.8</java.version> <scala.version>2.11.12</scala.version>