Spark's Workcount

Source: Internet
Author: User
Tags lzf shuffle zookeeper glassfish log4j

Using Idea+maven to build Spark's development environment, encounter a little pit, fortunately finally completed successfully, using MAVEN to manage the project is still very necessary ~ ~ ~

1. Create a new MAVEN project, select the Scala class project, and Next

2. Fill in the Groupid,artifactid,projectname, continue next, next, and fill in the project name

3. After the project has been generated, delete test class Myspec.scala, if not deleted, may report a test error when running

4. Set Scala to the appropriate version in the Pom file

1 < Properties > 2   < scala.version >2.10.5</scala.version>3</  Properties>

5. Convention first on a Hello Word program, normal operation

Package com.scalatest/** * Hello world! * */object App  {  def main (args:array[string]): Unit = {    println ("Hello Scala")  }}

  

6. Continue to build the spark development environment, add MAVEN dependencies, here Select the spark1.6.1 version, if you do not know the specific dependencies how to write, you can query

1 <Dependency>2       <groupId>Org.apache.spark</groupId>3       <Artifactid>spark-core_2.11</Artifactid>4       <version>1.6.1</version>5     </Dependency>

7. Writing a wordcount program

Object Test {  def main (args:array[string]): Unit = {    val conf = new sparkconf (). Setappname ("Sparktest"). Setmaste R ("local")    val sc = new Sparkcontext (conf)    val Filerdd = Sc.textfile ("F:\\data\\wordcount.txt")    Filerdd.flatmap (_.split ("")). Map ((_, 1)). Reducebykey (_+_). Collect (). foreach (println)  }}

  

8. Run the test procedure with the following error.

1 "C:\Program Files\java\jdk1.7.0_79\bin\java"-didea.launcher.port=7538 "-didea.launcher.bin.path=c:\program Files ( x86) \jetbrains\intellij idea 2016.2.5\bin "-dfile.encoding=utf-8-classpath" C:\Program files\java\jdk1.7.0_79\jre\ Lib\charsets.jar; C:\Program Files\java\jdk1.7.0_79\jre\lib\deploy.jar; C:\Program Files\java\jdk1.7.0_79\jre\lib\ext\access-bridge-64.jar; C:\Program Files\java\jdk1.7.0_79\jre\lib\ext\dnsns.jar; C:\Program Files\java\jdk1.7.0_79\jre\lib\ext\jaccess.jar; C:\Program Files\java\jdk1.7.0_79\jre\lib\ext\localedata.jar; C:\Program Files\java\jdk1.7.0_79\jre\lib\ext\sunec.jar; C:\Program Files\java\jdk1.7.0_79\jre\lib\ext\sunjce_provider.jar; C:\Program Files\java\jdk1.7.0_79\jre\lib\ext\sunmscapi.jar; C:\Program Files\java\jdk1.7.0_79\jre\lib\ext\zipfs.jar; C:\Program Files\java\jdk1.7.0_79\jre\lib\javaws.jar; C:\Program Files\java\jdk1.7.0_79\jre\lib\jce.jar; C:\Program Files\java\jdk1.7.0_79\jre\lib\jfr.jar; C:\Program Files\java\jdk1.7.0_79\jre\lib\jfxrt.jar; C:\Program FilEs\java\jdk1.7.0_79\jre\lib\jsse.jar; C:\Program Files\java\jdk1.7.0_79\jre\lib\management-agent.jar; C:\Program Files\java\jdk1.7.0_79\jre\lib\plugin.jar; C:\Program Files\java\jdk1.7.0_79\jre\lib\resources.jar; C:\Program Files\java\jdk1.7.0_79\jre\lib\rt.jar; F:\IDEAworkspace\scalaproject\target\classes; F:\repo\org\apache\spark\spark-core_2.11\1.6.1\spark-core_2.11-1.6.1.jar; F:\repo\org\apache\avro\avro-mapred\1.7.7\avro-mapred-1.7.7-hadoop2.jar; F:\repo\org\apache\avro\avro-ipc\1.7.7\avro-ipc-1.7.7.jar; F:\repo\org\apache\avro\avro\1.7.7\avro-1.7.7.jar; F:\repo\org\apache\avro\avro-ipc\1.7.7\avro-ipc-1.7.7-tests.jar; F:\repo\org\codehaus\jackson\jackson-core-asl\1.9.13\jackson-core-asl-1.9.13.jar; F:\repo\org\codehaus\jackson\jackson-mapper-asl\1.9.13\jackson-mapper-asl-1.9.13.jar; F:\repo\com\twitter\chill_2.11\0.5.0\chill_2.11-0.5.0.jar; F:\repo\com\esotericsoftware\kryo\kryo\2.21\kryo-2.21.jar; F:\repo\com\esotericsoftware\reflectasm\reflectasm\1.07\reflectasm-1.07-shaded.jar; F:\repo\com\esoTericsoftware\minlog\minlog\1.2\minlog-1.2.jar; F:\repo\org\objenesis\objenesis\1.2\objenesis-1.2.jar; F:\repo\com\twitter\chill-java\0.5.0\chill-java-0.5.0.jar; F:\repo\org\apache\xbean\xbean-asm5-shaded\4.4\xbean-asm5-shaded-4.4.jar; F:\repo\org\apache\hadoop\hadoop-client\2.2.0\hadoop-client-2.2.0.jar; F:\repo\org\apache\hadoop\hadoop-common\2.2.0\hadoop-common-2.2.0.jar; F:\repo\commons-cli\commons-cli\1.2\commons-cli-1.2.jar; F:\repo\org\apache\commons\commons-math\2.1\commons-math-2.1.jar; F:\repo\xmlenc\xmlenc\0.52\xmlenc-0.52.jar; F:\repo\commons-configuration\commons-configuration\1.6\commons-configuration-1.6.jar; F:\repo\commons-collections\commons-collections\3.2.1\commons-collections-3.2.1.jar; F:\repo\commons-digester\commons-digester\1.8\commons-digester-1.8.jar; F:\repo\commons-beanutils\commons-beanutils\1.7.0\commons-beanutils-1.7.0.jar; F:\repo\commons-beanutils\commons-beanutils-core\1.8.0\commons-beanutils-core-1.8.0.jar; F:\repo\org\apache\hadoop\hadoop-auth\2.2.0\hadoop-auth-2.2.0.jar; F:\repo\org\apache\commons\commons-compress\1.4.1\commons-compress-1.4.1.jar; F:\repo\org\tukaani\xz\1.0\xz-1.0.jar; F:\repo\org\apache\hadoop\hadoop-hdfs\2.2.0\hadoop-hdfs-2.2.0.jar; F:\repo\org\mortbay\jetty\jetty-util\6.1.26\jetty-util-6.1.26.jar; F:\repo\org\apache\hadoop\hadoop-mapreduce-client-app\2.2.0\hadoop-mapreduce-client-app-2.2.0.jar; F:\repo\org\apache\hadoop\hadoop-mapreduce-client-common\2.2.0\hadoop-mapreduce-client-common-2.2.0.jar; F:\repo\org\apache\hadoop\hadoop-yarn-client\2.2.0\hadoop-yarn-client-2.2.0.jar; F:\repo\com\google\inject\guice\3.0\guice-3.0.jar; F:\repo\javax\inject\javax.inject\1\javax.inject-1.jar; F:\repo\aopalliance\aopalliance\1.0\aopalliance-1.0.jar; F:\repo\com\sun\jersey\jersey-test-framework\jersey-test-framework-grizzly2\1.9\ Jersey-test-framework-grizzly2-1.9.jar; F:\repo\com\sun\jersey\jersey-test-framework\jersey-test-framework-core\1.9\jersey-test-framework-core-1.9.jar ; F:\repo\javax\servlet\javax.servlet-api\3.0.1\javax.servlet-api-3.0.1.jAr F:\repo\com\sun\jersey\jersey-client\1.9\jersey-client-1.9.jar; F:\repo\com\sun\jersey\jersey-grizzly2\1.9\jersey-grizzly2-1.9.jar; F:\repo\org\glassfish\grizzly\grizzly-http\2.1.2\grizzly-http-2.1.2.jar; F:\repo\org\glassfish\grizzly\grizzly-framework\2.1.2\grizzly-framework-2.1.2.jar; F:\repo\org\glassfish\gmbal\gmbal-api-only\3.0.0-b023\gmbal-api-only-3.0.0-b023.jar; F:\repo\org\glassfish\external\management-api\3.0.0-b012\management-api-3.0.0-b012.jar; F:\repo\org\glassfish\grizzly\grizzly-http-server\2.1.2\grizzly-http-server-2.1.2.jar; F:\repo\org\glassfish\grizzly\grizzly-rcm\2.1.2\grizzly-rcm-2.1.2.jar; F:\repo\org\glassfish\grizzly\grizzly-http-servlet\2.1.2\grizzly-http-servlet-2.1.2.jar; F:\repo\org\glassfish\javax.servlet\3.1\javax.servlet-3.1.jar; F:\repo\com\sun\jersey\jersey-json\1.9\jersey-json-1.9.jar; F:\repo\org\codehaus\jettison\jettison\1.1\jettison-1.1.jar; F:\repo\stax\stax-api\1.0.1\stax-api-1.0.1.jar; F:\repo\com\sun\xml\bind\jaxb-impl\2.2.3-1\jaxb-impl-2.2.3-1.jar; F:\rePo\javax\xml\bind\jaxb-api\2.2.2\jaxb-api-2.2.2.jar; F:\repo\javax\activation\activation\1.1\activation-1.1.jar; F:\repo\org\codehaus\jackson\jackson-jaxrs\1.8.3\jackson-jaxrs-1.8.3.jar; F:\repo\org\codehaus\jackson\jackson-xc\1.8.3\jackson-xc-1.8.3.jar; F:\repo\com\sun\jersey\contribs\jersey-guice\1.9\jersey-guice-1.9.jar; F:\repo\org\apache\hadoop\hadoop-yarn-server-common\2.2.0\hadoop-yarn-server-common-2.2.0.jar; F:\repo\org\apache\hadoop\hadoop-mapreduce-client-shuffle\2.2.0\hadoop-mapreduce-client-shuffle-2.2.0.jar; F:\repo\org\apache\hadoop\hadoop-yarn-api\2.2.0\hadoop-yarn-api-2.2.0.jar; F:\repo\org\apache\hadoop\hadoop-mapreduce-client-core\2.2.0\hadoop-mapreduce-client-core-2.2.0.jar; F:\repo\org\apache\hadoop\hadoop-yarn-common\2.2.0\hadoop-yarn-common-2.2.0.jar; F:\repo\org\apache\hadoop\hadoop-mapreduce-client-jobclient\2.2.0\hadoop-mapreduce-client-jobclient-2.2.0.jar; F:\repo\org\apache\hadoop\hadoop-annotations\2.2.0\hadoop-annotations-2.2.0.jar; F:\repo\org\apache\spark\spark-launCher_2.11\1.6.1\spark-launcher_2.11-1.6.1.jar; F:\repo\org\apache\spark\spark-network-common_2.11\1.6.1\spark-network-common_2.11-1.6.1.jar; F:\repo\org\apache\spark\spark-network-shuffle_2.11\1.6.1\spark-network-shuffle_2.11-1.6.1.jar; F:\repo\org\fusesource\leveldbjni\leveldbjni-all\1.8\leveldbjni-all-1.8.jar; F:\repo\com\fasterxml\jackson\core\jackson-annotations\2.4.4\jackson-annotations-2.4.4.jar; F:\repo\org\apache\spark\spark-unsafe_2.11\1.6.1\spark-unsafe_2.11-1.6.1.jar; F:\repo\net\java\dev\jets3t\jets3t\0.7.1\jets3t-0.7.1.jar; F:\repo\commons-codec\commons-codec\1.3\commons-codec-1.3.jar; F:\repo\commons-httpclient\commons-httpclient\3.1\commons-httpclient-3.1.jar; F:\repo\org\apache\curator\curator-recipes\2.4.0\curator-recipes-2.4.0.jar; F:\repo\org\apache\curator\curator-framework\2.4.0\curator-framework-2.4.0.jar; F:\repo\org\apache\curator\curator-client\2.4.0\curator-client-2.4.0.jar; F:\repo\org\apache\zookeeper\zookeeper\3.4.5\zookeeper-3.4.5.jar; F:\repo\jline\jline\0.9.94\jline-0.9.94.jar; F:\repo\com\google\guava\guava\14.0.1\guava-14.0.1.jar; F:\repo\org\eclipse\jetty\orbit\javax.servlet\3.0.0.v201112011016\javax.servlet-3.0.0.v201112011016.jar; F:\repo\org\apache\commons\commons-lang3\3.3.2\commons-lang3-3.3.2.jar; F:\repo\org\apache\commons\commons-math3\3.4.1\commons-math3-3.4.1.jar; F:\repo\com\google\code\findbugs\jsr305\1.3.9\jsr305-1.3.9.jar; F:\repo\org\slf4j\slf4j-api\1.7.10\slf4j-api-1.7.10.jar; F:\repo\org\slf4j\jul-to-slf4j\1.7.10\jul-to-slf4j-1.7.10.jar; F:\repo\org\slf4j\jcl-over-slf4j\1.7.10\jcl-over-slf4j-1.7.10.jar; F:\repo\log4j\log4j\1.2.17\log4j-1.2.17.jar; F:\repo\org\slf4j\slf4j-log4j12\1.7.10\slf4j-log4j12-1.7.10.jar; F:\repo\com\ning\compress-lzf\1.0.3\compress-lzf-1.0.3.jar; F:\repo\org\xerial\snappy\snappy-java\1.1.2\snappy-java-1.1.2.jar; F:\repo\net\jpountz\lz4\lz4\1.3.0\lz4-1.3.0.jar; F:\repo\org\roaringbitmap\RoaringBitmap\0.5.11\RoaringBitmap-0.5.11.jar; F:\repo\commons-net\commons-net\2.2\commons-net-2.2.jar; F:\repo\com\typesafe\akka\akka-rEmote_2.11\2.3.11\akka-remote_2.11-2.3.11.jar; F:\repo\com\typesafe\akka\akka-actor_2.11\2.3.11\akka-actor_2.11-2.3.11.jar; F:\repo\com\typesafe\config\1.2.1\config-1.2.1.jar; F:\repo\io\netty\netty\3.8.0.Final\netty-3.8.0.Final.jar; F:\repo\com\google\protobuf\protobuf-java\2.5.0\protobuf-java-2.5.0.jar; F:\repo\org\uncommons\maths\uncommons-maths\1.2.2a\uncommons-maths-1.2.2a.jar; F:\repo\com\typesafe\akka\akka-slf4j_2.11\2.3.11\akka-slf4j_2.11-2.3.11.jar; F:\repo\org\json4s\json4s-jackson_2.11\3.2.10\json4s-jackson_2.11-3.2.10.jar; F:\repo\org\json4s\json4s-core_2.11\3.2.10\json4s-core_2.11-3.2.10.jar; F:\repo\org\json4s\json4s-ast_2.11\3.2.10\json4s-ast_2.11-3.2.10.jar; F:\repo\org\scala-lang\scalap\2.11.0\scalap-2.11.0.jar; F:\repo\org\scala-lang\scala-compiler\2.11.0\scala-compiler-2.11.0.jar; F:\repo\org\scala-lang\modules\scala-xml_2.11\1.0.1\scala-xml_2.11-1.0.1.jar; F:\repo\org\scala-lang\modules\scala-parser-combinators_2.11\1.0.1\scala-parser-combinators_2.11-1.0.1.jar; F:\repo\com\sun\jErsey\jersey-server\1.9\jersey-server-1.9.jar; F:\repo\asm\asm\3.1\asm-3.1.jar; F:\repo\com\sun\jersey\jersey-core\1.9\jersey-core-1.9.jar; F:\repo\org\apache\mesos\mesos\0.21.1\mesos-0.21.1-shaded-protobuf.jar; F:\repo\io\netty\netty-all\4.0.29.Final\netty-all-4.0.29.Final.jar; F:\repo\com\clearspring\analytics\stream\2.7.0\stream-2.7.0.jar; F:\repo\io\dropwizard\metrics\metrics-core\3.1.2\metrics-core-3.1.2.jar; F:\repo\io\dropwizard\metrics\metrics-jvm\3.1.2\metrics-jvm-3.1.2.jar; F:\repo\io\dropwizard\metrics\metrics-json\3.1.2\metrics-json-3.1.2.jar; F:\repo\io\dropwizard\metrics\metrics-graphite\3.1.2\metrics-graphite-3.1.2.jar; F:\repo\com\fasterxml\jackson\core\jackson-databind\2.4.4\jackson-databind-2.4.4.jar; F:\repo\com\fasterxml\jackson\core\jackson-core\2.4.4\jackson-core-2.4.4.jar; F:\repo\com\fasterxml\jackson\module\jackson-module-scala_2.11\2.4.4\jackson-module-scala_2.11-2.4.4.jar; F:\repo\org\scala-lang\scala-reflect\2.11.2\scala-reflect-2.11.2.jar; F:\repo\com\thoughtworks\paranaMer\paranamer\2.6\paranamer-2.6.jar; F:\repo\org\apache\ivy\ivy\2.4.0\ivy-2.4.0.jar; F:\repo\oro\oro\2.0.8\oro-2.0.8.jar; F:\repo\org\tachyonproject\tachyon-client\0.8.2\tachyon-client-0.8.2.jar; F:\repo\commons-lang\commons-lang\2.4\commons-lang-2.4.jar; F:\repo\commons-io\commons-io\2.4\commons-io-2.4.jar; F:\repo\org\tachyonproject\tachyon-underfs-hdfs\0.8.2\tachyon-underfs-hdfs-0.8.2.jar; F:\repo\org\tachyonproject\tachyon-underfs-s3\0.8.2\tachyon-underfs-s3-0.8.2.jar; F:\repo\org\tachyonproject\tachyon-underfs-local\0.8.2\tachyon-underfs-local-0.8.2.jar; F:\repo\net\razorvine\pyrolite\4.9\pyrolite-4.9.jar; F:\repo\net\sf\py4j\py4j\0.9\py4j-0.9.jar; F:\repo\org\spark-project\spark\unused\1.0.0\unused-1.0.0.jar; F:\repo\org\scala-lang\scala-library\2.10.5\scala-library-2.10.5.jar; C:\Program Files (x86) \jetbrains\intellij idea 2016.2.5\lib\idea_rt.jar "Com.intellij.rt.execution.application.AppMain com.scalatest.Test2Exception in thread "main" Java.lang.NoSuchMethodError:scala. predef$. $conforms () lscala/predef$ $less $colon$less;3At Org.apache.spark.util.utils$.getsystemproperties (utils.scala:1546)4At Org.apache.spark.sparkconf.<init> (sparkconf.scala:59)5At Org.apache.spark.sparkconf.<init> (sparkconf.scala:53)6At Com.scalatest.test$.main (test.scala:12)7 At Com.scalatest.Test.main (Test.scala)8 At sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)9At Sun.reflect.NativeMethodAccessorImpl.invoke (nativemethodaccessorimpl.java:57)TenAt Sun.reflect.DelegatingMethodAccessorImpl.invoke (delegatingmethodaccessorimpl.java:43) OneAt Java.lang.reflect.Method.invoke (method.java:606) AAt Com.intellij.rt.execution.application.AppMain.main (appmain.java:147)

Change the Scala version to 2.11.0, run again, no more errors. At this point, the Idea+maven-based Spark infrastructure development environment, even if the basic build is complete

Next time, explore the file read implementation of Spark, which draws on many of the basic classes of Hadoop

Spark's Workcount

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.