Local boot Spark-shell

Because spark-1.3 as a milestone release, add a lot of features, so, it is necessary to study a, spark-1.3 need to scala-2.10.x version support, and the system on the default Scala version of 2.9, need to upgrade, You can refer to Ubuntu to install the 2.10.x version of Scala. After configuring the Scala environment, download the CDH version of Spark and click I download.

Once downloaded, unzip directly and then run directly in the bin directory./spark-shell:

The logs are as follows:

[Email protected]:~/spark-evn/spark-1.3.0-bin-cdh4/bin$./spark-Shell Spark Assembly have been built with Hive, including DataNucleus jars on Classpathlog4j:warn No appenders could be Found forLogger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory). Log4j:warn Please initialize the log4j system Properly.log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.Using Spark's default log4j profile:org/apache/spark/log4j-defaults.properties the/Geneva/ - xx:Geneva: -INFO securitymanager:changing View ACLs To:zhangchao3 the/Geneva/ - xx:Geneva: -INFO securitymanager:changing Modify ACLs To:zhangchao3 the/Geneva/ - xx:Geneva: -INFO SecurityManager:SecurityManager:authentication Disabled; UI ACLs disabled; Users with view Permissions:set (Z HANGCHAO3); Users with Modify Permissions:set (ZHANGCHAO3) the/Geneva/ - xx:Geneva: -INFO httpserver:starting HTTP Server the/Geneva/ - xx:Geneva: -INFO server:jetty-8. y.z-SNAPSHOT the/Geneva/ - xx:Geneva: -INFO abstractconnector:started [email protected]0.0.0.0:45918 the/Geneva/ - xx:Geneva: -INFO Utils:successfully started service'HTTP Class Server'On port45918. Welcome to ____ __/ __/__  ___ _____/ /__    _\ \/ _ \/ _ `/ __/'_//___/. __/\_,_/_//_/\_\ Version1.3.0/_/Using Scala Version2.10.4(OpenJDK --bit Server VM, Java1.7. 0_75) Typeinchexpressions to has them evaluated. Type:help for  Moreinformation. the/Geneva/ - xx:Geneva: -WARN Utils:yourhostname, hadoop01 resolves to a loopback address:127.0.1.1; Using172.18.147.71instead (on interface em1) the/Geneva/ - xx:Geneva: -WARN Utils:set Spark_local_ipifYou need to bind to another address the/Geneva/ - xx:Geneva: -INFO sparkcontext:running Spark Version1.3.0 the/Geneva/ - xx:Geneva: -INFO securitymanager:changing View ACLs To:zhangchao3 the/Geneva/ - xx:Geneva: -INFO securitymanager:changing Modify ACLs To:zhangchao3 the/Geneva/ - xx:Geneva: -INFO SecurityManager:SecurityManager:authentication Disabled; UI ACLs disabled; Users with view Permissions:set (en ANGCHAO3); Users with Modify Permissions:set (ZHANGCHAO3) the/Geneva/ - xx:Geneva: -INFO Slf4jlogger:slf4jlogger started the/Geneva/ - xx:Geneva: -INFO remoting:starting Remoting the/Geneva/ - xx:Geneva: -INFO Remoting:remoting started; Listening on addresses: [AKKA.TCP://[email protected]:51629] the/Geneva/ - xx:Geneva: -INFO Utils:successfully started service'Sparkdriver'On port51629. the/Geneva/ - xx:Geneva: -INFO sparkenv:registering Mapoutputtracker the/Geneva/ - xx:Geneva: -INFO sparkenv:registering Blockmanagermaster the/Geneva/ - xx:Geneva: -INFO diskblockmanager:created Local Directory at/tmp/spark-d398c8f3-6345-41f9-a712-36cad4a45e67/blockmgr-255070a6-19a9-49a5-a117-e4e8733c250a the/Geneva/ - xx:Geneva: -INFO Memorystore:memorystore started with capacity265.4MB the/Geneva/ - xx:Geneva: -INFO httpfileserver:http File Server directory is/tmp/spark-296eb142-92fc-46e9-bea8-f6065aa8f49d/ httpd-4d6e4295-dd96-48bc-84b8-c26815a9364f the/Geneva/ - xx:Geneva: -INFO httpserver:starting HTTP Server the/Geneva/ - xx:Geneva: -INFO server:jetty-8. y.z-SNAPSHOT the/Geneva/ - xx:Geneva: -INFO abstractconnector:started [email protected]0.0.0.0:56529 the/Geneva/ - xx:Geneva: -INFO Utils:successfully started service'HTTP File Server'On port56529. the/Geneva/ - xx:Geneva: -INFO sparkenv:registering outputcommitcoordinator the/Geneva/ - xx:Geneva: -INFO server:jetty-8. y.z-SNAPSHOT the/Geneva/ - xx:Geneva: -INFO abstractconnector:started [email protected]0.0.0.0:4040 the/Geneva/ - xx:Geneva: -INFO Utils:successfully started service'Sparkui'On port4040. the/Geneva/ - xx:Geneva: -INFO sparkui:started Sparkui at http://172.18.147.71:4040 the/Geneva/ - xx:Geneva: -INFO executor:starting Executor ID <driver>On host localhost the/Geneva/ - xx:Geneva: -INFO executor:using REPL class Uri:http://172.18.147.71:45918 the/Geneva/ - xx:Geneva: -INFO akkautils:connecting to HeartbeatReceiver:akka.tcp://[Email Protected]:51629/user/heartbeatreceiver the/Geneva/ - xx:Geneva: -INFO Nettyblocktransferservice:server created on55429 the/Geneva/ - xx:Geneva: -INFO blockmanagermaster:trying to register Blockmanager the/Geneva/ - xx:Geneva: -INFO blockmanagermasteractor:registering block manager localhost:55429With265.4MB RAM, Blockmanagerid (<driver>, localhost,55429) the/Geneva/ - xx:Geneva: -INFO blockmanagermaster:registered Blockmanager the/Geneva/ - xx:Geneva: theINFO sparkiloop:created Spark context. Spark context available as SC. the/Geneva/ - xx:Geneva: theINFO sparkiloop:created SQL context (with Hive support): SQL context available as Sqlcontext.scala>

http://172.18.147.71:4040/jobs/, you can see the spark running:

Local boot Spark-shell