At the same time, two SECURECRT terminal interfaces have been opened, one of which has entered the Spark-shell and executed in another SECURECRT interface.
"Spark-shell--master yarn--deploy-mode client < Cha02.scala" times wrong, as follows:
[[email protected] test_code]# spark-shell--master yarn--deploy-mode client <cha02.scala
Setting Default log level to "WARN".
To adjust logging level use Sc.setloglevel (Newlevel). For Sparkr, use Setloglevel (Newlevel).
17/05/07 20:41:35 WARN util. nativecodeloader:unable to load Native-hadoop library for your platform ... using Builtin-java classes where APPLICABLE
17/05/07 20:41:37 WARN component. abstractlifecycle:failed [email protected]{http/1.1}{0.0.0.0:4040}: Java.net.BindException:Address already in Use
at sun.nio.ch.Net.bind0 (Native Method)
at Sun.nio.ch.Net.bind (net.java:433)
at Sun.nio.ch.Net.bind ( net.java:425)
17/05/07 21:00:46 WARN util. Utils:service ' Sparkui ' could not bind on port 4040. Attempting port 4041.
17/05/07 21:00:59 WARN yarn. Client:neither Spark.yarn.jars nor Spark.yarn.archive is set, falling back to uploading libraries under spark_home.
17/05/07 21:01:37 WARN Metastore. objectstore:failed to get Database global_temp, returning nosuchobjectexception
Spark context Web UI available at http://192.168.1.200:4041
Spark context available as ' sc ' (master = yarn, App id = application_1494142860645_0005).
Spark session available as ' spark '.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \ _/_ '/__/' _/
/___/. __/\_,_/_//_/\_\ version 2.1.0
/_/
Using Scala version 2.11.8 (Java HotSpot (TM) 64-bit Server VM, Java 1.8.0_121)
Type in expressions to has them evaluated.
Type:help for more information.
Scala>
This is because when you start a spark context, Sparkui uses 4040 ports by default, and when 4040 ports are occupied, you try to use a different port, the one in the log
Service ' Sparkui ' could not bind on port 4040. Attempting port 4041.
Spark-shell start warn---java.net.BindException:Address already in use