Spark-shell Start spark Error

Objective

　　After installing CDH and Coudera Manager offline, all of your own apps are installed through Coudera Manager, including HDFs, hive, yarn, Spark, hbase, and so on, and the process is a twist, so don't complain and go straight to the subject.

Describe

　　In the installation of Spark node, through the Spark-shell start Spark, full of anticipation of the start Spark,but, came a thunderbolt, error, Error! The error message is as follows:

18/06/11 17:40:27ERROR Spark. Sparkcontext:error initializing SparkContext.java.lang.IllegalArgumentException:Required executor memory (1024+384 MB) is above the max Threshold (1024x768) of Thiscluster! Please check the values of ' yarn.scheduler.maximum-allocation-mb ' and/or ' YARN.NODEMANAGER.RESOURCE.MEMORY-MB '. At Org.apache.spark.deploy.yarn.Client.verifyClusterResources (Client.scala:281) at Org.apache.spark.deploy.yarn.Client.submitApplication (Client.scala:140) at Org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start (Yarnclientschedulerbackend.scala: 57) at Org.apache.spark.scheduler.TaskSchedulerImpl.start (Taskschedulerimpl.scala:158) at Org.apache.spark.SparkContext.<init> (sparkcontext.scala:538) at Org.apache.spark.repl.SparkILoop.createSparkContext (Sparkiloop.scala:1022) at $line 3. $read $ $iwC $ $iwC.<init> (<console>:15) at $line 3. $read $ $iwC.<init> (<console>:25) at $line 3. $read.<init> (<console>:27) at $line 3. $read $.<init> (<console>:31) at $line 3. $read $.<clinit> (<console>) at $line 3. $eval $.<init> (<console>:7) at $line 3. $eval $.<clinit> (<console>) at $line 3. $eval. $print (<console>) at Sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method) at Sun.reflect.NativeMethodAccessorImpl.invoke (Nativemethodaccessorimpl.java:62) at Sun.reflect.DelegatingMethodAccessorImpl.invoke (Delegatingmethodaccessorimpl.java:43) at Java.lang.reflect.Method.invoke (Method.java:498) at Org.apache.spark.repl.sparkimain$readevalprint.call (Sparkimain.scala:1045) at Org.apache.spark.repl.sparkimain$request.loadandrun (Sparkimain.scala:1326) at org.apache.spark.repl.sparkimain.loadandrunreq$1 (sparkimain.scala:821) at Org.apache.spark.repl.SparkIMain.interpret (Sparkimain.scala:852) at Org.apache.spark.repl.SparkIMain.interpret (Sparkimain.scala:800) at org.apache.spark.repl.sparkiloop.reallyinterpret$1 (sparkiloop.scala:857).................... There's a lot of error messages back there .

Spark start error 1

After careful review of the error message found that the original yarn configuration is not enough memory, spark boot requires 1024+384 MB of memory, but my yarn configuration only a few megabytes, not enough to meet the spark boot requirements, so throw an exception, the key error message as shown:

Workaround

Log in to Cloudera Manager, find yarn (MR2 Included), click into (do not care about my cluster has so many warnings and errors, solve the spark problem is the key), as shown in:

Locate the configuration options in the navigation bar, as shown in:

Click on the go to Configuration page and enter YARN.SCHEDULER.MAXIMUM-ALLOCATION-MBin the search field as shown in:

As you can see, the value of this configuration parameter is as shown in the exception thrown at spark startup, 1GB, modified to 2GB, click Save Changes as shown in:

Follow the steps above to continue modifying the value of the yarn.nodemanager.resource.memory-mb parameter to 2GB, as shown in, click Save changes, restart yarn to make the settings effective.

Return to the Spark node command line inside Execute Spark-shell command, strange, still error, but wrong for other, no longer is the above error, error message for

18/06/11 17:46:46ERROR Spark. Sparkcontext:error initializing SparkContext.org.apache.hadoop.security.AccessControlException:Permission denied: User=root, Access=write, inode= "/user": hdfs:supergroup:drwxr-xr-x at Org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission ( Defaultauthorizationprovider.java:279) at Org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check ( Defaultauthorizationprovider.java:260) at Org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check ( Defaultauthorizationprovider.java:240) at Org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission ( Defaultauthorizationprovider.java:162) at Org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission (Fspermissionchecker.java: 152) at Org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission (Fsdirectory.java:3530) at Org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission (Fsdirectory.java:3513) at Org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess (Fsdirectory.java:3495) at Org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess (Fsnamesystem.java:6649) at Org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal (Fsnamesystem.java:4420) at Org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt (Fsnamesystem.java:4390) at Org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs (Fsnamesystem.java:4363)........................... There's a lot of things that aren't important back there.

Spark start error 2

The critical error message is as follows:

The reason is that the user right to start spark is not enough, I started spark with the root command, I need the HDFS user to start Spark (note: HDFs is the Super user of Hadoop), so error, switch to HDFs user, start Again is spark, success.

Add

　　YARN.SCHEDULER.MAXIMUM-ALLOCATION-MB Parameter: This parameter is configured in the Yarn-site.xml configuration file, setting the maximum allocated memory of the yarn container, in megabytes, if Yarn Explorer (rm/resourcemanager The container in the request has a resource greater than the value set here, and an invalid resource request exception (Invalidresourcerequestexception) is thrown.
　　YARN.NODEMANAGER.RESOURCE.MEMORY-MB Parameter: This parameter is configured in the Yarn-site.xml configuration file to set the physical memory available on the Yarn node, and the default size is 8192 (MB), which can be used to assign to yarn containers.

Spark-shell Start spark Error