Open idea under the SRC under main under Scala right click to create a Scala class named Simpleapp, the content is as followsOrg.apache.spark.SparkContext org.apache.spark.sparkcontext._ org.apache.spark.SparkConf"a"). Count () numbs = logdata.filter (line = Line.contains ("B")). Count () println ("Lines with a:%s, Lines with B:%s". Format (Numas, numbs))}}
Packaging files:File-->>projectstructure-click artificats-->> click the Green Plus-click jar-->> Select from module with Depe
1. After installing Spark, enter spark in the bin directory: Bin/spark-shell scala> val textfile = Sc.textfile ("/users/admin/spark/ Spark-1.6.1-bin-hadoop2.6/readme.md ") scala> Textfile.flatmap (_.split (" ")). Filter (!_.isempt
1, first download the image to local. https://hub.docker.com/r/gettyimages/spark/~$ Docker Pull Gettyimages/spark2, download from https://github.com/gettyimages/docker-spark/blob/master/docker-compose.yml to support the spark cluster DOCKER-COMPOSE.YML fileStart it$ docker-compose Up$ docker-compose UpCreating spark_master_1Creating spark_worker_1Attaching to Sp
Step 2: Use the spark cache mechanism to observe the Efficiency Improvement
Based on the above content, we are executing the following statement:
650) This. width = 650; "src =" http://s3.51cto.com/wyfs02/M00/49/AF/wKioL1QY8tmiGO95AAG6MKKe5vI885.jpg "style =" float: none; "Title =" 1.png" alt = "wkiol1qy8tmigo95aag6mkke5vi885.jpg"/>
650) This. width = 650; "src =" http://s3.51cto.com/wyfs02/M00/49/AD/wKiom1QY8sLjnB_KAAHXbDhuD_I646.jpg "style =" float
Step 1: Test spark through spark Shell
Step 1:Start the spark cluster. This is very detailed in the third part. After the spark cluster is started, webui is as follows:
Step 2: Start spark shell:
In this case, you can view the shell in the following Web console:
S
Step 2: Use the spark cache mechanism to observe the Efficiency Improvement
Based on the above content, we are executing the following statement:
It is found that the same calculation result is 15.
In this case, go to the Web console:
The console clearly shows that we performed the "count" Operation twice.
Now we will execute the "Sparks" variable for the "cache" Operation:
Run the Count operation to view the Web console:
At this tim
Step 2: Use the spark cache mechanism to observe the Efficiency Improvement
Based on the above content, we are executing the following statement:
It is found that the same calculation result is 15.
In this case, go to the Web console:
The console clearly shows that we performed the "count" Operation twice.
Now we will execute the "Sparks" variable for the "cache" Operation:
Run the Count operation to view the Web console:
At this time, we found
Quick StartThis article describes how to use scala, java, and python to compile a spark click Mode Program. First, you only need to successfully build Spark on a machine. Practice: Enter the Spark root directory and enter the command: $ sbt/sbt package(Because of the Great Firewall of tianchao, the mainland China cannot succeed, unless you can smoothly flip the w
it.
External Datasets:
javardd
Spark can create distributed datasets (distributed datasets) from any storage source supported by Hadoop, including local file systems, Hdfs,cassandra,hbase,amazon S3, and so on. Spark supports text files, sequencefiles, and any other Hadoop inputformat.
You can use the Sparkcontext Textfile method to create an RDD for a text fil
Label:This article explains the structured data processing of spark, including: Spark SQL, DataFrame, DataSet, and Spark SQL services. This article focuses on the structured data processing of the spark 1.6.x, but because of the rapid development of spark (the writing time o
Step 5: test the spark IDE development environment
The following error message is displayed when we directly select sparkpi and run it:
The prompt shows that the master machine running spark cannot be found.
In this case, you need to configure the sparkpi execution environment:
Select Edit configurations to go to the configuration page:
In program arguments, enter "local ":
This configuration i
Next package, use Project structure's artifacts:Using the From modules with dependencies:Select Main Class:Click "OK":Change the name to Sparkdemojar:Because Scala and spark are installed on each machine, you can delete both Scala and spark-related jar files:Next Build:Select "Build Artifacts":The rest of the operation is to upload the jar package to the server, and then execute the
Reason:Running the spark code with the root userWorkaround: Run spark with a non-administrator account[[Email protected] Bin]$./Add-User.ShWhatType of userDoYou wish to add?A) Management User (Mgmt-Users.Properties)B) Application User (Application-Users.Properties)(A):BEnterThe details of theNewUser to add.Realm (Applicationrealm) : Applicationrealm ---->> Careful Here . YouNeed to typeThisor leave it blank
RDD into a new rdd via the transform operator Fliter, etc., triggering the spark submission job through the action operator. If the data needs to be reused, the data can be cached to memory through the cache operator.• Output: Program run end data outputs spark runtime space, stored in distributed storage (e.g. Saveastextfile output to HDFs) or Scala data or collections (collect output to Scala collection,
Spark Communication Module
1, Spark Cluster Manager can have local, standalone, mesos, yarn and other deployment methods, in order to
Centralized communication mode
1, RPC remote produce call
Spark Communication mechanism:
The advantages and characteristics of Akka are as follows:
1, parallel and distributed: Akka in design with asynchronous communication and dis
= Sc.parallelize (Array (1 to 10)) splits multiple slice based on the number of executor that can be started, and each slice initiates a task for processing.
Val Rdd = Sc.parallelize (Array (1 to 10), 5) specifies the number of partition
(2). Hadoop Data Set
Spark can convert any of the storage resources supported by Hadoop into an rdd, such as a local file (requiring a network file system, all nodes must be accessible), HDFS, Cassandra, HBase, Amaz
/49/D5/wKioL1QbpNKDWXo_AAElnZLjV4U229.jpg "style =" float: none; "Title =" 14.png" alt = "wkiol1qbpnkdwxo_aaelnzljv4u229.jpg"/>
Select "yes" to enable automatic installation of scala plug-in idea.
650) This. width = 650; "src =" http://s3.51cto.com/wyfs02/M00/49/D3/wKiom1QbpLijqttNAAE3LTevJ5I077.jpg "style =" float: none; "Title =" 15.png" alt = "wkiom1qbplijqttnaae3ltevj5i077.jpg"/>
In this case, it takes about 2 minutes to download and install the SDK. Of course, the download time varies depen
; "src =" http://s3.51cto.com/wyfs02/M02/4A/13/wKioL1QiJJPzxOm0AAFxk_FS8AU762.jpg "style =" float: none; "Title =" 51.png" alt = "wkiol1qijjpzxom0aafxk_fs8au762.jpg"/>
We found that we fully used the new background and correctly ran the program, which is much faster than the first operation.
This article is from the spark Asia Pacific Research Institute blog, please be sure to keep this source http://rockyspark.blog.51cto.com/2229525/1557591
[
Step 1: software required by the spark cluster;
Build a spark cluster on the basis of the hadoop cluster built from scratch in Articles 1 and 2. We will use the spark 1.0.0 version released in May 30, 2014, that is, the latest version of spark, to build a spark Cluster Based
The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion;
products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the
content of the page makes you feel confusing, please write us an email, we will handle the problem
within 5 days after receiving your email.
If you find any instances of plagiarism from the community, please send an email to:
info-contact@alibabacloud.com
and provide relevant evidence. A staff member will contact you within 5 working days.