Azure HDInsight and Spark Big Data Combat (ii)

HDInsight Cluster on Linux

Sign in to Azure portal (https://manage.windowsazure.com)

Click the NEW button in the lower left corner, then click the DATA SERVICES button, click HDINSIGHT, and select HADOOP on LINUX, as shown in.

Enter the cluster name, select the cluster size and account number, set the cluster password and storage account, the following table is the meaning of each parameter and configuration instructions.

Name	Value
Cluster Name	Name of the cluster.
cluster size	number of data nodes you want to deploy. The default value is 4. But the option to use 1 or 2 data nodes are also available from the Drop-down. Any number of cluster nodes can is specified by using the Custom create option. Pricing details on the billing rates for various cluster sizes is available. Click the ? symbol just above the Drop-down box and follow the link on the pop-up.
password	the password for the HTTP account (Default user name: Admin) and SSH account (Default User name:hdiuser). Note that these is not the administrator accounts for the virtual machines on which the clusters is provisioned.
Storage Account	Select the Storage account to created from the Drop-down box. Once a Storage account was chosen, it cannot be changed. If the Storage account is removed, the cluster would no longer be available for use. The HDInsight cluster is co-located in the same datacenter as the Storage account.

Clicking Create HDINSIGHT CLUSTER creates a Hadoop cluster running on Azure.

This process quickly creates a Linux cluster running Hadoop, and the default SSH user name is the Hdiuser,http account with the default name of admin. To create a cluster with custom options, such as using SSH keys for authentication, or to use additional storage space, see provision Hadoop Linux clusters in HDInsight using custom options (https://a zure.microsoft.com/en-us/documentation/articles/hdinsight-hadoop-provision-linux-clusters/).

Installing Spark

Click on the created Hadoop cluster in HDInsight (in this case the cluster name is Hadooponlinux) and go to dashboard as shown in.

Copy the value of Cluster Connection String in the quick glance , which is the address for logging in to the Hadoop on Linux configuration console ambari, paste Cluster in the browser the value of the Connection String at which the login user name and password are verified. The user name at this time is the default HTTP username admin When you quickly create a Hadoop cluster in the previous step, and the password is the one set when you quickly create a Hadoop cluster.

After the user name and password are entered correctly, Ambari login username and password Authentication appears, then enter user name Admin password for Hadoop to enter Ambari management console.

Shows the process of installing spark using Ambari.

The following diagram shows the Spark installation process using Ambari.

1. Select the Ambari "Services" tab.

In the Ambari "Actions" drop-down menu, select "Add Service." This launches the Add Services Wizard.

Select Spark, and then click Next.

(For HDP 2.2.4, Ambari would install Spark version 1.2.1, not 1.2.0.2.2.)

1. Ambari will display a warning message confirming that the cluster is running HDP 2.2.4 or later, and then click Proceed.

	Note
	You can reconfirm component versions in Step 6 before finalizing the upgrade.

1. Select the Spark History server node and click click "Next" to continue.

1. To specify Spark's slaves, click "Next" to continue.

2. In the custom service interface It is recommended that you use the default values for your initial configuration and then click "Next" to continue.
3. Ambari Display the confirmation screen, click "Deploy" to continue.

	Important
	On the Review screens, make sure all HDP is version 2.2.4 or later.

1. Ambari Displays the install, start, and test interfaces, with status bars and messages indicating progress.

2. When the Ambari installation is complete, click "Complete" to finish the whole Spark installation process.

Run Spark

To log in to Hadoop's Linux cluster via SSH, execute the following Linux instructions to download the document and run it for later spark programs.

wget Http://en.wikipedia.org/wiki/Hortonworks

Copy the data to HDFs in the Hadoop cluster,

Hadoop fs-put ~/hortonworks/user/guest/hortonworks

In many spark examples using Scala and Java application Demonstrations, this example uses Pyspark to demonstrate the use of the Python voice-based spark method.

Pyspark

The first step is to create an RDD using Spark Context, SC, as follows:

Mylines = Sc.textfile (' Hdfs://sandbox.hortonworks.com/user/guest/hortonworks ')

Now we instantiate the RDD, and here we do the conversion of the RDD. For this we use Python lambda expressions for filtering.

mylines_filtered = Mylines.filter (lambda x:len (x) > 0)

Note that the above Python statement does not cause any rdd to execute, and only the count () behavior of the following code will cause the true RDD operation to occur.

Mylines_filtered.count ()

The result of the final spark job operation is as follows.

341 .

Data Science with Spark

For data scientists, spark is a highly effective data processing tool. Data scientists often have tools like notebook (such as IPython http://ipython.org/notebook.html) to quickly create prototypes and share their work. Many data scientists prefer to use the R language, and it is gratifying that the integration of Spark and R-Sparkr has become the spark's emerging capabilities. Apache Zeppelin (https://zeppelin.incubator.apache.org/) is an emerging tool that provides Spark-based Notebook capabilities, which are available in Apache Zeppelin for Sp The user interface view of Ark.

Cedar

Microsoft MVP--Windows Platform development,

Hortonworks Certified Apache Hadoop 2.0 Developer

Azure HDInsight and Spark Big Data Combat (ii)