Example of configuring Rhadoop and running WordCount

1. Install R language Environment

Su-c ' RPM-UVH http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm '

Su-c ' yum install foo '

Yum List r-\*

Yum Install R

2. Install Rstudio Desktop and server

Desktop is the RPM package, double-click Execute

Server installation command:

Yum Install openssl098e # Required only for Redhat/centos 6 and 7

wget http://download2.rstudio.org/rstudio-server-0.98.1091-x86_64.rpm

Yum Install--nogpgcheck rstudio-server-0.98.1091-x86_64.rpm

Add R-user User

3. Install GCC, git, pkg-config

Yum install gcc git pkg-config

4, Installation thrift0.9.0

Yum Install Automake Libtool Flex Bison pkgconfig gcc-c++ boost-devel libevent-devel zlib-devel python-devel ruby-devel

To compile the installation steps:

Update the System

???? Yum-y Update

Install The Platform development Tools

???? Yum-y Groupinstall "Development Tools"

Upgrade Autoconf/automake/bison

???? Yum install-y wget

Upgrade autoconf

???? wget http://ftp.gnu.org/gnu/autoconf/autoconf-2.69.tar.gz

???? Tar xvf autoconf-2.69.tar.gz

???? CD autoconf-2.69

????. /configure--PREFIX=/USR

???? Make

???? Make install

Upgrade Automake

???? wget http://ftp.gnu.org/gnu/automake/automake-1.14.tar.gz

???? Tar xvf automake-1.14.tar.gz

???? CD automake-1.14

????. /configure--PREFIX=/USR

???? Make

???? Make install

Upgrade Bison

???? wget http://ftp.gnu.org/gnu/bison/bison-2.5.1.tar.gz

???? Tar xvf bison-2.5.1.tar.gz

???? CD bison-2.5.1

????. /configure--PREFIX=/USR

???? Make

???? Make install

Install C + + Lib Dependencies

???? Yum-y Install Libevent-devel zlib-devel openssl-devel

Upgrade Boost

???? wget http://sourceforge.net/projects/boost/files/boost/1.55.0/boost_1_55_0.tar.gz

???? Tar xvf boost_1_55_0.tar.gz

???? CD Boost_1_55_0

????. /bootstrap.sh

????. /B2 Install

Build and Install the Apache Thrift IDL Compiler

???? git clone https://git-wip-us.apache.org/repos/asf/thrift.git

???? CD Thrift

????. /bootstrap.sh

????. /configure--with-lua=no

???? Modify/THRIFT-0.9.1/LIB/CPP/THRIFT.PC's Includedir=${prefix}/include/thrift

???? Make

???? Make install

Update Pkg_config_path:

???? Export Pkg_config_path= $PKG _config_path:/usr/local/lib/pkgconfig/

Verifiy pkg-config path is correct:

???? Pkg-config--cflags Thrift

???? Returns

???? -i/usr/local/include/thrift

Copy files to Lib folder

???? cp/usr/local/lib/libthrift-1.0.0-dev.so/usr/lib/

5. Setting Linux Environment variables

Export Hadoop_prefix=/usr/lib/hadoop

Export Hadoop_cmd=/usr/lib/hadoop/bin/hadoop

Export Hadoop_streaming=/usr/lib/hadoop-mapreduce/hadoop-streaming.jar

6, the root user to open the R Environment installation dependency package

Install.packages (C ("Rjava", "Rcpp", "Rjsonio", "Bitops", "Digest",

????????????????????" Functional "," Stringr "," Plyr "," Reshape2 "," Dplyr ",

????????????????????" R.methodss3 "," Catools "," Hmisc "," data.table "," memoise "))

7, the root user to open the R environment installation Rhadoop package

Install.packages ("/root/rhadoop/rhdfs_1.0.8.tar.gz", Repos=null, type= "source")

Install.packages ("/root/rhadoop/rmr2_3.3.0.tar.gz", Repos=null, type= "source")

Install.packages ("/root/rhadoop/plyrmr_0.5.0.tar.gz", Repos=null, type= "source")

Install.packages ("/root/rhadoop/rhbase_1.2.1.tar.gz", Repos=null, type= "source")

8. Configuring Ant and Maven

Export maven_home=/root/apache-maven-3.2.5

Export Path=/root/apache-maven-3.2.5/bin: $PATH

Export ant_home=/root/apache-ant-1.9.4

Export path= $ANT _home/bin: $PATH

9. Test Rhadoop

Sys.setenv ("Hadoop_prefix" = "/usr/lib/hadoop")

Sys.setenv ("hadoop_cmd" = "/usr/lib/hadoop/bin/hadoop")

Sys.setenv ("hadoop_streaming" = "/usr/lib/hadoop-mapreduce/hadoop-streaming.jar")

?

???? Library (RMR2)

???? bp = rmr.options ("Backend.parameters")

???? Trans <-list (d= "mapreduce.map.java.opts=-xmx400m",

???????????????? D= "mapreduce.reduce.java.opts=-xmx400m",

???????????????? D= "mapreduce.map.memory.mb=4096",

???????????????? D= "mapreduce.reduce.memory.mb=4096",

???????????????? D= "mapreduce.task.io.sort.mb=100")

???? bp <-list (Hadoop=trans)

???? # # # # # # no code started #######################

???? bp$hadoop[1]= "mapreduce.map.java.opts=-xmx400m"

???? bp$hadoop[2]= "mapreduce.reduce.java.opts=-xmx400m"

???? bp$hadoop[3]= "mapreduce.map.memory.mb=1024"

???? bp$hadoop[4]= "mapreduce.reduce.memory.mb=2048"

???? bp$hadoop[5]= "Mapreduce.task.io.sort.mb=100"

???? # # # # no end of code used #######################

???? Rmr.options (backend.parameters = BP)

???? Rmr.options ("Backend.parameters")

???? # # Map function

???? Map <-Function (k,lines) {

???????? Words.list <-Strsplit (lines, ' \\s ')

???????? Words <-unlist (words.list)

???????? Return (Keyval (words, 1))

????}

???? # # Reduce function

???? Reduce <-function (Word, counts) {

???????? Keyval (Word, sum (counts))

????}

???? WordCount <-function (input, output=null) {

???????? MapReduce (Input=input, Output=output, input.format= "text",

Map=map, Reduce=reduce)

????}

?

???? # # Delete previous result if any

???? System ("/usr/lib/hadoop/bin/hadoop fs-rm-r/tmp/zhengcong/out")

????

???? # # Submit Job

???? Hdfs.root <-'/tmp/zhengcong '

???? Hdfs.data <-File.path (hdfs.root, ' HP ')

???? Hdfs.out <-File.path (hdfs.root, ' out ')

???? Out <-WordCount (Hdfs.data, Hdfs.out)

?

???? # # Fetch results from HDFS

???? Results <-From.dfs (out)

?

???? # # Check Top frequent words

???? RESULTS.DF <-As.data.frame (results, stringsasfactors=f)

???? Colnames (RESULTS.DF) <-C (' word ', ' count ')

???? Head (Results.df[order (Results.df$count, decreasing=t),], 30)

?

10. Error resolution

???? Rjava cannot load, the root user runs R CMD javareconf-e

???? Add Export ld_library_path= $JAVA _home/lib/amd64: $JAVA _home/jre/lib/amd64/server

Example of configuring Rhadoop and running WordCount