Rstudio running Sparkr method under Windows environment (Local mode)

Source: Internet
Author: User

The related functions in Sparkr are used in Rstudio by loading the SPARKR package, but because everyone has different versions of Hadoop and Spark, it is necessary to compile the SPARKR package to accommodate the different versions. 1. Resource preparation      rtools,r,rstudio,sparkr-pkg-master source package, maven (used to compile source code)  2. Software Installation and source access     " Rtools: Install the Rtools and gcc-4.6.3 corresponding to the installation bin directory (such as: Rtools\bin, and Rtools\gcc-4.6.3\bin), into the system environment variable path. Enter GCC--help in the Command Prompt window and the appropriate information appears to install successfully.     maven: Download the MAVEN package and configure its bin to path. Command prompt input mvn-version can see the relevant information is configured successfully .    R and Rstudio installation     on GitHub's official website (https://github.com/ amplab-extras/sparkr-pkg) Download the zip package and unzip it.  3. Compile the Sparkr-pkg-master source code in the window environment so that it can be used by R.     Because the bat file is executed under the window System. So you need to make simple changes to the Sparkr-pkg-master install-dev.bat file:          record line R.exe CMD INSTALL--library= ". Lib "pkg\   in R.exe to the installed location on the local machine, point to the executable R.exe, such as:e:\stall_palce\r-3.3.0\r\bin\r.exe     " Sparkr-pkg-master By default corresponds to the Hadoop,spark version of 1.0.4 and 1.1.0, so when the version is inconsistent, you can modify the sparkr-pkg-master/pkg/ SRC directory in the BUILD.SBT file, change the corresponding version value to its own version: If Hadoop is changed to 2.6.0,spark 1.5.1 o'clock. Val defaulthadoopversion =  "2.6.0" val defaultsparkversion =  "1.5.1"
"Opens a command prompt, enters into the Sparkr-pkg-master directory, enters the Install-dev, it executes, and finally generates a new directory in the Sparkr-pkg-master directory Lib, which has the compiled Sparkr, Copy the compiled Sparkr to the library of R, and you can start the SPARKR process by importing the library (SPARKR) in Rstudio. (The process is a bit long and a bit patient). Simple WordCount Sample Test-"can refer to the blog http://www.cnblogs.com/hseagle/p/3998853.html Note: At compile time, the program will automatically go to the C drive to find the Gcc.exe under Rtools. Could not find the error that the compilation failed, but I do not know where to set the parameters, so the installation of Rtools copied to the C drive to solve the error.

Rstudio running Sparkr method under Windows environment (Local mode)

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.