Install Apache Spark

Spark can read and write data directly to HDFS and also supports Spark on YARN. Spark runs in the same cluster as MapReduce, shares storage resources and calculations, borrows Hive from the data warehouse Shark implementation, and is almost completely compatible with Hive. Spark's core concepts 1, Resilient Distributed Dataset (RDD) flexible distribution data set RDD is ...

This article describes how to deploy Apache to Hadoop 2.2.0, http://www.aliyun.com/zixun/aggregation/14417.html". If your Hadoop is another version, such as CDH4, you can refer directly to the official Explain the operation. Need to pay attention to two points: (1) the Hadoop must be 2.0 series, such as 0.23.x, 2.0.x, 2.xx or CDH4, CDH5 ...

Currently, the Hadoop distribution has an open source version of Apache and a Hortonworks distribution (HDP Hadoop), MapR Hadoop, and so on. All of these distributions are based on Apache Hadoop.

1. Kyoto Buffer protocal Buffer is a library of Google Open source for data interchange, often used for cross-language data access, and the role is generally serialized/deserialized for objects. Another similar open source software is Facebook open source Thrift, their two biggest difference is that thrift provides the function of automatically generating RPC and protocal buffer needs to implement itself, but protocal buffer one advantage is its preface ...

The operating language of the data is SQL, so many tools are developed with the goal of being able to use SQL on Hadoop. Some of these tools are simply packaged on top of the MapReduce, while others implement a complete data warehouse on top of the HDFs, while others are somewhere between the two. There are a lot of such tools, Matthew Rathbone, a software development engineer from Shoutlet, recently published an article outlining some common tools and scenarios for each tool and not ...

The Docker has been in use for more than a year and has been installed on both local Linux systems and cloud platforms. At that time learned a lot about managing mirroring, the flexibility to create mirrors for any platform, learn to write some of your own programs that are not relevant to Docker. I've tried to summarize my experience to the following five points, providing references for those who just started using Docker. When making a mirror you need to be particularly clear I try to run my application without the root user. Most Linux distributions have one advantage: when you install a service, the operating system ...

"Editor's note" Shopify is a provider of online shop solutions company, the number of shops currently serving more than 100,000 (Tesla is also its users). The main frame of the website is Ruby on rails,1700 kernel and 6TB RAM, which can respond to 8,000 user requests per second. In order to expand and manage the business more easily, Shopify began to use Docker and CoreOS technology, Shopify software engineer Graeme Johnson will write a series of articles to share their experience, this article is the department ...

The "Editor's note" Container technology is a lightweight virtualization technology that allows you to virtualize a stand-alone system's operating environment with fine-grained granularity. This means that a physical server, or a virtual server above a cloud, can run hundreds of separate containers at the same time. Based on the distinctive features of "fast" in the gaming industry, every business logic, such as the control of character walking, fighting, scene, chatting, archiving, database, etc., can be run in a separate container. This series of containers constitutes the autonomous system of an independent gaming world. ...