Hadoop Learning for the first time: Hadoop concept

1. Big Data Learning Direction : One is the system construction technology, two, the massive data application.

First of all, the system construction, now the mainstream technology is Hadoop, mainly based on MapReduce distributed framework. You can learn this now. But my point is that before the distributed system comes out, it is mainly a centralized architecture, such as Db2,oracle. Why now with distributed architecture, that is because the centralized architecture is limited to IO performance, come out slow, if another hardware technology, can quickly deal with the massive data, performance can meet the demand, then the centralized architecture is superior to the distributed architecture, because the centralized architecture is stable, the operation and maintenance pressure is small. Today's centralized architectures are either performance-less or too expensive. I'm looking forward to a technology that can transfer and process data very quickly, and then the centralized architecture will come into the eye again. Again, huge data applications. Mass data applications are mainly data mining and machine algorithms. There are different application scenarios, such as personalized search and recommendation, social network discovery, precision marketing, precision advertising, real-time optimal path, artificial intelligence and so on. See if you want to do system support technology or a combination of applications with business.

2.hadoop Learning route:① Select a Hadoop version and read the documentation to learn about Hadoop:what's Hadoop, why Hadoop Exists;② installs Hadoop, three ways to try ③ the Hadoop command in the Hadoop document, I.hdfs command,ii.job command, try these two commands; ④hadoop files, look at the concept of Hadoop files, focus on its distributed features, Then take a look at the files of the reduce function output; ⑤ write WordCount and advanced Wordcount;⑥ write HDFs io, this example is in "Hadoop in action", and it's good to speak. such as Copy,sequencefile, ⑦ write sort program, ⑧ write Mrbench program (this online has many examples), understand mrbench is what; ⑨ use Randomtextwriter ; 10. Imitate Sequencefileinputformat, Sequencefileoutputformat, Sequencefilerecordreader write their own 11.yahoo has a Hadoop tutorial, the English version of the content is very good; 12. The "Hadoop authoritative guide" as a reference book, the actual combat.

Hadoop:

1. It's a distributed computing platform that allows users to easily architect and use it, where users can develop applications that run massive amounts of data. 2. The project and its structure:

(1). HDFS: Distributed File system good extensibility, fault tolerance.

Namenode:master manages the namespace of HDFs, manages block mapping information, configures replica policies, and handles client read and write requests

Datanode:slave stores actual blocks of data and performs actual block read and write operations

Client: File Segmentation (default size 64M), interacting with Namenode,datanode

Secondary Namenode: Auxiliary Namenode

(2) MapReduce: Distributed computing framework is easy to program, good scalability, high fault tolerance, suitable for off-line processing of petabytes of data

Jobtracker:master Manage all jobs, decompose and assign jobs to Tasktracker

Tasktrack:slave running map task and reduce task

(3) Hive Data Warehouse based on Hadoop, defines the query language hql of class SQL for offline processing of data

(4) Pig: Hadoop-based data stream processing system provides a AD-HOC data analysis tool based on MapReduce

(5) Mahout: Data Mining Algorithm Library

(6) HBase: Distributed Database

(7) ZooKeeper: Distributed writing Service

(8) Sqoop: Data synchronization tool connecting Hadoop to a traditional database, essentially a mapreduce program

(9) Flume: Log Collection Tool

Hadoop Learning for the first time: Hadoop concepts