Greenplum + Hadoop learning notes-11-distributed database storage and query processing, hadoop-11-
3. 1. Distributed Storage Greenplum is a distributed database system. Therefore, all its business data is physically stored in the database of all Segment instances in the cluster. In the Greenplum database, all tables are distributed, therefore, each table is sliced, and each Segment instance database stores
Label:Workaround:Change to the following:Directory/usr/local/hadoop/tmp/tmp/hadoop-root/dfs/name is in a inconsistent state:storage directory does not exist or is not accessible
Install and deploy Apache Hadoop 2.6.0
Note: For this document, refer to the official documentation for the original article.
1. hardware environment
There are three machines in total, all of which use the linux system. Java uses jdk1.6.0. The configuration is as follows:Hadoop1.example.com: 172.20.115.1 (NameNode)Hadoop2.example.com: 172.20.1152 (DataNode)Hadoop3.example.com: 172.115.20.3 (DataNode)Hadoop4.example.com: 172.20.115.4Correct resolution
Overview:
The file system (FS) shell contains commands for various classes of-shell, directly interacting with Hadoop Distributed File System (HDFS), and support for other file systems, such as: Local file system fs,hftp Fs,s3 FS, and others. Calls to the FS shell:
Bin/hadoop FS
All FS shell commands have URI paths as parameters, and the URI forma
Notes on Hadoop single-node pseudo-distribution Installation
Lab EnvironmentCentOS 6.XHadoop 2.6.0JDK 1.8.0 _ 65
PurposeThe purpose of this document is to help you quickly install and use Hadoop on a single machine so that you can understand the Hadoop Distributed File System (HDFS) and Map-Reduce framework, for example, run the sample program or simple job on H
High-availability Hadoop platform-Hadoop Scheduling for Oozie Workflow1. Overview
In the "high-availability Hadoop platform-Oozie Workflow" article, I will share with you how to integrate a single plug-in such as Oozie. Today, we will show you how to use Oozie to create related workflows for running and Hadoop. You mu
Several Hadoop daemon and Hadoop daemon
After Hadoop is installed, several processes will appear when jps is used.
Master has:
Namenode
SecondaryNameNode
JobTracker
Slaves has
Tasktracker
Datanode
1.NameNode
It is the master server in Hadoop, managing the file system namespace and accessing the files stored in the
resourcesMaster-Slave structureMaster node, there can be 2: ResourceManagerFrom the node, there are a number of: NodeManagerResourceManager is responsible for:Allocation and scheduling of cluster resourcesFor applications such as MapReduce, Storm, and Spark, the Applicationmaster interface must be implemented to be managed by RMNodeManager is responsible for:Management of single node resourcesVII: The architecture of MapReduceBatch computing model with disk IO dependentMaster-Slave structureMas
Big data: Massive dataStructured data: Data that can be stored in a two-dimensional tableunstructured data: Data cannot be represented using two-dimensional logic of the data. such as word,ppt, picture Semi-structured data: a self-describing, structured and unstructured data that stores the structure with the data itself: XML, JSON, HTMLGoole paper: mapreduce:simplified Date processing on Large Clusters Map: Small data that maps big data to multiple nodes that are segmented
First on the correct run display:Error 1: The variable is intwritable and is receiving longwritable, such as:Reason, write more parameters reporter, such as:Error 2: The array is out of bounds, such as:Cause: The Combine class is set up, such as:Error 3:nullpointerexception exception, such as:Cause: The static variable is null and can be assigned, such as:Error 4: Entering map, but unable to enter reduce, and direct map data output, and no error promptCause: The new and older version of
1 access to Apache Hadoop websitehttp://hadoop.apache.org/2.2. Click image to downloadWe download the 2.6.0 third in the stable version of stableLinux Download , here is an error, we download should be the bottom of the second, which I did not pay attention to download the above 17m .3. Install a Linux in the virtual machineFor details see other4. Installing the Hadoop environment in Linux1. Installing the
Tags: bit success tmp BASHRC Mon core [1] dpkg folderTo create a new user:
$sudo useradd-m hadoop-s/bin/bashTo set the user's password:$sudo passwd HadoopTo add Administrator privileges:$sudo adduser Hadoop sudo
Install SSH, configure SSH login without password:To install SSH Server:
$ sudo apt-get install Openssh-serverUse SSH to log in to this machine:$ ssh localhostLaunched Shh Loc
Hadoop is a distributed filesystem (Hadoop distributedfile system) HDFS. Hadoop is a large amount of data that can beDistributed Processingof theSoftwareFramework. Hadoop processes data in a reliable, efficient, and scalable way. Hadoop is reliable because it assumes that
Run Hadoop WordCount. jar in Linux.
Run Hadoop WordCount in Linux
Enter the shortcut key of Ubuntu terminal: ctrl + Alt + t
Hadoop launch command: start-all.sh
The normal execution results are as follows:
Hadoop @ HADOOP :~ $ Start-all.sh
Warning: $ HADOOP_HOME is deprecate
Read file:is the process by which HDFs reads files:Here is a detailed explanation:1. When the client begins to read a file, the client first obtains the Datanode information for the first few blocks of the file from Namenode. (steps)2. Start calling read (), the Read () method, first to read the first time from the Namenode to obtain a few blocks, when the read is completed, then go to Namenode take a block of datanode information. (Step 3,4,5)3. Call the Close method to complete the read. (Step
[Hadoop] 5. cloudera manager (3) and hadoopcloudera installed on HadoopInstall
Http://blog.sina.com.cn/s/blog_75262f0b0101aeuo.html
Before that, install all the files in the cm package
This is because CM depends on postgresql and requires postgresql to be installed on the local machine. If it is installed online, it is automatically installed in Yum mode. Because it is offline, postgresql cannot be installed automatically.
Check whether postgresql
1. By default, the Yarn log only displays info and above level information, and it is necessary to display the necessary debug information when the system is developed two times.
2. Configure yarn to print debug information to the log file, just modify its startup script sbin/yarn-daemon.sh, and change the info to debug (this step only).
Export Yarn_root_logger=${yarn_root_logger:-debug,rfa}
3. For HDFs, the modification method is similar, only need to modify the sbin/
Original articles, reproduced please mark from http://blog.csdn.net/lsttoy/article/details/53406840.First, go to Apache to see the official support version
You can see that hadoop2.4.x later versions basically support hbase1.2.4.The installation starts next.
The first step is to download the latest version from the Apache Foundation
Https://mirrors.tuna.tsinghua.edu.cn/apache/hbase/1.2.4/hbase-1.2.4-bin.tar.gz
If you can not go to csdn and other major sites to download.
Step two , unzip to the
Configuration
The following properties should is in the core-site.xml of all the nodes in the cluster.
Hadoop.http.filter.initializers:add to the Org.apache.hadoop.security.AuthenticationFilterInitializer Initializer class.
Hadoop.http.authentication.type:Defines authentication used for the HTTP web-consoles. The Supported values Are:simple | Kerberos | #AUTHENTICATION_HANDLER_CLASSNAME #. The Dfeault value is simple.
Hadoop.http.authentication.token.validity:Indicates how long (in s
The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion;
products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the
content of the page makes you feel confusing, please write us an email, we will handle the problem
within 5 days after receiving your email.
If you find any instances of plagiarism from the community, please send an email to:
info-contact@alibabacloud.com
and provide relevant evidence. A staff member will contact you within 5 working days.