Today, nothing to do, so the basic operation of HDFs with Java to write a simplified program to give you some small help! PackageCom.quanttech;Importorg.apache.hadoop.conf.Configuration;ImportOrg.apache.hadoop.fs.FileSystem;ImportOrg.apache.hadoop.fs.Path;/*** @topic HDFs file Operation Tool class *@authorZhouj **/ Public classHdfsutils {/** Determine if the HDFs
file.2.NameNode Storage block number is limited, a block meta-information consumes about four bytes of memory, store 100 million blocks, about 20GB of memory, if a file size of 10K, The 100 million file size is only 1TB (but consumes NAMENODE20GB memory)Six, HDFs access modeHDFS shell command;HDFS Java API;HDFS REST API;HD
Hadoop distributed FileSystem (Hadoop Distributed File System, HDFS)A distributed File system is a file system that consents to file sharing on multiple hosts over a network. Allows multiple users on multiple machines to share files and storage space.HDFs is just one of them. applies to the case of one write, multiple queries. Concurrent write scenarios are not supported. Small files are not appropriate. 2.HDFS
HDFS FederationNamenode saves the reference relationship for each file in the file system and each block of data in memory, which means that for an oversized cluster with a large number of files, memory becomes the bottleneck that limits the scale of the system. The Federation HDFS introduced in the 2.0 release series allowsThe system is extended by adding namenode, where each namenode manages a portion of
Need to prepare 2 filesApache-tomcat-5.5.25.zip (recommended to use TOMCAT6)Hdfs-webdav.war Unzip Tomcat# Unzip Apache-tomcat-5.5.25.zip Copy War to WebApps# CD apache-tomcat-5.5.25# Cp/soft/hdfs-webdav.war./webapps Start Tomcat to start deployment and unzip# CD Bin# chmod 777 Startup.sh#./startup.sh # CD./hdfs-webdav/linux_mount_lib # TAR-XZVF Neon-0.28.3.tar.gz
HDFS file system provides an API for an abstract File System Based on hadoop, which supports stream-based access to data in the file system.Features:1. Support for ultra-large files2. Detect and quickly respond to hardware faults (fault detection and Automatic Recovery)3. Streaming Data Access focuses on data throughput rather than data response speed4. Simplified consistency model with one write and multiple reads.Not Suitable:5. Low-latency data acc
Install times wrong: Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (site) on project Hadoop-hdfs:an Ant B Uildexception has occured:input file/usr/local/hadoop-2.6.0-stable/hadoop-2.6.0-src/hadoop-hdfs-project/ Hadoop-hdfs/target/findbugsxml.xml does not existThe reason for the mistake has been written very clearly.Workaround: Remove Docs, execute: mvn package-pdist,native-dski
Using Anaconda to install Python HDFs package Python-hdfs 2.1.0 PackageFrom HDFs Import *Import timeClient = Client ("http://192.168.56.101:50070")ll = client.list ('/home/test ', status=true)For I in LL: table_name = i[0] #表名 table_attr = i[1] #表的属性 #修改时间1528353247347, 13 bits to milliseconds, need to be converted to a timestamp of 10 bits to seconds (f
Sudo addgroup hadoop # Add a hadoop GroupSudo usermod-a-g hadoop Larry # Add the current user to the hadoop GroupSudo gedit ETC/sudoers # Add the hadoop group to sudoerHadoop all = (all) All after root all = (all) All
Modify hadoop Directory PermissionsSudo chown-r Larry: hadoop/home/Larry/hadoop
Sudo chmod-r 755/home/Larry/hadoop
Modify HDFS PermissionsSudo bin/hadoop DFS-chmod-r 755/Sudo bin/hadoop DFS-ls/
Modify the
Add a Hadoop group
sudo addgroup Hadoop
Add the current user Larry to the Hadoop groupsudo usermod-a-G Hadoop Larry
Add Hadoop Group to Sudoersudo gedit etc/sudoersHadoop all= (All) after Root all= (all)
Modify the permissions for the Hadoop directorysudo chown-r larry:hadoop/home/larry/hadoop
Modify permissions for HDFssudo chmod-r 755/home/larry/hadoopsudo bin/hadoop dfs-chmod-r 755/sudo bin/hadoop dfs-ls/
Modify the owner of the HDFs filesudo bin/
Not much to say, directly on the code.CodePackage zhouls.bigdata.myWholeHadoop.HDFS.hdfs7;Import java.io.IOException;Import Java.net.URI;Import java.net.URISyntaxException;Import org.apache.hadoop.conf.Configuration;Import Org.apache.hadoop.fs.FSDataInputStream;Import Org.apache.hadoop.fs.FSDataOutputStream;Import Org.apache.hadoop.fs.FileStatus;Import Org.apache.hadoop.fs.FileSystem;Import Org.apache.hadoop.fs.FileUtil;Import Org.apache.hadoop.fs.Path;Import Org.apache.hadoop.fs.PathFilter;Impo
HA-Federation-HDFS + Yarn cluster deployment mode
After an afternoon's attempt, I finally set up the cluster, and it didn't feel much necessary to complete the setup. So I should study it and lay the foundation for building the real environment.
The following is a cluster deployment of Ha-Federation-hdfs + Yarn.
First, let's talk about my Configuration:
The four nodes are started respectively:
1. bkjia117:
The term Federation was the first company to use the DB2 federal database.
First generation Hadoop HDFS:
The structure consists of a namenode and multiple datanode.
The functions are divided into namespace and block storage service.
HDFS Federation involves multiple namenode (or namespace ).
Here we have the concept of block pool. Each namespace has a pool. datanodes stores all the pools in the cluste
Requirements are as follows:Generate an approximately 100-byte text file on your local filesystem, write a program (which can take advantage of the Java API or C API), read the file, and write its 第101-120 byte content to HDFs as a new file.ImportJava.io.File;ImportJava.io.FileOutputStream;Importjava.io.IOException;ImportJava.io.OutputStream; Public classShengchen { Public Static voidMain (string[] args)throwsIOException {//TODO auto-generated Method
1. Problem: When the input of a mapreduce program is a lot of mapreduce output, since input defaults to only one path, these files need to be merged into a single file. This function copymerge is provided in Hadoop.
The function is implemented as follows:
public void Copymerge (string folder, string file) {
path src = new Path (folder);
Path DST = new path (file);
Configuration conf = new configuration ();
try {
Fileutil.copymerge (src.getfilesystem (conf), SRC,
dst.getfilesys
contentsHadoop Fs-tail/user/trunk/test.txt #查看 The last 1000 lines of the/user/trunk/test.txt fileHadoop fs-rm/user/trunk/test.txt #删除/user/trunk/test.txt fileHadoop fs-help ls #查看ls命令的帮助文档Two HDFS deployment The main steps are as follows:1. Configure the installation environment for Hadoop;2. Configure the configuration file for Hadoop;3. Start the HDFs service;4. Verify that the
The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion;
products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the
content of the page makes you feel confusing, please write us an email, we will handle the problem
within 5 days after receiving your email.
If you find any instances of plagiarism from the community, please send an email to:
info-contact@alibabacloud.com
and provide relevant evidence. A staff member will contact you within 5 working days.