cdh support

Use Windows Azure VM to install and configure CDH to build a Hadoop Cluster This document describes how to use Windows Azure virtual machines and NETWORKS to install CDH (Cloudera Distribution Including Apache Hadoop) to build a Hadoop cluster. The project uses CDH (Cloudera Distribution Including Apache Hadoop) in the private cloud to build a Hadoop cluster for

1> removing the UUID of the agent node# rm-rf/opt/cm-5.4.7/lib/cloudera-scm-agent/*2> emptying the master node cm databaseGo to the MySQL database of the master node, and then drop db cm;3> Removing Agent node Namenode and Datanode node information# rm-rf/opt/dfs/nn/*# rm-rf/opt/dfs/dn/*4> re-initializing the CM database on the primary node#/opt/cm-5.4.7/share/cmf/schema/scm_prepare_database.sh MySQL cm-hlocalhost-uroot-p123456--scm-host localhost SCM SCM SCM5> Execute startup scriptMaster node:

Recently, a Hadoop cluster was installed, so the HA,CDH4 that configured HDFS supported the quorum-based storage and shared storage using NFS two HA scenarios, while CDH5 only supported the first scenario, the Qjm ha scenario. About the installation deployment process for Hadoop clusters You can refer to the process of installing CDH Hadoop clusters using Yum or manually installing Hadoop clusters. Cluster Planning I have installed a total of three no

Encounter a problem, because the default is to install CDH System/var/log directory, because it is a virtual instance, the system disk smaller only 50G, is the use of the system cm will be alerted to the alarm log directory space is not enough, if the script is deleted periodically, although it can solve the current problem, but not a good way. The other is to directly modify the configuration file, all the/var/log/* manually changed to/home/var/log/*

1, it is in the installation of CDH can not be installed successfully, only restart, the following to share an artifact, according to this script should be almost able to uninstall clean, and then reinstall, write a script, the content is as follows, life-saving artifact:#!/bin/Bashsudo/usr/share/cmf/uninstall-cloudera-Manager.shsudo Service Cloudera-scm-Server Stopsudo Service Cloudera-scm-server-db Stopsudo Service Cloudera-scm-Agent Stopsudo Yum re

Original address: http://www.cnblogs.com/qiaoyihang/p/6995146.html1. Download Source: http://archive-primary.cloudera.com/cdh5/cdh/5/2. Prepare the Environment for compilationA. Install MavenB, Installation Protobuffer./configure--prefix=/usr/local/protobufNote the dependency packages required to install the compilationsudo yum install gcc-c++sudo yum-y install CMakesudo yum-y install zlibsudo yum-y install Openssl-develOld process:Config > Make >make

CDH: Full name Cloudera ' s distribution including Apache HadoopCDH version-derived Hadoop is an open source project, so many companies are commercializing this foundation, and Cloudera has made a corresponding change to Hadoop.Cloudera Company's release, we call this version CDH (Cloudera distribution Hadoop).　　So far, there are 5 versions of CDH, of which the f

Carbondata is a new type of tabular file format for distributed computing, this time using Spark-thrift mode to operate Carbondata, briefly describes how to start Spark-carbondata-thriftserver. version CDH 5.10.3 spark 2.1.0 carbondata 1.2.0 download spark https://archive.apache.org/dist/spark/spark-2.1.0 /spark-2.1.0-bin-hadoop2.6.tgz Carbondata https://dist.apache.org/repos/dist/release/carbondata/1.2.0/ Apache-carbondata-1.2.0-source-release.zip ca

CM Add hive Service after installing CDH, error message appearsWhen adding a service, hive is configured as follows: Error message: Error log:Xec/opt/cloudera/parcels/cdh-5.4.7-1.cdh5.4.7.p0.3/lib/hadoop/bin/hadoop jar/opt/cloudera/parcels/ Cdh-5.4.7-1.cdh5.4.7.p0.3/lib/hive/lib/hive-cli-1.1.0-cdh5.4.7.jar Org.apache.hive.beeline.HiveSchemaTool- Verbose-dbtype My

1. Create the lib121 directory under the hive0.13.1 version Cd/opt/cloudera/parcels/cdh/lib/hive;mkdir lib1212. Download the hive1.2.1 version and copy all files from this version of Lib to lib121 3. Modify the Hive_lib variable in/opt/cloudera/parcels/cdh/lib/hive/bin/hive hive_lib=${hive_home}/lib121 4. Update the JLine jar package on Hadoop and remove the old Jlien jar package RM-RF Jline-0.9.94.jar

I. risks are classified into internal and externalFirst, internal:During the deployment of CDH Big Data clusters, users named after services are automatically created,Username (login_name): Password location (passwd): User ID (UID): User Group ID (GID): annotation description (users): Home directory ): log on to Shell)CAT/etc/shadowThe format of the second column in the shadow file. It is the encrypted password. This column is "!! ", That is ":!! : ",

Recent projects need to use Oozie Workflow scheduling hivesql, found unable to execute query statements, see: https://community.cloudera.com/t5/Batch-Processing-and-Workflow/ oozie-hive-action-failed-with-wrong-tmp-path/td-p/37443 this, the culprit is CDH bug, need to upgrade the version.Upgrade steps:1. Querying a service on a single nodeService--status-allFound only cloudera-scm-agent, no cloudera-scm-server, indicating that this is not the primary

We know that Namenode's single-machine failure is cumbersome, and CDH offers high-availability options.The operation is as follows:Click on "HDFS"Select NamenodeClick "Action" and select:Set your own name.Click "Continue"Click "Continue"This keeps the default and then continues with the problemReturn, write a valueGo onIndicates that the operation is being processed,Start successfully!Go back and look at the Overview interface:You can see that Seconda

."s/eth1/eth0/g"$net _rule_fileElseNew_mac_str=$(sed-n-e'/eth0/p'$net _rule_file) #new_mac_1=${NEW_MAC_STR: -: -} New_mac=$(Echo$new _mac_str|awk-F','{'Print $4'}|awk-F'=='{'Print $'}|sed 's/\ "//g') Echo "Done 70-persistent-net.rules file!"fi#===================================#将新的网络配置入写网卡文件, restart the networkif(Cat$net _conf_file|grep$netmask _conf); Then Echo "Done/etc/sysconfig/network-scripts/ifcfg-eth0"elif[!-N" $"] ; Then Echo "You had not input a IP address!"Else sed-I."/$old _ma

1. Configure the yum SourceLs-L/dev | grep CDMkdir/mnt/CDROMMount/dev/CDROM/mnt/CDROMCD/etc/yum. Repos. dBack up and delete other Yum sourcesVI media. Repo[Rh6-media] Name = rh6-mediaautorefresh = 0 baseurl = file: // MNT/CDROM/gpgcheck = 0 enabled = 1Yum clean allYum makecacheIi. Install KerberosRefer to another article:Add an RPM package: rpm-IVH krb5-server-ldap-1.10.3-65.el6.x86_64.rpm3. Install LDAPYum install OpenLDAP openldap-servers openldap-clients openldap-devel compat-OpenLDAPInstall

Recently CDH cluster frequent alarm, because some host frequent swapping, greatly affected the performance of the cluster.Later found a setting (/proc/sys/vm/swappiness) needs to be modified, the default value of 60Setting the vm.swappiness Linux Kernel Parametervm.swappinessis a Linux Kernel Parameter This controls how aggressively memory pages are swapped to disk. It can set to a value between 0-100; The higher the value, the more aggressive the ker

Error when executing hive count query:Error:java Heap SpaceThe solution is set io.sort.mb=10;Error when executing Hadoop exeample, also Java heap space problemDiagnostic Messages for this Task:Error:java Heap SpaceFailed:execution Error, return code 2 from Org.apache.hadoop.hive.ql.exec.mr.MapRedTaskMapReduce Jobs Launched:Stage-stage-1: map:1 reduce:1 hdfs read:0 HDFs write:0 FAILTotal MapReduce CPU time spent:0 msecHive execution hql prompt error Error:java heap spaceJava Heap Space

configuring CDH and Managing servicesTuning of HDFs before closing DatanodeRole requirements: Configurator, Cluster Administrator, full Administratorwhen a datanode is closed, Namenode ensures that each block in each Datanode is still available based on the replication factor (the replication factor) across the cluster. This process involves the block duplication of small batches between datanode. In this case, a datanode has thousands of blocks, and

How to do integration, in fact, especially simple, online is actually a tutorial.http://blog.csdn.net/fighting_one_piece/article/details/40667035 look here.I'm using the first integration. When you do, there are a variety of problems. Probably from from 2014.12.17 5 o'clock in the morning to 2014.12.17 night 18 o'clock 30 summed up in fact very simple, but do a long time AH Ah!!! This kind of thing, a fall into your wit. Question 1, need to refer to a variety of packages, these packages to bre

1Create a collection SSH connects remotely to the CDH node that has SOLR installed. Running the solrctl instancedir--generate/solr/test/gx_sh_tl_tgryxx_2015 command generates the default configuration for the Gx_sh_tl_tgryxx_2015 collection. Enter the/solr/test/gx_sh_tl_tgryxx_2015/conf directory, first edit the Schema.xml configuration field information, the specific online search one piece. Solrconfig.xml file in the other The following