hbase performance monitoring

Original address HBase Test Report This article will introduce the details of our tests for Aliyun HBase and HBase1.1.12, and will probably introduce the environment of the test, the test tool analysis and our selection of tools, the case of the test, and the analysis of the results of the test. 1. Test environment 1.1. Physical environment: Service end: Single regionserver:8core,32g,4x250g SSD Client:16c

for a single request. When setting this value, the main reference is monitoring memory. Note that if the number of region servers is small and a large number of requests are on the same region, the read/write lock caused by memstore triggering flush will affect the global TPS, the higher the number of Io threads, the better. Enabling RPC-level logging is enabled during stress testing to monitor the memory consumption and GC status of each request at

Document directory Configuration Optimization Others Hbase client Optimization Link: http://kenwublog.com/hbase-performance-tuning Because the performance tuning section of the official book does not index by configuration item, quick query cannot be achieved. So I re-organized the original text with the configu

RS is manually recovered, this balance action is meaningless, which will result in uneven load and bring more burden to Rs. In particular, for scenarios with fixed regions allocation. Hbase. regionserver. handler. CountDefault Value: 10Description: The number of I/O threads that the regionserver requests process.Optimization:The optimization of this parameter is closely related to the memory.A small number of Io threads are suitable for big put scena

manually recovered, this balance action is meaningless, which will result in uneven load and bring more burden to RS. In particular, for scenarios with fixed regions allocation. Hbase. regionserver. handler. countDefault Value: 10Description: The number of I/O threads that the RegionServer requests process.Optimization:The optimization of this parameter is closely related to the memory.A small number of IO threads are suitable for Big PUT scenarios w

discarded.Load testingPEHBase has its own stress testing tool named PE (Performance Evaluation)YCSBCloud service benchmarking tool launched by Yahoo. It is easier to use than PE and can perform stress testing on hbase.YCSB provides more options and can mix read/write loads togetherHBase performance optimization notes1 hbase. hregion. max. filesizeDefault value:

balance (so that other RS can be restored Based on the wal logs recorded by the faulty machine ). When the faulty RS is manually recovered, this balance action is meaningless, which will result in uneven load and bring more burden to Rs. In particular, for scenarios with fixed regions allocation. Hbase. regionserver. handler. CountDefault Value: 10Description: The number of I/O threads that the regionserver requests process.Optimization:The optimizat

Label:Transferred from: http://www.ibm.com/developerworks/cn/java/j-lo-HBase/index.html High Performance HBase Database This paper first introduces the basic principles and terminology of hbase database, then introduces the operation API and some examples of hbase database r

Speaking of Hadoop and HBase Cluster Monitoring, we all know and use third-party monitoring tools, such as cacti, ganglia, and zabbix. Zenoss is used for playing more deeply. These tools are indeed good and can play a major role, but after a long time, I always feel that the monitoring granularity is still relatively c

be directly associated with a Region of HBase. You need to check whether the Region on the Table is available. 2) read/write of Region is associated with the state of underlying HDFS. This association determines the monitoring of read/write status through Region, and also reflects the HDFS status.2. Practical Tools ? Org. apache. hadoop. hbase. tool. Canary moni

1 ganglia Introduction Ganglia is an open-source monitoring project initiated by UC Berkeley designed to measure thousands of nodes. Each computer runs a gmond daemon that collects and sends metric data (such as processor speed and memory usage. It is collected from the operating system and the specified host. Hosts that receive all metric data can display the data and pass the simplified form of the data to the hierarchy. Ganglia can be well expande

not drop the setmaxversions, only the latest version will be taken.Scan.settimerange (): Specifies the maximum timestamp and minimum timestamp, which can only be obtained by the Cell in this range;Scan.settimestamp (): Specifies the timestamp;Scan.setfilter (): Specify filter to filter out unwanted information;Scan.setstartrow (): Specifies the start line. If not called, start from the table header;Scan.setstoprow (): Specifies the end of the line (not including this line);Scan. Setcaching ():

1 hbase. hregion. Max. filesize 2 autoflush = false 3. Concerning the settings of family and qualifier in table from the perspective of performance 4 hbase. regionserver. handler. Count 1 hbase. hregion. Max. filesizeDefault Value: 256 mDescription: Maximum hstorefile size. If any one of a column families 'hstorefiles

single request. When setting this value, the main reference is monitoring memory. Note that if the number of region servers is small and a large number of requests are on the same region, the read/write lock caused by memstore triggering flush will affect the global TPS, the higher the number of Io threads, the better. Enabling RPC-level logging is enabled during stress testing to monitor the memory consumption and GC status of each request at the sa

HBase Ganglia monitoring HBase configuration Hadoop-metrics2-hbase.properties *. Sink. Ganglia. Class=org. Apache. Hadoop. Metrics2. Sink. Ganglia. Gangliasink31 *. Sink. Ganglia. Period=TenHBase. Sink. Ganglia. Period=TenHBase. Sink. Ganglia. Servers=239.2. One.:8649HBase. Class=org. Apache. Hadoop. Metrics2. Sink. Ganglia. Gangliasink31hbase. Peri

HBase Ganglia Monitor HBase construction Hadoop-metrics2-hbase.properties *. Sink. Ganglia. Class=org. Apache. Hadoop. Metrics2. Sink. Ganglia. Gangliasink31 *. Sink. Ganglia. Period=TenHBase. Sink. Ganglia. Period=TenHBase. Sink. Ganglia. Servers=239.2. One.:8649HBase. Class=org. Apache. Hadoop. Metrics2. Sink. Ganglia. Gangliasink31hbase. Period=TenHBase. Servers==239.2. One.:8649Jvm. Class=

= "_blank" >.*If [-Z $dead _region_servers];thenecho "[OK] all regionservers are healthy."echo "[OK] all regionservers are healthy." >> $outputElseecho "[Abnormal] the dead regionserver list:" >> $outputecho $dead _region_servers >> $outputExit $STATE _criticalFi# HDFS UsageDfs_remaining= ' curl-s http://${hadoop_web_interface}:50070/dfshealth.jsp |egrep-o "DFS Remaining%.*%" | Egrep-o "[0-9]*\. [0-9]* "'Dfs_remaining_word= "DFS remaining%: ${dfs_remaining}%"echo "$dfs _remaining_word" >> $outpu

60000 (60s), we recommend that according to the actual Regionserver log monitoring found the exception for reasonable settings, such as we set to 900000, The modification of this parameter requires simultaneous changes to the Hdfs-site.xml19.dfs.datanode.socket.write.timeout: Default 480000 (480s), sometimes regionserver when merging, Datanode write timeout may occur, 480000 Millis timeout while waiting for channel to is ready for write, the modifica

Apache Ambari is a Web-based tool that supports the supply, management, and monitoring of Apache Hadoop clusters. Ambari currently supports most Hadoop components, including HDFS, MapReduce, Hive, Pig, Hbase, Zookeper, Sqoop, and Hcatalog.Apache Ambari supports centralized management of HDFS, MapReduce, Hive, Pig, Hbase, Zookeper, Sqoop, and Hcatalog. It is also

Summary of HBase Performance optimization Method (i): Table Design This article is mainly from the HBase application design and development point of view, summed up several commonly used performance optimization methods. About the HBase system configuration level optimizati