Opentsdb+grafana monitoring System by use summary

I. Introduction of OPENTSDBThe open source monitoring system, OPENTSDB, uses hbase to store all the timings (without sampling) to build a distributed, scalable time-series database. It supports the second-level data acquisition of all metrics, supports permanent storage, can do capacity planning, and easy access to existing alarm systems. Opentsdb can get the appropriate metrics and storage, indexing, and service from large-scale clusters (including network devices, operating systems, applications) in a cluster, making the data easier to understand, such as web-based, graphical, and so on. for Operations engineers, OPENTSDB can obtain real-time status information on infrastructure and services, demonstrating various hardware and software errors, performance changes, and performance bottlenecks in the cluster. For managers, OPENTSDB can measure the SLA of a system, understand the interactions between complex systems, and demonstrate the consumption of resources. The overall operation of the cluster can be used to support budget and cluster resource coordination. For developers, OPENTSDB can focus on key issues by demonstrating the main performance bottlenecks of the cluster, frequently occurring errors. Here are a few words for Opentsdb's introduction very detailed:http://liubin.org/blog/2016/03/05/tsdb-opentsdb/http://www.jianshu.com/p/0bafd0168647https://yq.aliyun.com/articles/54785This is a detailed description of the table structurehttps://zhengheng.me/2016/03/07/opentsdb-hbase-tuning/Optimizationsecond, the installation configuration of Opentsdb 1. Construction of basic Environmentfirst make sure that your system meets the following conditions:JDK 1.6 and configure environment variables.hbase 0.92 or latergnuplot 4.2 or higher (yum install bar)Autotools (as if only autoconf and Automake are needed)

#1. Reinstall the new version autoconf/automake# Deleting older versionsYumErase autoconf# Install new version autoconfwgethttp//mirrors.ustc.edu.cn/gnu/autoconf/autoconf-2.68.tar.gzTar-ZXVF autoconf-2.68.Tar. GZCD autoconf-2.68./Configure Make Make InstallLN-s/usr/local/bin/autoconf/usr/bin/autoconf# Installing the new version Automakewgethttp//mirrors.ustc.edu.cn/gnu/automake/automake-1.11.2.tar.gzTar-ZXVF automake-1.11.2.Tar. GZCD Automake-1.11.2./Configure Make Make InstallLN-s/usr/local/bin/automake/usr/bin/AutomakeLN-s/usr/local/bin/aclocal/usr/bin/aclocal#2. Installing gnuplot-4.4.0wgethttp//nchc.dl.sourceforge.net/project/gnuplot/gnuplot/4.4.0/gnuplot-4.4.0.tar.gzTar-ZXVF gnuplot-4.4.0.Tar. GZCD gnuplot-4.4.0./Configure Make Make InstallLN-s/usr/local/bin/gnuplot/usr/bin/gnuplot

2, the installation of Opentsdband then the following is the process I installed through the source code, I hope that can be useful for reference:

1 git clone https://github.com/opentsdb/opentsdb.git2cd opentsdb  3 ./build. SH 4 #当build. You should see Opentsdb.jar under the Opentsdb/build folder when the SH is finished running

Finally, the installation is successful, here is summed up: First, Java to be installed, the variables must be configured, and secondly, the autotools of the several tools to be installed, Opentsdb must be gnuplot to install; Extra: Because Opentsdb's backend is hbase, So here also add my configuration of HBase simple introduction: 1, hbase installation configuration, after decompression, in the/etc/profile add hbase_home=path/to/hbase-0.94.x not add words, The following command needs to be done manually plus the source/etc/profile takes effect after completion, and then, the Zookeeper property of HBase is configured,This is related to the hbase-env.sh file, where the HBASE_MANAGES_ZK environment variable is used to set whether to use HBase's default zookeeper or to use an independent zookeeper. Hbase_manages_zk=false when using standalone, true when using the default self-contained. Manually start the build directory for HBASE:./START-HBASECD opentsdb && hbase_home=/home/xxx/hbase-1.2.5./src/create_table.sh./tsdb Tsd

3. Configuration files

Configuration files and Parameters:

Zookeeper (HBase relies on zookeeper so ensure that there is hbase while guaranteed to have zookeeper)
There is a opentsdb.conf configuration file under the Opentsdb/src folder with 3 main properties that need to be configured:tsd.http.cachedir OPENTSDB Cache Data Storage directory, find a path configurationtsd.http.staticroot Opentsdb The root path of a Web static file, typically in Opentsdb/build/staticrootTsd.network.port-Bound portsThe following is configured as a common optional configurationTsd.network.bind= 0.0.0.0 Web page binding IP, typically configured to 0.0.0.0 to ensure that other IPs can access OPENTSDB Web pages. Tsd.storage.hbase.zk_quorum = localhost This is the address of the zookeeper. General standalone HBase does not need to be configured by default on this machine. If zookeeper is independent or distributed on other machines, specify IP multiple IP comma delimited.There are other configurations, please see the official documentation:http://opentsdb.net/docs/build/html/user_guide/configuration.html after the configuration file has been filled out and copied to/etc/, the OPENTSDB will search for the configuration file from the following places:./opentsdb.conf/etc/opentsdb.conf/etc/opentsdb/opentsdb.conf/opt/opentsdb/opentsdb.confIf you have configuration files in multiple places, make sure that the contents of these multiple profiles are consistent. That is to say, modify one to synchronize all, or you may search for different profiles depending on the path that you start the command to execute. it is advisable to put only one under the/etc/. Watch out! To automatically update the results obtained by the OPENTSDB tag_values () function, you need to set four parameters in the configuration file

• tsd.core.meta.enable_realtime_uid
• tsd.core.meta.enable_tsuid_tracking
• tsd.core.meta.enable_tsuid_incrementing
• tsd.core.meta.enable_realtime_ts

opentsdb configuration file Description:http://opentsdb.net/docs/build/html/user_guide/stats.html

Name of parameter	Type	is required	Default value	Detailed Introduction
Tsd.core.auto_create_metrics	Boolean	Optional	False	Whether the new metric data point is assigned a UID. When false, data points in the database that do not have metric are rejected and an exception is thrown.
TSD.CORE.AUTO_CREATE_TAGKS (2.1)	Boolean	Optional	True	Whether a data point with a new label name assigns a UID to TAGK. When false, data points with label names that are not in the database are rejected and an exception is thrown.
Tsd.core.auto_create_tagvs (2.1)	Boolean	Optional	True	Whether the data point with the new label value is assigned to the UID of the TAGV. When false, data points with tag values that are not in the database are rejected and an exception is thrown.

4. Start Opentsdb:first create an hbase table:There's a create_tab.sh script under OPENTSDB/SRC.Execution:env compression=none hbase_home=path/to/hbase-0.94.x./src/create_table.shCompression=none Specifies the compression method for HBase tables, Optional: NONE, LZO, GZIP, or SNAPPYHbase_home not much to say, if configured in the environment variable can not write this. the last one is to execute the Build table script. Once the table is completed, it can be started:There is a TSDB script execution under the Opentsdb/build folder../tsdb TSD to start opentsdb open the corresponding IP and port page to see the page, such as to perform in the background: Nohup./tsdb TSD & 5, the simple use of Opentsdbtelnet client sends data to OPENTSDBCreate Metric:./tsdb mkmetric metricnametelnet IP port connected to OPENTSDThe syntax is as follows: Putsuch as:put Sys.cpu.user 1356998400 42.5 host=webserver01 cpu=0Post send data: Send the POST request to Ip:port/api/put, the parameters are JSON data, as follows: TD style= "width:70px; height:14px; font-size:14px; Vertical-align:middle; Color: #333333 ">integer, Float, String

name	data Type	required	description	qs
Metric	string	required	the name of the metric you is storing	sys.cpu.nice
Timestamp	integer	required	a Unix epoch style timestamp in seconds or milliseconds. The timestamp must not contain non-numeric characters.	1365 465600
value	required	the value to record for this data point. It May is quoted or not quoted and must conform to the OPENTSDB value rules:writing Data	42.5
Tags	Map	Required	A Map of tag Name/tag value pairs. At least one pair must is supplied.	{"Host": "Web01"}

Example:{"metric": "Sys.cpu.nice", "timestamp": 1346846400, "value": "tags": {"host": "Web01", "DC": "LGA"}}send success will return 204 statusFor details, see:http://opentsdb.net/docs/build/html/api_http/put.html-------------------------------------------Scan command scanning commands are useful for debugging and exporting data points. Provides a start time, an optional end time, and one or more queries that respond to raw byte data for storage or data points that are acceptable for the text format of the import command. Scanning also provides a basic way to delete data. The Scan command accepts common CLI parameters. Data is sent to standard output. Note that although the query requires a aggregator, it is effectively ignored. If the query contains many time series, the scan output can be very large, so be careful when making queries. Scan [--delete|--import] start-date [end-date] query [queries ...] TD style= "WIDTH:148PX; Vertical-align:top; White-space:normal ">1h-ago

name	data Type	description	default	example
--delete	flag	Optional Flag That deletes data on any row that matches the query. See warning below. An optional tag that removes data from any row that matches the query.	not set	--delete
--import	flag	Optional flag That outputs results in a text format useful for importing or storing as a backup.	not set	--import
start-date	string or Integer	Starting time for the query. This is a absolute or relative time. See Dates and times for details	 
end-date	string or Integer	Optional end time for the query. If not provided, the current time is used. This is a absolute or relative time. See Dates and times for details	current Timestamp	2014/01/01-00:00:00
Query	String	One or more command line queries		Sum Tsd.hbase.rpcs type=put

Example./TSDB scan 2014/05/01 sum test_uid ED=ENDP--delete./tsdb scan 2014/05/01 sum tcpext.arpfilter--deleteThird, Grafana installation (Grafana specific introduction and use of the method will be a separate article introduction)Grafana Installation1, in the official website Download Grafana installation package, after installation, according to the installation output log prompts to start Grafana (grafana-server start)2, start Grafana after opening the machine ip:3000 port can login Grafana interface, initial password admin admin3, add a data source in DataSource, select Opentsdb Fill in the ip+ port. OPENTSDB defaults to Port 4242.

Opentsdb+grafana monitoring System by use summary