Build your own Go Runtime metrics environment

This is a creation in Article, where the information may have evolved or changed.

Since go 1.5, every go release, Gopher Brian Hatfield has been basking on Twitter for the performance data of the new Go runtime (compared to previous go versions). Even when the Go team staff do speaking around the world, Brian's pictures are also quoted in slide. Later, Brian Hatfield packaged its code for measuring Runtime performance data into a library and put it on GitHub, and we could use the library to build our own go Runtime metrics facility. Here is a brief description of the steps to build.

I. Environment and principles

Brian Hatfield's Go-runtime-metrics Library is simple to implement, with runtime data from Memstats in the Go runtime package, Numgoroutine and Numcgocall and so on. The test target program only needs to import the library to output runtime states data:

import _ "github.com/bmhatfield/go-runtime-metrics"

The Go-runtime-metrics library will launch a separate goroutine and periodically escalate the runtime data. Currently, the library only supports output data to STATSD, which can be configured to import STATSD data into graphite and use graphite web to view the process such as:

The experimental environment for Ubuntu 16.04.1:

$ uname -rmntonybai-ThinkCentre-M6600s-N000 4.4.0-83-generic x86_64

Second, the construction steps

1. Install Go-runtime-metrics Library

We can download the Go-runtime-metrics library directly from the go get:

$ go get github.com/bmhatfield/go-runtime-metrics

We write a target program:

//main.gopackage mainimport (    "flag"    "log"    "net/http"    "os"    _ "github.com/bmhatfield/go-runtime-metrics")func main() {    flag.Parse()    cwd, err := os.Getwd()    if err != nil {        log.Fatal(err)    }    srv := &http.Server{        Addr:    ":8000", // Normally ":443"        Handler: http.FileServer(http.Dir(cwd)),    }    log.Fatal(srv.ListenAndServe())}

There are four go versions installed on my Ubuntu host, namely Go 1.5.4, go 1.7.6, go 1.8.3 and go1.9beta2, so we use these four versions of the server as the test program to report the go runtime data for comparison.

$ GOROOT=~/.bin/go154 ~/.bin/go154/bin/go build -o server-go154 main.go$ GOROOT=~/.bin/go174 ~/.bin/go174/bin/go build -o server-go174 main.go$ GOROOT=~/.bin/go183 ~/.bin/go183/bin/go build -o server-go183 main.go$ GOROOT=~/.bin/go19beta2 ~/.bin/go19beta2/bin/go build -o server-go19beta2 main.go$ ls -l-rwxr-xr-x 1 tonybai tonybai 6861176 7月   4 13:49 server-go154-rwxrwxr-x 1 tonybai tonybai 5901876 7月   4 13:50 server-go174-rwxrwxr-x 1 tonybai tonybai 6102879 7月   4 13:51 server-go183-rwxrwxr-x 1 tonybai tonybai 6365648 7月   4 13:51 server-go19beta2

2. Installation, configuration and Operation STATSD

STATSD This tool collects statistical information and sends aggregated information to backend services (e.g., graphite). STATSD is a service implemented with JS, so you need to install Nodejs, NPM, and related modules:

$ sudo apt-get install nodejs$ sudo apt-get install npm

Next, we will clone the STATSD project locally and configure a goruntimemetricconfig.js (basically the default configuration) that we use based on the exampleconfig.js template:

// goruntimemetricConfig.js{  graphitePort: 2003, graphiteHost: "127.0.0.1", port: 8125, backends: [ "./backends/graphite" ]}

Start STATSD:

$ nodejs stats.js goruntimemetricConfig.js3 Jul 11:14:20 - [7939] reading config file: goruntimemetricConfig.js3 Jul 11:14:20 - server is up INFO

Start successfully!

3. Installation, configuration and operation Graphite

Graphite is a tool for storing time-series monitoring data and displaying data graphically as required by the user, including three components:

• Whisper

Whisper is a file-based sequential database format, while Whisper also provides the appropriate commands and APIs for other components to invoke to manipulate the timing database;

• Carbon

Carbon is used to read the metrics information of external push, to aggregate and write to DB, it also supports cache hotspot data and improves access efficiency.

• Graphite-web.

Graphite-web is a user-oriented graphical system for customizing the display of monitoring data.

The installation and configuration of the graphite is a bit cumbersome, and we take it one step at a time.

A) Install graphite

$sudo apt-get install graphite-web graphite-carbonwhisper将作为依赖自动被安装。

b) local_settings.py

Graphite The main configuration file in/etc/graphite/local_settings.py, the file contains many configuration items, here only the relevant, and this effective configuration:

// /etc/graphite/local_settings.pyTIME_ZONE = 'Asia/Shanghai'LOG_RENDERING_PERFORMANCE = TrueLOG_CACHE_PERFORMANCE = TrueLOG_METRIC_ACCESS = TrueGRAPHITE_ROOT = '/usr/share/graphite-web'CONF_DIR = '/etc/graphite'STORAGE_DIR = '/var/lib/graphite/whisper'CONTENT_DIR = '/usr/share/graphite-web/static'WHISPER_DIR = '/var/lib/graphite/whisper'LOG_DIR = '/var/log/graphite'INDEX_FILE = '/var/lib/graphite/search_index'  # Search index fileDATABASES = {    'default': {        'NAME': '/var/lib/graphite/graphite.db',        'ENGINE': 'django.db.backends.sqlite3',        'USER': '',        'PASSWORD': '',        'HOST': '',        'PORT': ''    }}

c) Synchronizing the database

Next, execute the following two commands to do database sync (sync):

$ sudo graphite-manage migrate auth. .... Operations to perform:apply all migrations:authrunning migrations:rendering model states ... Done Applying contenttypes.0001_initial ... OK Applying Contenttypes.0002_remove_content_type_name ... OK Applying auth.0001_initial ... OK Applying auth.0002_alter_permission_name_max_length ... OK Applying auth.0003_alter_user_email_max_length ... OK Applying auth.0004_alter_user_username_opts ... OK Applying Auth.0005_alter_user_last_login_null ... OK Applying auth.0006_require_contenttypes_0002 ... ok$ sudo graphite-manage syncdboperations to perform:synchronize unmigrated Apps:account, CLI, Render, whitelist, Metri CS, Url_shortener, dashboard, composer, events, browser Apply all migrations:admin, contenttypes, tagging, auth, session    Ssynchronizing apps without migrations:creating tables ... Creating table Account_profile Creating table account_variable Creating table Account_view Creating table Account _window Creating TableAccount_mygraph Creating table Dashboard_dashboard Creating table events_event Creating table Url_shortener_link  Running Deferred SQL ... Installing Custom SQL ... Running migrations:rendering Model states ... Done Applying admin.0001_initial ... OK Applying sessions.0001_initial ... OK Applying tagging.0001_initial ... Okyou has installed Django's auth system, and don ' t has any superusers defined. Would to create one now? (yes/no): Yesusername (leave blank to use ' root '): Email Address:xx@yy.compassword:password (again): Superuser created suc Cessfully.

Here we create a superuser:root for use when logging in to Graphite-web.

d) Configuration Carbon

The configuration file involving carbon is as follows, and we keep the default configuration fixed:

/etc/carbon/carbon.conf（内容太多，这里不列出来了）/etc/carbon/storage-schemas.conf[carbon]pattern = ^carbon\.retentions = 60:90d[default_1min_for_1day]pattern = .*retentions = 60s:1d[stats]pattern = ^stats.*retentions = 10s:6h,1min:6d,10min:1800d

Carbon has a cache feature that we can turn on by following these steps:

打开carbon-cache使能开关：$ vi /etc/default/graphite-carbonCARBON_CACHE_ENABLED=true启动carbon-cache：$ sudo cp /usr/share/doc/graphite-carbon/examples/storage-aggregation.conf.example /etc/carbon/storage-aggregation.conf$ systemctl start carbon-cache

e) Start Graphite-web

Graphite-web supports a variety of major Web servers, and here apache2, for example, Graphite-web the Mod-wsgi way under Apache2:

$sudo apt-get install apache2 libapache2-mod-wsgi$ sudo service apache2 start$ sudo a2dissite 000-defaultSite 000-default disabled.$ sudo service apache2 reload$ sudo cp /usr/share/graphite-web/apache2-graphite.conf /etc/apache2/sites-available$ sudo  a2ensite apache2-graphiteEnabling site apache2-graphite.To activate the new configuration, you need to run:  service apache2 reload$ sudo systemctl reload apache2

Because APACHE2 worker process runs by default with Www-data:www-data user rights, the database file access is: _graphite:_graphite:

$ ll /var/lib/graphite/graphite.db-rw-r--r-- 1 _graphite _graphite 72704 7月   3 13:48 /var/lib/graphite/graphite.db

We need to modify the user of the Apache worker:

$ sudo vi /etc/apache2/envvarsexport APACHE_RUN_USER=_graphiteexport APACHE_RUN_GROUP=_graphite

Restart Apache2 in effect! Open with browser: http://127.0.0.1, if no surprises, you will see the following Graphite-web homepage:

III. implementation of the benchmarking

Here I will use wrk this HTTP benchmarking tool for the previous four versions of the target program (server-go154 server-go174 server-go183 server-go19beta2) For benchmarking test, each target program receives a 10-minute request:

$ ./server-go154$ wrk -t12 -c400 -d10m http://127.0.0.1:8000$ ./server-go174$ wrk -t12 -c400 -d10m http://127.0.0.1:8000$ ./server-go183$ wrk -t12 -c400 -d10m http://127.0.0.1:8000$ ./server-go19beta2$ wrk -t12 -c400 -d10m http://127.0.0.1:8000

Iv. Display of results

Open the Graphite-web with your browser, and open the tree structure on the left side of the Trees tab: Metrics, stats, gauges, go, Your_host_name- > Pause, if it goes well, you will see the line chart in the Graphite composer window, and we also take GC pause as an example, GC pause is also the gopher of concern:

With this picture (the unit of the left axis is nanoseconds), we can see roughly:

Go 1.5.4 's GC pause is about 600μs;
Go 1.7.4 's GC pause is about 300μs;
The GC pause for Go 1.8.3 and go 1.9beta2 is basically below 100μs. The GC improvements to Go 1.9 seem to be small. But here my program is not enough typical.

Other results:

Go Routines Number:

GC Count:

Memory Allocations:

In addition to viewing a single indicator curve, you can also customize the panel you want to monitor with the dashboard function provided by Graphite-web, which is not discussed here.

V. References

• How to Setup Graphite with STATSD in an Ubuntu 16.04

Weibo: @tonybai_cn
Public Number: Iamtonybai
Github.com:https://github.com/bigwhite

, Bigwhite. All rights reserved.