mtools-The MongoDB artifact that you might not use.

Objective

has been in contact with MongoDB for some time, from the beginning of not understanding, to now has slowly adapted to this NoSQL field leader, or experienced a lot of twists and turns.
When it comes to database sizing, many people like to compare MongoDB with MySQL, Oracle, and summarize a very detailed analysis of the results.
But this analysis often does not change the decision of the manager (or the architect). Because managers may already have an answer before deciding to use a technology, the analysis contrasts to make decisions more certainty and more evidence.
Although this statement is somewhat frustrating, it is true in reality.

It is well known thatMongoDB is not a relational database, does not follow the classic "Three paradigms", and does not have the characteristics of a traditional database such as a table association, a transaction (until the latest version 4.0 is implemented) .
However, MongoDB's own non-schema, copy set, Auto-sharding and other capabilities to obtain a lot of favor, but also worth mentioning is that the database itself provides a large number of monitoring, analysis Tools command, the operation and maintenance management provides a great convenience.

Now that we've talked about tools, this article is about a tool called Mtools. During this period, the author has been in the organization of online database log collection and analysis, because there is no better tool support, the analysis of each time the total need to invest a lot of manpower.
In this case, Mtools can help with a significant increase in efficiency, and the following space begins to introduce this tool.

What is Mtools?

Mtools is a set of tools implemented by MongoDB's official engineers, which enables fast log query analysis and statistical functions, and also supports local cluster deployment management, making it easy for novice learners to learn.
GitHub address, the set of tools non-official company maintenance, only by the author to do open source maintenance, the current project update frequency is not high, but already have a large number of users.

Mtools is written using Python and can be obtained through the PIPY website;
The tool contains several key components:

Mlaunch
Support the rapid building of local test environment, can be single machine, replica set, shard cluster.
Mlogfilter
The log filtering component supports retrieving slow queries, full table scan operations by time, supports information filtering through multiple attributes, and supports output in JSON format.
Mplotqueries
Supports converting log analysis results to chart form, depending on tkinter (Python graphics module), and Matplotlib module.
Mlogvis
Supports converting log analysis results to a separate HTML page for the same functionality as mplotqueries.

There are a lot of information about Mtools on the Internet, including the description of its official documents are more detailed, here is only a brief introduction to the use of tools

What can be done 1. Simple Cluster Management

You can start a single-node Mongod process by executing the following command.

# mlaunch init --singlelaunching: "mongod" on port 27017

Can be compared with single-node hand-built

To perform another slightly more complex command:

  # mlaunch init--sharded 2--replicaset--nodes 3--config 3--csrs--mongos 3--port 27050--auth--username A DMin--password [email protected]--auth-db admin Launching: "Mongod" on port 27053launching: ' Mongod ' on port 27054l Aunching: "Mongod" on port 27055launching: ' Mongod ' on port 27056launching: ' Mongod ' on port 27057launching: ' Mongod ' on p ORT 27058launching:config Server on port 27059launching:config server in Port 27060launching:config server on port 2706 1replica set ' Configrepl ' Initialized.replica set ' shard01 ' Initialized.replica set ' shard02 ' initialized.launching: MONGOs on port 27050launching:mongos in Port 27051launching:mongos on port 27052adding shards. Can take up to seconds ... Username "admin", password "[email protected]"  

What the? has completed a double copy set of shard cluster build! 27050 is the starting port,--sharded 2 means there are two shards,--replicaset means the replica set is enabled,
--config 3--csrs indicates that Config uses a 3-node replica set of the CSRS structure,--auth means enable authentication,--username--password is the user initialized, and the user will have administrative rights to all the libraries by default.
The architecture of the entire cluster is as follows:

Can be managed with the following command

]# mlaunch listPROCESS PORT STATUS PID()mongos 27050 running 13017mongos 27051 running 13059mongos 27052 running 13093()config server 27059 running 12134config server 27060 running 12217config server 27061 running 12261()shard01    primary 27053 running 12404    secondary 27055 running 12559    mongod 27054 running 12509()shard02    secondary 27057 running 12793    secondary 27058 running 12845    mongod 27056 running 12697()() auth: "admin:[email protected]"

Start stop

# mlaunch stopsent signal 15 to 12 processes.# mlaunch startlaunching: config server on port 27059...

This is quite handy and can be compared to the process described in this article of distributed cluster construction, the tool can be reduced by dozens of times times compared to manual construction.

2. Log statistics

Mloginfo is a tool for logging information statistics, enter the following command:

# mloginfo mongo.log     source: mongo.log       host: MongoDB_1:10001        start: 2018 May 18 16:33:11.692        end: 2018 May 19 01:13:08.290date format: iso8601-local     length: 144480     binary: mongod    version: 3.4.10    storage: wiredTiger

You can see the log start and end time range, host port, version, Database engine, and other profile information.

Number of connections
When we want to check the number of connections for a client, you can execute the following command:

# mloginfo mongo.log --connectionsCONNECTIONS     total opened: 14282     total closed: 14358    no unique IPs: 4socket exceptions: 0127.0.0.1 opened: 12886 closed: 12889172.21.0.29 opened: 658 closed: 716172.21.0.28 opened: 461 closed: 490172.21.0.27 opened: 277 closed: 263

Through this information, further determine whether there is a connection overload and other anomalies.

Event statistics
Or, you want to count the frequency of certain events that are currently occurring.

# mloginfo mongo.log --distinctDISTINCT   14358 end connection ... ( ... now open)   14281 connection accepted from ... # ... ( ... now open)   13075 received client metadata from ... :    5340 Successfully authenticated as principal ... on    1194 Use of the aggregate command without the 'cursor'     338 build index on: ... properties:     244 building index using bulk method; build may temporarily use up to ... megabytes of RAM     234 ns: ... key: ... name:     219 Refreshing chunks for collection ... based on version     218 Refresh for collection ... took ... ms and found version     179 Index ... :

Slow query
In business problem analysis, slow query is the most common problem.

# mloginfo Mongo.log--queries--sort countqueriesnamespace operation pattern Count min (ms) max (ms) mean (ms) 95%-ile (ms) sum (ms) Nsspace.             Statisticshour Find {"$and": [{"Recordtime": 1}]..} 22331 276 747 345 414.0 7720736nsspace.    Statisticshour Getmore {"Aggregate": 1, "Cursor": ...}]} 231 304 227 272.0 52587dmspace. DeviceInfo Remove {"_id": 1} 109 205 1786 420 771 .0 45860cmspace.            Devicedata Update {"AppId": 1, "DeviceId": 1} 95 201 1802 431 824.5 40966dmspace. TaskHistory Update {"_id": 1} 54 268 2643 692 2 019.0 37413nsspace. Statisticsday Find {"$and": [{"RecordtIME ": 1}",..} 31 201 348 241 345.0 7472

As the above command, all slow queries are displayed and sorted by the number of occurrences.

Restart information

# mloginfo mongo.log --restartRESTARTS   May 18 21:37:51 version 3.4.10   May 18 21:48:33 version 3.4.10

The potential failure of the system is evaluated and analyzed by detecting the restart information.

Replica set switching
Similarly, primary and standby switching can lead to a certain business failure that requires regular monitoring.

# mloginfo mongo.log --rsstateRSSTATEdate host state/message()May 18 21:48:53 172.21.0.29:10001 ARBITERMay 18 21:49:26 172.21.0.28:10001 SECONDARY

3. Log filtering

Mlogfilter is a powerful log filtering module that can parse the log content and filter it according to the results we want, compared to the Grep/egrep text filter for Linux.

View slow operations over 10s

# Mlogfilter Mongo.log--slow 10000--shorten 2002018-05-18t21:49:04.069+0800 I REPL [Replicationexecutor] Starting an ele Ction, since we ' ve seen no PRIMARY in the past 10000ms2018-05-18t21:50:22.988+0800 I command [conn31] command DMSPACE.FS.C Hunks appName: "Mong...quirecount: {w:46}}, Oplog: {acquirecount: {w:46}}} Protocol:op_command 10804ms2018-05-18 t21:50:22.988+0800 I command [conn44] command dmspace. DeviceInfo command:inse...quirecount: {w:16}}, Oplog: {acquirecount: {w:16}}} Protocol:op_command 10931ms2018-0 5-18t21:50:22.988+0800 I command [conn157] command dmspace. Lwm2mdevice command...quirecount: {w:16}}, Oplog: {acquirecount: {w:16}}} Protocol:op_command 10762ms2018-05-18t 21:50:22.988+0800 I command [conn156] command dmspace. TaskHistory command:in ... Quirecount: {w:16}}, Oplog: {acquirecount: {w:16}}} protocol:op_command 10927ms2018-05-18t21:50:50.104+0800 I CO Mmand [conn31] command dmspace. DeviceInfo appName: "Mon ... Quirecount: {w:+}}, Oplog: {acquirecount: {w:16}}} protocol:op_command 10020ms2018-05-18t21:50:51.203+0800 I command [conn156] C  Ommand dmspace.fs.chunks command:inse...quirecount: {w:51}}, Oplog: {acquirecount: {w:51}}} Protocol:op_command 10823ms

viewing slow scan operation
Slow scan means that the operation needs to scan too many records (more than 1w rows), and returns 1/100 of the number of low-volume scans, which is typically CPU intensive and inefficient,

# mlogfilter Mongo.log--scan--shorten 2002018-05-18t21:57:09.123+0800 I command [conn683] command cmspace. User_login_history Command:find ... e: {acquirecount: {r:95}}, Collection: {acquirecount: {r:95}}} protocol:op_ Command 556ms2018-05-18t21:57:17.381+0800 I command [conn784] Getmore nsspace.  Statisticsday query: {aggre...nt: {r:10}, Timeacquiringmicros: {r:1667}}, Collection: {acquirecount: {r:890}} } 214ms2018-05-18t22:06:16.148+0800 I COMMAND [conn764] Getmore nsspace.  Statisticshour query: {aggr ...}}, Database: {acquirecount: {r:69128}}, Collection: {acquirecount: {r:69128}}} 12053ms2018-05-18t22:06:24.962+0800 I COMMAND [conn764] Getmore nsspace. Statisticshour query: {aggr ...}}, Database: {acquirecount: {r:69106}}, Collection: {acquirecount: {r:69106}} } 8782ms2018-05-18t22:06:33.787+0800 I COMMAND [conn764] Getmore nsspace. Statisticshour query: {aggr.}}, Database: {acquirecount: {r:69111}}, Collection: {acquirecount:{r:69111}} } 8822ms

Filter by namespace

# mlogfilter mongo.log --namespace dmspace.DeviceInfo2018-05-18T21:50:58.105+0800 I COMMAND [conn31] command dmspace.DeviceInfo appName: "MongoDB Shell...adata: { acquireCount: { w: 16 } }, oplog: { acquireCount: { w: 16 } } } protocol:op_command 2963ms2018-05-18T21:50:59.195+0800 I COMMAND [conn31] command dmspace.DeviceInfo appName: "MongoDB Shell...tadata: { acquireCount: { w: 16 } }, oplog: { acquireCount: { w: 16 } } } protocol:op_command 936ms2018-05-18T21:51:00.173+0800 I COMMAND [conn44] command dmspace.DeviceInfo command: insert { inser...tadata: { acquireCount: { w: 16 } }, oplog: { acquireCount: { w: 16 } } } protocol:op_command 745ms2018-05-18T21:51:00.433+0800 I COMMAND [conn44] command dmspace.DeviceInfo command: insert { inser...tadata: { acquireCount: { w: 16 } }, oplog: { acquireCount: { w: 16 } } } protocol:op_command 252ms

Filter by Operation type

# mlogfilter Mongo.log--operation update2018-05-18t21:56:25.114+0800 I WRITE [conn156] Update dmspace. Policytask query: {_id: "# # #"} pla ... Count: {w:2}}, Metadata: {acquirecount: {w:1}}, Oplog: {acquirecount: {w:1}}} 2630ms2018-05-18t21:56:25.114 +0800 I WRITE [conn92] Update nsspace. TimerTask query: {_id: "# # #"} plans ... Count: {w:2}}, Metadata: {acquirecount: {w:1}}, Oplog: {acquirecount: {w:1}}} 1264ms2018-05-18t21:56:25.125 +0800 I WRITE [conn43] Update dmspace. TaskHistory query: {_id: "# # #"} pla ... Count: {w:2}}, Metadata: {acquirecount: {w:1}}, Oplog: {acquirecount: {w:1}}} 2643ms2018-05-18t21:56:30.027 +0800 I WRITE [conn532] Update dmspace. TaskHistory query: {_id: "# # #"} pl ... Ecount: {w:2}}, Metadata: {acquirecount: {w:1}}, Oplog: {acquirecount: {w:1}}} 868ms2018-05-18t21:56:32.115 +0800 I WRITE [conn517] Update dmspace. TaskHistory query: {_id: "# # #"} pl ... Ecount: {w:2}}, Metadata: {acquirecount: {w:1}},Oplog: {acquirecount: {w:1}}} 497ms 

to get a log of 1 hours after a point in time

# mlogfilter Mongo.log--from Apr 6 0:00--to "+1h" | tail-n32018-05-19t00:59:59.876+0800 I command [conn16386] command nsspace.  Statisticshour Command:find {find: "# #", filter: {User: "# # #", Region: "# # #", AppKey: "# #", Recordtime: {$lte: "# # #" }, $and: [{recordtime: {$gte: ' # # # '}}]}, Shardversion: [' # # # ', ' # # # ']} plansummary: # # keysexamined:249767 Doc  sexamined:249767 cursorexhausted:1 numyields:1952 nreturned:84 reslen:29748 locks:{Global: {acquireCount: {r:3906}}, Database: {acquirecount: {r:1953}}, Collection: {acquirecount: {r:1953}}} Protocol:op_command 319ms2018-05-19t 00:59:59.879+0800 I command [conn15998] command nsspace.  Statisticshour Command:find {find: "# #", filter: {User: "# # #", Region: "# # #", AppKey: "# #", Recordtime: {$lte: "# # #" }, $and: [{recordtime: {$gte: ' # # # '}}]}, Shardversion: [' # # # ', ' # # # ']} plansummary: # # keysexamined:249767 Doc sexamined:249767 cursorexhausted:1 numyields:1954 nreturned:84 reslen:29833 locks:{ Global: {acquirecount: {r:3910}}, Database: {acquirecount: {r:1955}}, Collection: {acquirecount: {r:1955}} } Protocol:op_command 321ms

Mlogfilter provides very flexible date condition settings that can be offset by offsets, in addition to starting and ending times.

Time zone Conversion

# mlogfilter mongo.log --tiemzone 2 > mongo-correct.log

The above command will increase the date by 2 time zones and output to Mongo-correct.log, which is useful in scenarios that deal with internationalized systems.

More details stamp here

4. Chart rendering

Mplotqueries is a chart component based on the Tkinter implementation that converts flat text information in a log into a chart form.
Enter the following command:

mplotqueries mongo.log --group operations --output-file operations.png

You can get a scatter plot that is output grouped by operation , such as:

The y-axis on the left is duration, that is, the execution duration of the operation, and the x-axis below is the time. Each operation will have a stroke in the diagram, so there are many overlaps in the scatter plot.
Of course, you can also group output by the collection name, such as the following command:

mplotqueries mongo.log --group namespace --group-limit 20 --type scatter --yaxis nscanned --output-file namespace_nscan.png

The output chart is grouped by namespace (limited to 20), and the y-axis is the nscanned value, which is the number of records scanned .

By default, the y-axis is rendered long (during) and can be specified as a different metric:

Indicator name	Description
nscanned	Number of scans
nupdated	Number of updates
ninserted	Insert Number
Ntoreturn	return number
Numyields	Number of concessions
R	Read lock
W	Write lock

Sometimes you don't have to relate to a specific operation, but you want to see a time period, a certain type of action, or a collection of operations.
such as hourly, each set of operation scale distribution, at this time can use the histogram

mplotqueries mongo.log --group namespace --bucketsize 3600 --group-limit 10 --type histogram --output-file namespaces_perhour.png

As already mentioned, the monitoring of the number of connections is very important, Mplotqueries also provides the connection change statistics type

mplotqueries mongo.log --type connchurn --bucketsize 3600 --output-file connchurn_perhour.png

In most cases, inefficient operations often come from a large number of scan scans, especially if the return number is much smaller than scan.
You can output a chart of that dimension by specifying the nscanned/n parameter, which is the number of scans/returns

mplotqueries mongo.log --type nscanned/n --group-limit 20 --output-file nscan.png

Output Event Persistence graph
It is often necessary to track a certain type of time-consuming operation or event, such as Oplog synchronization, creating an index, etc., and we want to see the execution window of the event.
At the same time also need to accompany with some operational statistics, used to do resource monitoring and analysis. Specifically, you perform a time-consuming operation in which, in some cases, business access may
Impacts, such as business access timeouts and Alerts with DB server resources, such as high CPU, are expected to be analyzed in subsequent analyses through time-consuming operations and the distribution of simultaneous segment service accesses.

The following command shows a typical usage

# 创建一个overlaygrep "index" mongo.log | mplotqueries --type durline --overlay# 叠加overlay输出图表mplotqueries mongo.log --group operation --output-file duration.png

By the means, if you don't want to generate so many charts, Mtools also offers a lazy tool mlogvis. HTML page can be generated directly, built-in powerful script
It basically covers most of Mplotqueries's charting capabilities.

# mlogvis mongo.log

See more usage

Hope you have begun to like Mtools, and have been tempted. The following provides a simple installation method

How to Install

Due to space limitations, this is only available under CentOS installation

• Preparing for the Python environment (2.7 or 3.5), most of the current releases of Linux contain Python.
  If you do not have Python, run the yum install Python python-devel to complete the installation
  Make sure PIP is already installed, perform PIP for detection, and if not installed, refer to the official instructions for installation.

• Installing PYTHON-TK
  The following commands are executed:

  yum install python-tools

  Other Linux systems (such as ubuntu/opensuse) need to install the PYTHON-TK package

• Installing dependent modules

  pip install psutilpip install pymongopip install matplotlibpip install numpy

  Description
  The function of each module can be referred to the following table:

Module Name	function
Psutil	Tools for managing Processes
Pymongo	MongoDB python Driver
Matplotlib	Python's 2D chart rendering module
NumPy	Tools to support scientific computing

• Installing Mtools

  pip install mtools

  At this point, you should have an installed Mtools environment, if you want to install the latest unstable version, can be installed through the source code

Reference documents

MongoDB Dry Series-Mtools of regular inspection
Introduction to Mtools
Mtools Installation Instructions

mtools-The MongoDB artifact that you might not use.