Mtools for mongodb logs analyzer

mtools is a collection of helper scripts to parse and filter MongoDB log files (mongod, mongos), visualize log files and quickly  for performance tuning and debugging the mongodb logs easily.

The following tools are in the mtools collection:

mlogfilter

slices log files by time, merges log files, filters slow queries, finds table scans, shortens log lines, filters by other attributes, convert to JSON

To get the collection scan collections 

 mlogfilter mongod.log --planSummary=COLLSCAN

To get the slow queries more than 1000ms 

mlogfilter mongod.log --slow 1000

To get the errors,warnings 

 mlogfilter mongod.log  --word assert warning errors

To get the how many queries execute the queries on namespace

mlogfilter mongod.log --namespace admin.\$cmd --slow 1000

mloginfo

returns info about log file, like start and end time, version, binary, special sections like restarts, connections, distinct view

mplotqueries

visualize logfiles with different types of plots (requires matplotlib)

mlogvis

creates a self-contained html file that shows an interactive visualization in a web browser (as an alternative to mplotqueries)

mlogvis mongod.log --out mongo.html

mlaunch

a script to quickly spin up local test environments, including replica sets and sharded systems (requires pymongo)

mgenerate

generates structured pseudo-random data based on a template for testing and reproduction

Usage of the tools :

Mloginfo reads logs generated by MongoDB and returns informations about the usage of the database. In our case, we wanted to monitor and profile our request to define the ones which will be take a long amount of time and the ones which will be used the most.

Ex:

--queries 

The queries section will go through the log file and find all queries (including queries from updates) and collect some statistics for each query pattern. A query pattern is the shape or signature of a query (similar to an index definition) without the actual query values for each field.

mloginfo --queries logs.mongo/mongod.log

namespace              operation        pattern                              count     min (ms)    max (ms)    mean (ms)    95%-ile(ms)    sum (ms)

myCol.$cmd               findandmodify    {"ID": 1}                          916493         101       10486          277          453.0    254423325
myCol.User               count            {"activeSeg": 1, "updatedAt": 1}   68           30135     1413998       419353      1350776.4    28516024
myCol.InactiveUser       count            {"inactiveSeg": 1}                 32

--sort 

This option can be used to sort the results of the --queries

mloginfo mongod.log --queries --sort count

mloginfo mongod.log --queries --sort sum 

--restarts 

The restarts section will go through the log file and find all server restarts. It will output a line per found restart, including the date and time and the version.

mloginfo mongod.log --restarts

The distinct section goes through the log file and group all the lines together by the type of message (it uses the "log2code" matcher). It will then output a line per group, sorted by the largest group descending. This will return a good overview of the log file of what kind of lines appear in the file.

mloginfo mongod.log --distinct

The connections section returns general information about opened and closed connections in the log file, as well as statistics of opened and closed connections per unique IP address.

To ge the open and closing connections info 

mloginfo mongod.log --connections 

Outputs information about every detected replica set state change.

mloginfo mongod.log --rsstate