CairoPlot makes log files on Linux servers more intuitive (1)

Some Linux server administrators really enjoy reading and checking log files, but why not create a beautiful list and graphic system to highlight those faults and problems, but do you have to suffer from this crime? Try this excellent tool-CairoPlot, which provides you with a beautiful and visualized Server Log File Analysis Path.

As a practitioner who needs to deal with data all day, I have been striving to find better ways to display complex data as lists and graphs, especially using Python to achieve this goal. There are a lot of integrated software packages made using Python, but if you want to output the results, it will not be laughed at by those Apple users because of the rough visual effect, then I strongly recommend CairoPlot to you.

CairoPlot is not packaged as most of the released software, but its installation process is still simple. The latest version is 1.1 on the CairoPlot Launchpad page (CairoPlot official homepage. You can download the cairoplot-1.1.tar.gz file there, or search for BZR based on your preferences (once version 1.2 is released, the CairoPlot project may be transferred to Sourceforge.net as a whole ).

Decompress the package:

$ tar xvf cairoplot-1.1.tar.gz

Copy the following file: cairoplot-1.1/CairoPlot. py and paste it to the directory where the Python script you want to develop resides.

The Slice chart shows: who is sending spam?

Finding a good data source is always our top priority before we start surveying and mapping. For this project, let's first analyze a Postfix log file,/var/log/mail.info, to observe the many sources of a series of spam.

Through random checks on files, we will find that many emails that receive requests come from an objectively nonexistent address. For example:

Mar 5 15:05:45 mailserver postfix/smtpd [29764]: NOQUEUE: reject:RCPT from 212.199.94.45.static.pdf 12.net.il [212.199.94.45]: 450 4.7.1 <ex02.maccabiworld.org>: Helo command rejected: Host not found; from = <> to = <aiglance@mydomain.com> proto = ESMTP helo = <ex02.maccabiworld.org>

Our posifix servers usually reject such emails because they are usually spam. Correctly configured email servers should not fabricate these fake addresses-of course, this may happen on servers with incorrect configurations.

But where do these fake receiving requests come from? Are they from specific countries? In specific countries, how many. com websites exist and how many. net websites exist?

To find the answer, I will create a Python index system and then use the CairoPlot tool to plot a sector chart. Each keyword in the index will cover a top-level domain, such as ". com", and its value is the number of rejected emails from this type of domain.

Profiling log files

To fill in the entry items in the index system, we need to analyze the/var/log/mail.info file. The real Sending address of each mail can be queried from RCPT, and the result is applied to the re module of Python. Because this process is for CairoPlot, we do not have to follow the Python description method, but simply follow the form of the following code:

#! /usr/bin/env pythonimport CairoPlot, reMAIL_INFO = "/var/log/mail.info"# Dictionary to store the results as (domain : number of rejects)rejected = {}# Parse mail.info to find all the 'NOQUEUE: reject' lines and# figure out what top-level domains (TLDs) they're coming from.f = open(MAIL_INFO)for line in f :if line.find('status=sent') > 0 :passelif line.find('NOQUEUE: reject') > 0 :# An attempt we rejected. Look for a pattern like# RCPT from foo.example.com[nnn.nnn.nnn.nnn]rcpt = re.search("RCPT from ([^[]*)\[([0-9\.]+)\]", line)if not rcpt :continue# Now rcpt.group(1) is the reverse-DNS hostname (if any)# from the log file, rcpt.group(2) is the IP address.if rcpt.group(1) and rcpt.group(1) != 'unknown' :hostname = rcpt.group(1)else :hostname = None# Find the part after the last "."tld = "Unknown" # default there's no "." in the hostnameif hostname :dot = hostname.rfind(".")if dot >= 0 :tld = hostname[dot+1:]if tld in rejected :# We've seen this TLD before; add 1.rejected[tld] += 1else :# First time we've seen this TLD.rejected[tld] = 1f.close()

At the end, the "rejection" standard in the index system is transmitted to CairoPlot through the following content.

{'ru': 3, 'ch': 1, 'ma': 2, 'rs': 2, 'it': 4, 'hu': 1, 'cz': 1, 'ar': 2, 'il': 35, 'br': 16, 'es': 1, 'co': 2, 'net': 4, 'com': 24, 'pl': 7, 'at': 2}