Automated web security O & M

You may also encounter this situation. The company does not have any security equipment, nor has any funds to purchase and deal with layer-by-layer attacks. Therefore, the company has only one security engineer, how can we automate the protection of the company's web security?
Next, let's talk about some practical operations based on my experience. Because all of them are manual and web security is maintained to filter security attacks in real time, you must create a proxy in front of the web server or have a proxy at the code layer.
Layer, real-time detection of requests, and then to the application, this involves writing web application firewall, relatively speaking, an independent security engineer is more complex, simpler?
In fact, I took two steps. The first step was automated inspection. For the first server security inspection, there are some checklists on the network, which will not be listed. This is not the focus of this article.
The server performs a comprehensive and detailed checklist check at the beginning. After the repair, the baseline security can be basically achieved. Next, we need to perform daily inspection. What is the topic of inspection? Key document changes. The ruby I used at the time,
The core code is adapted from the Internet:
# Ruby compares two files
Def cmpfile (source_file, tmp_file, security_type)
Num = 0
 
F1 = File. open (source_file)
F2 = File. open (tmp_file)
If! File. zero? ("Source_file") and! File. zero? ("Tmp_file") then
Num1, f1array = getFileLine (f1)
Num2, f2array = getFileLine (f2)
 
# Ensure the numberof loop
If num1> num2
Num = num1
Else
Num = num2
End
 
For I in (0 .. num-1)
Mesg1 = "Exception: Maybe file"
Mesg2 = "was not enough lines. Cant find the data when check line"
 
If f1array [I]! = Nil and f2array [I]! = Nil
If f1array [I]! = F2array [I]
$ Rrp. write ("\ n -- # {security_type} -- num:" + I. to_s + "\ n" + "source_file:" + f1array [I]. to_s + "\ n" + "new_file:" + f2array [I]. to_s)
End
Else
If f1array [I] = nil
$ Rrp. write ("\ n -- # {security_type} -- num:" + I. to_s + "" + mesg1 + '1' + mesg2 + (I + 1 ). to_s)
Break
Else if f2array [I] = nil
$ Rrp. write ("\ n -- # {security_type} -- num:" + I. to_s + "" + mesg1 + '2' + mesg2 + (I + 1 ). to_s)
Break
Else
Break
End
End
End
End
End
F1.close
F2.close
End
 
Def getFileLine (f)
Farray = []
Num = 0
F. each do | fi |
Num + = 1
Farray + = [fi. strip]
End
Return num, farray
End
My idea is to keep all the items to be monitored as an original file and store it in a special folder. Each inspection is performed through the following:
# Execute commands to extract source information
Def resource_source (cmd, source_path_file)
Cont = '# {cmd }'
If! File. exist? (Source_path_file)
File. new (source_path_file, "w"). write (cont)
End
End
# Execute commands to extract Temporary Information
Def pai_tmp (cmd, tmp_path_file)
Cont = '# {cmd }'
File. new (tmp_path_file, "w"). write (cont)
End
In this way, the results of each manual command execution are stored in a special file and compared with the original file (see the content comparison function ). Then, we optimized and sorted it out a little more, and an automated detection step was executed,
You can change the original file within one week or in special circumstances, and replace the new standard file with the original file for comparison. Perform a scheduled task and perform a check every five minutes. Once a file change or Trojan is detected
It can be detected in the generated change report file immediately.
In the early stage, I had worked like this for one month, and it was much easier to work. Later, a buddy in the company was familiar with ruby, so the company needed a ruby risk monitoring system, this article focuses on the Risk Monitoring System.
The key point is to send the information about the detection to the backend for display. The message platform we use is amqp. Let's talk about the overall idea first, and then refer to the core code explanation below. The risk monitoring system can manually detect the generated packets.
Extract key information from logs and send it to the backend. the backend displays images, tables, and other data based on the obtained data. This is divided into three parts. Source of information: raw logs and information automatically generated during the early stage;
Information Analysis: sort the analysis, retrieve the special information containing keywords for classification and classification, and place it in different pipelines for sending to the background; Information Display: the background first stores the sent information in the database, then retrieve the database as needed
Displays data in graphs or tables.
In fact, this detailed adaptation and optimization is a complete soc system. At present, the domestic soc system is nothing more than this, but it has to be bought for hundreds of thousands or even millions of dollars, so we despise it very much. In fact, the key to soc is rule Association and
Device interaction means that in our information analysis part, we define a large number of rules for various log association analysis. I did not do this in my own program. At that time, I only engaged in apache and resin log analysis, all of them are simple. The whole system
It took a week or two to work with another engineer to go online and run normally. After that, all the situations were put on hold. However, some engineers could still save some time by providing some reference ideas, hope to help some people. See below
Code, not to mention:
#! /Usr/local/ruby/bin/ruby
= Begin
Author: kn1ghtc
Describe: that's true
= End
Require 'rubygems'
Require 'amqp'
# Ruby was a bit of a whim, but I still don't know much about it. I wrote it after reading a book, so there are a lot of shanzhai usage. Don't laugh, logs stored on the server are automatically appended to the following files every day by time,
# Let's first define several variables to get the source file path
$ Con_source_path_ssl = ".../access_ssl _ # {Time. now. strftime (" % Y % m % d ")}. log"
$ Con_source_path_access = "/../access _ # {Time. now. strftime (" % Y % m % d ")}. log"
$ Con_source_path_error = "/../error _ # {Time. now. strftime (" % Y % m % d ")}. log"
# Define several variables and save the file content filtered by rules. The figure here is simple. Four levels and directions are allocated manually. You can optimize the rules and categories by yourself.
$ Tmp_scan = []
$ Tmp_attack = []
$ Tmp_app = []
$ Tmp_abnormal = []
# The log format on the server has its own generated rules. It doesn't matter. We have standardized them into our format as needed to facilitate subsequent analysis of rule application and storage presentation.
# Log_format
Def message_format (str_line, str_type, str_risk, str_format)
@ Str_time = ""
@ Str_ip = ""
@ Str_data = ""
@ Str_result = ""
@ Str_format = str_format
@ Str_line = str_line
 
If @ str_format = "error"
@ Str_time =/[a-zA-Z] {3 }. [a-zA-Z] {3 }. \ d {2 }. \ d {2 }:\ d {2 }:\ d {2 }. \ d {4 }/. match (@ str_line)
@ Str_ip =/\ d {1, 3} \. \ d {1, 3} \. \ d {1, 3} \. \ d {1, 3}/. match (@ str_line)
@ Str_data = @ str_line.gsub (/^ \ [[a-zA-Z] {3 }. [a-zA-Z] {3 }. \ d {2 }. \ d {2 }:\ d {2 }:\ d {2 }. \ d {4} \]. \ [[a-zA-Z] {3, 5} \]. \ [[a-zA-Z] {6 }. \ d {1, 3 }\. \ d {1, 3 }\. \ d {1, 3 }\. \ d {1, 3} \]/, ""). strip
Else
@ Str_time =/\ d {2} \/[a-zA-Z] {3} \/\ d {4 }:\ d {2 }: \ d {2 }:\ d {2 }/. match (@ str_line)
@ Str_ip =/^ \ d {1, 3}. \ d {1, 3}. \ d {1, 3}. \ d {1, 3}/. match (@ str_line)
@ Str_data = @ str_line.gsub (/^ \ d {1, 3 }. \ d {1, 3 }. \ d {1, 3 }. \ d {1, 3 }..... \ [\ d {2} \/[a-zA-Z] {3} \/\ d {4 }:\ d {2 }:\ d {2 }: \ d {2 }. \ + \ d {4} \]/, ""). strip
End
@ Str_result = @ str_ip.to_s + "=>" + @ str_data.to_s + "=>" + @ str_time.to_s + "=>" + str_type + "=>" + str_risk + "\ n"
End
# Retrieve each row in the log, put it into the following function, set rules, and put matching rules into the corresponding array.
# Line_rule_match
Def line_rule (rule_line, rule_log_type)
@ Rule_str_line = rule_line
@ Rule_log_type = rule_log_type
@ New_line = ""
@ Rule_data = []
 
If @ rule_str_line = ~ /Error/
$ Tmp_abnormal <message_format (@ rule_str_line, "abnormal", "M", @ rule_log_type)
Else
@ New_line = message_format (@ rule_str_line, "unknown", "unknown", @ rule_log_type). split (/=>/) [1]
@ Rule_data = @ new_line.split (/"/)
If @ rule_data.size> 6
$ Tmp_scan <message_format (@ rule_str_line, "scan", "M", @ rule_log_type)
Else
If @ rule_data.size> 1
If @ rule_data [3] = ~ /Script> | <script | passWord | password/
$ Tmp_attack <message_format (@ rule_str_line, "attack", "H", @ rule_log_type)
# Some status code checks
Elsif @ rule_data [1] = ~ /'| Alert | script> | cat | union | order by |;| \. \. \/| and | --/or @ rule_data [2]. lstrip. split (/\ s/) [0] = ~ /400 | 403 | 404 | 500/
$ Tmp_scan <message_format (@ rule_str_line, "scan", "M", @ rule_log_type)
End
 
End
 
End
End
 
End
# Log_ids
Def invade_log (source_path_file, log_type)
@ Path_source = source_path_file
@ Log_type = log_type
I = 0
@ Line_num_tmp =-1 www.2cto.com
# Shanzhai idea. At that time, various server performance restrictions and causes were imposed. All scripts were run once every one minute in order to get the log Content in real time (web logs were added ).
# Run our script every minute. The script will find a special file from the created special folder, generate the row number of the original file in the file, compare the row number, and start scanning down from the new row number, original row number
If @ log_type = "ssl"
File. open ("/tmp/num_ssl", "r") do | file |
@ Line_num_tmp = file. gets. to_ I
End
Elsif @ log_type = "access"
File. open ("/tmp/num_access", "r") do | file1 |
@ Line_num_tmp = file1.gets. to_ I
End
Else
File. open ("/tmp/num_error", "r") do | file2 |
@ Line_num_tmp = file2.gets. to_ I
End
End
File. open (@ path_source) do | files |
Files. each_line do | line |
Begin
Line. force_encoding ("gbk ")
If I> @ line_num_tmp
Line_rule (line, @ log_type)
Else
I = I + 1
Next
End
I = I + 1
Rescue
I = I + 1
$ Tmp_app <message_format (line, "app", "L", @ log_type)
End
End
End
@ Line_num_tmp = I
@ File_log = File. open ("/tmp/num _ # {@ log_type}", "w ")
@ File_log.write (@ line_num_tmp)
@ File_log.close
End
# The row number file solves the problem of implementing detection.
# Time_question
# Implement_method
Def implement_meth (source_path_file, log_type)
@ Im_path_source = source_path_file
@ Im_log_type = log_type
@ Filename_old = ""
 
If File. exist? (@ Im_path_source)
File. open ("/tmp/time _ # {@ im_log_type}") do | file |
@ Filename_old = file. gets. to_s.strip
End
If @ im_path_source! = @ Filename_old
@ Im_file_log = File. open ("/tmp/num _ # {@ im_log_type}", "w ")
@ Im_file_log.write ("-1 ")
@ Im_file_log.close
@ Im_file_log = File. open ("/tmp/time _ # {@ im_log_type}", "w ")
@ Im_file_log.write (@ im_path_source)
@ Im_file_log.close
End
Invade_log (@ im_path_source, @ im_log_type)
End
End
# The application of mq message defines channel-related information. The basic format here is as follows, which can be modified as your own.
# Send_message_method_scan
Def send_to_exchange_scan (message)
Exchange = MQ. direct ('guofubao ')
Exchange. publish message,: key => 'scan'
End
# Send_message_method_atack
Def send_to_exchange_attack (message)
Exchange = MQ. direct ('guofubao ')
Exchange. publish message,: key => 'attack'
End
# Send_message_method_app
Def send_to_exchange_app (message)
Exchange = MQ. direct ('guofubao ')
Exchange. publish message,: key => 'app'
End
# Send_message_method_abnormal
Def send_to_exchange_abnormal (message)
Exchange = MQ. direct ('guofubao ')
Exchange. publish message,: key => 'abnormal'
End
 
# Test_data
# Tmp_test = "127.0.0.1 => kn1ghtc_test => 201:12:31 => unknown" + "\ n"
# Implement
Implement_meth ($ con_source_path_access, "access ")
Implement_meth ($ con_source_path_error, "error ")
Implement_meth ($ con_source_path_ssl, "ssl ")
# Conn
# Application message sending and method body
AMQP. start: host => '127. 0.0.1 ',: port => 127 do
Event_loop = Thread. new do
EM. run do
EM. add_timer (1) do
EM. stop
End
End
End
Send_to_exchange_abnormal $ tmp_abnormal.join (). to_s
Send_to_exchange_app $ tmp_app.join (). to_s
Send_to_exchange_attack $ tmp_attack.join (). to_s
Send_to_exchange_scan $ tmp_scan.join (). to_s
 
Event_loop.join
End
 
 
End: I was too lazy. At that time, I was too hasty to write and thought about the subject. Later, the system went offline for other reasons. I wrote it a few months ago. Now I only remember my thoughts, I think it's useful to look at the official website.

Author: knightc