Web site active probing tool use

?

Unit of the project is Ibatis do, each query SQL inside have a lot of judgment
After the last optimization of SQL, one of the branches is an error, but as a DBA, it is not possible to troubleshoot each branch.
So, simply crawl through all the Web pages, actively detect the program anomalies.
There are two benefits to this. 
1. Can proactively see if the webpage is abnormal (500 error, 404 error) 
2. You can screen slow pages, in this direction you can also locate the slow SQL bar. (there is also a shortage of server resources, resulting in a network timeout situation) 
Premise 
Must be an Internet company, most Web pages do not have to log in or browse 
First, build the table 
CREATE SEQUENCE seq_probe_id INCREMENT by 1 START with 1 nomaxvalue nocycle CACHE 2000; 
CREATE TABLE Probe ( 
ID int PRIMARY KEY, 
Host varchar (+) is not NULL, 
Path varchar (.) is not NULL, 
The state int is not NULL, 
Tasktime int NOT NULL, 
Type varchar (TEN) is not NULL, 
Createtime date default sysdate NOT NULL 
) ; 
Where host is the domain name, path is the relative path of the Web page, State is the HTTP status code, Tasktime is the Web Capture time, the unit is milliseconds, type is the types (html,htm,jpg, etc.) 
　　  Program Structure

The program is divided into three main steps, and then the producer consumer model is implemented with three queues respectively.
1. Connect. Based on the destination of the connection queue, use the socket to get the Web page and then put the parsing queue
2. Parse. Based on the contents of the parsing queue, a regular expression is used to obtain a legitimate connection to the Web page and then put it into the connection queue. Then put the parsed page into the persistence queue
3. Persist the contents of the persisted queue into the database for querying.  
The program uses three steps in parallel, and each step can be in a concurrent manner.

$new _page$

However, in general, parsing and persistence can be performed in a single thread, respectively.
Import Java.io.BufferedReader;
Import Java.io.BufferedWriter;
Import Java.io.InputStreamReader;
Import Java.io.OutputStreamWriter;
Import java.net.InetAddress;
Import Java.net.Socket;
Import java.sql.Connection;
Import Java.sql.DriverManager;
Import java.sql.PreparedStatement;
Import java.sql.SQLException;
Import java.util.ArrayList;
Import Java.util.Iterator;
Import java.util.List;
Import Java.util.Set;
Import Java.util.concurrent.BlockingQueue;
Import Java.util.concurrent.ConcurrentSkipListSet;
Import java.util.concurrent.CopyOnWriteArrayList;
Import Java.util.concurrent.ExecutorService;
Import java.util.concurrent.Executors;
Import Java.util.concurrent.LinkedBlockingQueue;
Import Java.util.concurrent.atomic.AtomicInteger;
Import Java.util.regex.Matcher;
Import Java.util.regex.Pattern;
public class Probe {
private static final blockingqueue<task> connectlist = new linkedblockingqueue<task> ();
private static final blockingqueue<task> parselist = new linkedblockingqueue<task> ();
private static final blockingqueue<task> persistencelist = new linkedblockingqueue<task> ();
private static Executorservice Connectthreadpool;
private static Executorservice Parsethreadpool;
private static Executorservice Persistencethreadpool;
private static final list<string> DomainList = new copyonwritearraylist<> ();
static {
Connectthreadpool = Executors.newfixedthreadpool (200);
Parsethreadpool = Executors.newsinglethreadexecutor ();
Persistencethreadpool = Executors.newfixedthreadpool (1);
Domainlist.add ("domain name");
}
public static void Main (String args[]) throws Exception {
Long start = System.currenttimemillis ();
Connectlist.put (New Task ("Domain name", "/static/index.html"));
for (int i = 0; i <; i++) {
Connectthreadpool.submit (New Connecthandler (Connectlist, parselist));
}
Parsethreadpool.submit (New Parsehandler (Connectlist, Parselist, Persistencelist, domainlist));
Persistencethreadpool.submit (New Persistencehandler (persistencelist));
while (true) {
Thread.Sleep (1000);
Long end = System.currenttimemillis ();
Float interval = ((End-start)/1000);
int connecttotal = Connecthandler.getcount ();
int parsetotal = Parsehandler.getcount ();
int persistencetotal = Persistencehandler.getcount ();
int Connectps = Math.Round (connecttotal/interval);
int parseps = Math.Round (parsetotal/interval);
int persistenceps = Math.Round (persistencetotal/interval);
System.out.print ("Total connections:" + connecttotal + "connections per Second:" + Connectps + "Connection Queue remaining:" + connectlist.size ()
+ "Total Resolution:" + Parsetotal + "Resolution per Second:" + parseps + "Resolve queue remaining:" + parselist.size () + "Total number of persisted:"
+ Persistencetotal + "persistence per Second:" + persistenceps + "Persistent queue remaining:" + persistencelist.size ());
}
}
}
Class Task {
Public Task () {
}
public void init (string host, int port, string path) {
This.setcurrentpath (path);
This.host = host;
This.port = port;
}
Public Task (string host, int port, string path) {
Init (host, port, path);
}
Private String host;
private int port;
Private String Currentpath;
Private long tasktime;
Private String type;
Private String content;
private int state;
public int getState () {
return state;
}
public void SetState (int.) {
This.state = State;
}
Public String Getcurrentpath () {
return currentpath;
}
public void Setcurrentpath (String currentpath) {
This.currentpath = Currentpath;
This.type = currentpath.substring (Currentpath.indexof (".") + 1,
Currentpath.indexof ("?")! =-1? Currentpath.indexof ("?"): Currentpath.length ());
}
Public long Gettasktime () {
return tasktime;
}
public void Settasktime (long tasktime) {
This.tasktime = Tasktime;
}
Public String GetType () {
return type;
}
public void SetType (String type) {
This.type = type;
}
Public String GetHost () {
return host;
}
public int Getport () {
return port;
}
Public String getcontent () {
return content;
}
public void SetContent (String content) {
this.content = content;
}
}
Class Parsehandler implements Runnable {
private static set<string> Set = new concurrentskiplistset<string> ();
public static int GetCount () {
return Count.get ();
}
private static final Atomicinteger COUNT = new Atomicinteger ();
Private blockingqueue<task> connectlist;
Private blockingqueue<task> parselist;
Private blockingqueue<task> persistencelist;
List<string> DomainList;

This article was selected from: http://www.spasvo.com/news/html/2014122144836.html

?

Web site active probing tool use