Blinded and forgotten-using application delivery to dialysis large web site logs

Memories like a machine that's been on the open
While I'm not paying attention
Slowly, clearly and repeatedly.

--"30,000 feet" Dick Cowboy

when the flight climbed and fell at a height of 30,000 feet, across the stream, sinking into the seabed, no one knew what was going on in the clouds, or even knowing where it was eventually sleeping, and over time, it might have been forgotten. To sort out everything and regain the memory, the only hope is to find the black box with a faint electric wave that remembers everything.

650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M00/7C/E8/wKioL1bb5-_Qev40AARt2Vu4KYM629.png "/>

Server Web site logging is the case, it is the Web server's black box, the daily operation of our failures are often very dependent on the analysis of statistical logs to find problems. Like the protagonist of the suspense film, every few minutes will forget their identity of their own name, need to keep writing in the book, posted around the wall, in the record when we have to look around, once omitted did not see can write down, it may be the end of tomorrow.

Fortunately we have a log, all of this is automated!

Load balancer devices are at the forefront of the task of unifying the delivery of business data at the front-end of the site and are the core of the focus, and we use the widely used NetScaler as an example to explore the help that can be brought in the website log operation.

One: Record website high latency slow response time via NetScaler

Typically, a site has a standard format similar to

192.35.100.21--[19/jan/2016:14:47:37-0400] "get/netscaler.html http/1.1" 200 6553

Such logs can always record access to the source address, access time, destination URL, and so on. But this is often not enough: in addition to the user side of the problem, the server backend is a part of our concern. For example, server performance is slow, user access to a webpage is delayed, and so on. When the user reported the failure of this access delay has occurred in the past, if it is the above log format, how to find the moment there is a problem?

Record a lot, can not be recorded the key! This moment feels like losing the light!

650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M01/7C/E9/wKiom1bb53OCDoi4AASu67pSRbE876.png "/>

To lock down a fault you must be able to record the server-side response time before you can identify the culprit.

We need to record the response time and write the log as follows

Sep 7 03:04:21 <local0.alert> 127.0.0.2 09/06/2015:19:04:21 GMT myns 0-ppe-0: Default REWRITE Message 10782 0: "My log:clientip:192.168.20.1:62716 serverip:192.168.50.68:80 host:192.168.20.67 URL:/index.asp restime:90ms "

This is a very easy job for NetScaler, NetScaler can directly see through the content of the transmission and make analysis rankings, can be in accordance with the request of the unit time, bandwidth, or response time ranking, at a glance

650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M02/7C/EA/wKiom1bb53SBTwFgAAE0S5pKw6g777.png "/>

Of course, such statistics can also be reflected in the text log. We can specify the format of the log in detail and check the user-defined log information at the log output.

650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M02/7C/E8/wKioL1bb5_OwmX4BAAAkXrongnU378.png "/>

While our test logs are simple and straightforward, the log generated is very small, but in fact a large number of users of the huge web site log is huge, a few hours may be calculated at the G level. If each record records the response time, it is still difficult to find a high latency, and you need to use various tools to query the filter. NetScaler can also directly define the standard response time , which is generated and recorded in the log only after the delay that is above our definition has occurred.

Log format content according to the requirements of any definition, not only in English, we can also define the Chinese journal on the NetScaler directly view

650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M01/7C/E8/wKioL1bb5_Sy4JmhAANEvpVk-jc265.png "/>

the cumulative log, preferably with a dedicated log server to accept, NetScaler can also send high-latency logs to any defined remote log server, the custom address is as follows:

650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M00/7C/EA/wKiom1bb53aCYLzwAAA8DLnI9qQ760.png "/>

High-latency alert logs can be received on the server, using the free Kiwi Syslog server

650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M02/7C/E8/wKioL1bb5_aTSSoVAAMl7OI0_e0774.png "/>

Because NetScaler naturally handles HTTP traffic in a seven-tier manner, dynamic statistics like the above do not affect performance at all.

Two: Centralized statistical log through NetScaler

By default, server logs are recorded on each server. The inspection of each device log is extremely time-consuming and laborious. We need to configure each server separately, send logs to the central log server to record, regular file cutting, log analysis.

650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M00/7C/E8/wKioL1bb5_eS8m87AAFlXlq9OBQ334.png "/>

as mentioned above, in fact, most of the front end of the site are deployed load balancing system, all VIP virtual sites are on load balancer, the browser access server must be load balancer device, then you have to let load balancer directly generate log send log, and eliminate the hassle of managing many server configurations on the backend?

NetScaler itself has this function, called NetScaler Web Logging (NSWL), which can be transported through a single sending point. Http/https logs can be generated directly in the standard format of W3C/NCSA, and the log generated by Nginx Apache is exactly the same. The format can be adjusted according to the specified variables. Logs can be sent to our common LINUX,WINDOWS,FREEBSD system.

650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M00/7C/EA/wKiom1bb53mTUuFAAAFKj_T5-gA571.png "/>

@Netscaler_Insight

We can also define the build cycle for each log file (e.g. hourly per day), generate file size (e.g. 100M 1G), log file name (e.g. Exmmyydd.log), virtual host name (e.g. www.netscaler.com ) does not require you to cut logs with your own edited script.

Three: Through appflow Visualization log

In the 90 's, Cisco developed a statistical protocol for network traffic Analysis (NetFlow) for operators and enterprises, requiring no probes and low and functional CPU and network requirements. After multiple version upgrade optimizations, Netflow V9 was identified by the IETF organization as the Ipfix (IP flowinformation Export) standard from 5 candidate scenarios. NetFlow is widely used in network monitoring, Citrix developed a new variety of extensions based on this standard, such as application layer parameters, Web page performance parameters, database parameters, etc.

650) this.width=650; "title=" 1.png "alt=" Wkiol1bb6zhc_jetaakuaubidsq930.png "src=" http://s1.51cto.com/wyfs02/M00/ 7c/e8/wkiol1bb6zhc_jetaakuaubidsq930.png "/>

Using these collected data, NetScaler Insight collects statistics on application layer information and displays it with powerful visual graphs.

For example, access to geographic locations, bandwidth consumption, and average response time

650) this.width=650; "title=" 2.png "style=" Float:none; "alt=" wkiol1bb6lmwgnwmaagxk38fxla931.png "src=" http:/ S1.51cto.com/wyfs02/m00/7c/e8/wkiol1bb6lmwgnwmaagxk38fxla931.png "/>

Access Source IP

650) this.width=650; "title=" 3.png "style=" Float:none; "alt=" wkiom1bb6dvgyzjfaabewohbhag045.png "src=" http:/ S4.51cto.com/wyfs02/m01/7c/ea/wkiom1bb6dvgyzjfaabewohbhag045.png "/>

Request Quantity Curve

650) this.width=650; "title=" 4.png "style=" Float:none; "alt=" wkiom1bb6dzhwugcaacdkq2lnnw208.png "src=" http:/ S4.51cto.com/wyfs02/m01/7c/ea/wkiom1bb6dzhwugcaacdkq2lnnw208.png "/>

You can also make statistics for your specific business

650) this.width=650; "title=" 5.png "style=" Float:none; "alt=" wkiol1bb6luw7a9baabecvov0mg589.png "src=" http:/ S4.51cto.com/wyfs02/m01/7c/e8/wkiol1bb6luw7a9baabecvov0mg589.png "/>

Enter the specific business, can be specific URL and client analysis, you can see the number of hits per URL, rendering time, load time, etc.

650) this.width=650; "title=" 6.png "style=" Float:none; "alt=" wkiom1bb6d3alpbiaacd3_wtvjo174.png "src=" http:/ S4.51cto.com/wyfs02/m02/7c/ea/wkiom1bb6d3alpbiaacd3_wtvjo174.png "/>

Enter a specific URL to see the time curve

650) this.width=650; "title=" 7.png "style=" Float:none; "alt=" wkiol1bb6nzanvm0aadydnrsyvo538.png "src=" http:/ S3.51cto.com/wyfs02/m01/7c/e8/wkiol1bb6nzanvm0aadydnrsyvo538.png "/>

Detailed analysis of the entire page

650) this.width=650; "title=" 8.png "style=" Float:none; "alt=" wkiom1bb6f6svo4saadlanr-gva347.png "src=" http:/ S1.51cto.com/wyfs02/m02/7c/ea/wkiom1bb6f6svo4saadlanr-gva347.png "/>

Access to the operating system, browser type, response type, etc.

650) this.width=650; "title=" 9.png "style=" Float:none; "alt=" wkiol1bb6n2bxoj8aab9bojhzw8700.png "src=" http:/ S4.51cto.com/wyfs02/m02/7c/e8/wkiol1bb6n2bxoj8aab9bojhzw8700.png "/>

Summarize

Although it has been more than 10 years since the application was delivered to the device, many users simply equate it with load balancing and even just the polling algorithm. In fact, with the development of technology, ADC has become the network to see through and control the application layer of the brain and eyes, the use of the ADC can be from the network's three or four-tier helicopter into the application layer of the world.

Finally found that we have no amnesia, nor blindness, we have always had eyes, to do is just remove the eye patch, open eyes.

The memory of open eyes is light, and the memory of closing your Eyes is darkness.

650) this.width=650; "title=" 10.png "style=" Float:none; "alt=" wkiol1bb6ocsjupoaayrkmsrfoe567.png "src=" http:/ S4.51cto.com/wyfs02/m02/7c/e8/wkiol1bb6ocsjupoaayrkmsrfoe567.png "/>

This article is from the "netscaler_insight" blog, make sure to keep this source http://netscaler.blog.51cto.com/9136680/1748082

Blinded and forgotten-using application delivery to dialysis large web site logs