"Go" IIS Logs-a good helper for website operations

For a long-term maintenance of the site, how to make the site long-term stable operation is a very meaningful thing. It is also normal that some problems that are not exposed during the development phase are likely to occur in the operations phase. There are also times when we want to constantly optimize our website and make it more responsive to user requests, all of which occur in the operational phase after development.

Unlike the development phase, the operations phase is not likely to allow you to debug the program, to find all kinds of problems, we can only through a variety of system logs to analyze the health of the site, for the site deployed on IIS, IIS logs provide the most valuable information, we can use it to analyze the response of the site, To determine whether the site has performance issues, or what needs to be improved.

Back to top what information does the IIS log contain

I said earlier that "IIS logs provide the most valuable information", what are the information? Let's look at this:

Here's the record:
1. At what point the request occurred,
2. Which client IP has access to the port of the server IP,
3. What is the client tool type, what version,
4. The URL of the request and what the query string parameter is,
5. Whether the requested method is get or post,
6. What is the processing result of the request: the HTTP status code, and the underlying status code of the operating system,
7. How much data is uploaded by the client during the request and how much data is sent by the server?
8. How long the request takes up the server, and so on.

What is the use of this information in analysis, I'll say it later. Let's make an impression on it first.

Back to the top of the IIS log configuration

By default, IIS generates log files, but there are some parameters that deserve our attention. The Setup interface for IIS is as follows (this article takes the IIS 8 interface as an example).

In IIS Manager, select a Web site, and double-click the log icon, refer to:

At this point (main section) the interface is as follows:

In, the log is created every day to generate a new file, and the file name is generated by date (this is the default value).
Description: IIS uses UTC time, so I ticked the bottom check box to tell IIS to generate the file name with local time.

Click the "Select Field" button and the following dialog box will appear:

Note:the number of fields sent and number of bytes received is not selected by default. It is recommended to check them.
As for the other fields, you can decide whether or not to tick them as needed.

Back to top how to parse IIS logs

If you set the IIS log parameters in the way that I described earlier, IIS logs are generated after the request is processed (after a period of time).
We can quickly navigate to the root of the IIS log by tapping "View log File" in the "Actions" section on the right area of the "Log Interface", and then look for the appropriate log file in the directory (by default, the directory is differentiated according to the application pool ordinal).

For example: I found the log I needed:

This file is a whole bunch of characters, how can I analyze it now?

A tool called log Parser can specifically parse the IIS log, which we can use to view the information in the log.
For example, I can run the following command line (note: In order not to affect the width of the page I wrap the command text):

"C:\Program files\log Parser 2.2\logparser.exe"-i:iisw3c-o:datagrid "select C-ip,cs-method,s-port,cs-uri-stem, Sc-status,sc-win32-status,sc-bytes,cs-bytes,time-taken from U_ex130615.log "

You can now read the IIS logs in tabular form:

Description: I do not recommend this method for parsing IIS logs for two reasons:
1. Slow: When the log file is slightly larger, it is a waste of time to analyze it (especially when multiple statistics are required).
2. Inconvenient: The query syntax it supports is not rich enough to be as comprehensive as SQL Server for data table queries.

Back to top recommended IIS Log parsing methods

Although log Parser supports the parsing of IIS logs in tabular form for people to read, sometimes we need to do some detailed analysis, may be different ways of "multiple" query, for this need, if each query directly run log Parser, you will waste a lot of time. Fortunately, Log parser supports exporting parsing results in multiple formats (Help documentation below):

Here, I recommend selecting the output format for SQL.
Note: The SQL here does not refer to SQL Server, but to all databases that provide an ODBC access interface.
I can use the following command to import the IIS logs into SQL Server (note: In order to not affect the width of the page I wrap the command text):

"C:\Program files\log Parser 2.2\logparser.exe"  "Select  *  from  ' D:\Temp\u_ex130615.log '  to Mymvc_weblog "-i:iisw3c-o:sql-oconnstring:" Driver={sql server};server=localhost\sqlexpress;database=mytestdb;i ntegrated Security=sspi "-createtable:on

After the import is complete, we can use the familiar SQL Server to do various queries and statistical analysis, such as the following query:

SELECT Cip,csmethod,sport,csuristem,scstatus,scwin32status,scbytes,csbytes,timetaken from dbo. Mymvc_weblog

If this is the case:

Attention:
1. IIS logs characters that do not conform to the identifier specification in the field name are deleted when the results are exported to SQL Server.
For example: C-IP will become CIP and S-port will become sport.
2. The time recorded in the IIS log is UTC time, and the date and time are separated, and when exported to SQL Server, two fields are generated:

Date, time these two fields look uncomfortable, right?
I am also disgusted with this result, the following two ways to solve:

1. Add a column to SQL Server, and then change the UTC time to the local time zone, as in the T-SQL script:

ALTER TABLE Mymvc_weblog add RequestTime datetimegoupdate mymvc_weblog set Requesttime=dateadd (hh,8,convert (varchar (10 ), date,120)             + "+ CONVERT (varchar (), time,114))

2. When you export the IIS log, convert the time to change the command:

"C:\Program files\log Parser 2.2\logparser.exe"  "Select To_localtime (To_timestamp (ADD (to_string (date, ' Yyyy-mm-dd '), to_string (Time, ' Hh:mm:ss ')), ' Yyyy-mm-dd hh:mm:ss ') as RequestTime, * from  ' D:\Temp\u_ Ex130615.log '  to  mymvc_weblog2 "-i:iisw3c-o:sql-oconnstring:" Driver={sql server};server=localhost\ sqlexpress;database=mytestdb;integrated Security=sspi "-createtable:on

Look at these three columns again:

Select RequestTime, date, time from MYMVC_WEBLOG2

Once this is done, you can simply delete the date, time two columns (you can also omit them when you export the IIS logs, but explicitly specify each field name).

UTC time in the IIS log the problem is here, I hope everyone understands ~~~~~~~~~~~

Go back to the top of the IIS log for exception logging

Each requested information is logged in the IIS log, including a normal response request and an exception request.

The "exception" described here is not related to exceptions in the. NET Framework.
For an ASP, if an uncaught exception is thrown, it is logged to the IIS log (500), but the exception I'm talking about is not limited to this.

The anomalies mentioned in this article can be divided into four parts:
1. An uncaught exception thrown by the program (ASP. NET) causes the server to produce a 500 response output.
2. There are no errors for request resources such as 404.
3. Server error greater than 500, for example: 502,503
4. System error or network transmission error.

The first three types of exceptions can be obtained using the following query:

Select Scstatus, COUNT (*) as Count, sum (Timetaken * 1.0)/1000.0 as Sum_timetaken_secondfrom Mymvc_weblog with (NOLOCK) Grou P by Scstatusorder by 3 desc

There is a column in the IIS log: Sc-win32-status, which records system-level errors, such as network transmission errors, that occur during request processing.
Normally, 0 is normal, and a non-0 value means an error occurred. We can count such errors like this:

DECLARE @recCount bigint;select @recCount = count (*) from Mymvc_weblog with (nolock) Select Scwin32status, COUNT (*) as Count , (COUNT (*) * 100.0/@recCount) as [percent] from Mymvc_weblog with (NOLOCK) where Scwin32status > 0group by Scwin32stat Usorder by 2 desc

The following table lists the more common network-related errors and explanations:

Scwin32status	Meaning
64	Client connection is down (or disconnected)
121	Transfer timeout
1236	Local network outage

All status codes can be explained by the following command:

D:\temp>net HELPMSG 64 The specified network name is no longer available.

With regard to Scwin32status and scstatus, I would also like to add that they are not related.
For example, request this address: http://www.abc.com/test.aspx
It is possible that scstatus=200, but scwin32status=64, indicates that ASP. NET has successfully processed the request, but when IIS sends the response result, the client's connection is broken.
Another scenario is: scstatus=500, but scwin32status=0, which indicates that an uncaught exception occurred during the processing of the request, but the exception result was successfully sent to the client.

Go back to the top and talk about scwin32status=64.

Remember to see scstatus=200,scwin32status=64 this situation is not understand, so search the Internet, all kinds of answers have, and some even say with the web crawler related. In order to verify the various answers, I did an experiment. I write a ashx file that uses it to simulate long-time network transmissions with the following code:

public class Test_iis_time_taken:ihttphandler {public        void ProcessRequest (HttpContext context) {        context. Response.ContentType = "Text/plain";        System.Threading.Thread.Sleep (+ 2);                Context. Response.Write (String. Format ("{0}, {1}\r\n", "Start", DateTime.Now));        Context. Response.Flush ();                System.Threading.Thread.Sleep (+ 2);        for (int i = 0; i <; i++) {            context. Response.Write (String. Format ("{0}, {1}\r\n", I, DateTime.Now));            Context. Response.Flush ();            System.Threading.Thread.Sleep (+ 1);        }        Context. Response.Write ("End");    }

This code is very simple, I do not want to do too much explanation, just want to say: I use Thread.Sleep and Response.Flush these two methods to simulate a long duration of the continuous transmission process.

We can see this output in the browser (show me when it's not completely over)

I have done this test 8 times, only 2 times is full display completed, the remaining 6 times I close the browser window early.
We then look at the contents of the IIS logs:

Depending on the IIS logs and in combination with my own actions, you can discover:
1. When I close the browser window early, I see scstatus=200,scwin32status=64
2. If the requested content is fully displayed, I will see scstatus=200,scwin32status=0
From this experiment we can also find: Timetaken contains the network transmission time.

Based on the results of this experiment, have you ever thought of a problem:
If there is a lot of scstatus=200,scwin32status=64 in the IIS logs of your site, and the request is initiated by the user's browser.
What is the cause of this?
My "guess" is that users have been reluctant to wait while they visit the site, and they have turned off the browser window.
In other words: You can see from the statistical results of scwin32status=64 whether the response speed of the website can make users satisfied.

Back to top for performance issues

The IIS log has a column called: Timetaken, which shows what it means in the IIS interface: all the time.
This time is defined as the time at which the first byte of the request received from the server starts, until all responses have been sent out.
Microsoft's website has made a description of this field: http://support.microsoft.com/kb/944884

Once we know the definition of timetaken, we can use it to analyze the processing time of some requests, that is, performance analysis.

For example, I want to see the load of the slowest 20 pages, so you can query:

Select Top Csuristem,scstatus,scwin32status,scbytes,csbytes,timetakenfrom dbo. Mymvc_weblog with (NOLOCK) where Csuristem like '/pages/% ' ORDER by timetaken Desc

Or I would like to look at the slowest 20 of the response of the Ajax situation, you can query:

Select Top Csuristem,scstatus,scwin32status,scbytes,csbytes,timetakenfrom dbo. Mymvc_weblog with (NOLOCK) where Csuristem like '/ajax/% ' ORDER by timetaken Desc

In summary, the way to look for performance problems is to select the Timetaken field in the query and use it to sort in descending order.

Note:scbytes,csbytes These two fields are also worthy of our attention:
1. If the csbytes is too large, we need to analyze whether the form can be split because it contains too much useless data.
Csbytes There is another possibility: the cookie is too large, but it will behave as a large number of requests csbytes, so it is easy to differentiate.
2. Scbytes if it is too large, we have to check whether the page is not paged, or you can consider the load-on-demand way to implement.
Typically, these two values become larger when ViewState is used in a large number of cases. So we can find the ViewState abuse problem through IIS logs.
Another special case is: Uploading the download file will also cause these two values to become larger, why I do not explain.

Scbytes,csbytes, regardless of which value is large, will occupy the network transmission time, for the user, it will take longer waiting time.

I said three fields at a time, which one should I refer to when looking for performance problems?
I think the timetaken should be given priority because its value directly reflects the user's wait time (excluding the front-end rendering time).
If the timetaken is too large, it is necessary to check whether the scbytes,csbytes is too large,
If the latter is too large, then the direction of optimization is to reduce the amount of data transfer, otherwise the program processing takes up a lot of time, should consider optimizing program code.

Go back to the top for an improved goal

In addition to discovering performance issues from the IIS logs, you can use it to find targets that can be improved.
For example:
1. Are there 404 errors?
2. Is there a large number of 304 requests?
3. Is there a large number of duplicate requests?

When we find that there are 404 responses, we should analyze the reason for generating 404:
1. Is the user entering the wrong URL address?
2. Or does the developer refer to a resource file that does not exist?
If the latter, you should remove the invalid reference as soon as possible, because the 404 response is also a page response, and they also occupy the network transmission time, especially such a request can not be cached, it will always appear, wasting network resources.

If you want to be able to easily discover 404 errors during the development phase, refer to my blog: Some of the errors that the program should have found before it was released.

If you find that a large number of 304 requests should also be carefully analyzed:
1. is the 304 request resulting from the ASP. NET cache response?
2. Is the 304 request generated when requesting a static resource file?
If it is the latter, it may be related to the browser's settings, or it may be related to IIS settings.

IIS has a "Enable Content expiration" feature that can be used to set the cache header for output responses and reduce the number of requests.
This feature is useful for static files, ASP. NET processing results are unaffected.
The specific setup method can be consulted: some ways to optimize the performance of ASP. NET Web site without modifying the code

We can use this query to analyze how often the page is loaded:

Select Top Csuristem, COUNT (*) as [count],    avg (Timetaken) as Avg_timetaken, Max (Timetaken) as Max_timetakenfrom MyM Vc_weblog with (NOLOCK) where Csuristem like '/pages/% ' GROUP by Csuristemorder by 2 desc

If you find a large number of duplicate requests, you also need to carefully analyze:
1. Does the requested response vary with different parameters?
2. The requested URL is fixed and the response content is rarely changed.
In the latter case, you might consider using the page caching feature. Example: ASP. OutputCache

My blog does not modify the code to optimize the performance of ASP. Some ways to cache the request without having to modify the code, you can try it if you need to.

Back to top the impact of the program architecture on the IIS log parsing process

Earlier I introduced some methods for parsing IIS logs, which are used without queries. Most of the time, we need to output the URL information in the query (Cs-uri-stem) and according to their grouping to statistics, so the reasonable design URL will be convenient for later statistics, but also to get more accurate statistical results. An extreme example is the use of the WebForms default development method, the page load and each button submission is the same URL, you will find it difficult to count the user's each operation spent a lot of time.

What is the URL design to meet the statistical needs?
I think: Each User action (page display or submit) should have a URL corresponding to it, and different URLs can reflect different actions.
It is also suggested that different user actions can be clearly separated in the URL, which makes it easier to do more statistics (for example: page loading, AJAX requests, report display).

While we can use Timetaken to do performance statistics, when you use FRAMESET or IFRAME extensively in your program, it's difficult to count how long it takes to load a page (the page containing the IFRAME). Because the entire page is divided into multiple requests, they are not contiguous in the IIS log, and you cannot accurately count on user requests. For example: A1.aspx with iframe embedded b1.aspx, b2.aspx, b3.aspx, when you count a1.aspx load time, you get the results will always be different from the user feel, Because A1.aspx's timetaken does not contain b1.aspx, b2.aspx, b3.aspx these three requests timetaken!

Therefore, if you want to use IIS logs to analyze program performance, the IFRAME will not be used again.

Reprinted from: http://www.cnblogs.com/fish-li/p/3139366.html

"Go" IIS Logs-a good helper for website operations