IIS Hang Troubleshoot

Your website Maybe stop working and response very lowly. How do I find out the reason? Below is the guide, hope it'll help you out!

Identify It's a hang

What website hang really means? An IIS website hangs whenever it appears to stop serving incoming requests, with requests either taking a very long time O R timing out. It ' s generally caused by all available application threads becoming blocked, causing subsequent requests to get queued (or Sometimes by the number of active reuqests exceeding configured concurrency limits).

There is three types of hangs which we should differentiate fist.

1. Full Hang. All the requests is very slow or timeout.

Symptoms include detectable request queuing and sometimes 503 Service unavailable errors when queue limits is reached. Most hang doesn ' t involve high CPU, and is often called "Low CPU Hangs". Also Most of the time, high CPU does not itself cause a. In the rare cases, you could also get a "high CPU hang", which we don ' t cover here.

2. Rolling Hang. Most request is slow, but eventually load.

This usually occurs before a full-hang develops, and may also represent a stable state for an application that's overload Ed.

3. Slow requests. Only specific URLs in your application is slow.

Generally a true hang, but rather just a performance problem with as specific part of your application.

Below Three signs is the hints of hang

1. Performance counter "Http Service Request queues\maxqueueitemage"

It means IIS is falling behind on request processing if this counter keep increasing, so all incoming requests are waiting At least this long to begin getting processed.

2. Performance Counter "Http Service Request queues\arrivalrate" and "w3wp_w3svc\requests/sec"

If "Http Service Request queues\arrivalrate" exceeds "w3wp_w3svc\requests/sec" for the application poo L ' s worker process over a period of time. This basically implies, requests is coming into the system than is being processed, and this is always eventually results in queuing.

3. snapshotting currently executing requests.

It's the best-of-the-detect a hang. If the number of currently executing requests is growing, this can reliably tell you that requests be piling up which ALW Ays leads to high latencies and request queuing. Most importantly, this can also tell you which Urls is causing the hang and which requests is queued. You can use the GUI of InetMgr (IIS Manager) or APPCMD tool to get the this information.

Using Inetmgr:open IIS Manager and navigate to server node, going to Worker Processes, and picking your application pool ' S worker process:

Using Appcmd:this Tool is located at the folder%windir%\system32\inetsrv, you run below command to get requets that has run At least ten seconds.

Appcmd List requests/elapsed:10000

If you see mutiple requests that is taking a long time to execute and you is seeing more and more requests begin to ACCU Mulate, you likely has a hang. If you don't see requests accumulating, it's likely that's slow requests to some parts of your application, but yo U do not has a hang.

Diagnose the Hang

1. Eliminate IIS as the source of hang

IIS hangs happen when all available IIS threads is blocked, causing IIS to stop dequeueing additional requests. This was rare these days, because IIS request threads almost never block. Instead, IIS hands off request processing to a ASP, Classic ASP, or FastCGI application, freeing up it threads to de Queue more requests. We can use below performance counter eliminate IIS as source of the hang quickly:

"Http Service Request queues\currentqueuesize", IIS is has no problems dequeuing requests if it is 0

"w3wp_w3svc\active Threads" This would almost always be 0, or 1 because IIS Threads almost never block. If It is significantly higher, you likely has IIS thread blockage due to a custom module or because you explicitly config ured ASP. NET to run on IIS threads. Consider increasing your MaxPoolThreads registry key.

2. Snapshot the currently executing request to identify where blockage was taking place

From the resulting list of executing request (* * APPCMD or Inetmgr to get these information), we can know which UR L was causing the blockage, and which requests is queued.

What code is causing the hang? (For Developers)

At this point, you ' ve confirmed the hang, and determined where in your application its located (e.g. URL). The next and final step is for the developer to figure out what's in the application code was causing the hang.

Is that developer? Then, what do you know? Hard it's-to-make this final leap, because most of the time hangs be very hard-reproduce in the Te St Environment. Because of this, you'll likely need to analyze the hang in production while its still happening.

Here's how:

1. Make sure are Windows debugging Tools installed on the server (takes longer), or get ProcDump (fast ER).

Expert tip #3: It always pays to has these tools available on each production server ahead of time. Taking the dump approach is usually faster and poses less impact to your production process, letting you analyze it Offlin E. However, taking a dump could be a problem if your process memory was many Gbs in size.

2. Identify the worker process for the application pool has the hang. The executing request list would show you the process ID if your run it with the/xml switch.
   
   
4. Attach the debugger to the process, OR, snapshot a dump using procdump and load it in a debugger later.

   Attach Debugger Live (if you is fast)        ntsd-p [PID]        //or take a dump to attach later        procdump-ma-w [PID ] C:\dump.dmp        ntsd-z c:\dump.dmp                    

6. Snapshot the thread stacks, and exit. Make sure to detach before closing the debugger, to avoid killing the process!

   . Loadby SOS CLR. Loadby SOS Mscorwks ~*e!clrstack. Detach QQ 

8. The output would show you the code in where each of the thread is currently executing. It'll look like this:

   OS Thread id:0x88b4 (7)        retaddr call          Site        000007fed5a43ec9 asp.test_aspx. Page_Load (System.Object, System.EventArgs)        000007fee5a50562 System.Web.UI.Control.OnLoad (System.EventArgs)        000007FEE5A4CAEC System.Web.UI.Control.LoadRecursive ()        000007fee5a4beb0 System.Web.UI.Page.ProcessRequest ()        000007ff001b0219 System.Web.UI.Page.ProcessRequest ( System.Web.HttpContext)        000007fee5a53357 asp.test_aspx. ProcessRequest (System.Web.HttpContext)        000007fee61fcc14 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification (IntPtr, IntPtr, INTPTR, Int32)                    

10. Wait 10-20 seconds, and do it again. If you is taking a dump, just take the dumps seconds or so apart.

Alright. Once you have the lists stack, your objective are to find thread IDs, which has the same stack in both snapshots. These stacks show the code that's blocking the threads, and thereby causing the hang.

Note:if you is only seeing a couple threads or no threads with the same stack, it likely because you either a) has A rolling hang where requests be taking awhile but be still moving, or b) your application is asynchronous. If It async, debugging hangs is-harder because its nearly impossible-tell where requests is blocked without stack Need to implement custom application tracing across async boundaries to help you debug hangs.

References

Https://www.leansentry.com/Guide/IIS-AspNet-Hangs

IIS Hang Troubleshoot