VmWare平台Windows Server 2012 無響應宕機

標籤：系統管理員   check   enc   step   think   console   沒有   右鍵   etc   

我們生產伺服器都部署在VMware ESXi 5.5平台上，最近大半年的時間，偶爾就會出現作業系統為Windows Servre 2012的伺服器出現沒有任何響應（unresponsive）的情況，出現問題的時候，伺服器有下面一些現象：

 

1： 應用程式無法訪問SQL Server資料庫，使用Microsoft SQL Server Management Sutdio去測試連接資料庫，也會返回串連錯誤。

2： 網路有時候能Ping通，有時候是Ping不通的情況。

3： 遠端連線無法訪問伺服器，從VMware vSphere Client進入後，嘗試從該伺服器的控制台進入系統，無任何響應。無法進入系統，實質上就是一個宕機的情況。

4： 出現問題是隨機的，沒有任何規律可言，有時候隔了個把月出現一次，有時候好長時間才出現。

 

碰到這種情況，只能在VMware vSphere Client裡面，選擇該伺服器，單擊右鍵選擇“電源”選項，然後關閉電源，重啟電源來解決。分析過伺服器和虛擬機器的日誌，均無法擷取有價值的錯誤資訊。後面同事在VMware官方論壇發現很多人都遇到過這種情況 Windows Server 2012 VM becomes unresponsive / VW Tools "Not Running" 。官方暫時沒有給出結論，有個人反饋是因為賽門鐵克的殺毒軟體（Symantec Endpoint Protection anti-virus）引起的，如下所示：

 

Question to all that are having the problem.  Do you have SEP (Symantec Endpoint Protection) anti-virus installed on these servers?

I had similar issue and after doing a lot of tracing and log reviewing I noticed that every one of my servers that froze had a SEP definition update and within 3 minutes the server froze and had to be hard power off and back on.  If you have SEP installed check the SEP client log under applications and services logs in event viewer and see if you notice a gap between when the server froze and when you rebooted the server.  This log entry time will correspond with time entries in the system and application logs within 3 minutes or so when you have no entries until you rebooted the server.

My resolution was to uninstall SEP from the servers and I have not had anymore freeze since.  I don‘t know if something change in SEP but my servers has had SEP on them for years and never encounter this problem until early February and then I was getting 1-2 frozen servers each week until I uninstalled SEP and I have not had another freeze since early March.

If somebody thinks it‘s something else I‘m all ears but SEP was the only commonality (within 3 minutes of a SEP update) my servers had in common.  The one thing I was to point out is that all my unresponsive servers were still pingable but nothing else was responding, no cntl-alt-del, no rdc, nothing.

 

一個人反饋找過VmWare和微軟公司，都沒有找出原因和解決方案，後面發現出現問題的伺服器都安裝了SEP Client 12.1.2.x,後面通過Symantec的支援人員，要求其更新最新的SEP Client 12.1.6.x，後面也確實沒有出現過問題。如下所示：

 

16. Re: Windows Server 2012 VM becomes unresponsive / VW Tools "Not Running"

copelsimo1 2016-5-10 上午6:09 （回複 Robby68）

Hi to all.

In my company we have the same issue: random unresponsive server (2012/2012r2)

We have ESXI 6.0 up.2

We opened different support request (VmWare, Microsoft, etc) but no one tell us why this happened, and no solution.

Then crossing different tables from different console, i noticed that all unresponsive server had same sep version (12.1.2.x).

So i open a technical call to Symantec, and meantime i started to distribute last update of sep client (at time 12.1.6.x). This update require a system reboot, so only 30-40% of systems have been updated in the first step.

Symantec tell me we had old version of SEP, and requested us full Microsoft dump to analize (but this require reboot,too!) as well as update all client version.

No one server with last SEP version (21.1.6.x) got unresponsive.

At the end, Symantec confirm us problem was right in SEP version:

@- Fix ID: 3590578

@ Symptom: System freezes due to a deadlock in File System Auto-Protect driver after updating virus definitions.

@ Solution: Modified File System Auto-Protect driver to avoid this deadlock. 

So, UPGRADING SEP TO LAST VERSION, PROBLEM SOLVED.

I hope to have helped.

Simone

Alba(CN)

 

我們系統管理員將所有遇到過這種情況的伺服器的Symantec Endpoint Protection anti-virus Client都升級了，暫時也不能確定就能真正解決了這個問題，還需通過時間來驗證。

 

------------------------------------------------------------PS 2017-01-06 追加下面部分內容------------------------------------------------------------

 

自伺服器的Symantec Endpoint Protection anti-virus Client都升級後，從2016-10-20號到今天2017-01-06已經幾個月，都沒有出現過宕機情況，看來確實就是這個問題所致。可以下定論了！

VmWare平台Windows Server 2012 無響應宕機