Error creating bean with name 'memcachedClient'...java.lang.OutOfMemoryError

1，Tomcat啟動報錯如下：

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'memcachedClient' defined in file [/usr/local/apache-tomcat-6.0.37_6500/webapps/trade_service/WEB-INF/classes/META-INF/spring/springContext_memcached.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.factory.BeanDefinitionStoreException: Factory method [public net.rubyeye.xmemcached.MemcachedClient net.rubyeye.xmemcached.XMemcachedClientBuilder.build() throws java.io.IOException] threw exception; nested exception is java.lang.OutOfMemoryError        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:581)        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1015)        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:911)        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:485)        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:294)        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:225)        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:291)        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193)        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:322)        ... 44 moreCaused by: org.springframework.beans.factory.BeanDefinitionStoreException: Factory method [public net.rubyeye.xmemcached.MemcachedClient net.rubyeye.xmemcached.XMemcachedClientBuilder.build() throws java.io.IOException] threw exception; nested exception is java.lang.OutOfMemoryError        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:169)        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:570)        ... 53 moreCaused by: java.lang.OutOfMemoryError

,

2，看到建立memcache clent報錯，馬上去memcache伺服器查看一下，memcache正常運行，記憶體也足夠剩餘。

[root@localhost ~]# tail -f /tmp/memcache_error.log <287    0x00 0x00 0x00 0x00<287    0x00 0x00 0x00 0x00<287    0x00 0x00 0x00 0x00>287 Writing bin response:>287   0x81 0x0b 0x00 0x00>287   0x00 0x00 0x00 0x00>287   0x00 0x00 0x00 0x06>287   0x00 0x00 0x00 0x00>287   0x00 0x00 0x00 0x00>287   0x00 0x00 0x00 0x00...[root@localhost ~]# free -m             total       used       free     shared    buffers     cachedMem:          5938       3368       2570          0        195        296-/+ buffers/cache:       2876       3061Swap:         7935          0       7935[root@localhost ~]# 

問題在哪裡？再看看日誌裡面有誤error或者warning資訊：

[root@localhost ~]# cat  /tmp/memcache_error.log  |grep warning[root@localhost ~]# cat  /tmp/memcache_error.log  |grep error[root@localhost ~]#

一點錯誤資訊也沒有，判斷問題不是出在memcache這一塊。

3，去check tomcat伺服器

[root@localhost logs]# free -m             total       used       free     shared    buffers     cachedMem:         18022      16943       1078          0        104        675-/+ buffers/cache:      16162       1859Swap:        20031       8012      12019[root@localhost logs]# 

只剩餘1個G的記憶體，去修改一下tomcat的記憶體配置：

[root@localhost bin]# vim catalina.sh

#JAVA_OPTS="-Xms2048m -Xmx2048m -Xss1024K -XX:PermSize=256m -XX:MaxPermSize=768m"# add by wangqi on 20140614 JAVA_OPTS="-Xms512m -Xmx512m -Xss128K -XX:PermSize=64m -XX:MaxPermSize=256m"

再重啟tomcat，看到還是報一樣的錯誤。

4，回到應用本身去查問題

跟開發人員一起排查，看到是trade_service啟動了過多的多餘的事務，把tomcat的記憶體耗光了，所以關閉掉多餘的事務，再啟動tomcat，就不會再報錯。

感慨：就跟大部分web應用卡，頁面登入不上去，相應過慢很多是出在效率低下的sql上面，tomcat啟動不起來，記憶體溢出大部分也是在java代碼上面，大家需要寫代碼或者sql需要謹慎，且寫且謹慎。