Tuning process for one SOLR index update

Lucene-based SOLR has always been known for its stability and high performance, although it has high CPU requirements, but it can solve complex queries and return search results in such a fast speed, which is a great tool to develop search.     SOLR, the Master-slave architecture deployed on Linux servers, has been running more stably over the past 1 years. In the last week, developers have been slow to update index updates, resulting in a lot of data not waiting for index operations, and the logical optimizations of client-submitted indexes have been ineffective many times. For the index performance check, mainly from the client and server two places to start: 1, the client, check the results found that multiple threads do not appear deadlock, send Index command through SOLRJ because the execution speed is too slow to be blocked in the thread pool,     The client is using the officially recommended concurrentupdatesolrserver, updating the index in bulk adds (an average of 30,000 updates per second), using a more efficient binary request (replacing the XML request), and the client is not the source of the problem.     2, the service side, the IO pressure is not big, the memory is abundant, the CPU occupies the high rate to reach above 40%; viewing the SOLR logs finds a significant amount of time for adds 300 indexes qtime a staggering 10 seconds; index bottlenecks appear on the server side. The developer is then asked to analyze whether the index pressure caused by the increase in the amount of data (130 million index data, or about 50G of disk space) is causing the speed to be dragged down. By adjusting a series of parameters of solrconfig.xml, such as increasing rambuffersizemb, increasing mergefactor, reducing the number of autocommit, minimizing disk operation and testing index speed, it has little effect.     GC logs analyze garbage collection as normal, and there is no significant room for improvement even if you replace the garbage collection policy. Finally can only go to dump thread analysis of what the CPU spends, in the dump log see tomcat Many threads are waiting state, in the running state is often participle, so the stack copy out to the word breaker developer analysis, Developers from the stack see Code execution in the simple traditional conversion, after the rapid repair of the word breaker after the bug update to the SOLR server, index speed recovery.