apache spark performance tuning

Recently, due to Project Memory leakage, large-scale JVM performance tuning has been performed. , Now we will record the experience. I. JVM Memory Model and garbage collection Algorithm 1. According to the Java Virtual Machine specification, JVM divides the memory: New (young generation) Tenured (elder generation) Permanent generation (Perm) Among them, new and tenured are heap memory. The heap memory

22 criteria for performance tuning of large-scale and high-traffic Internet sites (zz)Reprint from http://icyriver.net /? P = 26. The author should be yahoo China's engineer. Among the six newly added rules, the flush head method is quite interesting. yahoo changed its own yapache, so it is better to do these things, several other rules I think are also summarized by the research efforts of this jj team. ya

, faster disk.2.5.1. Network performance evaluation（1）通过ping命令检测网络的连通性（2）通过netstat –i组合检测网络接口状况（3）通过netstat –r组合检测系统的路由表信息（4）通过sar –n组合显示系统的网络运行状态Three Linux server performance Tuning 1. Tuning the Linux kernel elevator algorithm for disk I/OWhen the file system is selected, the algorithm balances low latency requireme

This article transferred from: http://www.cnblogs.com/chen77716/archive/2010/06/26/2130807.htmlRecent memory leaks due to the project, so large-scale JVM performance tuning, now the experience to do a record.First, JVM memory model and garbage collection algorithm1. According to the Java Virtual Machine specification, the JVM divides the memory into: New (Young Generation) Tenured (old generati

, you only have limited number of links, memory problems, file handles, and so on.And, Apache runs in an extremely parallel environment, creating a lot of processes. This is why this "permanent link" mechanism is not working well. Before you decide to use permanent link, you need to think about the architecture of your entire system.22. Using an Object-relational mapper (relational Mapper)With ORM (Object relational Mapper), you can gain reliable

used in WebSphere application Server V7.0 (and previous releases), and how to tune them. Unlike the tuning recommendations provided in other related articles, this article uses the Apache DayTrader performance Benchmark Sample case study as the context for this article. With DayTrader applications, you can clearly identify the primary server components used, foc

function. Instead of storing the results in a loop every time the iterator is traversed, the queue is maintained outside the loop, and the results are stored in batches According to spark cluster resources, the resources of spark cluster can be rationally utilized, such as the more resources, the more the cluster computing power is. The more reasonable relationship between machine resources and task para

. A similar method call takes a time close to 15 local variable increment operations.26, Apache parsing a PHP script time is 2 to 10 times times slower than parsing a static HTML page. Use static HTML pages as much as possible and use fewer scripts.27. Unless the script can be cached, it will be recompiled every time it is called. Introducing a set of PHP caching mechanisms can typically improve performance

extensive database connection pool components are c3p0, DBCP, PROXOOL;C3P0 is a database connection pool that accompanies hibernate and is closely connected with hibernate;4. In the JDK, the new operation is quite efficient and does not need to worry about the performance impact of the frequent new operation on the system. However, the class constructor called by the new operation may be very time consuming, and for these objects, pooling can be cons

for new TCP connections, which defaults to 0, which means shutdown; Net.ipv4.tcp_tw_recycle = 1 indicates a fast recovery of time-wait sockets in a TCP connection, and the default is 0, which means close; NET.IPV4.TCP_FIN_TIMEOUT=30 modifies the default timeout time for the system . If the above configuration tuning performance is not ideal, you can continue to modify the configuration:Vi/etc/sysctl.co

Performance optimization ParametersThe tuning parameters for the spark SQL performance are as follows:code exampleimportjava.util.list;importorg.apache.spark.sparkconf;import Org.apache.spark.api.java.javasparkcontext;importorg.apache.spark.sql.api.java.javasqlcontext;import org.apache.spark.sql.api.java.row;importorg.

and optimized the performance of Hadoop clusters deployed in virtualization and Physical Mode on the same scale. The experiment shows that the virtualized Hadoop cluster can well support the production environment.Performance Comparison between a virtualized environment and a physical environment Figure 1 shows the deployment style of the Performance Tuning Test

One, summing up the study of the day before From the "Third Day" performance Test section, we learned several important metrics for the deterministic test: • throughput Üresponsetime Ücpuload Ümemoryusage We have also done a certain optimization on the third day of study in Apache, to optimize the above 4 key readings, then our Apache

mount option can effectively improve performance if external storage is used.13. Adjust Linux file descriptor limitsLinux limits the number of file descriptors that any process can open, with the default limit of 1024 per process, which may hinder benchmarking clients (such as Httperf and Apachebench) and the Web server itself from getting the best performance, with one process per connection for

recommended to copy the library file tcnative-1.dll to the bin directory of Tomcat. In linux, you can directly decompress and install the tomcat_native.tar.gz file under the bindirectory. Before compiling, make sure that the APR library has been installed.How can I determine whether the APR library is enabled for Tomcat? The method is to view the Tomcat startup log:If APR is not enabled, the startup log generally contains the following:Org. Apache. C

described in the previous article, and the same bufferedwrtier and bufferedreader efficiency is also very high. The buffer is used preferentially. Using static methods: Static methods can be used directly without building an instance, and because the method area GC is rarely recycled, and the JVM caches commonly used classes, some common tool classes are more likely to encapsulate static performance. And it is more expressive than function overlo

15 useful MySQL/MariaDB performance tuning and Optimization Techniques MySQL is a powerful open-source Relational Database Management System (RDBMS ). It was released in 1995 (20 years ago ). It uses Structured Query Language (SQL), which may be the most popular choice in database content management. The latest MySQL version is 5.6.25, which was released on July 25, May 29, 2015. An interesting fact about M

Apache or IIS (server) to open gzip, you can read this article.Intermediate Performance ImprovementAre you sure you have mastered the above basic-level optimization schemes? But the reality is that sometimes some solutions are not as obvious as those, and they often depend heavily on how you structure and write your app. The following recommendations are for these scenarios.9. Reuse and lazy load (lazy loa

instead of dynamic content whenever possible, and if you are generating weather forecasts or other data that must be updated every 1 hours, it is best to write a program that generates a static file every 1 hours instead of letting the user run a CGI to generate the report dynamically. Choosing the fastest and most suitable api,cgi for a dynamic application can be the easiest to program, but it will produce a process for each request, which is usually a costly and unnecessary process, fastcgi i

sysstat.x86_64Perform:$sar-P All 1 100Note:-P all means monitoring all cores;1 indicates the acquisition every 1 seconds;100 means acquisition 100 times;5. Turn on slow logSlowlog = log/$pool. Log.slowRequest_slowlog_timeout = 2Note:The above is to open php-fpm slow log, Time threshold value of 2 seconds;Perform:Grep-v "^$" Php.slow.log | Cut-d ""-F 3,2 | Sort |uniq-c | SORT-K1,1NR | Head-n 50Note:Sort: Sorting wordsUniq-c: Displays unique rows and the number of occurrences of the bank in the f