apache spark performance tuning

One, the Spark runtime schema:The Spark distributed architecture takes a master/slave architecture pattern. The Master is the drive (Driver) node, which is responsible for central coordination and scheduling of each work (actuator executor) node.From the actuator (executor) node.Spark drive nodes and executor nodes are collectively known as spark applications. Th

I. Language performance optimization 1. Using the AB tool under Apache for performance testingTest: ab-n100-c100 https://www.baidu.com/(Request 100 times, concurrency is 100)Focus on two volumes:Requests per second (number of requests per second) and time per request (average response times)2. Try to use PHP built-in variables, constants, functions, Reason: PHP

will store intermediate results in the/tmp directory while computing, Linux now supports TMPFS, in fact, it is simply to mount the/tmp directory into memory.Then there is a problem, the middle result is too much cause the/tmp directory is full and the following error occurredNo Space left on the deviceThe workaround is to not enable TMPFS for the TMP directory, modify the/etc/fstabQuestion 2Sometimes you may encounter Java.lang.OutOfMemory, unable to create new native thread error, which causes

I will share 11 practical tips on Java performance tuning and java Tuning Most developers think that performance optimization is a complicated problem and requires a lot of experience and knowledge. Yes, it is not wrong. It is not easy to optimize the application to achieve the best

that the response of community problems should also be relatively fast.Individuals on the Flink is more optimistic, because the original flow processing concept, in the premise of ensuring low latency, performance is relatively good, and more and more easy to use, the community is also evolving.NetEase has: Enterprise-class Big Data visualization analysis platform. Self-service Agile analysis Platform for business people, using PPT mode report making

Hu Xi, "Apache Kafka actual Combat" author, Beihang University Master of Computer Science, is currently a mutual gold company computing platform director, has worked in IBM, Sogou, Weibo and other companies. Domestic active Kafka code contributor.ObjectiveAlthough Apache Kafka is now fully evolved into a streaming processing platform, most users still use their core functions: Message Queuing. For how to ef

Linux server performance tuning skills, linux Server TuningPerformance tuning skills for 20 Linux servers Guide Linux is an open-source operating system that supports various hardware platforms. Linux servers are world-renowned. The main difference between Linux and Windows is that, by default, a Linux server does not provide a GUI (graphical user int

11 simple Java performance tuning techniques, java Tuning Most developers naturally think that performance optimization is complex and requires a lot of experience and knowledge. Well, it cannot be said that this is completely wrong. Optimizing applications to achieve optimal perfo