Spark tuning experience (Part 2)

FAQ

1. Spark performance Configuration

My current environment is five machines, each with 8 cores. If there are two configuration schemes:

()

Spark_worker_instances = 8

Spark_worker_cores = 1

(B)

Spark_worker_instances = 1

Spark_worker_cores = 8

What should I do?

A: In Solution A, each node starts eight workers to run eight JVMs, and each worker starts an excutors. In solution B, a worker starts to run one JVM. If the data is small, select solution B, because it can save the JVM startup overhead. If the data is large, the JVM startup time can be ignored. Select solution.

If the memory of all five machines is 48 gb, the following configuration is added to the and B schemes:

(A) spark_worker_memory = 6 GB // allocate 6 GB memory to each worker

(B) spark_worker_memory = 48g

Note the following When configuring:

Spark_worker_cores * spark_worker_instances = total cores of each machine

 

 

2. hadoop configuration details:

1. core-site.xml

Common settings:

FS. Default. Name HDFS: /mastername: 9000

This is a URI describing the namenode node in the cluster (including the protocol, host name, and port number). Each machine in the cluster needs to know the namenode address. Datanode nodes are first registered on namenode so that their data can be used. An independent client program interacts with datanode through this URI to obtain the file block list. The name of the default file system. The default value is to set the single-host configuration. If it is a pseudo-distributed file system, set HDFS: // localhost: 9000. If it is a fully distributed file system, set it to HDFS: // mastername: 9000

 

Hadoop. tmp. DIR/tmp/hadoop-$ {user. name}

Hadoop. TMP. DIR is the basic setting of hadoop File System dependency, many paths depend on it, if the hdfs-site.xml does not configure the storage location of namenode and datanode, by default in this path, so it is best to set, and set it to the persistent directory. The default path will generate different temporary directories based on different user names.

 

Optimization parameters:

Hadoop. logfile. Size 10000000 the maximum log file size is 10 MB.

Hadoop. logfile. Count 10. The number of log files is 10.

After hadoop runs many tasks, you can set hadoop. logfile. Size and hadoop. logfile. Count to set the maximum log size and quantity to automatically identify the logs.

 

Io. file. Buffer. Size 4096 the buffer of the stream file is 4 K

This is the buffer size for reading and writing sequence files, which can reduce the number of I/O operations. We recommend that you use a larger value in a large hadoop cluster.

Hdfs-site.xml

Common settings:

DFS. Replication determines the number of data backups of file blocks in the system. For a practical application, it should be set to 3 (This number has no upper limit, but more backups may not work and will occupy more space ). Less than three backups may affect data reliability (data loss may occur in case of system failure)

DFS. Data. dir this is the path of the local file system where the datanode node is specified to store data. The path on the datanode node does not need to be identical, because the environment of each machine is likely to be different. However, if the path on each machine is configured in a unified manner, the work will be easier. By default, its hadoop. tmp. dir path can only be used for testing, because it may lose some data. Therefore, it is best to overwrite this value.

DFS. Name. dir this is the local system path where the namenode node stores hadoop file system information. This value is only valid for namenode and does not need to be used for datanode. The above warning for the/temp type also applies here. In practical applications, it is best to overwrite it.

 

DFS. permissions is the identifier of the permission check when the file is operated. It controls the read and write permissions. False does not control the read and write permissions.

 

DFS. Block. Size 67108864 the default file block size is 64 MB.

Hadoop fsck/file directory-files-locations-blocks View File storage Information

Usage: hadoop fsck [generic_options] <path> [-move |-delete |-openforwrite] [-files [-blocks [-locations |-racks]

<Path> Check Start directory.

-Move the damaged file to/lost + found

-Delete: Delete the damaged file.

-Openforwrite: prints the files opened by the write operation.

-Files: prints the file being checked.

-Blocks: print the block information report.

-Locations: prints the location information of each block.

-Racks prints the network topology of the data-node.

 

 

DFS. namenode. Logging. level info output Log Type

 

Optimization settings:

DFS. df. interval 60000 disk space statistics interval: 6 seconds

DFS. Client. Block. Write. retries

DFS. Heartbeat. interval 3 data node heartbeat detection Interval

DFS. Balance. bandwidthpersec 1048576 the maximum bandwidth available for data nodes that start Server Load balancer is 1 MB.

 

DFS. datanode. Failed. volumes. tolerated 0

Determines the number of errors that occur when the data node Stops providing services. If no errors occur, the data node is stopped.

 

DFS. datanode. Data. dir. Perm 755 set the directory access permission for the storage block of the Data Node

For more information, see:

Http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml

Mapred-site.xml

Mapreduce. Framework. Name Yarn

Mapred. Job. Tracker mastername: 9001 whether the job tracking manager is in the same process as the Mr task. The default value is local, which is a single node. The host and port number of the job.

Directory for storing mediation data files of mapred. Local. dir $ {hadoop. tmp. dir}/mapred/local Mr

Directory for storing control files of mapred. system. dir $ {hadoop. tmp. dir}/mapred/system Mr

Mapred. Temp. dir $ {hadoop. tmp. dir}/mapred/temp Mr temporary shared file storage Zone

Common Terms:

Driver: There are many distributed frameworks using the concept of driver, such as HIVE. The driver in Spark runs the main () function of the application and creates sparkcontext, the purpose of creating sparkcontext is to prepare the runtime environment of the spark application. In Spark, sparkcontext is responsible for communicating with clustermanager for resource application, task allocation, and monitoring. When the excutor part is completed, the driver is also responsible for disabling sparkcontext. Generally, sparkcontext represents the driver.

 

Excutor: a process where an appliction runs on a worker node. This process is responsible for running certain tasks and storing data in memory or disk. Each application has its own excutor. In spark on Yarn mode, the process name is coarsegrainedexcutorbackend. A coarsegrainedexcutorbackend process has only one excutor object. It encapsulates tasks into taskrunner and extracts an idle thread from the thread pool to run tasks, the number of tasks that each coarsegrainedexcutorbackend can run in parallel depends on the number of CPUs allocated to it.

 

Cluster Manager: an external service that obtains resources from a cluster. Currently, the following services are available:

Standalone: spark native resource management. A master is responsible for resource allocation and can run on Amazon EC2.

Apache mesos: A resource scheduling framework with good compatibility with hadoop mapreduce;

Hadoop yarn: mainly refers to the yarn resourcemanger;

Spark tuning experience (Part 2)