Spark log Configuration

Spark log Configuration

When testing spark computing, submit the job to yarn (Mode-master yarn-cluster). To view the print to the console, this is imposible, because the job is submitted to the yarn cluster, so reading logs on the yarn cluster is very troublesome, but I want to view the print information for debugging or other purposes.

Modify log4j. properties. template to log4j. properties under the conf directory of Spark. The original content is as follows:

# Set everything to be logged to the console
Log4j. rootCategory = INFO, console
Log4j. appender. console = org. apache. log4j. leleappender
Log4j.appender.lele.tar get = System. err
Log4j. appender. console. layout = org. apache. log4j. PatternLayout
Log4j. appender. console. layout. ConversionPattern = % d {yy/MM/dd HH: mm: ss} % p % c {1}: % m % n

# Settings to quiet third party logs that are too verbose
Log4j.logger.org. spark-project.jetty = WARN
Log4j.logger.org. spark-project.jetty.util.component.AbstractLifeCycle = ERROR
Log4j.logger.org. apache. spark. repl. SparkIMain $ exprTyper = INFO
Log4j.logger.org. apache. spark. repl. SparkILoop $ SparkILoopInterpreter = INFO

Change log4j. rootCategory = INFO to log4j. rootCategory = WARN, And the console can suppress Spark from hitting the console with INFO logs. If you want to display all the information, change INFO to DEBUG.

If you want to print the println in the Code to the console, and keep the logs output by spark itself, you can output it to the log file.

Log4j. rootCategory = INFO, console, FILE
Log4j. appender. console = org. apache. log4j. leleappender
Log4j.appender.lele.tar get = System. err
Log4j. appender. console. layout = org. apache. log4j. PatternLayout
Log4j. appender. console. layout. ConversionPattern = % d {yy/MM/dd HH: mm: ss} % p % c {1}: % m % n

# Settings to quiet third party logs that are too verbose
Log4j.logger.org. eclipse. jetty = WARN
Log4j.logger.org. eclipse. jetty. util. component. AbstractLifeCycle = ERROR
Log4j.logger.org. apache. spark. repl. SparkIMain $ exprTyper = INFO
Log4j.logger.org. apache. spark. repl. SparkILoop $ SparkILoopInterpreter = INFO

Log4j. appender. FILE = org. apache. log4j. DailyRollingFileAppender
Log4j. appender. FILE. Threshold = DEBUG
Log4j. appender. FILE. file =/home/Hadoop/spark. log
Log4j. appender. FILE. DatePattern = '. 'yyyy-MM-dd
Log4j. appender. FILE. layout = org. apache. log4j. PatternLayout
Log4j. appender. FILE. layout. conversionPattern = [%-5 p] [% d {yyyy-MM-dd HH: mm: ss}] [% C {1 }:% M: % L] % m % n
# Spark
Log4j.logger.org. apache. spark = INFO

In the preceding operations, spark logs are printed to the console and/home/hadoop/spark. log. This is the log inheritance feature and will be improved later. Currently, log4j is used. rootCategory = INFO, console, and FILE to log4j. rootCategory = INFO, FILE

 

-------------------------------------- Split line --------------------------------------

Spark1.0.0 Deployment Guide

Install Spark0.8.0 in CentOS 6.2 (64-bit)

Introduction to Spark and its installation and use in Ubuntu

Install the Spark cluster (on CentOS)

Hadoop vs Spark Performance Comparison

Spark installation and learning

Spark Parallel Computing Model

-------------------------------------- Split line --------------------------------------

Spark details: click here
Spark: click here

This article permanently updates the link address: