Spark Basic Essay: Setting the log output level in the Spark application

We typically develop spark applications using the IDE (for example, IntelliJ idea), while the program debug runtime prints out all the log information in the console. It describes all the behavior of the (pseudo) cluster operation and execution of the program.

In many cases, this information is irrelevant to us, and we are more concerned with the end result, whether it is a normal output or an abnormal stop.

Fortunately, we can actively control the level of log output through log4j. Introduce log4j. Logger and Log4j.level, and set the Logger.getlogger ("org") in the object. SetLevel (Level.error)

Import org.apache.log4j. {level, Logger}

Object Example {
  logger.getlogger ("org"). SetLevel (Level.error)

  def main (args:array[string]) {
    ...
  }
}

After this operation, the console only outputs the error level information and does not miss the output and debug errors.

Reprinted from: http://www.cnblogs.com/kevingu/p/5580838.html