Hadoop Monitoring Analysis Tool dr.elephant

Company infrastructure this side wants to extract slow jobs and learn about the waste of resources, so install a Dr Elephant look. Linkin Open Source System provides performance analysis and tuning recommendations for yarn-based Mr and spark jobs.

Most of Dre is based on Java development, and the Spark Monitoring section is developed using Scala, using the play stack-up framework. This is a python-like framework, based on Java?scala? Not too thin to understand, directly down can be used, need to java1.8 above.

Prerequest list:

Java 1.8

Playframework+activator

nodejs+npm

Scala+sbt

Compiled server is set up in the United States Silicon Valley, a cloud host, before the Bigtop has been installed JAVA,MAVEN,ANT,SCALA,SBT and other compile tools, so download activator decompression put to/usr/local and add path.

Then from GitHub clone a copy of Dr-elephant down, open compile.conf, modify the Hadoop and spark versions for the current version,: Wq Save exit, Run compile.sh to compile, after a short wait, because the United States server, under dependence fast. There will be a dist folder, inside will pack a dr-elephant-2.0.x.zip, copy out decompression can be used.

The DRE itself requires more than 5.5 MySQL support, or mariadb the latest 10.1 stable version. There is a problem here, which is the three lines in the Dre/conf/evolutions/default/1.sql:

Create INDEX YARN_APP_RESULT_I4 on Yarn_app_result (flow_exec_id), CREATE index yarn_app_result_i5 on Yarn_app_result ( JOB_DEF_ID); Create index yarn_app_result_i6 on Yarn_app_result (flow_def_id);

Because in some databases, the index length exceeds the limits of the database itself, the index length needs to be modified to avoid a failure to start.

Create INDEX YARN_APP_RESULT_I4 on Yarn_app_result (flow_exec_id), CREATE index yarn_app_result_i5 on Yarn_app_ Result (job_def_id); create index yarn_app_result_i6 on Yarn_app_result (flow_def_id (150));

Then there should be no problem.

Create a database called Drelephant in the database and give the relevant access rights to the user

The next step is to configure Dre:

Open app-conf/elephant.conf

# Play Application Server port# launch Dre after the play framework listens to the Web port port=8080# database configuration# The DB Host, username password Vault name Db_url=localhostdb _name=drelephantdb_user= "Root" db_password=

Other default, no change required

And then the Generalconf.xml.

<configuration>  <property>    <name> drelephant.analysis.thread.count</name>    <value>3</value>     <description>Number of threads to analyze the completed  jobs</description>  </property>  <property>    < Name>drelephant.analysis.fetch.interval</name>    <value>60000</value >    <description>interval between fetches in milliseconds </description>  </property>  <property>    <name >drelephant.analysis.retry.interval</name>    <value>60000</value>     <description>interval between retries in milliseconds</ description>  </property>  <property>    <name>drelephant.application.search.match.partial</name >    <value>true</value>    <description>If  this property is  "false", search will only make exact matches< /description>  </property></configuration>

Modify Drelephant.analysis.thread.count, the default is 3, the proposed modification to 10, 3 words from the Jobhistoryserver reading speed is too slow, higher than 10 words and read too fast, will cause great pressure on jobhistoryserver. The following two one is the time period of the read, and the other is the interval between retry reads.

Then execute the start.sh boot under the bin. And then, show smile to the yellow elephant.

Loaded up to look at this thing, in fact, its own principle is not complex, is to read all kinds of jmx,metrics, log information, write one of their own is not possible. The main function is to put the contents of the job information in a screen inside the display, the province in the JHS page one point.

650) this.width=650; "Src=" Http://s5.51cto.com/wyfs02/M01/8C/74/wKiom1hsx8jRByDsAAZdBTE75J0031.png-wh_500x0-wm_3 -wmp_4-s_2404359343.png "style=" Float:none "title=" screenshot 2017-01-04 pm 5.57.29.png "alt=" Wkiom1hsx8jrbydsaazdbte75j0031.png-wh_50 "/>

650) this.width=650; "Src=" Http://s5.51cto.com/wyfs02/M01/8C/71/wKioL1hsx8yDxMmTAAa6YRfEAs0611.png-wh_500x0-wm_3 -wmp_4-s_1193186933.png "style=" Float:none "title=" screenshot 2017-01-04 pm 5.58.05.png "alt=" Wkiol1hsx8ydxmmtaaa6yrfeas0611.png-wh_50 "/>

That's it, so easy

This article is from the "Practice Test Truth" blog, please be sure to keep this source http://slaytanic.blog.51cto.com/2057708/1889037

Hadoop Monitoring Analysis Tool dr.elephant