SPARK App--cleaning check-related shells (1)

Pre-stress testing requires cleaning up large data environments, logs, and storage databases, checking the environment for availability, and saving the state of the environment before testing.

During the stress testing process, regular checking of parquet file generation, report generation, hbase data generation, spark APP FAILED, and error logging are required.

After stress testing, you need to check Parquet file generation, report generation, hbase data generation, whether there are spark app FAILED, collect error logs, view APP execution intervals, CPU memory network disk status, and so on.

Basic scripts, eea_2_all.sh, distribution commands to the HDP cluster node host execution, as follows, note that the underlying script runs on the HDP first host (with other hosts do not handle). #!/bin/bash
#数组中用主机名亦可
hosts= (192.168.1.4 192.168.1.5 192.168.1.6 192.168.1.8 192.168.1.9 192.168.1.10 192.168.1.11 192.168.1.12 192.168.1.13 )
Command=$1
Prompt=$2
if [$# = 0]; Then
command= "/bin/rm-rf/var/log/rmslog/*;/bin/echo" >/var/log/rmslog/debug.log;/bin/chmod 666/var/log/rmslog/ Debug.log "prompt=" Clear Debug Log "
Fi
Count=1
For host in ${hosts[@]}
Todo
echo "-----------------------eea-younge-${count}.test--$host: ${prompt}----------------------"
SSH $host ${command}
Let count++
Done

1. Preparatory work before the test includes,

A. Make sure the HBase Thrift server above the Hmaster is running, as follows.

. SSH 192.168.1.6 "if test $ (Ps-ef | grep ' hbase.thrift.ThriftServer start ' | WC-L)-eq 2 then;  Echo ' Hbase Thrift Server is running. ' else Nohup hbase Thrift Start & Fi "

B. Cleaning up large data environments, local logs, and storage databases to check yarn applications.

#清理Hbase表

#其中cmd. HBase stores hbase shell commands stored in rows, and the last line is exit for exiting the HBase shell. Nohup hbase Shell hbase/cmds.hbase > Result/clear_hbase_${now}.log

#清理Hdfs parquet file, shell/clear_parquets.sh #!/bin/bash
Deletehdfsfolder () {sudo-u spark Hadoop fs-rm-r-F "[$-ne 0]; then LogMessage" Can not create Delete files a nd folders $ "fi} Createhdfsfolder () {sudo-u spark Hadoop fs-mkdir-p $ [$-ne 0]; then LogMessage" Can not Create HDFS Folders $ "fi} Deletehdfsfolder"/user/spark/eea/alarm/* "Deletehdfsfolder"/user/spark/eea/fine/* " Deletehdfsfolder "/user/spark/eea/trip/*" Deletehdfsfolder "/user/spark/eea/event/*" DeleteHDFSFolder "/user/spark/" eea/collector/* "Deletehdfsfolder"/user/spark/eea/rawdata/* "sudo-u HDFs hdfs dfs-expunge createhdfsfolder"/user/ spark/eea/alarm/enrichment/"Createhdfsfolder"/user/spark/eea/fine/enrichment/"CreateHDFSFolder"/user/spark/eea /trip/enrichment/"

#清理各主机的本地日志

. ${shell_path}/eea_2_all.sh

#清理Redis数据库, where Cmds.redis stores commands in rows, only one of the commands in this article Flushall Nohup/bin/cat Redis/cmds.redis | Redis-cli-h 192.168.1.5-a Runoob > Result/clear_redis_${now}.log

#清理PostgreSQL工作日志表 export pgpassword=pass123456 nohup psql-h 192.168.1.155-p 5432-u test-d test-f sql/query_truncate_ App_orchestration.sql > Result/test_clear_work_log_${now}.log

#query_truncate_work_log. SQL content is as follows, select * from App_orchestration; Truncate app_orchestration;

#检查磁盘挂载情况 nohup shell/eea_2_all.sh "Mount | grep \ "Data/eea\" "Mount directory/data/eea Check" > Result/mount_check_${now}.txt

#检查各主机磁盘空间情况 nohup shell/eea_2_all.sh "Df-ht" "Disk Usage" 2>&1 > result/"Vms_disk_usage_${now}.txt"

#检查数据库空间情况 psql-h 192.168.1.155-p 5432-u test-d test-f sql/check_databases_size.sql > Result/rms_eea_databases_siz E_${now}.txt

#check_databases_size. SQL content as follows,SELECTD.datname asName, Pg_catalog.pg_get_userbyid (D.DATDBA) asOwner, Case whenPg_catalog.has_database_privilege (d.datname, ' CONNECT ')THENPg_catalog.pg_size_pretty (Pg_catalog.pg_database_size (d.datname))ELSE' No Access ' End asSIZE fromPg_catalog.pg_database d WHERE d.datname in (' Test ') Order byCase whenPg_catalog.has_database_privilege (d.datname, ' CONNECT ')THENPg_catalog.pg_database_size (D.datname)ELSENull EndDESC--nulls-LIMIT 20;

#检查数据库表空间情况

Psql-h 192.168.1.155-p 5432-u test-d test-f sql/check_tables_size.sql > Result/rms_eea_tables_size_${now}.txt

#check_tables_size. SQL content is as follows, SELECT Table_schema | | '.' || TABLE_NAME as Table_full_name, Pg_size_pretty (pg_total_relation_size) (' | | Table_schema | | '"."' || table_name | | ' ")) as size from information_schema.tables WHERE table_schema= ' public ' order by pg_total_relation_size (' "' | | Table_schema | | '"."' || table_name | | ' "') DESC;

#检查YARN中失败的应用列表

Nohup yarn application-list-appstates FAILED > Result/yarn_app_failed_list.log

#检查YARN中KILLED的应用列表

Nohup yarn application-list-appstates killed > Result/yarn_app_killed_list.log

Inspection in the process of testing see follow-up articles ...