Shell/hadoop/hive useful command collection

Some commands are often used and recorded in an article for reference. This article is often updated. Shell Command: linux Command for counting the number of lines of all files in a directory findhomecrazyant-typef-name * | xargscat | wc-l use find to find the total number of lines of all text files in the crazyant directory. However, the command execution is slow.

Some commands are often used and recorded in an article for reference. This article is often updated. Shell Command linux Command to count the number of lines of all files in a directory find/home/crazyant-type f-name "*" | xargs cat | wc-l use find to find all the sum of the number of lines in a text file. However, the command execution is slow.

Some commands are often used and recorded in an article for reference. This article is often updated.

Shell Command: linux Command for counting the number of lines of all files in a directory

find /home/crazyant -type f -name "*" | xargs cat | wc -l

Use find to find the total number of lines of all text files in the crazyant directory. However, the command execution is slow.

Linux Command for counting the size of all directories and total directories in a directory

du -h --max-depth=1 /home/crazyant/

Count the size of all files in the crazyant directory. Here I only want to see the size of a directory. Therefore, if-max-depth = 1 is added, this command recursively lists the file sizes of all subdirectories.

Use of the scp command:

Copy from local to remote: scp-r? Logs_jx pss@crazyant.net/home/pss/logs

Hive command hive creates and executes Indexes

create index table02_index on table table02(id) as 'org.apache.hadoop.hive.ql.index.compact.CompactIndexHandler' with deferred rebuild; alter index table02_index on table02 rebuild;

In the first sentence, hive creates an index on a table but does not take effect. In the second sentence, the process of creating an index is actually a map-reduce process.

Hive Load data command

hive -e "use crazyant_net; LOAD DATA INPATH '/app/crazyant/student/*' INTO TABLE student;"

Inpath indicates the input path, so load data from local or hdfs must be carried.

Hadoop command hadoop distcp command with username and password method

hadoop distcp -su user1,pass1 -du user1,pass1 hdfs://src1 hdfs://dest1

Sometimes there is no distcp permission between two clusters. In this case, the account password of the two clusters is required, and the account password of the first cluster is required after-su, add the account and password of the target cluster to the end of-du.

Mysql Command? View the most detailed field descriptions of a data table

SHOW FULL FIELDS FROM student

This command lists comments, permissions, default values, types, and other table fields.

View the mysql thread being executed

Mysql> show processlist;

+ --- + ---- + ------- + ----- + --- + -- + --- + ------ +

| Id ????? | ??? User ?? | Host ?????????????? | Db ????????? | Command | Time | State | Info ???????????? |

+ --- + ---- + ------- + ----- + --- + -- + --- + ------ +

| 2153201 | crazyant? | 127.0.0.1: 25357 ??? | Pulse ?????? | Sleep ?? |? 914 | ?????? | NULL ???????????? |

| 2153733 | crazyant? | 127.0.0.1: 48814 ??? | Hive ??????? | Query ?? | ??? 0 | NULL? | Show processlist |

| 2153735 | crazyant? | 127.0.0.1: 39639 ??? | Pulse ?????? | Sleep ?? | ?? 13 | ?????? | NULL ???????????? |

| 2153736 | crazyant? | 127.0.0.1: 39640 ??? | Pulse ?????? | Sleep ?? | ?? 13 | ?????? | NULL ???????????? |

+ --- + ---- + ------- + ----- + --- + -- + --- + ------ +

4 rows in set (0.01 sec)

Mysql>

You can use the kill command to delete one of the processes.

Mysql> kill 2153474;

Query OK, 0 rows affected (0.00 sec)

To LOAD data to MYSQL:

Load data local infile 'd:/workbench/python/result.txt 'replace into table my_urlvisit fields terminated by '\ t' lines terminated by' \ n' IGNORE 0 LINES (url, pdate, COUNT );

Vim command

If an editing error occurs, Press ESC to return to the command mode, and press u to cancel the editing;

Garbled characters appear when vim edits Chinese Text

VIM text is often garbled with Chinese characters, because fileencoding and termencoding encoding are inconsistent. You can keep setting them;

: Set termencoding

Termencoding = cp936

: Set fileencoding

Fileencoding = UTF-8

: Set termencoding = utf8

In this way, vim can display Chinese normally;

Reprinted please indicate Source: http://www.crazyant.net/1209.html