Flume Configuring |shell Scripts |python| Sql

Flume is a highly available, highly reliable, distributed mass log collection, aggregation and transmission system.

You can look at the model:

Each flume agent can provide a flume service. Each agent has three members: source, channel, sink

As shown, fetching data from source and sending it to Channel,channel is like a buffer, from which sink reads data from the channel.

----------------------------------------------

There is no picture to write a thought first:

Model 1: Multiple SOURCE---1channel---1sink

Model 2: Single SOURCE---Multi-channel---multi-sink

-----------------------------------------------

A JVM environment can only run one agent

But.. If you have multiple JVM environments on the same machine, you can run multiple agents.

Application scenario words multiple data source access and to import into the same type of sink and the same address of the different paths (well I don't know how to describe the syntax, follow-up map)

A more in-depth understanding of the configuration of the flume is required for multi-JVM environments to run. Follow up with time to understand.

==========================

About flume source code, in the Flume official web site can download its source package, where to read where, the configuration and use of the help is very large.

=============================================

About shell scripts:

if[[$?-ne 0]]

$? : Refers to the result of the previous script execution, such as the previous script is a basic ls-l command, if the successful query, of course, the return is 0, but if there is an error, the return value is not 0.

If you want to modify one of the fields in a file, or a row or lines, the SED command will do you a favor.

If you want to get the path: then pwd, dirname These are your good helpers.

The script can assign the result you want directly to the variable, like this:

Hadoop_path= ' Hadoop fs-ls/path1/path2/path3/path4 '

Note here the "but the number 1 left the key yo."

After getting this path, I can tell if the path exists.

Ingenious live memory. A script can do a lot of things.

====================================

SQL, if someone else wrote a very long, very long, very round SQL command.

Do not feel that you can not understand, do not understand the database to run a, you understand. Do not stare at the line of the dead, look at half a day will not see it. Waste of time.

Write slowly.

Flume Configuring |shell Scripts |python| Sql