nifi-process-oriented large data processing framework

Any big data analysis software needs a powerful data pull component, data warehousing system, processing engine, task scheduling engine and process design interface. The focus of Hadoop and Spark is on data storage and task scheduling, and R focuses on the data analysis engine. Data pull components and processes are the main strengths of Nifi.

What is Nifi?

Apache NiFi is an easy-to-use, powerful and reliable data extraction, processing and distribution system. Apache NiFi is designed for data flow. It supports data routing, transformation, and system mediation logic for highly configurable indicators, enabling the dynamic pull of data from a variety of data sources. Nifi was originally a project of the NSA and is now open source and managed by the Apache Foundation.

Nifi is a Java-based, build management using MAVEN support packages. Nifi works on a web-based basis and is scheduled on the server in the background. You can define a process for data processing and then process it, with components such as data processing engines, task scheduling, and so on.

Second, the installation of Nifi

First of all, you need to install the latest version of Maven and Java, OpenJDK on Ubuntu will encounter some problems, it is recommended to download from Oracle's official website http://java.com, and the SDK instead of the JRE. After decompression, set the path to the Java_home in the/etc/profile file so that other programs can access it. Then install the maven,ubuntu15.04 maven version too long, go to http://maven.apache.org/to download the latest version, and then set the path of Maven_home, and add to the PATH environment variable. When you're done, it looks like this:

Export Java_home=/home/supermap/jdk1.8.0_51export Maven_home=/home/supermap/giscript/apache-maven-3.3.3export Path= $JAVA _home/bin: $MAVEN _home/bin: $PATH

Now, go to http://nifi.apache.org/to download the Nifi installer. Because Nifi uses MAVEN for package management, a large number of support libraries are automatically downloaded during compilation and need to keep the network unblocked. If there is an error in the middle, it is usually caused by a network connection interruption, run the compilation again.

Third, the use of Nifi

Run (it's a good idea to restart the system in order for the environment variable to work):

MVN clean Install

Then, wait ... After the end, into the/home/supermap/giscript/nifi-0.2.1/nifi-assembly/target, find nifi-xxx-bin.zip This file, copy to their own running directory, unzip, go in, execute:

./bin/nifi.sh Start

Then, through the browser access address: http://localhost:8080/nifi/, under normal circumstances, you can see the main interface of Nifi.

Stop service use:./bin/nifi.sh stop

The current version is 0.2.X, not very mature, but considering the NSA has been used, there should be some usability, but the current documentation and tutorials are still very few, mainly by reading the source code to understand the logic of the program.

nifi-process-oriented large data processing framework