Fetching data from the Web API to HDFs with Nifi

1. Panorama Figure 2. Generate a dynamic date parameter with Executescript in order to generate only one Flowfile:groovy code:

Import Org.apache.commons.io.IOUtils
Import java.nio.charset.*
Import Java.text.SimpleDateFormat;
Import Java.lang.StringBuilder;
Import Java.util.Calendar;

def flowfile = Session.create ()

Flowfile = Session.write (Flowfile, {inputstream, OutputStream-
SimpleDateFormat SDF = new SimpleDateFormat ("YyyyMMdd");
Calendar cal = Calendar.getinstance ();
StringBuilder sb = new StringBuilder ();
for (int i = 0; i < 9500; i++) {
Cal.add (Calendar.date,-1);
Sb.append (Sdf.format (Cal.gettime ()) + "\ n");
}

println (SB);

Outputstream.write (Sb.tostring (). GetBytes (Standardcharsets.utf_8))
} as Streamcallback)

Flowfile = Session.putattribute (flowfile, ' filename ', ' get_date ')
Session.transfer (Flowfile, rel_success)

3. Use Splittext to generate a date for each line

Line Split Count 1

4. Take Extracttext to date parameters

5. Generate URLs and filename with Updateattribute

Be sure to set filename here, otherwise, all filenames are the same, and finally only one record can be inserted successfully into HDFS.

6. Get Data with Invokehttp

7. Inserting data into HDFs using Puthdfs

Note that the directory here must be added/, otherwise it will be inserted into the User/root/nifi, rather than the Nifi under files.

Fetching data from the Web API to HDFs with Nifi