Leverage one of the most common interprocess communication mechanisms in Linux and Unix

DataStage jobs are typically used to process data by batches, and they are scheduled to run at specific intervals. When there are no specific plans to follow, the DataStage operator can manually start the job through DataStage and Qualitystage Director clients or on the command line. If you run a job at the command line, you can execute it as follows.

Dsjob-run-http://www.aliyun.com/zixun/aggregation/12616.html ">param Input_file=/path/to/in_file-param output_ File=/path/to/out_file Dstage1 Job1

Figure 1 shows a chart that represents this command.

In a normal environment, in_file and Out_file are stored in a file system on a machine running DataStage. However, in Linux or UNIX, input and output can be transferred to a series of commands. For example, when a program needs to be sorted, the executable command command|sort |uniq >/path/to/out_file. In this case, Figure 2 has an explicit data flow in which the output of one command becomes the input of the next command, and the final output is in the file system.

Assuming that the intermediate process generates millions of lines of code, you can avoid getting intermediate files, saving space in the file system and the time it takes to write those files. Unlike many programs or commands that are executed in UNIX, DataStage jobs do not get standard input through pipelines. This article describes a method that demonstrates how to use a script to accomplish this task and how it is actually used.

If the job should accept standard input and generate standard output, like a regular UNIX command, you must call it through a wrapper script in the following manner: Command1|piped_ds_job.sh|command2 >/path/to/out_file.

Or you may have to send the output to a file, such as using the command command1|piped_ds_job.sh >/path/to/out_file.

The diagram in Figure 3 shows how the script should be structured.

The script must convert the standard input to the specified pipe, and it must also convert the output file of the DataStage job to standard output. In the following sections, you will learn how to accomplish this task.