FME2010 Case Study: Dynamic Batch Conversion

link:http://blog.163.com/[email protected]/blog/static/140492492201022545726452/?suggestedreading& Wumii

Keywords: Dynamic fan-out

FME2010 Case Study: Dynamic Batch Conversion ??

2010-04-22 14:20:16|?? Category: fme application case ?|?? tags: fan out ?? | report | font size subscription

?????

Download Lofter Client

? walh

Contact Information: [email protected]

Finishing: Horse

? this case is by a user in Fmetalk questions raised by the user group:

customers in a directory with a large number of Shape formatted file. All of these files are to be re-projected into another coordinate system. We are considering how to convert the batch automation. The main problem that can be foreseen is that I cannot Unify the attribute structure of shape. However, the transformations do not require any action on the properties, just copy them to the destination file.

ability to handle unknown properties for FME2010 dynamic functionality is an easy thing to do. However, what the dynamic workspace does not do is recreate the directory structure. In fact, The default behavior of FME combines all the source data into a single output dataset.

Therefore, this article describes how to FME integrated dynamic and batch processing tools.

? Source Data

This case uses a FME the standard sample data FME Sample Dataset .

in the default installation directory C:fmedatadata includes a series of shp formatted data. We will re-project this data into the c:fmedatareprojecteddata directory.

? Create a work space

creating a workspace that reads a multi-source dataset is very simple ---- It is also easy to set it to read a dataset folder with an unknown number.

, create a workspace:

first set up the source and target data (both Shape format), and then select " Add Directory " icon: Cross Fork Shape

? in the Settings dialog box, select all the source data directories by selecting the options for all subdirectories.

writing data Set parameters is not important - and in fact it's optional, so I don't even need to set it up. Finally, confirm that you are using a workflow for dynamic schemas, and then click OK to create the workspace. It looks like this .

（）：

? It is important to note that there are no attributes on the target features that are not needed in the dynamic workspace. The source data feature type merges all the attributes into a shape 's dataset, which is not important to us.

? get the source data folder

we need to know which directory all the source data came from and write the same relative path when writing to the target data. You can do this by setting the Fme_dataset property of the feature , so you first need to open the Properties Definition dialog box for the source feature class in the workspace and select the Format Attributes Options page , select the fme_dataset property, such as:

? now this Fme_dataset The property is ready to use, but what we need is the folder name, not the folder + the full name of the file name. So use the filenamepartextractor Converter for folder extraction, such as:

? Here we extract Fme_dataset name of the directory, put it into a new property New_fme_dataset .

? Set output Folder

in Span style= "font-family:arial" >new_fme_data , pointing to a directory of c:\fmedata\data\xxxx xxxx contains the source data shp file. First you need to replace " Reprojecteddata " "c \"

Use Stringreplacer converters, such as:

??

Note that in the path setting, double slashes are required because the single slash is a reserved character, so you need to use the escape character to preserve it. Now New_fme_dataset points to the directory fmedata\reprojecteddata\xxxx.

? set fan out

Now for the fan-out setting of the DataSet , the fan-out of the dataset needs to be categorized by an attribute, which uses New_fme_dataset as a fan-out attribute.

in the navigation form, locate Shape in the settings of the Write module fanout parameter, and double-click it, such as:

In the dialog box, set:

? as you can see, the settings C Packing Catalogue C + to fan out the directory (that's why it's removed from the C + ). Then set the fan-out property condition to new_fme_dataset.

? converting Data

once all the settings are complete, you can make any conversions that we need. In the case, a re-projection transformation is performed. If the shape Format data has an engineering prj, we do not need to set the coordinate system parameters of the read module, just need to set the Write module (target dataset).

? Run the work space

The log file will record how many elements are read and how many are written, such as:

? through Windows Explorer, you can view the converted results directory:

Use FME Viewer the converted data can be viewed without any changes to the properties, except that the coordinate system has changed:

? More Considerations

It is very useful to be able to read a set of datasets from folders and subfolders, where any file can be in any directory location. What's more, this feature is dynamic, and in this sense, it's only going to search the dataset at runtime, not when creating a workspace. So, you can add and delete any file before the transition is ready to begin. Fanout is another method for multi-target output (or batch processing). You can also Use the menu file->batch Deploy feature in Worbench, or combine FilePath Reader (file path read module) and Workspacerunner function to run the workspace and pass the source data parameters to this workspace.

The more you have all the source data, the better the fan-out function, as all the data is read into and cached into a single conversion process. With other methods, you will degrade performance because each source data is handled separately.

? FME Server Implementation

Although we did not use a Web service to run the data transformation, but it is possible for this scenario to exploit FME Server to complete. Because the FME Server Architecture is scalable, the processing of large amounts of data can be better.

in this case, you can define the use of the Workspacerunner method (more of the case is Fmeserverjobsubmitter method), because each dataset, during the batch process, can be in a separate FME Conversion Engine ( FME Engine ), which can greatly improve performance and use system resources.

We want you to be interested in this case, which shows why the dynamic workspace is important, as well as other tasks, such as using the fan-out feature in batch processing.

FME2010 Case Study: Dynamic Batch Conversion