kettle etl

Some things to note about using Java code modules in kettle in case you forget. First you need to inherit the Processrow method, similar to the Java Main method, where the code starts and reads and saves variables inside. public boolean Processrow (Stepmetainterface SMI, Stepdatainterface SDI) throws Exception { object[] r = GetRow () ; if (r = = null) { setoutputdone (); return false; } R = Createoutputrow (R, Data.outputRowMeta.size

Kettle Condition Judgment Flow operation 1. Background According to the position information that the customer uploads, download the standard product information in the position to the customer local database, in order to determine whether the id_instrument is an incremental update or a full amount, according to the different results returned by the query client, perform different operations, if the return record data is greater than 0, then increme

Chinese garbled problem solution when extracting data using kettle When using kettle to extract data from different databases, there are sometimes garbled Chinese problems: The solution is as follows: 1. See if the character set of the database is UTF-8 (the most commonly used character set) 2. If the database settings are still garbled in Chinese, it may be because the default encoding set for the clien

Error connecting database [MySQL]: org.pentaho.di.core.exception.KettleDatabaseException:Error occurred while trying to connect to the databaseDriver class ' Org.gjt.mm.mysql.Driver ' could not being found, make sure the ' MySQL ' Driver (jar file) is installed.Org.gjt.mm.mysql.DriverOrg.pentaho.di.core.exception.KettleDatabaseException:Error occurred while trying to connect to the databaseDriver class ' Org.gjt.mm.mysql.Driver ' could not being found, make sure the ' MySQL ' Driver (jar file) i

Tags: style class blog Java http comRecent study of Big data processing, due to the needs of project development, using an easy way to import data from Excel into the database, the development of the use of kettle tools. Kettle Tool installation is very simple, after downloading from the official website, directly extracted to the set of the drive letter can be (if you configure the Java environment variabl

Kettle is a data cleaning tool, it may be a lot of functions, so far I understand the feeling is to extract data, the most useful is used for the database of scheduled tasks. Kettle is a free installation, decompression can be tried. 1.spoon.bat is used under the Windows operating system 2.spoon.sh is used under the Linux operating system Next is to create the resource pool Click on the ' + ' sign and selec

Kettle itself has three main components: Spoon,kitchen,pan. Where spoon is a graphical interface for Windows, set the environment variables first: Pentaho_java_home, for example: C:\Program Files\java\jdk1.7.0_25, is actually your Java installation directory, more than 1.6 can. Under Windows Double-click on Spoon.bat, the interface is as follows: Here I set up a library, in fact, can be stored in the form of files, the structure of the storage is XML

JavaScript Step error HandlingIf you are familiar with the error features of the kettle conversion, you may want to know how to use it in JavaScript steps. Steps to support the error mechanism the user interface mechanism is the same, right-click on the JavaScript step, select "Define error Handling" To enable error handling and configure. The JavaScript code can identify the error lines and forward them to the configured error handling steps, which r

Things are getting more and more complex ... The new staff asked me what I was doing, I said it was a handyman, basically not the development of things are dry. Background: It is now necessary to synchronize the development library data to the test environment in accordance with the business code, registration sequence (development test pass), and so on, with kettle. The Development library and test library are the same MySQL database server u

Kettle Download Address: community.pentaho.com/projects/data-integration/ One, preparation work 1. Unzip the downloaded compression package such as: Pdi-ce-6.1.0.1-196.zipAnd then open Spoon.bat.2. Create a conversion.(When you connect to the database, you may encounter problems with the database driver package, download the corresponding driver package into the Kettle Lib folder restart software on it.) )

Before the tutorial starts, do some preparation work, look at the picture should know how to do it. This is the preparation for the kettle. 1: New file, size for 1000*1100. With a resolution of 300 pixels, pull the ellipse with the Ellipse tool and add two strokes below the ellipse. Only adjusts the three dots in the red circle, moving a few pixels each. 2: Enter the channel, perform the selected modifications-shrink. The shrinkage is 25. Reverse-S

Installation complete kettle, start spoon.sh times wrong as follows: [Root@kettle data-integration]#./spoon.sh org.eclipse.swt.SWTError:No More handles [Gtk_init_check () FAI LED] at Org.eclipse.swt.SWT.error (Unknown Source) at Org.eclipse.swt.widgets.Display.createDisplay (Unknown SOURCE) at Org.eclipse.swt.widgets.Display.create (Unknown source) at org.eclipse.swt.graphics.device.This is because the gr

Tags: oracle error message generation SQL statement blog Data Conversion 9.png1. System Version Information System:windows flagship Service Pack1 Kettle version:6.1.0.1-196 JDK version:1.8.0_72 2. Connect to the database This instance uses global variables when connecting to the database. 2.1 Create a new transformation: After spoon startup, click CTRL + N to create a new transformation 　　 2.2 In the new conversion interface, right click on the DB con

Use javascript steps and FIRETODB functions in kettle to define your own database queriesIf you need to implement non-traditional database query operations. To discuss this scenario, if you need to read the regular table in the database, then check the number of field match expressions for each row entered.Run a database query in a JavaScript stepWhen the JavaScript step is initialized, query the database. Gets the regular table-set of records. The in

Kettle incremental update design skills First, you need to determine whether you are processing a dimension table. If it is a dimension table, it may beSCDSituation, you can useKettleOfDimension LookupTo solve this problem, if you are dealing with fact tables, the methods may be different. The main difference between them is that the primary key judgment method is different. Fact tables generally have a large amount of data. You must first determin

Package kettle; Import java. SQL. resultset;Import java. SQL. sqlexception;Import java. SQL. statement;Import java. util. arraylist;Import java. util. List; Import org. Apache. log4j. Logger;Import org. pentaho. Di. Core. kettleenvironment;Import org. pentaho. Di. Core. database. databasemeta;Import org. pentaho. Di. Core. Exception. kettledatabaseexception;Import org. pentaho. Di. Core. Exception. kettleexception;Import org. pentaho. Di. Core. util.

Environment Description:Now a project has a lot of homework, need to know which ktr run successfully after each run, which failedProblem solving:The following is a specific operating procedureFirst create a database tableCREATE TABLE test_1 (id int,name VARCHAR), INSERT into Test_1 VALUES (1, ' HSJ '); CREATE TABLE test_2 (id INT PRIMARY KEY not null,name VARCHAR (10));After the database table is established, the new KTR,KTR is as follows:In the interface right--"conversion settings", you can s

Today encountered a problem: different databases need to use full connectivity, so do not hesitate to merge join Plug-in, but in the process of using a lot of problems encountered. After you connect to get the field, a repeating field appears. Workaround: Change the field you want to associate to a different name.2. The data obtained is not the data we want:For example:Table A:1 A2 bTable B:1 of3 placesFull connection expected effect:1 A's2 b NULL3 null PlacesThe result is:1 a 1 of2 b NU

For any two buildings, the minimum spanning tree is obtained by the shortest path between them.The answer to the query (x, y) is the maximum value of the top right of the X-to-y path on the minimum spanning tree.The BFS finds the closest building to each point and the distance to it, and can find only the edges of the junction to use, using these edges to seek MST.#include 　　BZOJ4242: Kettle

Use the app Send mail icon: The information needed in this icon is not manually filled, but is taken from the previous stream.This generates data using the Generate records chart, which is given to the Send mail control using:In fact, the overall process is equivalent to kettle know your mailbox user name and password, using the standard interface to call the form of operation of your mailbox.Copyright NOTICE: This article for Bo Master original artic