OGG synchronizes Oracle data to Kafka

Environment:
SOURCE side: oracle12.2 ogg for Oracle 12.3
Target side: KAFKA ogg for Bigdata 12.3
Synchronizing data from Oracle to Kafka via OGG

Source-side configuration:
1. Add additional logs for the tables to be synchronized
dblogin USERID [email protected], PASSWORD ogg
Add Trandata scott.tab1
Add Trandata scott.tab2

2. Add Extraction Process
Ggsci>add extract ext_kaf1,integrated Tranlog, begin now
Ggsci>add exttrail/dirdat/k1, extract ext_kaf1,megabytes 200

To edit the extraction process parameters:
ggsci> Edit Params EXT_KAF1

Extract EXT_KAF1
UserID C # #ggadmin, PASSWORD ggadmin
Logallsupcols
Updaterecordformat COMPACT
Exttrail./dirdat/k1,format RELEASE 12.3
Sourcecatalog orclpdb-(Specify PDB)
Table SCOTT.TAB1;
Table scott.tab2;

Registration process
Ggsci> dblogin USERID C # #ggadmin, PASSWORD ggadmin
Ggsci> Register Extract EXT_KAF1 Database Container (orclpdb)

3, add the delivery process:
Ggsci>add extract PMP_KAF1, Exttrailsource./dirdat/k1
Ggsci>add rmttrail./dirdat/f1,extract pmp_kaf1,megabytes 200

To edit the delivery process parameters:
Ggsci>edit param PMP_KAF1

EXTRACT PMP_KAF1
USERID C # #ggadmin, PASSWORD ggadmin
PASSTHRU
Rmthost 10.1.1.247, Mgrport 9178
Rmttrail./dirdat/f1,format Release 12.3
Sourcecatalog orclpdb
TABLE SCOTT.TAB1;
Table scott.tab2;

4. Adding the data initialization process (Oracle initial load) can initialize multiple tables separately or together, choosing to initialize separately
Ggsci> Add extract ek_01, sourceistable

Edit parameters:
ggsci> EDIT PARAMS ek_01

EXTRACT ek_01
USERID C # #ggadmin, PASSWORD ggadmin
Rmthost 10.1.1.247, Mgrport 9178
Rmtfile./dirdat/ka,maxfiles 999, megabytes 500,format release 12.3
Sourcecatalog orclpdb
Table SCOTT.TAB1;

Ggsci> Add extract ek_02, sourceistable

EDIT PARAMS ek_02
EXTRACT ek_02
USERID C # #ggadmin, PASSWORD ggadmin
Rmthost 10.1.1.247, Mgrport 9178
Rmtfile./dirdat/kb,maxfiles 999, megabytes 500,format release 12.3
Sourcecatalog orclpdb
Table scott.tab2;

5. Generate DEF file:
ggsci> edit param Defgen1

USERID C # #ggadmin, PASSWORD ggadmin
Defsfile/home/oracle/ogg/ggs12/dirdef/defgen1.def,format Release 12.3
Sourcecatalog orclpdb
Table SCOTT.TAB1;
Table scott.tab2;

Under Ogg_home, execute the following command to generate the Def file
Defgen Paramfile DIRPRM/DEFGEN1.PRM

Upload the generated DEF file to the target end $ogg_home/dirdef

Target-side configuration:
1. Copy all files under $ogg_home/adapterexamples/big-data/kafka to $OGG_HOME/DIRPRM
CD $OGG _home/adapterexamples/big-data/kafka
CP * $OGG _HOME/DIRPRM

2, the $oracle_home/adapterexamples/trail under the file tr000000000 copy to $ogg_home/dirdat under
CD $ORACLE _home/adapterexamples/trail
CP tr000000000 $OGG _home/dirdat

3. Add initialization process: (can be initialized with multiple tables or can be initialized separately, here choose to initialize separately)
Ggsci> ADD replicat rp_01, Specialrun

ggsci> EDIT PARAMS rp_01
Specialrun
End Runtime
Setenv (nls_lang= "American_america. ZHS16GBK ")
Targetdb libfile libggjava.so Set Property=./dirprm/kafka1.props
Sourcedefs./dirdef/defgen1.def
Extfile./dirdat/ka
Reportcount Every 1 minutes, rate
Grouptransops 10000
MAP orclpdb.scott.tab1, TARGET scott.tab1;

Ggsci> ADD replicat rp_02, Specialrun
ggsci> EDIT PARAMS rp_02

Specialrun
End Runtime
Setenv (nls_lang= "American_america. ZHS16GBK ")
Targetdb libfile libggjava.so Set Property=./dirprm/kafka2.props
Sourcedefs./dirdef/defgen1.def
Extfile./dirdat/kb
Reportcount Every 1 minutes, rate
Grouptransops 10000
MAP orclpdb.scott.tab2, TARGET scott.tab2;

4. Add recovery process:
Ggsci>add replicat R_kaf1,exttrail./dirdat/f1
Ggsci>edit params R_KAF1

Replicat R_KAF1
Setenv (nls_lang= "American_america. ZHS16GBK ")
Handlecollisions
Targetdb libfile libggjava.so Set Property=./dirprm/kafka1.props
Sourcedefs./dirdef/defgen1.def
Reportcount Every 1 minutes, rate
Grouptransops 10000
MAP orclpdb.scott.tab1, TARGET scott.tab1;

ggsci> add Replicat r_kaf2,exttrail./dirdat/f2
ggsci> Edit Params r_kaf2

Replicat r_kaf2
Setenv (nls_lang= "American_america. ZHS16GBK ")
Handlecollisions
Targetdb libfile libggjava.so Set Property=./dirprm/kafka2.props
Sourcedefs./dirdef/defgen1.def
Reportcount Every 1 minutes, rate
Grouptransops 10000
MAP orclpdb.scott.tab2, TARGET scott.tab2;

5, parameter configuration:
The contents of the Custom_kafka_producer.properties file are as follows:

BOOTSTRAP.SERVERS=10.1.1.246:9200,10.1.1.247:9200--Just change this line, specify the address and port number of the Kafka
Acks=1
reconnect.backoff.ms=1000
Value.serializer=org.apache.kafka.common.serialization.bytearrayserializer
Key.serializer=org.apache.kafka.common.serialization.bytearrayserializer
batch.size=16384
linger.ms=10000

The contents of the Kafka1.props file are as follows:
Gg.handlerlist = Kafkahandler
Gg.handler.kafkahandler.type=kafka
Gg.handler.kafkahandler.kafkaproducerconfigfile=custom_kafka_producer.properties
#The following resolves the topic name using the short table name
Gg.handler.kafkahandler.topicmappingtemplate= Topic1
#gg. Handler.kafkahandler.format=avro_op
Gg.handler.kafkahandler.format =json--changes are made here to specify the format in JSON format
Gg.handler.kafkahandler.format.insertopkey=i
Gg.handler.kafkahandler.format.updateopkey=u
Gg.handler.kafkahandler.format.deleteopkey=d
Gg.handler.kafkahandler.format.truncateopkey=t
Gg.handler.kafkahandler.format.prettyprint=false
Gg.handler.kafkahandler.format.jsondelimiter=cdata[]
Gg.handler.kafkahandler.format.includeprimarykeys=true--Contains primary key
Gg.handler.kafkahandler.schematopicname= Topic1--specified here as the target topic name to sync to
Gg.handler.kafkahandler.BlockingSend =false
Gg.handler.kafkahandler.includetokens=false
Gg.handler.kafkahandler.mode=op
Goldengate.userexit.timestamp=utc
Goldengate.userexit.writers=javawriter
Javawriter.stats.display=true
Javawriter.stats.full=true
gg.log=log4j
Gg.log.level=info
Gg.report.time=30sec
#Sample Gg.classpath for Apache Kafka
gg.classpath=dirprm/:/opt/cloudera/parcels/kafka/lib/kafka/libs/--Specify CLASSPATH, here is important, must have Kafka installation files of the class library.
#Sample Gg.classpath for HDP
#gg. classpath=/etc/kafka/conf:/usr/hdp/current/kafka-broker/libs/
Javawriter.bootoptions=-xmx512m-xms32m-djava.class.path=ggjava/ggjava.jar

Start Process Recovery:
1, start the source-side crawl process
ggsci> Start EXT_KAF1
2. Start the source-side delivery process
ggsci> Start PMP_KAF1
3. Start the source-side initialization process
ggsci> Start ek_01
4. Start the target-side initialization process
Under $ogg_home, execute the following command:
./replicat paramfile./DIRPRM/RP_01.PRM reportfile./dirrpt/rp_01.rpt-p initialdataload
5. Start the target-side recovery process
ggsci> Start R_KAF1

Error encountered:
1. Error OGG-15050 Error loading Java VM Runtime Library (2 No such file or directory)

Cause: The class library could not be found (after configuring the environment variable, OGG's MGR process did not restart, resulting in)
Workaround: Restart the MGR Process

2. ERROR OG-15051 Java or JNI exception

Reason: Instead of using ogg12.3.1.1.1 's own kafka.props, the ogg12.2 kafka.props was copied, resulting in an exception.
Workaround: Use ogg12.3.1.1.1 to bring the Kafka.props, and specify the relevant properties to resolve.

OGG synchronizes Oracle data to Kafka