OGG synchronizes Oracle Data to Kafka

OGG synchronizes Oracle Data to Kafka

Environment:
Source: Oracle12.2 ogg for Oracle 12.3
Target: Kafka ogg for bigdata 12.3
Synchronize Oracle Data to Kafka through OGG

Source Configuration:
1. Add additional logs to the table to be synchronized
Dblogin USERID ogg @ orclpdb, PASSWORD ogg
Add trandata scott. tab1
Add trandata scott. tab2

2. Add extraction process
GGSCI> add extract EXT_KAF1, integrated tranlog, begin now
GGSCI> add EXTTRAIL./dirdat/k1, extract EXT_KAF1, MEGABYTES 200

Edit extraction process parameters:
GGSCI> edit params EXT_KAF1

Extract EXT_KAF1
Userid c # ggadmin, PASSWORD ggadmin
LOGALLSUPCOLS
UPDATERECORDFORMAT COMPACT
Exttrail./dirdat/k1, format release 12.3
SOURCECATALOG orclpdb -- (specify pdb)
Table scott. tab1;
Table scott. tab2;

Registration Process
GGSCI> dblogin userid c # ggadmin, PASSWORD ggadmin
GGSCI> register extract EXT_KAF1 database container (orclpdb)

3. Add a shipping process:
GGSCI> add extract PMP_KAF1, exttrailsource./dirdat/k1
GGSCI> add rmttrail./dirdat/f1, EXTRACT PMP_KAF1, MEGABYTES 200

Edit shipping process parameters:
GGSCI> edit param PMP_KAF1

EXTRACT PMP_KAF1
USERID c # ggadmin, PASSWORD ggadmin
PASSTHRU
RMTHOST 10.1.1.247, MGRPORT 9178
RMTTRAIL./dirdat/F1. format release 12.3
SOURCECATALOG orclpdb
TABLE scott. tab1;
Table scott. tab2;

4. Add the data initialization process (Oracle initial load). Multiple tables can be initialized separately or together. Separate Initialization is selected here.
GGSCI> add extract ek_01, sourceistable

Edit parameters:
GGSCI> edit params ek_01

EXTRACT ek_01
USERID c # ggadmin, PASSWORD ggadmin
RMTHOST 10.1.1.247, MGRPORT 9178
RMTFILE./dirdat/ka, maxfiles 999, megabytes 500, format release 12.3
SOURCECATALOG orclpdb
Table scott. tab1;

GGSCI> add extract ek_02, sourceistable

Edit params ek_02
EXTRACT ek_02
USERID c # ggadmin, PASSWORD ggadmin
RMTHOST 10.1.1.247, MGRPORT 9178
RMTFILE./dirdat/kb, maxfiles 999, megabytes 500, format release 12.3
SOURCECATALOG orclpdb
Table scott. tab2;

5. Generate the def file:
GGSCI> edit param defgen1

USERID c # ggadmin, PASSWORD ggadmin
Defsfile/home/oracle/ogg/ggs12/dirdef/defgen1.def, format release 12.3
SOURCECATALOG orclpdb
Table scott. tab1;
Table scott. tab2;

Run the following command under OGG_HOME to generate the def file:
Defgen paramfile dirprm/defgen1.prm

Upload the generated def file to the target end $ OGG_HOME/dirdef.

Target end Configuration:
1. copy all files under $ OGG_HOME/AdapterExamples/big-data/kafka to $ OGG_HOME/dirprm.
Cd $ OGG_HOME/AdapterExamples/big-data/kafka
Cp * $ OGG_HOME/dirprm

2. copy the tr000000000 file under $ ORACLE_HOME/AdapterExamples/trail to $ OGG_HOME/dirdat.
Cd $ ORACLE_HOME/AdapterExamples/trail
Cp tr000000000 $ OGG_HOME/dirdat

3. Add initialization process: (you can initialize multiple tables together or separately. Select separate initialization here)
GGSCI> ADD replicat rp_01, specialrun

GGSCI> edit params rp_01
SPECIALRUN
End runtime
Setenv (NLS_LANG = "AMERICAN_AMERICA.ZHS16GBK ")
Targetdb libfile libggjava. so set property =./dirprm/kafka1.props
SOURCEDEFS./dirdef/defgen1.def
EXTFILE./dirdat/ka
Reportcount every 1 minutes, rate
Grouptransops 10000
MAP orclpdb. scott. tab1, TARGET scott. tab1;

GGSCI> ADD replicat rp_02, specialrun
GGSCI> edit params rp_02

SPECIALRUN
End runtime
Setenv (NLS_LANG = "AMERICAN_AMERICA.ZHS16GBK ")
Targetdb libfile libggjava. so set property =./dirprm/kafka2.props
SOURCEDEFS./dirdef/defgen1.def
EXTFILE./dirdat/kb
Reportcount every 1 minutes, rate
Grouptransops 10000
MAP orclpdb. scott. tab2, TARGET scott. tab2;

4. Add a recovery process:
GGSCI> add replicat r_kaf1, exttrail./dirdat/f1
GGSCI> edit params r_kaf1

REPLICAT r_kaf1
Setenv (NLS_LANG = "AMERICAN_AMERICA.ZHS16GBK ")
HANDLECOLLISIONS
Targetdb libfile libggjava. so set property =./dirprm/kafka1.props
SOURCEDEFS./dirdef/defgen1.def
Reportcount every 1 minutes, rate
Grouptransops 10000
MAP orclpdb. scott. tab1, TARGET scott. tab1;

GGSCI> add replicat r_kaf2, exttrail./dirdat/f2
GGSCI> edit params r_kaf2

REPLICAT r_kaf2
Setenv (NLS_LANG = "AMERICAN_AMERICA.ZHS16GBK ")
HANDLECOLLISIONS
Targetdb libfile libggjava. so set property =./dirprm/kafka2.props
SOURCEDEFS./dirdef/defgen1.def
Reportcount every 1 minutes, rate
Grouptransops 10000
MAP orclpdb. scott. tab2, TARGET scott. tab2;

5. parameter configuration:
The contents of the custom_kafka_producer.properties file are as follows:

Bootstrap. servers = 10.1.1.246: 9200, 10.1.1.247: 9200 -- you only need to modify this line and specify the address and port number of kafka.
Acks = 1
Reconnect. backoff. milliseconds = 1000
Value. serializer = org. apache. kafka. common. serialization. ByteArraySerializer
Key. serializer = org. apache. kafka. common. serialization. ByteArraySerializer
Batch. size = 16384
Linger. ms = 10000

The content of the kafka1.props file is as follows:
Gg. handlerlist = kafkahandler
Gg. handler. kafkahandler. type = kafka
Gg. handler. kafkahandler. KafkaProducerConfigFile = custom_kafka_producer.properties
# The following resolves the topic name using the short table name
Gg. handler. kafkahandler. topicMappingTemplate = topic1
# Gg. handler. kafkahandler. format = avro_op
Gg. handler. kafkahandler. format = json -- here we have changed the format to json.
Gg. handler. kafkahandler. format. insertOpKey = I
Gg. handler. kafkahandler. format. updateOpKey = U
Gg. handler. kafkahandler. format. deleteOpKey = D
Gg. handler. kafkahandler. format. truncateOpKey = T
Gg. handler. kafkahandler. format. prettyPrint = false
Gg. handler. kafkahandler. format. jsonDelimiter = CDATA []
Gg. handler. kafkahandler. format. includePrimaryKeys = true -- contains the primary key
Gg. handler. kafkahandler. SchemaTopicName = topic1 -- specify the name of the target topic to be synchronized.
Gg. handler. kafkahandler. BlockingSend = false
Gg. handler. kafkahandler. includeTokens = false
Gg. handler. kafkahandler. mode = op
Goldengate. userexit. timestamp = utc
Goldengate. userexit. writers = javawriter
Javawriter. stats. display = TRUE
Javawriter. stats. full = TRUE
Gg. log = log4j
Gg. log. level = INFO
Gg. report. time = 30sec
# Sample gg. classpath for Apache Kafka
Gg. classpath = dirprm/:/opt/cloudera/parcels/KAFKA/lib/kafka/libs/-- specify classpath. It is very important to have a class library for the kafka Installation File.
# Sample gg. classpath for HDP
# Gg. classpath =/etc/kafka/conf:/usr/hdp/current/kafka-broker/libs/
Javawriter. bootoptions =-Xmx512m-Xms32m-Djava. class. path = ggjava/ggjava. jar

Start Process recovery:
1. Start the source capture process
GGSCI> start EXT_KAF1
2. Start the source delivery process
GGSCI> start PMP_KAF1
3. Start the source initialization process
GGSCI> start ek_01
4. Start the initialization process on the target end.
Run the following command under $ OGG_HOME:
./Replicat paramfile./dirprm/rp_01.prm reportfile./dirrpt/rp_01.rpt-p INITIALDATALOAD
5. Start the recovery process on the target end.
GGSCI> start R_KAF1

Errors:
1, ERROR OGG-15050 Error loading Java VM runtime library (2 no such file or directory)

Cause: the class library cannot be found. (after the environment variable is configured, The mgr process of OGG is not restarted)
Solution: restart the MGR process.

2, ERROR OG-15051 Java or JNI exception

Cause: the kafka. props that comes with ogg12.3.1.1.1 is not used, but the kafka. props of ogg12.2 is copied, resulting in exceptions.
Solution: Use kafka. props that comes with ogg12.3.1.1.1 and specify relevant attributes to solve the problem.

Https://www.bkjia.com/topicnews.aspx? Tid = 12

This article permanently updates link: https://www.bkjia.com/Linux/2018-03/151513.htm