71. GoldenGate
Architecture
Manager
The Manager
process must be running on both the source as well as target systems before the
Extract or Replicat process can be started and performs a number of functions
including monitoring and starting other GoldenGate processes, managing the
trail files and also reporting.
Extract
The Extract
process runs on the source system and is the data capture mechanism of
GoldenGate. It can be configured both for initial loading of the source data as
well as to synchronize the changed data on the source with the target. This can
be configured to also propagate any DDL changes on those databases where DDL
change support is available.
Replicat
The Replicat
process runs on the target system and reads transactional data changes as well
as DDL changes and replicates then to the target database. Like the Extract
process, the Replicat process can also be configured for Initial Load as well
as Change Synchronization.
Collector
The Collector is
a background process which runs on the target system and is started
automatically by the Manager (Dynamic Collector) or it can be configured to
start manually (Static Collector). It receives extracted data changes that are
sent via TCP/IP and writes then to the trail files from where they are
processed by the Replicat process.
Trails
Trails are
series of files that GoldenGate temporarily stores on disks and these files are
written to and read from by the Extract and Replicat processes as the case may
be. Depending on the configuration chosen, these trail files can exist on the
source as well as on the target systems. If it exists on the local system, it
will be known an Extract Trail or as an Remote Trail if it exists on the target
system.
Data Pumps
Data Pumps are
secondary extract mechanisms which exist in the source configuration. This is
optional component and if Data Pump is not used then Extract sends data via
TCP/IP to the remote trail on the target. When Data Pump is configured, the the
Primary Extract process will write to the Local Trail and then this trail is
read by the Data Pump and data is sent over the network to Remote Trails on the
target system.
In the absence
of Data Pump, the data that the Extract process extracts resides in memory
alone and there is no storage of this data anywhere on the source system. In
case of network of target failures, there could be cases where the primary
extract process can abort or abend. Data Pump can also be useful in those cases
where we are doing complex filtering and transformation of data as well as when
we are consolidating data from many sources to a central target.
Data
source
When processing
transactional data changes, the Extract process can obtain data directly from
the database transaction logs (Oracle, DB2, SQL Server, MySQL etc) or from a
GoldenGate Vendor Access Module (VAM) where the database vendor (for example
Teradata) will provide the required components that will be used by Extract to
extract the data changes.
Groups
To differentiate
between the number of different Extract and Replicat groups which can
potentially co-exist on a system, we can define processing groups. For
instance, if we want to replicate different sets of data in parallel, we can
create two Replicat groups.
A processing
group consisits of a process which could be either a Extract or Replicat
process, a corresponding parameter file, checkpoint file or checkpoint table
(for Replicat) and other files which could be associated with the process
72. What is
the default port for Goldengate manager process ?
7809
73. Information on all GoldenGate processes running on a system
GGSCI (devu007) 21> info all
Program Status Group Lag Time Since Chkpt
MANAGER RUNNING
EXTRACT RUNNING DPUMP 00:00:00 00:00:04
EXTRACT RUNNING EXT1 00:00:00 00:00:09
EXTRACT RUNNING EXT2 00:00:00 00:00:07
EXTRACT ABENDED GAVIN 00:00:00 73:29:25
EXTRACT STOPPED WORKPLAN 00:00:00 191:44:18
REPLICAT RUNNING MYLOAD2 00:00:00 00:00:09
REPLICAT RUNNING MYREP 00:00:00 00:00:08
74. status of a particular process
GGSCI (devu007) 23> status manager
Manager is running (IP port devu007.7809).
GGSCI (devu007) 24> status extract ext1
EXTRACT EXT1: RUNNING
Detailed information of a particular process
GGSCI (devu007) 6> info extract ext1, detail
EXTRACT EXT1 Last Started 2010-02-19 11:19 Status RUNNING
Checkpoint Lag 00:00:00 (updated 00:00:02 ago)
Log Read Checkpoint Oracle Redo Logs
2010-02-26 10:45:18 Seqno 786, RBA 44710400
Target Extract Trails:
Remote Trail Name Seqno RBA Max MB
/u01/oracle/software/goldengate/dirdat/lt 2 55644 10
Extract Source Begin End
/u02/oradata/apex/redo03.log 2010-02-19 11:13 2010-02-26 10:45
/u02/oradata/apex/redo02.log 2010-02-19 11:04 2010-02-19 11:13
/u02/oradata/apex/redo02.log 2010-02-18 10:42 2010-02-19 11:04
Not Available * Initialized * 2010-02-18 10:42
Current directory /u01/oracle/software/goldengate
Report file /u01/oracle/software/goldengate/dirrpt/EXT1.rpt
Parameter file /u01/oracle/software/goldengate/dirprm/ext1.prm
Checkpoint file /u01/oracle/software/goldengate/dirchk/EXT1.cpe
Process file /u01/oracle/software/goldengate/dirpcs/EXT1.pce
Stdout file /u01/oracle/software/goldengate/dirout/EXT1.out
Error log /u01/oracle/software/goldengate/ggserr.log
75. Monitoring an Extract recovery
GGSCI (devu007) 35> send extract ext1 status
Sending STATUS request to EXTRACT EXT1 ...
EXTRACT EXT1 (PID 1925238)
Current status: Recovery complete: At EOF
Sequence #: 786
RBA: 40549888
Timestamp: 2010-02-26 09:59:57.000000
Output trail #1
Current write position:
Sequence #: 2
RBA: 55644
Timestamp: 2010-02-26 09:59:54.337574
Extract Trail: /u01/oracle/software/goldengate/dirdat/lt
76. Monitoring processing volume - Statistics of the operations processed
GGSCI (devu007) 33> stats extract ext1
Sending STATS request to EXTRACT EXT1 ...
Start of Statistics at 2010-02-26 09:58:27.
DDL replication statistics (for all trails):
*** Total statistics since extract started ***
Operations 19.00
Mapped operations 2.00
Unmapped operations 9.00
Other operations 8.00
Excluded operations 17.00
Output to /u01/oracle/software/goldengate/dirdat/lt:
Extracting from GGS_OWNER.GGS_MARKER to GGS_OWNER.GGS_MARKER:
*** Total statistics since 2010-02-19 11:21:03 ***
No database operations have been performed.
Extracting from MONITOR.WORK_PLAN to MONITOR.WORK_PLAN:
*** Total statistics since 2010-02-19 11:21:03 ***
Total inserts 4.00
Total updates 46.00
Total deletes 0.00
Total discards 0.00
Total operations 50.00
*** Daily statistics since 2010-02-26 00:00:00 ***
Total inserts 0.00
Total updates 16.00
Total deletes 0.00
Total discards 0.00
Total operations 16.00
*** Hourly statistics since 2010-02-26 09:00:00 ***
No database operations have been performed.
*** Latest statistics since 2010-02-19 11:21:03 ***
Total inserts 4.00
Total updates 46.00
Total deletes 0.00
Total discards 0.00
Total operations 50.00
End of Statistics.
View processing rate - can use 'hr','min' or 'sec' as a parameter
GGSCI (devu007) 37> stats extract ext2 reportrate hr
Sending STATS request to EXTRACT EXT2 ...
Start of Statistics at 2010-02-26 10:04:46.
Output to /u01/oracle/ggs/dirdat/cc:
Extracting from SH.CUSTOMERS to SH.CUSTOMERS:
*** Total statistics since 2010-02-26 09:29:48 ***
Total inserts/hour: 0.00
Total updates/hour: 95258.62
Total deletes/hour: 0.00
Total discards/hour: 0.00
Total operations/hour: 95258.62
*** Daily statistics since 2010-02-26 09:29:48 ***
Total inserts/hour: 0.00
Total updates/hour: 95258.62
Total deletes/hour: 0.00
Total discards/hour: 0.00
Total operations/hour: 95258.62
*** Hourly statistics since 2010-02-26 10:00:00 ***
No database operations have been performed.
*** Latest statistics since 2010-02-26 09:29:48 ***
Total inserts/hour: 0.00
Total updates/hour: 95258.62
Total deletes/hour: 0.00
Total discards/hour: 0.00
Total operations/hour: 95258.62
End of Statistics.
77. View latency between the records processed by Goldengate and the timestamp in the data source
GGSCI (devu007) 13> send extract ext2, getlag
Sending GETLAG request to EXTRACT EXT2 ...
Last record lag: 3 seconds.
At EOF, no more records to process.
GGSCI (devu007) 15> lag extract ext*
Sending GETLAG request to EXTRACT EXT1 ...
Last record lag: 1 seconds.
At EOF, no more records to process.
Sending GETLAG request to EXTRACT EXT2 ...
Last record lag: 1 seconds.
At EOF, no more records to process.
78. Viewing the GoldenGate error log as well as
history of commands executed and other events
We can use the
editor depending on operating system – vi on Unix for example to view the ggserr.log
file which is located at the top level GoldenGate software installation
directory.
We can also use
the GGSCI command VIEW GGSEVT as well.
79. View the process report
Every Manager,
Extract and Replicat process will generate a report file at the end of each run
and this report can be viewed to diagnose any problems or errors as well as
view the parameters used, the environment variables is use, memory consumption
etc
For example:
GGSCI (devu007) 2> view report ext1
GGSCI (devu007) 2> view report rep1
GGSCI (devu007) 2> view report mgr
80. Information on Child processes started by the Manager
GGSCI (devu007) 8> send manager childstatus
Sending CHILDSTATUS request to MANAGER ...
Child Process Status - 6 Entries
ID Group Process Retry Retry Time Start Time
---- -------- ---------- ----- ------------------ -----------
0 EXT1 1925238 0 None 2010/02/19 11:07:54
1 DPUMP 2195496 0 None 2010/02/19 11:08:02
2 MSSQL1 422034 0 None 2010/02/22 13:54:59
4 MYREP 1302702 0 None 2010/02/23 09:08:34
6 MYLOAD2 1200242 0 None 2010/02/23 11:05:01
7 EXT2 2076844 0 None 2010/02/26 08:29:22
81. GoldenGate Directory structure
After successfully installing Oracle GoldenGate,
you must create the sub-directories structure used by the different components
of Oracle GoldenGate.
The directories are the default locations when
creating objects and parameter files.
Now, start GGSCI and run the below commands to create the directory structure.
Now, start GGSCI and run the below commands to create the directory structure.
GGSCI (edmr1s35) 1> create
subdirs
Creating subdirectories under current directory /u01/app/ogg/10.2.0
Parameter files
/u01/app/ogg/10.2.0/dirprm: created
Report files
/u01/app/ogg/10.2.0/dirrpt: created
Checkpoint files
/u01/app/ogg/10.2.0/dirchk: created
Process status files
/u01/app/ogg/10.2.0/dirpcs: created
SQL script files
/u01/app/ogg/10.2.0/dirsql: created
Database definitions files
/u01/app/ogg/10.2.0/dirdef: created
Extract data files
/u01/app/ogg/10.2.0/dirdat: created
Temporary files
/u01/app/ogg/10.2.0/dirtmp: created
Veridata files
/u01/app/ogg/10.2.0/dirver: created
Veridata Lock files
/u01/app/ogg/10.2.0/dirver/lock: created
Veridata Out-Of-Sync files
/u01/app/ogg/10.2.0/dirver/oos: created
Veridata Out-Of-Sync XML files /u01/app/ogg/10.2.0/dirver/oosxml: created
Veridata Parameter files
/u01/app/ogg/10.2.0/dirver/params: created
Veridata Report files
/u01/app/ogg/10.2.0/dirver/report: created
Veridata Status files
/u01/app/ogg/10.2.0/dirver/status: created
Veridata Trace files
/u01/app/ogg/10.2.0/dirver/trace: created
Stdout files
/u01/app/ogg/10.2.0/dirout: created
Successful
installation of Oracle GoldenGate 12c creates the directory structure. Use the GGSCI SHOW command to display the
instance directory structure.
The Goldengate
dirprm directory
This
directory is the default location for parameter files associated with group
names. When an object created such as extract, data pump extract and replicat,
a parameter file is looked at for the group name. Using non-default location is
referenced by the parameter PARAMS <parameter file>. Regardless of the parameter file location,
the parameter file name and the group name must match.
The Goldengate
dirrpt directory
This
directory is the default location for reports and discarded records. Whenever a
process ABENDED, a report file is created on this directory. The report file
indicates the reasons for failure, and if the failure is database related, it
also indicate the Oracle error code. Using non-default location is referenced
by the parameter REPORT <report file>.
The Goldengate
dirchk directory
This
directory is for storing extract, data pump and replicat checkpoint files.
Depending on the object type, a file is created using the group name with
proper a relevant extension. For extract process, the checkpoint file name is
<group name>.cpe. For replicat process, the checkpoint file name is
<group name>.cpr.
The Goldengate
dirpcs directory
This
directory stores Oracle GoldenGate process identifier (PID). It’s the PID
assigned by the operating system to running program, which is associated with
an Oracle GoldenGate process. To obtain more details, use the operating system
command ps –edf | grep <group name>.
The Goldengate
dirsql directory
This
directory is default location to store SQL Script used by Oracle GoldenGate.
The Goldengate
dirdef directory
This
directory is used to store definition files. Definition files are required when
the source and the target tables are not identical. Mostly used for
heterogeneous environment when mapping tables across dissimilar platforms.
The Goldengate
dirdat directory
This
directory is used to start trail files. The trail files are identified by a two
character prefix, followed by 6 digits. An extract or replicat processes trail
files are designated by the parameter EXTTRAIL followed by the trial file
location and name. For example, EXTTRAIL ./dirdat/sa.
The Goldengate
dirtmp directory
This
directory is the default location for Oracle GoldenGate temporary files due to
capturing large transactions that goes beyond default cache size used by the
extract process. The location and size is controlled by the CACHEMGR
sub-parameters CACHESIZE, CACHEDIRECTORY and CACHEPAGEOUTSIZE. Also, it’s
always recommended to dedicate a directory location to host the temporary
files, avoiding contention with other type of files.
To
confirm the current Oracle GoldenGate environment and subdirectories locations,
use the GGSCI SHOW command.
GGSCI
(ggs-source) 12> show
Parameter settings:
SET SUBDIRS ON
SET DEBUG OFF
Current directory: /u01/app/oracle/ggs
Using subdirectories for all process files
Editor: vi
Reports (.rpt) /u01/app/oracle/ggs/dirrpt
Parameters (.prm)
/u01/app/oracle/ggs/dirprm
Replicat Checkpoints (.cpr)
/u01/app/oracle/ggs/dirchk
Extract Checkpoints (.cpe)
/u01/app/oracle/ggs/dirchk
Process Status (.pcs)
/u01/app/oracle/ggs/dirpcs
SQL Scripts (.sql)
/u01/app/oracle/ggs/dirsql
Database Definitions (.def)
/u01/app/oracle/ggs/dirdef
Dump files (.dmp)
/u01/app/oracle/ggs/dirdmp
Masterkey wallet files (.wlt)
/u01/app/oracle/ggs/dirwlt
Credential store files (.crd)
/u01/app/oracle/ggs/dircrd
82. GoldenGate checkpoint table
GoldenGate
maintains its own Checkpoints which is a known position in the trail file from
where the Replicat process will start processing after any kind of error or
shutdown.
This ensures
data integrity and a record of these checkpoints is either maintained in files
stored on disk or table in the database which is the preferred option.
Oracle GoldenGate processes
extract and replicat processes perform checkpoint operations, in the event of
failure, the checkpoint file or database table ensures extract and replicat
re-start from the point failure avoiding re-capture and re-apply of
transactions.
We can also
create a single Checkpoint table which can used by all Replicat groups from the
single or many GoldenGate instances.
The default and
best-practice checkpoint for the delivery group is a database table, and we use
the GGSCI add checkpointtable command to create the checkpoint table.
An alternative is
the use of checkpoint file by specifying nocheckpointtble
parameter.
No comments:
Post a Comment