Need to get more details about Oracle GoldenGate CSN.
Following is the system architecture configured.
Source DataBase - Oracle
Target DataBase - Oracle
For each table on souce database, 2 tables (BASE table and DELETE table) are defined on target database.
Configured 2 replicas to transfer data from source to target database.
One replica moves the INSERT/UPDATES to target database and other replica moves the DELETE records to target.
Following is the view defined by GG which gives GoldenGate metadata information.
GoldenGate metadata info
Row with servername ends with 'CRN01A' represents the GG replica for BASE table.
Row with servername ends with 'CRN01D' represents the GG replica for DELETE table.
APPLIED_LOW_POSITION gives 'All messages with commit position less than this value have been applied'.
Our question is if the both replicas will have their own isolated CSN or in synchronization with respect to extract.
Example:
APPLIED_LOW_POSITION initial value - 100 for both BASE table replica and DELETE table replica.
100 INSERTS/UPDATES occured on source DB. BASE table replica APPLIED_LOW_POSITION value changed to 200.
After step2, 3 DELETEs occured on source system. Our question is at this point of time, what will be the APPLIED_LOW_POSITION value for DELETE replica?
Will it be 103 or 203?
Can you please provide your thoughts?
The CSN counter is connected with the CSN number on the source database. It always grows.
If the CSN was equal 100, then some transactions (INSERT) have occurred. It grew to 200.
Then 3 additional DELETE operations were made. And from 200 it would grow to 203.
Those 3 DELETE operations may not be replicated to the INSERT/UPDATE target, so the APPLIED_LOW_POSITION may not change on this target and stay at the 200 level.
But on the DELETE target the APPLIED_LOW_POSITION would rise to 203.
Related
Using the cordapp-example as a basis where the IOUstate is a queryableState and persisted. In the DB, you will see a new table IOU_States with column values as you have defined.
Build the project and start the nodes
Create a Tx from partyA to partyB
flow start ExampleFlow iouValue: 50, otherParty: "O=PartyB,L=New York,C=US"
Run vaultquery() on partyA and take note of the displayed output (label as display 1)
run vaultQuery contractStateType: com.example.state.IOUState
attach a H2 console to DB of partyA
Run search on IOU_States table
You will see the IOUState state object as a row item, note the value of 50
Run an update to change the value of 50 to 60
Run search on the IOU_States table to confirm the change
Run vaultquery() on partyA and take note of the displayed output (lable as display 2)
display 1 = display 2
Question:
1. When i have corrupted my persisted table, what exactly have i changed?
2. Does vaultQuery() query the node_transactions instead and de-serialise from the blob?
3. In the Vault_states table, we used to have a column Contract_states, but it is no longer there. That is the snapshot we tend to change to test data tampering previously. Where is the snap shot of the state kept now?
changing your state's table does not corrupt the transaction , the vault query acts on the vault, so even if you update the data , any transaction that has taken the manipulated state will not have inconsistencies
I have a situation where I have a Table in Cloudera Impala (Parquet Format),
The table statistcs are:
Size: 23GB
Rows: 67M
RowSize: Approx 5KB
Columns: 308
My Cloudera is
Total 6 Nodes Cloudera Cluster (Disk : 84TB Each, Ram: 251GB Each)
Kudu Master and Tablet Server
2 Master Nodes, 5 Tablet Servers (One Node acts as a Tablet Server as well as Master)
here is my table Schema(Structure)
CREATE TABLE SRV_REQ_X
PRIMARY KEY (row_id)
PARTITION BY HASH(row_id) PARTITIONS 5
STORED AS KUDU
TBLPROPERTIES ('kudu.table_name'='IMPALA_DATABASE.KUDU TABLE NAME','kudu.master_addresses'='host1:7051,host2:7051','kudu.num_tablet_replicas' = '3')
AS
Select columns* from table*
Different Properties tested
The properties I have checked and played with are
memory_limit_hard_bytes = Checked with 0 and 1 and 250GB (Same result Tablet
Server Crashes)
maintenance_manager_num = Checked with 1 as well as 4
Records are inserted but at some point this error Comes
Kudu error(s) reported, first error: Timed out: Failed to write batch of 94 ops to tablet 842e935e768f4a419b193e1fb18e3155 after 329 attempt(s): Failed to write to server: 2d35eb2445e747bea574a5e1af6e0b2a (bda-ptcl1node02.ptcl.net.pk:7050): Write RPC to 192.168.228.2:7050 timed out after 179.996s (SENT)
I need to insert other tables which are around 102M records and I cannot understand how to tweak Kudu Properties against my Cluster.
P.S The most records went into the Kudu Table were 13M with Following Properties and then the timeout happened.
memory_limit_hard_bytes = 250GB
maintenance_manager_num = 4
block_cache_capacity_mb = 130GB
Partitions: 4
Please Help!!
I am using the pi to record video surveillance data in a single, but highly populated table per camera. The table consists of 3 columns - Timestamp, offset and frame length. The video data is stored in a separate file on the filesystem. My code is written in C.
Timestamp is the date/time for a video frame in the stream, offset is the fseek/ftell offsets into the streaming data file and frame length is the length of the frame. Pretty self explanatory. The primary and only index is on the timestamp column.
There is one database writer forked process per camera and there could be multiple forked read-only processes querying the database at any time.
These processes are created by socket listeners in the classic client/server architecture which accept video streams from other processes that manage the surveillance cameras and clients that query it.
When a read-only client connects, it selects the first row in the database for a selected camera. For some reason, the select takes > 60 secs and subsequent selects on the same query are very snappy (much less than 1 sec). I've debugged the code to determine this is the cause.
I have these pragma's configured both for the reader and writer forked processes and have tried greater and lesser values with minimal if any impact:
pragma busy_timeout=7000
pragma cache_size=-4096
pragma mmap_size=4194304
I am assuming the cause is due to populating the SQLite3 caches when a read-only client connects, but I'm not sure what else to try.
I've implemented my own write caching/buffering strategy to help prevent locks, which helped significantly, but it did not solve the delay at startup problem.
I've also split the table by weekday in an attempt to help control the table population size. It seems once the population nears 100,000 rows, the problem starts occurring. The population for a table can be around 2.5 million rows per day.
Here is the query:
sprintf(sql, "select * from %s_%s.VIDEO_FRAME where TIME_STAMP = "
"(select min(TIME_STAMP) from %s_%s.VIDEO_FRAME)",
cam_name, day_of_week, cam_name, day_of_week);
(edit)
$ uname -a
Linux raspberrypi3 4.1.19-v7+ #858 SMP Tue Mar 15 15:56:00 GMT 2016 armv7l GNU/Linux
$ sqlite3
sqlite> .open Video_Camera_02__Belkin_NetCam__WED.db
sqlite> .tables VIDEO_FRAME
sqlite> .schema VIDEO_FRAME CREATE TABLE VIDEO_FRAME(TIME_STAMP UNSIGNED BIG INT NOT NULL,FRAME_OFFSET BIGINT, FRAME_LENGTH INTEGER,PRIMARY KEY(TIME_STAMP));
sqlite> explain query plan
...> select * from VIDEO_FRAME where TIME_STAMP = (select min(TIME_STAMP) from VIDEO_FRAME);
0|0|0|SEARCH TABLE VIDEO_FRAME USING INDEX sqlite_autoindex_VIDEO_FRAME_1 (TIME_STAMP=?)
0|0|0|EXECUTE SCALAR SUBQUERY 1 1|0|0|SEARCH TABLE VIDEO_FRAME USING COVERING INDEX sqlite_autoindex_VIDEO_FRAME_1 –
After some further troubleshooting, the culprit seems to be with the forked db writer process. I tried starting the r/o clients with no streaming data being written and the select returned immediately. I haven't found the root problem, but at least have isolated where it is coming from.
Thanks!
I would like to monitor 10 tables with 1000 records per table. I need to know when a record, and which record changed.
I have looked into SQL Dependencies, however it appears that SQL Dependencies would only be able to tell me that the table changed, and not which record changed. I would then have to compare all the records in the table to find the modified record. I suspect this would be a problem for me as the records constantly change.
I have also looked into SQL Trigger's, however I am not sure if triggers would work for monitoring which record changed.
Another thought I had, is to create a "Monitoring" table which would have records added to it via the application code whenever a record is modified.
Do you know of any other methods?
EDIT:
I am using SQL Server 2008
I have looked into Change Data Capture which is available in SQL 2008 and suggested by Martin Smith. Change Data Capture appears to be a robust, easy to implement and very attractive solution. I am going to roll CDC on my database.
You can add triggers and have them add rows to an audit table. They can audit the primary key of the rows that changed, and even additional information about the changes. For instance, in the case of an UPDATE, they can record the columns that changed.
Before you write/implement your own take a look at AutoAudit :
AutoAudit is a SQL Server (2005, 2008) Code-Gen utility that creates
Audit Trail Triggers with:
Created, CreatedBy, Modified, ModifiedBy, and RowVersion (incrementing INT) columns to table
Insert event logged to Audit table
Updates old and new values logged to Audit table
Delete logs all final values to the Audit table
view to reconstruct deleted rows
UDF to reconstruct Row History
Schema Audit Trigger to track schema changes
Re-code-gens triggers when Alter Table changes the table
What version and edition of SQL Server? Is Change Data Capture available? – Martin Smith
I am using SQL 2008 which supports Change Data Capture. Change Data Capture is a very robust method for tracking data changes as I would like to. Thanks for the answer.
Here's an idea.You can have a flag on each table that every time a record is created or updated is filled with current datetime. Then when you notice that a record has changed set its flag to null again.Thus unchanged records have null in their flag field and you can query not null values to see which record has changed/created and when (and set their flags to null again) .
I am new to Teradata. Can anyone tell me How exactly the AMPs going to helpful in creation of any table in Teradata.
Lets have a scenario.
I have a Teradata database with 4 AMPs. I learned that AMPs will usefull when we inserting the data into a table. Depending on the indexes it will distribute the data with the help of respected AMPs. But while creating the table, the command needs to execute through AMPs only. So i want to know which AMP will be used at that time??
The actual creation of the table in the data dictionary is a RowHash level operation involving a single AMP to store the record in DBC.TVM. Based on the other actions listed in the EXPLAIN there may be other AMPs involved as well but there is not single All-AMP operation. (This doesn't take into consideration the loading of the data and its distribution across the AMPs.)
Sample EXPLAIN:
1) First, we lock FUBAR.ABC for exclusive use.
2) Next, we lock a distinct DBC."pseudo table" for write on a RowHash
for deadlock prevention, we lock a distinct DBC."pseudo table" for
write on a RowHash for deadlock prevention, we lock a distinct
DBC."pseudo table" for read on a RowHash for deadlock prevention,
and we lock a distinct DBC."pseudo table" for write on a RowHash
for deadlock prevention.
3) We lock DBC.DBase for read on a RowHash, we lock DBC.Indexes for
write on a RowHash, we lock DBC.TVFields for write on a RowHash,
we lock DBC.TVM for write on a RowHash, and we lock
DBC.AccessRights for write on a RowHash.
4) We execute the following steps in parallel.
1) We do a single-AMP ABORT test from DBC.DBase by way of the
unique primary index.
2) We do a single-AMP ABORT test from DBC.TVM by way of the
unique primary index.
3) We do an INSERT into DBC.Indexes (no lock required).
4) We do an INSERT into DBC.TVFields (no lock required).
5) We do an INSERT into DBC.TVM (no lock required).
6) We INSERT default rights to DBC.AccessRights for FUBAR.ABC.
5) We create the table header.
6) Finally, we send out an END TRANSACTION step to all AMPs involved
in processing the request.
-> No rows are returned to the user as the result of statement 1.