Hi Oracle DB guys out there, Is there a way to propagate DML changes in database to some MQ server(Kafka in my case). I have read about following but coulnt find any concrete implementation
oracle streams:- which are for database to database replication
oracle strams AQ:- for integration with MQ clients
GoldenGate: which we cannot use because of cost
Related
I am very new to NoSQL world and wondering how connections are managed by NoSQL databases like Azure Cosmos DB.
I am designing a highly scalable solution for real-time application. And one of the concern is how to manage numerous connections/requests to Azure Cosmos DB from Azure Functions or my business tier?
Is Cosmos DB subjecting to similar limitations as SQL Server is in terms of number of available connections?
Azure functions connections limitation allied to all outbound connections irrespective of target service. Some services might optimize the connection usage (pooling, multi-plexing etc...) for higher concurrency and throughput.
Specifically for CosmosDB: 2.0.0-preview package has connection multiplexing and pooling, please check https://www.nuget.org/packages/Microsoft.Azure.DocumentDB/2.0.0-preview
NOTE: Azure functions V2 run-time is required for custom CosmosDB SDK version.
I have transaction scope in my code which move transaction to MSDTC. But when I run this code into AWS cloud where RDS is SQL server. It is not supporting MSDTC please how can I make this supportable or what will be alternative way for this. I need MSDTC in my code.
Well, there isn't much that you can do since AWS does not support it (https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_SQLServer.html).
Features Not Supported and Features with Limited Support
The following Microsoft SQL Server features are not supported on Amazon RDS:
Stretch database
Backing up to Microsoft Azure Blob Storage
Buffer pool extension
Data Quality Services
Database Log Shipping
Database Mail
Distribution Transaction Coordinator (MSDTC)
File tables
FILESTREAM support
Maintenance Plans
Performance Data Collector
...
...
The alternative is to deploy/host/manage your own MSSQL server on AWS.
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.SQLServer.Options.MSDTC.html
Looks like this is now supported by AWS.
I am looking for a solution to stream data from Oracle 11g to Kafka. I was hoping to use GoldenGate, but that only seems to be available for Oracle 12c. Is the Confluent platform the best way to go?
Thanks!
First, the general answer would be: The best way to connect Oracle (databases) to Kafka is indeed to use Confluent Platform with Kafka's Connect API in combination with a ready-to-use connector for GoldenGate. See the GoldenGate/Oracle entry in section "Certified Connectors" at https://www.confluent.io/product/connectors/. The listed Kafka connector for GoldenGate is maintained by Oracle.
Is the Confluent platform the best way to go?
Hence, in general, the answer to the above question is: "Yes, it is."
However, as you pointed out for your specific question about Oracle versions, Oracle unfortunately has the following information in the README of their GoldenGate connector:
Supported Versions
The Oracle GoldenGate Kafka Connect Handler/Formatter is coded and
tested with the following product versions.
Oracle GoldenGate for Big Data 12.2.0.1.1
Confluent IO Kafka/Kafka Connect 0.9.0.1-cp1
Porting may be required for Oracle GoldenGate Kafka Connect
Handler/Formatter to work with other versions of Oracle GoldenGate for
Big Data and/or Confluent IO Kafka/Kafka Connect
This means that the connector does not work with Oracle 11g, at least as far as I can tell.
Sorry if that doesn't answer your specific question. At least I wanted to give you some feedback on the general approach. If I do come across a more specific answer, I'll update this text.
Update Mar 15, 2017: The best option you have at the moment is to use the Confluent's JDBC connector. That connector can't give you quite the same feature set as Oracle's native GoldenGate connector though.
Oracle GoldenGate and Confluent Platform are not comparable.
Confluent Platform provides the complete streaming platform and is a collection of multiple software which can be used for streaming your data, where as GoldenGate is replication and data-integration software.
Also GoldenGate is highly reliable for db replication since it maintains transactional integrity, same cannot be said for Kafka Mirror Maker or Confluent's Replicator at this time.
If you want just pure transactions - please also consider using OpenLogReplicator. It supports Oracle database from version 11.2.0.1.
It can produce transactions to Kafka in 2 formats:
Classic format - when every transaction is one Kafka message (multiple DMLS per Kafka message)
Debezium style format - transactions are divided - every DML is one Kafka message
There is already a working version. You can try it.
Right now I am using ojdbc6 to connect to Oracle 11g. It is good enough but not perfect especially when using pooling mode to check if there are new updates on the original tables.
I tried also to read all tables using certain pattern but this did not work well.
The best mode to connect an Oracle DB to Kafka (especially when the tables are very wide, columns wise, is to use queries for the connectors. This way, you ensure that you pick the right fields and do some casting for numbers if you are using avro.
I'm developping a java application and i'm using JDBC to connect to MySQL Database, now i want to use ODBC to be able to get and retrieve data from any DBMS, of course if have access to it. Is there an API or tool to do this ?
What you are looking for is a JDBC-ODBC bridge. There are several available. It is not recommended, instead you should always use a native JDBC driver.
hoping someone has experience with both sql server and postgresql.
Between the two db's, which one is easier to scale?
Is creating a read only db that mirrors the main db easier/harder than sql server?
Seeing as sql server can get $$, I really want to look into postgresql.
Also, are the db access libraries written well for an asp.net application?
(please no comments on: do you need the scale, worry about scaling later, and don't optimize until you have scaling issues...I just want to learn from a theoretical standpoint thanks!)
Currently, setting up a read-only replica is probably easier with SQL Server. There's a lot of work going on to get hot standby and streaming replication part of the next release, though.
Regarding scaling, people are using PostgreSQL with massive databases. Skype uses PostgreSQL, and Yahoo has something based on PostgreSQL with several petabyte in it.
I've used Postgresql with C# and ASP.Net 2.0 and used the db provider from devart:
http://www.devart.com/dotconnect/postgresql/
The visual designer has a few teething difficulties but the connectivity was fine.
I have only used SQL Server and not much PostgreSQL, so I can only answer for SQL Server.
When scaling out SQL Server you have a couple of options. You can use peer to peer replication between databases, or you can have secondary read-only DB(s) as you mention. The last option is relatively straight-forward to set up using database mirroring or log shipping. Database mirroring also gives you the benefit of automatic switchover on primary DB failure. For an overview, look here:
http://www.microsoft.com/sql/howtobuy/passive-server-failover-support.mspx
http://technet.microsoft.com/en-us/library/cc917680.aspx
http://blogs.technet.com/josebda/archive/2009/04/02/sql-server-2008-database-mirroring.aspx
As for licensing, you only need a license for the standby server if it is actively used for serving queries - you do not need one for a pure standby server.
If you are serious, you can set up a failover cluster with a SAN for storage, but that is not really a load balancing setup in itself.
Here are some links on the general scale up/out topic:
http://www.microsoft.com/sqlserver/2008/en/us/wp-sql-2008-performance-scale.aspx
http://msdn.microsoft.com/en-us/library/aa479364.aspx
ASP.NET Libraries are obviously very well written for SQL Server, but I would believe there exists good alternatives for PostgreSQL as well.