I need to configure cross-region replication for my DynamoDb table. To try in manually I just used AWS Web Console where I configured Global Table with appropriate regions for replication through DynamoDb streams, it was pretty simple. Could somebody please advice me how can I do it using Java SDK?
Bear in mind that DyanmoDB now supports global tables as announced on 29 November 2017. So you can probably replace your multitude of regional tables with global ones.
Global Tables eliminates the difficult work of replicating data
between regions and resolving update conflicts, enabling you to focus
on your application’s business logic. In addition, Global Tables
enables your applications to stay highly available even in the
unlikely event of isolation or degradation of an entire region.
However to answer you queston directly, its not part of the AWS DynamoDB Java SDK. You have to use the dynamodb-cross-region-library from AWS labs.
Cross-Region Replication
Important
AWS previously provided a cross-region replication solution based on
AWS CloudFormation. This solution has now been deprecated in favor of
an open source command line tool. For more information, please refer
to the detailed instructions on
GitHub:
https://github.com/awslabs/dynamodb-cross-region-library/blob/master/README.md
The DynamoDB cross-region replication solution uses the Amazon
DynamoDB Cross-Region Replication Library. This library uses DynamoDB
Streams to keep DynamoDB tables in sync across multiple regions in
near real time. When you write to a DynamoDB table in one region,
those changes are automatically propagated by the Cross-Region
Replication Library to your tables in other regions.
You can leverage the Cross-Region Replication Library in your own
applications, to build your own replication solutions with DynamoDB
Streams. For more information, and to download the source code, go to
the following GitHub repository:
https://github.com/awslabs/dynamodb-cross-region-library
Related
I am new to DynamoDB and trying to learn, while going through the docs I got a doubt whether DynamoDB is version agnostic or not also I want to know wether it is backwards compatible or not?
In addition I also want to know how to download local DynamoDB of different versions if possible and how can I change the version of DynamoDB in the webservice of aws ?
dynamoDB is a cloud hosted datastore. It it not a piece of software that you can download or install. As far as I know, there is no version, and you can't/don't have to select any version anywhere.
At most you can use some docker containers that mimics the behavior of dynamoDB to some extend. They are useful for testing and dev purposes, but keep in mind they are just mocks.
Is there a framework/tool which I could use in order to separate the database table creation from the business logic for DynamoDB? In the RDBMS world a tool like Liquibase would do the trick.
Any thoughts ?
Many thanks
You can use AWS CloudFormation to create the DynamoDB table. Here is the link which has the DynamoDB create template.
AWS CloudFormation simplifies provisioning and management on AWS.
AWS CloudFormation gives developers and systems administrators an easy
way to create and manage a collection of related AWS resources,
provisioning and updating them in an orderly and predictable fashion.
Dynamobee is a fork of mongobee, like a Liquibase, but use annotations in Java Classes. Available at https://github.com/dynamobee/dynamobee
As we're developing a greedy real-time monitoring application on Meteor, we reached the limit of our single MongoDB instance.
We migrated the DB to a sharded cluster with 2 shards for now, but we might expand up to 6 shards. (I have 2 BladeE chassis with 28 servers)
How do we configure Meteor Oplog Tailing on a mongo db cluster with sharding enabled?
Thanks,
Now there is a good news :) You can use sharded MongoDB database with Meteor easily with a bit of tweaking
Although Meteor core development team hasn't yet add Oplog Tailing support for sharding to their RoadMap, the workaround is so simple. You just add these 2 Meteor packages: cultofcoders:redis-oplog and disable-oplog, add another Redis server, tweak your code a bit and you are good to go.
The reason why Oplog Tailing doesn't work with sharded MongoDB database is just because core development team hasn't yet planed to support it. In fact, it's now possible to add support for sharded database. If you add a bunch of new records and read the Oplogs with tailable cursors from all shards, you will notice that MognoDB balancer will move data, lets say, from shard01 to shard02 where record id 0001 got removed from shard01 and added to shard02. This situation seems to be confusing for Meteor as it doesn't know whether records are actually removed/added by users or by MongoDB balancer. However there is a way to know whether users or MongoDB balancer removed/added the data because we can distinguish by fromMigrate flag — read more about this at MongoDB official site blog — so what we can do for now is to wait for update from core development team or work around with few tricks.
And the most promising workaround I've found so far is Meteor package called cultofcoders:redis-oplog. It's opensource and available on Github (Please check up their repository for full documentation. It's very-easy-to-use). The idea behind this package is to use another Redis server as pub and sub system — it doesn't store any data. it's just for pub and sub — instead of Meteor's which heavily rely on Oplog. This way, we don't have to worry about Oplog for sharded database that Meteor hasn't yet supported. Redis is mature and has been being used in production by many big companies. It's fully compatible with Meteor and you don't have to change the way you use Meteor. However you have to tweak your code a bit when updating collection's data in order to publish changes to Redis, and then the package cultofcoders:redis-oplog will handle the rest.
Seems Meteor does not yet support sharded mongoDB setups with > 1 shard: https://forums.meteor.com/t/mongodb-sharding-and-livequery/3712
Are there scripts for exporting and importing all Apigee Edge objects, such as developers, users, apps, caches, key value maps, etc?
To clarify, it would be nice to have non-runtime objects as a priority vs. the runtime data contained within. E.g., the current content of caches are not as critical as just having the cache object available.
I have released a tool that can be used to retrieve Apigee organization settings. This tool has been in use internally at Apigee for some time, but this is the first time it has been released to the public. It uses the Apigee management API to pull configuration data, and that data to be pulled is configurable. The data is stored in a hierarchical directory structure, which can be archived, explored, or used to compare organizations. It can be used with both the Apigee Edge cloud and on-prem offerings.
A few caveats:
This tool does not retrieve all data from an organization. For example, it does not retrieve API proxies. Use the Apigee management UI or management API to retrieve API proxies.
The tool is composed of a few bash scripts. It has been successfully run on Linux and Mac OS X.
The tool does not write data back into the organization, although the files it retrieves can often be POSTed back to the organization using the management API.
This tool is released as-is. It is not officially supported by Apigee.
Find the tool at the api-platform samples site (https://github.com/apigee/api-platform-samples) in the tools/org-snapshot directory.
There is work planned to provide a tool that will export/import provisional data (such as apps, developer, products). Other aspects of an org's configuration require access to the production Cassandra database, which cannot be given out publicly. We have a provisional tool for in-house use that we are currently hardening. If the consumer tool (when it is available) doesn't provide all of the backup support you need, you will need to log a support ticket for them to run the in-house tool.
There are scripts for importing a set of objects (developers, apps, API products) that work with the sample proxies that you can find on GitHub:
https://github.com/apigee/api-platform-samples/tree/master/setup
For Perl programmers: see also Apigee::Edge on CPAN
Say I wanted to build a feed reader that downloads RSS and Atom feeds to your local computer and lets you view them locally. What are the respective advantages and disadvantages of using CouchDB or sqlite3 as the datastore for such an application?
SQLite and CouchDB are probably different in every respect but what you consider an advantage or disadvantage is a matter of preference and requirements.
SQLite is an SQL database where you store relations (tables).
CouchDB is a NoSQL database where you store JSON documents (objects of any structure).
SQLite has schemas.
CouchDB is schema-less.
SQLite is a library that you link with your application and use a C API.
CouchDB is a RESTful web service and its API is HTTP and JSON.
SQLite has no concept of a network.
CouchDB basically is a high performance web server.
SQLite is written in C.
CouchDB is written in Erlang.
Which of those are advantages and disadvantages? This is up to you. ;) Good luck.
Good summary from rsp. Without knowing more about your requirements, it's hard to say which one is better suited for your use case. One clear advantage that SQLite provides is that is has simpler installation and administration. As a library, it's linked into your application and installed along with your application. No separate database to install and configure. This is one of the features that makes database libraries, like SQLite and Berkeley DB very attractive, and possibly preferable to database servers.
Just to add to your list of considerations, Berkeley DB and Berkeley DB Java Edition are also database libraries that you may want to consider. Berkeley DB (written in C) offers you the choice of using a schema-free key-value pair API, a POJO-like Java API or a SQLite-compatible SQL API. Berkeley DB Java Edition (100% Java) offers you Java APIs for key-value pairs, Java Collections or POJO-like object persistence. Berkeley DB and SQLite tend to be the products of choice for people who are looking for embedding data management functionality inside of their application.
Disclaimer: I am the Product Manager for Berkeley DB, so I'm a little biased. :-)