I'm setting up an application which is a set of Mircroservices consuming a Cloud SQL DB in GCP. My queries are -
I want to set up HA for Cloud SQL in across regions(primary region and a secondary region with active replication enabled). I do not see any out of the box set up from Google Cloud to achieve the same. Out of the box HA for Cloud SQL 2nd Gen is to have a HA instance in the same region in another zone in the same region. Please provide the best practice to achieve the same.
All the microservices should be using private ip to do actions on this MySQL. How do set this up?
Is there any native support from MySQL to enable Active replication to another region?
Is it possible to set up manual backup as per customer requirements? I do understand automatic backup available.To meet RPO RTO requirements want to customize db backup frequency - is that possible?
I want to set up HA for Cloud SQL in across regions(primary region and a secondary region with active replication enabled)
You can use the external master feature to replicate to an instance in another zone.
All the microservices should be using private ip to do actions on this MySQL. How do set this up?
Instructions for Private IP setup are here. In short, your services will need to be on the same VPC as the Cloud SQL instances.
Is it possible to set up manual backup as per customer requirements?
You can configure backups using the SQL Admin API.
Please, let me list your questions along with their response:
I want to set up HA for Cloud SQL in across regions(primary region and a secondary region with active replication enabled). I do not see any out of the box set up from Google Cloud to achieve the same. Out of the box HA for Cloud SQL 2nd Gen is to have a HA instance in the same region in another zone in the same region. Please provide the best practice to achieve the same.
-According to the documentation [1], the configuration is made up of a primary instance (master) in the primary zone and a failover replica in the secondary zone, at the moment is not possible the HA for Cloud SQL across regions.
All the microservices should be using private ip to do actions on this MySQL. How do set this up?
-You can set up a cloud SQL instance to use private IP, please review the next information, you may find it helpful [2].
Is there any native support from MySQL to enable Active replication to another region?
-I would recommend to get in contact with mysql support [3], so that you get the help you need, in the meantime you could review the next link [4], and see if this fits your needs.
Is it possible to set up manual backup as per customer requirements? I do understand automatic backup available.To meet RPO RTO requirements want to customize db backup frequency - is that possible?
-You can create a backing up on demand, please review the next link [5] which helps to illustrate how to set this kind of backups.
Please let me know if this information helps to address your questions.
[1] https://cloud.google.com/sql/docs/mysql/high-availability
[2] https://cloud.google.com/sql/docs/mysql/private-ip
[3] https://www.mysql.com/support/
[4] https://dev.mysql.com/doc/mysql-cluster-excerpt/5.6/en/mysql-cluster-replication-conflict-resolution.html
[5] https://cloud.google.com/sql/docs/mysql/backup-recovery/backing-up#on-demand
Related
So I have set the location of my Firestore database to asia-south-1, let suppose now I start getting users from the US as well as from India, how will I serve both the groups with lower latency?
You can't change the properties of your project once it's set up, nor even simply pay more for better service. If you want improved service in different regions, you will need to buy computing resources in each of those regions. This requires setting up multiple projects, with each Firestore product configured for the region you want improved service. Note that not all Firebase and GCP products are available in all regions.
If you want all users in all regions to be using the same set of data with improved latency, that's not possible to configure. You will have to build a way to mirror data between the different projects. There is nothing terribly easy about that, nor are there any existing solutions offered by Firebase or GCP to do this for you.
Doug nailed it for Firestore. So 👍
Note that Firebase's Realtime Database (unlike Firestore) can have multiple databases per project, which means you can set up a database instances in each region (it support three right now) and redirect your user's traffic to the region closest to them. You may still have to replicate data between the database instances, similar to Doug's answer, but (unlike in Doug's answer) with Realtime Database this can happen within a single project.
I'm looking for options for failover in the case of a region outage that doesn't involve incurring the cost of having a hot second region and geo-redundancy. Is it possible to use the backup feature to manually create a new account in a new region and then upload the related documents?
I was looking at the restore feature and noticed these limitations:
"The restored account is created in the same region where your source account exists. You can't restore an account into a region where the source account did not exist."
I also saw this limitation which makes me wonder if backups work in case of region failure? Even with the geo-redundant backup storage option picked?:
"The backups are not automatically geo-disaster resistant. You have to explicitly add another region to have resiliency for the account and the backup."
Does this mean that the backups will also go down if geo-redundancy and multiple regions aren't set up?
Thanks.
Backups can only be restored into the same region in which the account was created. Backups are not intended or able to provide resiliency to regional outages. And certainly cannot provide anything approaching 99.999% availability. Regional replication is the only means to provide resiliency to regional events for mission critical workloads.
Backup is by default is geo-dr into the pair for that region. These are only restored into the source region where the regional event impacted the storage for that region which prevents it from recovering normally.
I am trying stream data from the AWS API Gateway to DynamoDB in AWS (directly, without something like lambda). I have looked for several tutorials, such as [1] and [2], who describe exactly this scenario. All of these assume that the right policies and roles are in place. Normally, I play and hack around till I get a working proof of concept, after which I rebuild a proper model, but with access rights I want to make sure I understand what I am doing. For [2], I also found a stack overflow question at [3] from somebody with the same problem that got solved, but not sure exactly how. I also looked at [4], describing API Gateway with Lambda.
Here is my guess:
Create a policy that allows calling from the API Gateway.
"AmazonAPIGatewayInvokeFullAccess" fits the name, but might not
be necessary + overkill with too much access
Create a policy that allows access to dynamoDB.
Here, "AmazonDynamoDBFullAccess" might be appropriate, even
though it might be overkill (too much access), and might only work
from the Management Console
Create a role that has those two policies attached.
Here, I run into the trouble that when I click create role, and
select AWS service, I can not find the correct "service that will use
this role" that has the policies I described above behind it. For
example, when clicking dynamoDB, I get the following "use-cases", none of which seem to relate to the dynamoDB full access policy:
Amazon DynamoDB Accelerator (DAX) - DynamoDB access
DynamoDB - Global Tables
DynamoDB Accelerator (DAX) - Cluster management
My main question is: How do I set the right minimal set of roles and policies to connect AWS API Gateway to DynamoDB (read and write), as described in [1]?
[1] https://sanderknape.com/2017/10/creating-a-serverless-api-using-aws-api-gateway-and-dynamodb/
[2] https://aws.amazon.com/blogs/compute/using-amazon-api-gateway-as-a-proxy-for-dynamodb/
[3] API Gateway does not have permission to assume the provided role DynamoDB
[4] https://docs.aws.amazon.com/apigateway/latest/developerguide/permissions.html
What you need to do is create an IAM Service Role that allows API Gateway to assume this role. You can easily do this through the UI. When you create a new role, the "Service Role" is selected by default and below the "Choose the service that will use this role" header, you can select API Gateway.
A role is a container of permissions that can be assumed by a certain entity (in our case, an API Gateway API resource). Your role needs "permissions" for the role to have any use. You add this permissions by adding policies to your role. This is explained more in depth here: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_terms-and-concepts.html
Be sure to read the AWS Service Role part. You say that you need to "Create a policy that allows calling from the API Gateway" but this is incorrect: you need to create a role that can be assumed by API Gateway.
In your case, you'll want specific DynamoDB permissions for your role. Following the least-privilege principle as you mention, you should only add the specific actions for the specific DynamoDB table. The list of possible permissions can be found here: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/api-permissions-reference.html
Let's say you only want API Gateway to get items from a specific table. Your policy might look something like this then:
{
"Effect": "Allow",
"Action": "dynamodb:GetItem",
"Resource": "arn:aws:dynamodb:eu-west-1:[aws_account_id]:table/[table_name]"
}
Hope this helps!
This recent new tutorial by ankana likhita sri priya includes starting off in high detail/screenshots of IAM (Policy, Role, etc.): https://medium.com/#likhita507/using-api-gateway-to-get-data-from-dynamo-db-using-without-using-aws-lambda-e51434a4f5a0
i'm configuring Standard Geo Replication for our Azure SQL database and need to validate Disaster recovery, but still have couple doubts and cannot find more details:)
I configured Standard Geo Replication and added Secondary database.
If i want to do DR Drill, i can just stop replication, make the second database as primary and re-point my ASP.NET applications to the new database server. Just change SQL server name.
My questions are:
What if Microsoft has some issues with data center, do they automatically just re-points DNS from main SQL to secondary so i don't need to do anything, and it would be done silently and everything would work, without my change?
I was trying to find out, if there is some kind of notification, that i can configure if SQL has an issue and i can stop replication and do this process by myself. Or will microsoft somehow notify us? email,... or only SQL AZURE dashboard?
What could be the case, that i need to do DR manually? Main SQL would stop working and AZURE cannot take care of that because...?
thanks a lot for your answers
Rado
We had this issues recently, we did not receive any email's or notification.
Luckily only SQL-Azure was affected. Hence we stopped the replication and updated connection string to point to secondary.
No they do not automatically re-point DNS from primary to secondary for databases
We have external services that keeps track of our systems and notifies us when things are not right.
We had to do a manual DR.
Would love to know if any one has a workflow in place to automate it or point to the right direction.
Hope that helps.
I am designing a multi-tenant system and am considering sharding by tenant at the application layer level instead of database.
Hypothetically, the way this should work is that for incoming request a router process has a global collection of tenants containing primary attributes to determine the tenant for this request as well as the virtual shard id. This virtual shard id is further mapped to an actual shard.
The actual shard contains both the code for application as well as whole data for this tenant. These shards would be LNMP (Linux, Nginx, MySQL/MongoDB, PHP) servers.
The router process should act as proxy. It should be able to run some code to determine the target shard for incoming request based on the collection stored in some local db or files. To be able to scale this better, i am considering making the shards themselves act as routers also so that they can run a reverse proxy that will forward the request to appropriate shard. Maybe the nginx instance running on shard can also act as that reverse proxy. But how will it execute the application logic needed to match up the request with the appropriate shard.
I will appreciate any ideas and suggestions for this router implementation.
Thanks
Another option would be to use a product such as dbShards. dbShards is the only sharding product that shards at the application level. This way you can use any RDMS (Postgres, MySQL, etc.) and still be able to shard your database without having to put some kind of proxy in-between. A lot of the other sharding products rely on a proxy to point the transactions to the correct shard, but dbShards knows where to go without having to "ask" anyone else.
Great product. dbshards
Unless you expect your tenants to generate approximately equal data volume, sharding by tenant will not be very efficient.
As to application level sharding in general, let me share my own experience:
Version 1 of our high-volume SaaS product sharded at the application level. You will find that resharding as you grow will be a major headache if you shard against a SQL type solution at the application level, or you will have to write significant tooling to automate the process.
We switched to MongoDB (after considering multiple alternatives including Cassandra) in no small part because of all of the built-in support for resharding / rebalancing as data grows.
If your application does not need the relational capabilities of MySQL, I would suggest concentrating your efforts on MongoDB (since you have already identified that as a possible data platform) if you expect more than modest data growth. Allow MongoDB to handle the data sharding.