Does anyone know how the following services take their time references from. In other words: with what source do they sync their time reference?
Firebase
MongoDB Atlas
Found out that AWS services sync their time with a service called Amazon Time Sync.
Amazon Time Sync is used by EC2 instances and other AWS services. It
uses a fleet of redundant satellite-connected and atomic clocks in
each Region to deliver time derived from these highly accurate
reference clocks. This service is provided at no additional charge.
Likewise I need information about Firebase and MongoDB Atlas specifically. Any help/source is appreciated.
What I found by myself.
AWS Services - AWS services sync time with Amazon Time Sync. It uses a fleet of redundant satellite-connected and atomic clocks in each Region to deliver time derived from these highly accurate reference clocks.
Google Services - Google services including Firebase use Google Public NTP. This is a free, global time service that you can use to synchronize to Google's atomic clocks.
MongoDB Atlas - MongoDB Atlas has been enhanced by a move to a global logical clock. Implemented as a hybrid logical clock and protected by encryption.
Since all the services use highly accurate time services, we can assume that they represent the exact time so that time is synchronized across all the services in one application.
References
https://aws.amazon.com/about-aw
https://developers.google.com/time/faq
https://www.mongodb.com/blog/post/transactions-background-part-4-the-global-logical-clock
Related
I am building a web app with the following stack:
UI - React
Backend framework - NestJS
Infrastructure - Google Firestore document DB, services deployed in Heroku
I need to calculate finance portfolio metrics on a daily basis for all users and display them when the user logs in. I am in a bit of a dilemma what approach to take and I have several ideas, so I hope you can give me some guidance.
Scheduled microservice
I can build and schedule a microservice in Python (the finance framework is in Python) that will run every day and calculate the needed metrics for the users and update the database. Seems straightforward but it might consume a lot of compute resources, especially when the user base grows large.
Cloud Functions
Google Firestore supports cloud functions that can trigger on specific events. I can leverage that and run the calculation microservice when the data is requested - that way I will calculate the information only on-demand. The downside is that if the data has not been requested for a long time, I will have to calculate the metrics for a larger period of time and this might take a while.
P.S. Just saw that there are also scheduled cloud functions - possible implementation might check if the data is calculated today (user has logged in at least once) and if not, calculate it.
I will be happy to discuss any other options that might be available.
Why does Azure Cosmos DB take so long to deploy? When I tried deploying Azure Cosmos DB (SQL API) via ARM template, it took ~20 min for deployment to complete. Why is that?
Deployment for a new Cosmos account requires involves multiple underlying resources and there are a number of steps involved that must be done in series to provision and connect these resources before the account is ready to accept requests.
Things that can impact provision time include, number child resources (databases, containers), number of regions and amount of throughput. In addition there can be other factors as well including the number of control plane operations overall in a region.
All that said, we are working on optimizations designed to reduce the amount of time it takes to provision new Cosmos DB accounts.
Hope that helps.
I just provisioned a new Cosmos DB today and it took a little less than 10 minutes. East US region.
I use Firebase for my web apps, not Google Cloud Platform -- though I'm aware that behind the scenes, every Firebase project is also a GCP project.
I just received an email from GCP, saying (excerpted):
[Action required]: Enable the Cloud Build API for your projects before April 20, 2020, to ensure Cloud Function is built and deployed correctly ... You must enable the Cloud Build API for your project(s) to ensure your Cloud Function is built and deployed correctly. Once you enable the API, you may incur charges if your Cloud Build, Container Registry, or Cloud Storage usage exceeds the free tier limits for these products.
Does this impact the deployment or development process for Firebase at all? (e.g., right now I use firebase deploy )
As far as I can tell, the answer is no -- it just allows for a greater maximum build-time-per-day. But the email didn't mention Firebase at all, so I want to be sure I understand any other implications well before April 20.
I had the same questions and got in touch with the firebase support team, here are my questions and the answers I received:
"Is that mandatory?"
Only if you are going to deploy Cloud Functions in the future, already
deployed functions will not be affected.
"How do I know if my Cloud Build, Container Registry, or Cloud Storage usage will exceed the free tier limits for these products? is there any way how can check what I've been using so I'll be prepared?"
Cloud Build provides a free tier where only usage above 120
build-minutes/day will be charged. When your usage is within the free
tier, you will not be charged for the Cloud Build portion of Cloud
Function deployments. See Cloud Build pricing for more information.
Similarly, Cloud Storage and Container Registry share a free tier
where only usage above 5GB-months will be charged. ( * Note : free
tier only limited to US regions - US-WEST1, US-CENTRAL1, and US-EAST1
and aggregated over all 3 regions )? For example, if you have a large
deployment that uses 100GB of storage, you will only be charged an
additional $2.47 for storage/month (based on these particular U.S.
regional storage prices). You can monitor your usage and see whether
you are getting close to hitting the free quotas.
"Lastly, what does all this stuff do for my app? Why do I need to add this other features? If you were to sell me this, how would you present it for my use-case?"
This is a best effort from Google to communicate information that is
necessary to the user’s continued use of the product or that is
considered a necessary legal update and keep customers away from
having future issues. Using Cloud Build, Container Registry and Cloud
Storage provides the following benefits: Detailed function build logs
will be available in the GCP Console, aiding in debugging and
increasing visibility. The ability to get build time that exceeds the
current build quota of 120 build-mins/day. The ability to view a built
container image for your function in Container Registry.
I received the following response from Firebase Support:
You will not be able to use cloud functions if you do not enable the Cloud Build API after this date [April 20, 2020].
And, in a follow-up exchange:
as far as I know in terms of deployment everything will be the same
So, my interpretation of those responses with respect to the original question (will this affect the mechanics of development or deployment) is:
Deployment process: no change necessary
Development process: no change necessary*
*if your build time exceeds 120 minutes per day, and you wish to avoid associated charges, you could voluntarily investigate changing your deployment/development process to reduce build minutes.
Thanks also to Fran for his answer, which sheds light on other aspects of the change.
Sorry if that has been answered - I couldn't find a complete answer - as there seems to be conflicting resources.
What I'm trying to achieve is the architecture where my backend 'communicates' in real time with millions of clients through Firebase DB (RTD or FS). Kind of like what's drawn here:
The architecture in a nutshell: millions of clients write 'events' to the Firebase DB, my java server listens to these 'events', processes them and writes 'responses' to the Firebase DB that are synced back to the clients.
The question: Is App Engine the best solution? Is it even suitable for the job?
On the one hand, in App Engine's documentation there's an example of it used that way:
https://cloud.google.com/solutions/mobile/mobile-firebase-app-engine-flexible
On the other hand, there are (seemingly) known issues with that approach:
1) App Engine instances awake on http requests, not on firebase events. https://stackoverflow.com/a/38357458/1806956
Jobs have a timeout, so even if we do a cron wakeup every minute, it doesn't ensure (or does it?) that the listener will keep living forever.
2) App Engine does not support the Firebase Admin SDK due to background threads? https://stackoverflow.com/a/45046605/1806956
3) App Engine limits the number of background threads. In a real app, we're talking about potential thousands of concurrent users, all throwing events...
Are the above issues not updated? Thank you...
I am starting to port one old desktop single tenant application into the cloud and wish to hear what would be your recommendation about the databases for my cloud-based multi-tenant application?
My basic requirement is simple:
For each tenant, its data is separate to any other tenants' data. I can easily backup, restore, export the data for one single tenant without affecting other tenants.
I don't really want to care about multi-tenancy in the business logic code. It should look like a single tenant application behind the security layer, no tenant ID pass around etc.
Easy to query using some mature technology like LINQ.
Availability and scalability, of course, easy to set up replicas, fail-over and scaling up and down etc.
I have gone through some investigations about multi-tenant application development. I have noticed SQL databases from Azure and AWS are both very expensive(the cost for just SQL database instance is close to the license fee of the original application), so I definitely can't use separate SQL database instances for tenants.
Now I'm reading this book Developing Multi-tenant Applications for the Cloud, 3rd Edition, and it uses Azure Storage Service to implement multi-tenancy. I haven't finished the book yet, it seems you still have to handle the multi-tenancy by yourself and the sample code is already out of date.
I have seen lots of SO questions compare Azure Table Storage with MongoDB. The MongoDB is very new to me, not sure whether it could be easily used to fulfill my requirements?
And I have seen RavenDB as well, it does support multi-tenancy out of box. But I didn't see some good sample code about how to use it in Azure app development.
Hope to hear some good advices from awesome SO guys.
I would better opt with RavenDB on top of MongoDB. Even Raven is a new comer in to the game, it supports most of the features which traditional SQL supports.
Also to make up a decisions the volume of data you are dealing is a also a key decision pointer. Also the amount of traffic you are expecting.
Also keep in mind that operational costs and development efforts. HA and DR scenarios can be problematic when you use Raven or Mongo because of the fact that you need to host them. But when it comes to Azure Storage, it by defaults protects you to a maximum extent by maintaining 3 copies of information.
So I would suggest you to carefully make the trade offs and opt wisely based on your business needs, cost optimization, development and operational effort.
Having a single instance of your application for each tenant is a very expensive way to implement an application, however I realise that if an application was developed with a single tenant in mind, then the costs of changing over can be high.
First can we start out with why you have a desktop application connecting to a database at another location. The latency can really slow down an application. Ideally you would want a locally installed database and have it sync with the cloud DB, or add in appropriate caching into your application.
However the DB would still need to differentiate the clients.
Why do you need this to go to a cloud database? Is it for backup purposes, not installing a DB locally on a clients machine, accessing the same data from many machines or something else?
Unless your application is extremely large, I would recommend rewriting it for multi-tenant to one SQL Azure database. The architecture chosen at the beginning of the project doesn't suit your requirements now. As you expand you will run into further issues.