We have an mobile web (ASP.NET + MS-SQL 2008) website that stores users' "personal" data. We are now developing an iPhone & Android apps that would sync mobile data periodically to the same backend. What is the best way of going about it for hosting in the cloud? My requirements in no particular order:
Speed/Performance
Security (we bcrypt the data etc.), but the database needs to be
secure, since a hint of database being "hacked" can affect perception
Availability: need HA
Finally $ - want to maintain costs low.
Some I have looked at are: Google data-store (problem: no RESTful API), Amazon SimpleDB (the cost seems quite prohibitive, has a WTF small 10GB "limit").
Any recommendations on how best to architect this solution?
We decided to go with Amazon RDS https://aws.amazon.com/rds/. It is still in beta and seems price reasonable. Hope this helps someone.
Related
I am starting to port one old desktop single tenant application into the cloud and wish to hear what would be your recommendation about the databases for my cloud-based multi-tenant application?
My basic requirement is simple:
For each tenant, its data is separate to any other tenants' data. I can easily backup, restore, export the data for one single tenant without affecting other tenants.
I don't really want to care about multi-tenancy in the business logic code. It should look like a single tenant application behind the security layer, no tenant ID pass around etc.
Easy to query using some mature technology like LINQ.
Availability and scalability, of course, easy to set up replicas, fail-over and scaling up and down etc.
I have gone through some investigations about multi-tenant application development. I have noticed SQL databases from Azure and AWS are both very expensive(the cost for just SQL database instance is close to the license fee of the original application), so I definitely can't use separate SQL database instances for tenants.
Now I'm reading this book Developing Multi-tenant Applications for the Cloud, 3rd Edition, and it uses Azure Storage Service to implement multi-tenancy. I haven't finished the book yet, it seems you still have to handle the multi-tenancy by yourself and the sample code is already out of date.
I have seen lots of SO questions compare Azure Table Storage with MongoDB. The MongoDB is very new to me, not sure whether it could be easily used to fulfill my requirements?
And I have seen RavenDB as well, it does support multi-tenancy out of box. But I didn't see some good sample code about how to use it in Azure app development.
Hope to hear some good advices from awesome SO guys.
I would better opt with RavenDB on top of MongoDB. Even Raven is a new comer in to the game, it supports most of the features which traditional SQL supports.
Also to make up a decisions the volume of data you are dealing is a also a key decision pointer. Also the amount of traffic you are expecting.
Also keep in mind that operational costs and development efforts. HA and DR scenarios can be problematic when you use Raven or Mongo because of the fact that you need to host them. But when it comes to Azure Storage, it by defaults protects you to a maximum extent by maintaining 3 copies of information.
So I would suggest you to carefully make the trade offs and opt wisely based on your business needs, cost optimization, development and operational effort.
Having a single instance of your application for each tenant is a very expensive way to implement an application, however I realise that if an application was developed with a single tenant in mind, then the costs of changing over can be high.
First can we start out with why you have a desktop application connecting to a database at another location. The latency can really slow down an application. Ideally you would want a locally installed database and have it sync with the cloud DB, or add in appropriate caching into your application.
However the DB would still need to differentiate the clients.
Why do you need this to go to a cloud database? Is it for backup purposes, not installing a DB locally on a clients machine, accessing the same data from many machines or something else?
Unless your application is extremely large, I would recommend rewriting it for multi-tenant to one SQL Azure database. The architecture chosen at the beginning of the project doesn't suit your requirements now. As you expand you will run into further issues.
I've been reading about Firebase and playing with it for a short while. The idea (BAAS) and implementation are impressive, and having programmed with Javascript it seems a viable choice. Not having to deal with scaling and other server side concerns makes it even more attractive.
My question is: generally speaking, is Firebase a first class back-end candidate for any average data-based application? e.g. billing, CRM, e-commerce, social, location based, etc. I do not include super light or heavy extremes such as a basic chat, or a nuclear plant monitor...
The answer may not be a clear yes/no, but was it built to support the general application space, or just stand out as a real-time read/write data service?
Would appreciate answers based on experience and existing production applications.
Thanks
Yes, Firebase is intended to be a first class back-end for any data based Web, iOS or Android application. The service offers real-time data reads and writes, but also comes with a powerful and flexible security system that allows you to write secure client-only apps, without needing any server code to enforce data boundaries.
There are several apps in production listed on the front page as customer and on the app showcase page on https://firebase.google.com/customers/
Firebase is now more capable and is considered as a full stand-alone back-end, especially after the introduction of cloud function. https://firebase.google.com/docs/functions/
Firebase may not have support for transaction spanning multiple business objects.
e.g. When a sales order is booked then it needs to update inventory for multiple items, update billing in receivables, give sales credit to multiple sales persons etc.
Firebase team is supposed to come up with a database trigger option which will make all these happen.
I have a general question about database hosting in relation to WCF and ASP.NET. We are currently developing a new online web application in ASP.NET, which gets/posts data to our MSSQL database with a WCF service (three tier infrastructure).
Now later in development we will be launching our website and hosting it on an external provider. We are unsure whether to keep the database for the website internally on our own servers, or host it externally along with our provider (they offer database hosting options as well).
If we hosted it externally, we would obviously back it up internally using batch scripts etc.
One major concern is the security of the database, as we are only a small business with not much experience in web security architecture. Due to this, we are leaning towards an external provider for both the website and database, who would obviously have experience and the equipment to manage such things.
Could you please offer some opinions on the matter?
Thanks!
There's always a risk associated with handing sensitive data off to an outside party, and trusting them to be as secure as you need.
There's no mystery here, someone at the provider will have enough access to look at your data if they really wanted to. So it all boils down to how sensitive is your data? Is there bank account info or social security numbers? For these reasons, our company cannot hand off such data to an outside party.
I'm a little confused though about one thing: if you could potentially host the database server when you go to production, why couldn't you host the website as well? Is it a matter of being able to handle high traffic?
Update in response to your comment:
It sounds like your data is somewhat sensitive, not highly sensitive. In which case if we're not being totally bonkers pedantic here, then you can reasonably assume a reputable hosting company will take the proper measures to secure your data, and from the sounds of it, they're probably more capable in this respect then your own company (not because you're careless or wet behind the ears, just because they would have considerable experience in this area where your company does not).
Now for the performance and hardware setup part if your comment... if you dont have the hardware or network infrastructure to meet your requirements, then you either a) upgrade your own infrastructure and hire the appropriate personnel to set it up and maintain it or b) you pay someone else to do it. Sounds like a no-brainer for you guys to go with option b.
Our company is thinking about moving to the cloud. Would we still be able to meet all our current requirements (below). We want to be able to easily scale in the future without high costs.
5 ASP.net 4.0 websites running (using sql databases, see below)
SQL Server 2008 Express (8 databases on this)
2 Scheduler services running (send nightly reports via email e.g. new orders in db)
MongoDB and Memcached are also installed on server
Currently the websites are on a separate server from the database server for security reasons.
We were thinking about Windows Azure and Amazon Web Services (AWS) as providers, which would best fit our requirements?
Are there any other factors we need to consider?
Re: SQL Databases: on Windows Azure this would map to SQL Azure. Costs start at $5/month for up to a 100 MB instance - and goes all the way up to 150 GB - and goes beyond that with Federations.
Re: 5 ASP.net 4.0 websites running: these map naturally into Windows Azure Web Roles. The "small" instance is $0.12/hour/instance, and you'll usually want two instances (to avoid single point of failure for a few scenarios). Depending on your load, you may be able to put all 5 sites on the same instances. If you have very low usage sites, consider the $0.05/hour/instance "extra small" instance.
Re: Currently the websites are on a seperate server from the database server for security reasons: of course this is also doable.
Re: 2 Scheduler services running: Running Windows Services is no problem.
Re: send nightly reports via email e.g. new orders in db: No problem doing, though is not baked into Windows Azure directly, but there are many simple ways to do this (even for free, such as via SendGrid).
Re: We want to be able to easily scale in the future without high costs: you will need to do the math regarding your actual costs, but Windows Azure can surely scale.
Re: MongoDB and Memcache are also installed on server: These can both be run on Azure. Check out https://github.com/mongodb/mongo for MongoDB. Also, the Azure Caching service is also avail (managed for you).
Re: We were thinking about Azure and Amazon as providers, which would best fit our requirements: These are functionally very similar (in capability and cost), with a few noteworthy differences.
Windows Azure is Platform as a Service meaning that you don't need to worry about Virtual Machines, but rather Applications. In other words, you upload your (basically) Zipped app package to the cloud for execution. With Amazon, you will be dealing with the Virtual Machine yourself. In Azure, you get a copy of Windows Server 2008 which is managed for you, but you can also do admin things to it if you need to. This is far less of an advantage if your app is an old messy install that isn't really clean (though may not be a good high-value cloud candidate anyway).
Windows Azure has an emulator that works great - F5 right from visual studio to work with storage system and VMs and more popular features.
Re: Are there any other factors we need to consider: Yes. With any cloud application, you need to be prepared to deal with scaling out (not up), dealing with transient retries (you may need to retry an operation to a cloud service - any cloud service). The benefits of this are much better (and more cost-effective) scalability and higher reliability (when you run across nodes, you don't have a single point of failure). Be sure to understand when/where storage on a VM is persistent vs. ephemeral. There are more considerations, but these are primary ones.
You may want to check out the Windows Azure Pricing calculator.
Good luck! And welcome to the cloud.
with the exception of the scaling question, and the 2 physical servers, you can move this functionality into a hosted environment and you will technically be in "the cloud". This could be a dedicated or VPS (Virtual Private Server), or even a shared server if you are small.
Those can allow for growth over time...you just need to upgrade what you have with the provider.
You also could use a colo-server with a hosting provider, which basically means you put your hardware in an hosting provider rack, and use their electricity and bandwidth. They charge based on bandwidth usage.
Since you are using SQL Express, remember that each database is limited to 8gb. So that will limit your growth at some point. That would entail an upgrade from Express to regular SQL if you don't want to re-engineer anything.
Have you considered AppHarbour? It has Memcached, MongoDB, SQL Server and so on, and is quicker to deploy to than Azure. I like Azure, but there is quite a learning curve and I have found the connection to SQL Azure to be pretty bad - which means re-engineering your DAL to use something like the SQL Transient Failure Library = a bit of a faff for existing projects.
AppHarbour does not have blob storage - so if you are uploading files you will need to use Azure Blob Storage or Amazon S3 or some equivalent as well.
Hope this helps.
Not an expert but being that Asp.net is a Microsoft product it should be easier to migrate to azure, although from what I have heard AWS shouldn't be difficult. Another thing you may want to consider is cost. Last time I checked AWS is significantly less costly unless you already pay for MSDN subscriptions.
All the requirements you sum up are not any issue to deploy in Windows Azure. You can find a lot of information on the internet on how to do this.
Keep in mind, if you want to deploy your services to windows azure, you'll need to do some code review of your applications to fix session state, output cache and so forth on your web applications.
Since you want to scale them out and they are sitting behind a non-sticky round-robin load balancer, you will run into issues with your session state if it is saved on the machine itself. You'll need to part session state to SQL Azure or to the Windows Azure table storage for example.
Installing MongoB and Memcache in Azure is not an issue, you'll find a lot of information on how to do it, but it'll require some to set up your role and the scripting
codingoutloud has given a very detailed answer. I would add two very key considerations to think about when moving any application to Azure (or, indeed, many other cloud providers).
Local state
With normal Azure, they reserve the right to shut down any one instance of a role at any time in order to move or upgrade it. This means you always need at least two instances of any one role and they will be transparently load balanced. If your current websites are currently running on individual servers then they may rely on session state or files in local directories etc. Now, there are ways around this (like putting session state in SQL, using the cookie provider for temp data, using a shared drive for files etc) or, indeed, bypassing a lot of the benefits of Azure and using their "virtual server" concepts which means you don't get the scale benefits etc.
But, sites that rely heavily on local state may be challenging to move to the cloud.
Time Zones
All Azure servers run on UTC time. If you are used to running on dedicated servers serving users from a single time zone then chances are that you use things like DateTime.Now() which won't really correspond to what the user wants.
I don't see any of the above as limitations of Azure, I find them very useful in forcing you to build global and scalable solutions from the start. However, when porting an existing application, the above may be quite a challenge to adapt to, even though there are workarounds.
As also mentioned elsewhere, there is a learning curve to Azure and somehow the documentation - plentiful as it is - just doesn't quite seem to help for some reason. Once you "get it", though, I find Azure really nice and there are a bunch of subtle features that will help you build scalable solutions, like the whole queuing infrastructure, the blob storage and the table storage. In some ways the learning is hampered by having too much choice.
Good luck!
First of all let me clear that I am not from a web background so if any of my understanding about how it works is not correct please feel free to correct me
Let's say I have a website which I would like to host on cloud because
- I don't want to take care of hardware
- I want to scale my website as needed
Now I am a bit confused between role of SQL Server vs role of SQL Azure in this case.
Normal Web Hosting
When I think of a normal website I know that I need a host/server on which my website will be hosted. The host should be able to support SQL Server. For scaling purpose I will have to host my website/ASP Pages on multiple servers. Similarly if I want to scale up my SQL Server I will have to host it on multiple servers and will have to make sure data is up to date in all servers through some mechanism.
Cloud Based Hosting
Now I think I can setup similar structure on Cloud/Azure as well. If yes, would I be using true capabilities of Cloud in this case?
Or should I use SQL Azure instead of SQL Server? What benefit would I get in that case? Would I be still be responsible for for scaling up and consistency of data? I know I can scale up the website by setting the number of VMs/instance but what about scaling of database?
Edit
Thanks to Florin Dumitrescu the terminology I wanted to use was Scaling Out because I am more concerned about the performance rather than how big my database is in terms of size. I am more concerned about how database would scale between different servers/systems to accommodate the load and hence result in better performance
SQL Azure, as Yossi mentioned, is a Database-as-a-Service. As such, you simply ask for it to be provisioned, magic happens, and you have a database that scales from 1GB to 5GB, 10GB, all the way to 50GB (soon to be 150GB as announced at SQL PASS). The nice thing about SQL Azure: you don't have to worry about any infrastructure, servers, licensing, etc. You simply connect with your connection string. SQL Azure is designed to be scalable to handle a considerable number of concurrent tenants, so you don't have to concern yourself with scaling.
SQL Azure also replicates its data in the data center, to provide "durable" storage. You still need to design a Disaster Recovery scheme, in case the data center becomes unavailable (and you can use the Data Sync service for that).
As far as your website itself: As you scale out to multiple instances, each instance runs the same code and uses the same resources. Taking this one step further, you can move your static (non-changing) web content, such as images and CSS, to Blob storage. This has several advantages over storing them with the website itself:
Ability to enable the Content Delivery Network, a worldwide edge-caching service providing better performance for your end users
Less strain on your web server instances, as requests for those images will now be directed to Blob storage, a completely separate URL than your website
Ability to update an image or stylesheet without having to re-deploy your application - simply upload a new file to Blob storage.
I highly recommend the Windows Azure Platform Training Kit, as there are labs that take you through the fundamentals of all of this, with complete code samples as well. This is updated almost monthly, staying in sync with the latest Windows Azure SDK and tools.
If you're hosting your web site in the cloud, and you need a database, than SQL Azure is almost certainly the best option.
SQL Azure is a database as a service, so you'll create your database and work against it from your code, but not have to worry about the provisioninig, there are no servers as such, it is all being taken care of.
From an application point of view it looks and behaves pretty much like SQL Server, so initially all that changes is the connecting string
As other noted SQL Azure takes away your concerns about setting up and taking care of the infrastructure. This is part of the premise of Azure in general which is to provide a platform rather than just Infrastructure.
The price you pay for that are some limitations on the capabilities (vs. regular SQL). Limitation on the size (at least until Federation will be available) and increased latency (since your database is not running on the same server of your app)
Microsoft Teched has as "SQL Azure Performance and Elasticity Guide" which you should probably take a look at