As we're developing a greedy real-time monitoring application on Meteor, we reached the limit of our single MongoDB instance.
We migrated the DB to a sharded cluster with 2 shards for now, but we might expand up to 6 shards. (I have 2 BladeE chassis with 28 servers)
How do we configure Meteor Oplog Tailing on a mongo db cluster with sharding enabled?
Thanks,
Now there is a good news :) You can use sharded MongoDB database with Meteor easily with a bit of tweaking
Although Meteor core development team hasn't yet add Oplog Tailing support for sharding to their RoadMap, the workaround is so simple. You just add these 2 Meteor packages: cultofcoders:redis-oplog and disable-oplog, add another Redis server, tweak your code a bit and you are good to go.
The reason why Oplog Tailing doesn't work with sharded MongoDB database is just because core development team hasn't yet planed to support it. In fact, it's now possible to add support for sharded database. If you add a bunch of new records and read the Oplogs with tailable cursors from all shards, you will notice that MognoDB balancer will move data, lets say, from shard01 to shard02 where record id 0001 got removed from shard01 and added to shard02. This situation seems to be confusing for Meteor as it doesn't know whether records are actually removed/added by users or by MongoDB balancer. However there is a way to know whether users or MongoDB balancer removed/added the data because we can distinguish by fromMigrate flag — read more about this at MongoDB official site blog — so what we can do for now is to wait for update from core development team or work around with few tricks.
And the most promising workaround I've found so far is Meteor package called cultofcoders:redis-oplog. It's opensource and available on Github (Please check up their repository for full documentation. It's very-easy-to-use). The idea behind this package is to use another Redis server as pub and sub system — it doesn't store any data. it's just for pub and sub — instead of Meteor's which heavily rely on Oplog. This way, we don't have to worry about Oplog for sharded database that Meteor hasn't yet supported. Redis is mature and has been being used in production by many big companies. It's fully compatible with Meteor and you don't have to change the way you use Meteor. However you have to tweak your code a bit when updating collection's data in order to publish changes to Redis, and then the package cultofcoders:redis-oplog will handle the rest.
Seems Meteor does not yet support sharded mongoDB setups with > 1 shard: https://forums.meteor.com/t/mongodb-sharding-and-livequery/3712
Related
I am wondering if it is feasible to deploy wordpress as a series of lambda functions on AWS API gateway. Any pointers on the feasibility/gotchas would be greatly appreciated!
Thanks in advance,
PKK
You'll have a lot of things to consider with persistence and even before that, Lambda doesn't support PHP. I'd probably look at Microsoft Azure Functions instead that do support PHP and do have persistent storage.
While other languages (such as Go, Rust, Swift etc.) can be "wrapped" to run in AWS Lambda with relative ease, compiling PHP targeting the same platform and running it is a bit different (and certainly more painstaking). Think about all the various PHP modules you'd need for starters. Moreover, I can't imagine performance will be as good as something like a Go binary.
If you can do something clever with the Phalcon framework and come up with an easy build and deploy process, then maayyyybee.
Though, you'd probably need to really overhaul something like WordPress which was not designed for this at all. It still uses some pretty old conventions due to the age of the project and while that is all well and good for your typical PHP server, it's a different ball game in the sense of this "portable" PHP installation.
Keep in mind that PHP sessions are relied upon as well and so you're going to need to move those elsewhere due to the lack of persistence with AWS Lambda. You can probably find some sort of plugin for WordPress that works with Redis?? I have to imagine something like that has been built by now... But there will be many complications.
I would seriously consider using Azure Functions to begin with OR using Docker and forgoing the pricing model that cloud functions offers. You can still find some pretty cheap and scalable hosting out there.
What I've done previously was use AWS ECS (Docker) with EFS (network storage) for persistence and RDS for the database. While this doesn't carry the same pricing model as Lambda, it is still cost efficient. You can set up your ECS Service to autoscale up and down. So that way you're running the bare minimum until you need more.
I've written a more in depth article about it here: https://serifandsemaphore.io/how-to-host-wordpress-like-a-boss-b5993fcfbd8e#.n6fbnf8ii ... but it's basically just the idea of running WordPress in Docker and using EFS to offload the persistent storage issues. You can swap many of the pieces of the puzzle out if you like. Use a database hosted in some other Docker service or Compose or where ever. That part need not be RDS for example. Even your storage could be handled in a different way, though EFS worked pretty well! The only major thing to note about EFS is the write speed. Most WordPress sites are read heavy though. Your mileage will vary depending on your needs.
Is it possible? Yes, anything is possible with enough time and effort. Is it worth it? That is a question best to ask yourself.
PHP can be run on Lambda as per the documentation located here: https://aws.amazon.com/blogs/compute/scripting-languages-for-aws-lambda-running-php-ruby-and-go/ .
The bigger initial problem as stated in other comments is a persistent file system. S3 for media storage is doable via Wordpress plugin (again from the comments) but any other persistent storage for the request / script execution is the initial biggest hurdle. Tackle one problem at a time till you get to the end!
I'm planning to work on a social site and I would like to leverage both a document and graph database for all of the desired features. Is there a way to get Meteor.js or Sail.js (or any better) to work with ArangoDB or OrientDB? Or should I just stick with the bundled MongoDB and integrate something like allegrograph DB?
Sails.js has support for both of the databases you mention:
https://www.npmjs.com/package/sails-orientdb
https://github.com/rosmo/sails-arangodb
In addition to MongoDB, Postgres, and dozens of others.
Sails.js is a classic mvc client-server web application framwork, basically its ruby on rails implemented for node.js + webscockets, so mostly all you need to make it work with any backend database is make changes to the orm.
Meteor is a very different beast, it is a very opinionated realtime end to end web framework including client server and database, by behing very very opinionated it solves many of the common issues in realtime application, where you need to implement a mechanisms for very quickly updating all your clients of each others actions and take care of things like latency compensation, data collision resolution, and real time client version managment, This is implemented by using web sockets and the mongoDB's write ahead logging for triggres of data updates, making meteor somewhat coupled with mongo.
But you can make orientdb work pretty well with meteor using the new orientDB live query api. It is a pubsub implementation for query results, and can be used for efficent updates pushed from the db through the server directly to the client with very little overhead on the server. This is far from production ready and only currently works on the orient db 2.1 rc5 version.
I have implemented a small meteor demo application as an example https://github.com/imdark/meteor-orientdb-demo
Please check on Meteorpedia the Alternative Database Post, they mention neo4j-livedata and minineo4j.
Here is the Atmosphere package: ostrio:neo4jdriver
Is there a specific economic or technological reason why you consider not to use the very well integrated mongoDB as the database of choice?
In addition to Travis answer I also recommend waterline-orientdb for Sails.js.
I've got an application where I'm populating a Mongo database. That code is not super easily portable to Meteor, so that I'm doing is running that code in a separate process and writing data to Mongo collections.
In my Meteor application, I point it at that shared Mongo instead of the default instance. The application works reasonably well, but the updates don't seem to happen as fast as I would have expected. When I write a value to Mongo from my external process, it seems to take a while before my client refreshes to show updated content. There is about 7MB of total data in my Mongo database.
Is this approach to share a Mongo and write to it from an external process feasible or is it not advised?
Thanks!
Eric.
As MongoDB doesn't have any live-queries Meteor polls for changes it missed every 10 seconds. Therefore it can take up to 10 seconds until changes are noticed by Meteor.
About the approach in general: it is fine to do so. If you need faster updates you might need to use an other approach or trigger Meteor to update itself (not sure if this is possible at all). Also note that meteor might not support every MongoDB feature yet.
Further readings
Multi-Server architecture and concurrency
Talk at the Dev-Workshop: How Meteor is Scaling Meteor
I’m using EE for the first time. I design/develop on two machines, so I use private github repos for all my client work. I’d like to run this EE project on SQLite locally while I’m developing and designing, and then switch it to MySQL once I’m ready to deploy to the staging environment for client review. That way, I can track DB changes easily between machines, as there is a lot of content on this particular site, and exporting/importing DB tables every time I switch machines (three or more times a day) is getting old.
I realize I could use the client’s MySQL DB remotely, but would rather keep everything local, if at all possible. So, how can I run EE with SQLite?
This question (or something similar) was made almost three years ago here. The accepted answer hints at possible support in EE v2.0+ (via CodeIgniter’s then-new database abstraction), but I can’t find any recent information (official or otherwise) on how to make it happen.
I short: no. Active Record is not universally-used throughout EE, and especially not in third party add-ons. In fact, the documentation for the Database class does not mention Active Record at all.
I'd suggest just installing MySQL locally.
I don't think you'll have much luck with non-MySQL DBs — MySQL statements are hard-coded into EE, despite the EE 2.0 promises of a fully-fledged CI app with various DB connectors.
I think your best results might be to find a centralized host for your dev DB. EE is especially painful to work with on multiple machines because of the DB — the whole thing is really built with "one server, one developer" as the primary use case.
You can simplify it a bit by keeping your templates saved as files, and rigging your settings to be server-agnostic. I also keep little scripts around to quickly dump/import the DB when I have to, which makes moving the site around in a git repo a little easier. A few details here: http://www.viget.com/inspire/expressionengine-on-multiple-machines/
I have got a Bizspark account from Microsoft and they are providing a basic Azure account. I have been told that it can run PHP, however I would like to use a more tested solution like WAMP. On top of that, I want to place a quite heavy WordPress / BuddyPress installation (that I hope will bring a lot of trafic :)
Has anyone done something similar to this? If so, what is your experience / pitfalls etc.?
Thanks
Stelios
Yes, you can do this. At the end of the day you are just using Windows Server, so anything that installs there will install in the cloud as well. I have done this myself for hosting WordPress in Windows Azure.
However, there are some pitfalls here. Mostly the pitfalls are around the M (MySQL). To setup MySQL in Windows Azure is not really that hard, but you have several considerations on how to make sure it is always available. You can:
Setup a single instance of MySQL in
a role and store the db on local
disk (this is a bad idea).
Setup a single instance of MySQL in
a role and store the db on a drive
(blob backed storage)
Setup 2 instances of MySQL to each
point to a shared drive
(hot-failover). Only one drives will
be able to mount. Now, you have reliability and failover, but a single instance at a time working for you.
Setup 1 writer of MySQL on a drive,
and multiple readers on a snapshot
of a drive. Put in some logic via
connection strings to make sure only
writes goto a single one and reads
to the others. Snapshot every X
mins to update readers.
Setup multiple instances of MySQL
and use native replication features
(each storing to local disk) and
rely on that if you lose an
instance.
There are probably more permutations, but the gist of the problem is how you scale out MySQL to be available and reliable. In Windows Azure, you don't get to rely on the fact that the local disk will always be around or that you will always have the same instance. In fact, you can guarantee that your instances will be down for some period of time each month and eventually, given enough time, you will lose the local disk.
Overall, with multiple instances however, you can guarantee they won't be down simultaneously (to the service SLA level at least). So, you need to make sure MySQL works with multiple instances (or live with single instance downtime) and that your data is backed by blob storage to guarantee it is persisted.
Or you can scrap all that crap and just use SQL Azure, which solves all those problems. So, it become WASP. SQL Azure can also be more economical as well for smaller DBs.
Or you can scrap all that crap and just use SQL Azure, which solves all those problems. So, it become WASP. SQL Azure can also be more economical as well for smaller DBs.
Ditto.
Installing MySQL on an Azure role is not a good idea for plenty of reasons, most notably (lack of) scalability and reliability. (That's just for deploying on Azure, MYSQL itself is great)
To set it up remotely reliably you're going to need a dedicated instance which will run you at least $40 a month, going with SQL Azure is $10/Gb, or free if you get an introductory offer or Bizspark.
If you're just looking to play around with a single instance app, I'd suggest you rather use SQLite or some other in memory db, it'll be a lot less painful.