From what I know it seems that Meteor Framework stores part of data on the client. It's clear how to do it for personal todo list - because it's small and you can just copy everything.
But how it works in case of let's say Q&A site similar to this? The collection of questions are huge, you can't possibly copy it to the client. And you need to have filtering by tags and sorting by date and popularity.
How Meteor Framework handles such case? How it partition data? Does it make sense to use Meteor for such use case?
Have a look at the meteor docs, in particular the publish and subscribe section. Here's a short example:
Imagine your database contains one million posts. But your client only needs something like:
the top 10 posts by popularity
the posts your friends made in the last hour
the posts for the group you are in
In other words, some subset of the larger collection. In order to get to that subset, the client starts a subscription. For example: Meteor.subscribe('popularPosts'). Then on the server, there will be a corresponding publish function like: Meteor.publish('popularPosts', function(){...}.
As the client moves around the app (changes routes), different subscriptions may be started and stopped.
The subset of documents are sent to the client and cached in memory in a mongodb-like store called minimongo. The client can then retrieve the documents as needed in order to render the page.
Related
I have just published an app that uses Firestore as a backend.
I want to change how the data is structured;
for example if some documents are stored in subcollections like 'PostsCollection/userId/SubcolletionPosts/postIdDocument' I want to move all this last postIdDocument inside the first collection 'PostsCollection'.
Obviously doing so would prevent users of the previous app version from writing and reading the right collection and all data would be lost.
Since I don't know how to approach this issue, I want to ask you what is the best approach that also big companies use when changing the data structure of their projects.
So the approach I have used is document versioning. There is an explanation here.
You basically version your documents so when you app reads them, it knows how to update those documents to get them to the desired version. So in your case, you would have no version, and need to get to version 1, which means read the sub-collections to the top collection and remove the sub collection before working with the document.
Yes it is more work, but allows an iterative approach to document changes. And sometimes, a script is written to update to the desired state and new code is deployed 😛. Which usually happens when someone wants it done yesterday. Which with many documents can have it's own issues.
I have two different nodes in database.
all posts
users
As per the fan-out model when a user adds a post , it gets updated at both all posts and users/uid/posts.
Each post consists of a like button which displays the number of likes.
When a user clicks on it the like should increase by +1.
According to the docs, we use transactionfor this kind of process.
But the problem with using transaction is that it updates only one node as far as i know
But my problem is how shall i update this transaction in both the nodes as mentioned above
Shall i use update method
What is the way to use transaction that gets updated at both the nodes
You can push all your logic for updating the database onto the server side with Cloud Functions for Firebase. Use can use a database trigger to respond to data being written in the database, then execute some JavaScript to make sure the fan-out finishes correctly. It will have the advantage of making sure all the changes happen without depending on the client.
Transactions can't modify data at two different locations at once, but you will still probably want to use them in your client and Cloud Functions to make sure concurrent writes will not have problems.
I know I can setup multiple namespaces for DoctrineCacheBundle in config.yml file. But Can I use one driver but with multiple namespaces?
The case is that in my app I want to cache all queries for all of my entities. The problem is with flushing cache while making create/update actions. I want to flush only part of my cached queries. My app is used by multiple clients. So when a client updates sth in his data for instance in Article entity, I want to clear cache only for this client only for Article. I could add proper IDs for each query and remove them manually but the queries are dynamically used. In my API mobile app send version number for which DB should return data so I don't know what kind of IDs will be used in the end.
Unfortunately I don't think what you want to do can be solved with some configuration magic. What you want it some sort of indexed cache, and for that you have to find a more powerful tool.
You can take a look at doctrines second level cache. Don't know how good it is now (tried it once when it was in beta and did not make the cut for me).
Or you can build your own cache manager. If you do i recommend using redis. The data structures will help you keep you indexes (Can be simulated with memcached, but it requires more work). What I meen by indexes.
You will have a key like client_1_articles where 1 is the client id. In that key you will store all the ids of the articles of client 1. For every article id you will have a key like article_x where x is the id the of article. In this example client_1_articles is a rudimentary index that will help you, if you want at some point, to invalidated all the caches of articles coming from client 1.
The abstract implementation for the above example will end up being a graph like structure over your cache, with possibly
-composed indexes 'client_1:category_1' => {article_1, article_2}
-multiple indexes for one item eg: 'category_1'=>{article_1, article_2, article_3}, 'client_1' => {article_1, article_3}
-etc.
Hope this help you in some way. At least that was my solution for a similar problem.
Good luck with your project,
Alexandru Cosoi
Apologies for this long winded question but I just want to walk through my research into trying to solve this problem myself.
My first stackoverflow question... here goes...
My meteor app was plugging along nicely until I wanted to have a standings table. This 30 years of basketball standings article was inspirational. I wanted to tap into that power and flexibility that Mongo provided and keep track of team and player standings I bought this book on Mongo (highly recommended). I have multiple collections and one collection called 'games' that keeps track of games and scores per game. Here is a same document:
{
"leagueId" : "6RtH74QbxGG7xbZXh",
"regionId" : "KDbqfAoKDx2iDDXSS",
"seasonId" : "b5HkGcWFNenpGGvCd",
"gameTime" : "15:00",
"gameDate" : "11/23/2014",
"gameNumber" : 4,
"gameStatus" : "played",
"homeTeam" : "MYBw2RiNwrBhfh9W8",
"homeTeamScore" : 4,
"awayTeam" : "fwx79JJFob5XbaAx6",
"awayTeamScore" : 2,
"gameType" : "regular_season",
"userId" : "4MKaZK84AdZ8j3xr2",
"author" : "league",
"submitted" : "Wed Dec 10 2014 09:51:48 GMT-0800 (PST)",
"_id" : "Gwsu6X6DXXzavdqZQ"
}
I determined the aggregate framework was the way to go but after digging into this I saw that Meteor had some client side limitations with the mongo aggregate framework (stackoverflow.com/questions/11266977/are-group-by-aggregation-queries-possible-in-meteor-yet).
I also thought about [denormalizing my document structure after reading articles like this][4]. And after reading the Denormalization chapter of Discover Meteor (good book!), it seems not denormalizing gives me a lot more flexibility but I can denormalize in Meteor/Mongo with other benefits. The NBA article mentioned earlier is very denormalized and not sure if I can replicate that structure using Meteor/Mongo.
I then jumped in the Aggregate rabbit hole and found some good reading (stackoverflow.com/questions/18520567/average-aggregation-queries-in-meteor/18884223#18884223 - I tried this using my data and I saw the output on the server but when I added the subscription to 'waiton' in my iron router there were no errors and it just hung. So it worked on the server, not the client?).
I then found this article which looked very promising. That led me to the atmosphere package ([meteorhacks:aggregate][8]... and it's dependencies) but the article seemed more directed at experts with mongo as I could not get my data client side.
From all my reading I surmise that I should create a new collection in my 'collections' folder (available on client and server) and then I publish that collection using my aggregate code ($group and $match and cool things like $sum). Then I'm really not sure what to do next. Do I create a server method and call that method from my client side code (tried that and couldn't get it to work)? Am I supposed to subscribe to the collection in iron-router (every time I did, my app just hung with no errors)?
If anyone gets how to use aggregate with meteor I would really appreciate some/any guidance. I think I just need a few more pieces of information before I can generate that standings table with wins, losses and draws.
note: I took a detour and thought MapReduce might be the solution(thebhwgroup.com/blog/2014/09/bi-nodejs-couchdb-mapreduce) but I couldn't get this to work in Meteor either.
You raise a number of topics and there are probably a lot of solutions to get where you want. One key point is that you cannot currently run mongo aggregations on the client side in Meteor. So any approach where you are aggregating game data into standings needs to first aggregate the game data server-side, and only then publish a cursor of the results to the client.
To break your problem down into more manageable simples, I would first focus on assembling an aggregation on the server side that looks the way you want. You will need a package such as meteorhacks:aggregate as you mentioned, or others that accomplish the same purpose.
Then, for starters, you should simply run the aggregation pipeline server-side on Meteor.startup a few times until it looks the way you want. To get a quick look at it, you could save it to a variable and log it in the console (again, server-side). Alternatively, you could save it into a new Collection such as Standings and inspect it using meteor mongo from the console. Part of the issue is that you may be getting ahead of things trying to publish and subscribe to your collections with Iron Router while you are still thinking about how to set up your collections.
This gets to the next point. Once the aggregation document looks the way you want on the server, you'll need to decide whether you want to save the results each time you aggregate a new set of standings, or assemble them on the fly each time you publish it. In the Owens article you linked to, he is aggregating data directly inside of a publish function. This means it is re-run each time it is published or subscribed to, which could be very fast or very slow depending on the size and complexity of the data. It is also important to consider that under this model you are not actually publishing a collection of standings to the client - you are publishing a mutated collection of games. If you expect your standings to acquire their own properties, change over time, or you want to poll them historically, you are best off creating a new Collection such as Standings and inserting a new standing into the collection each time you receive new game results.
Your publish function could then be fairly simple, e.g., Standings.find({}, {limit: 1}) with a query for the most recent standing. You can subscribe in Iron Router's waitOn as you suggested.
To insert new entries into the Standings collection, you could call a server side method like Meteor.Call("createStandings") each time a new game is introduced. This could result in a large document over time, which you could cull or even write over if you don't care about maintaining historical records.
Again, there are a lot of ways you could take this. Hope this helps.
I'm writing a simple Wordpress plugin for work and am wondering if using the Transients API is practical in this case, or if I should seek out another way.
The plugin's purpose is simple. I'm making a call to USZip Web Service (http://www.webservicex.net/uszip.asmx?op=GetInfoByZIP) to retrieve data. Our sales team is using a Lead Intake sheet that the plugin will run on.
I wanted to reduce the number of API calls, so I thought of setting a transient for each zip code as the key and store the incoming data (city and zip). If the corresponding data for a given zip code already exists, then no need to make an API call.
Here are my concerns:
1. After a quick search, I realized that the transient data is stored in the wp_options table and storing the data would balloon that table in no time. Would this cause a significance performance issue if the db becomes huge?
2. Is this horrible practice to create this many transient keys? It could easily becomes thousands in a few months time.
If using Transient is not the best way, could you please help point me in the right direction? Thanks!
P.S. I opted for the Transients API vs the Options API. I know zip codes don't change often, but they sometimes so. I set expiration time of 3 months.
A less-inflated solution would be:
Store a single option called uszip with a serialized array inside the option
Grab the entire array each time and simply check if the zip code exists
If it doesn't exist, grab the data and save the whole transient again
You should make sure you don't hit the upper bounds of a serialized array in this table (9,000 elements) considering 43,000 zip codes exist in the US. However, you will most likely have a very localized subset of zip codes.