for my new app i use this method
https://firebase.googleblog.com/2015/10/client-side-fan-out-for-data-consistency_73.html
i think that is a good method for a person that have a number of followers less than 1 million. i try and up to this number is fine. but for person that have 10kk
of followers the client get in crash because you get a big array of 10kk followers and short it to create another big array of 10kk of path activities.
I just wanted to point out this point, i think that this is a solution that work only with app that have a few numbers of users. finally We're forced to use server-side solutions. and this is bad for the general app efficency
would be a nice feature a function that allow this thing by firebase side with less cost in the client side. i think a feature like this. i make an example in javascript
var obj = { created: time } var path = "FollowersActivity/uid/" var followers = 'root.child("Followers").child("uid").val()' function massSaved(obj, path, followers)
by firebase server side the server get all childs by "followers" path and by foreach cycle append every follower name at the "path" string and save all objects. in this mode the client send only fews strings at firebase server without get all followers and make other big array of activity. probably my example not work because i not know the firebase infrastructure but is only an example to suggest an idea to conclude these operations entirely on the server side
Related
I am building chat application somewhat like whatsapp. I want to show registered app users list from user's device contact list while creating new group. Now in order to do that I have to compare each and every contact number with firebase firestore users. And any normal user can have more than 500 contacts in device. And moreover firestore has limitation to for querying the db so I can not compare more than one number at a time, the whole process takes almost 6-7 minutes as well as each read operation costs financially.
How can I overcome with this situation, or what is the better way to deal with this particular scenario?
You can store the contacts of the user on device and only send them to firestore as backup. You can then sync your local database with firestore on app start.
The operations you need are not possible to be robust in firebase. Even then if you want to do a search in firebase data, you need to use 3rd party search solution like Elastic Search with your firebase data to perform complex searching.
For local database you can use Room library: https://developer.android.com/topic/libraries/architecture/room
For using Elastic Search with Firebase have a look at this utility Flashlight: https://github.com/FirebaseExtended/flashlight .
The OP requested a structure and some code (Swift, Firebase Database) as a solution. I will present two options
If you want to use a Firebase Query to see if the phone numbers exist, a possible stucture would be
users
uid_0
contact_name: "Larry"
contact_phone: "111-222-3333"
uid_1
contact_name: "Joe"
contact_phone: "444-555-6666"
and then the swift code to query for existing numbers
let phoneNumbers = ["111-222-3333","444-555-6666"] //an array of numbers to look for
let myQueryRef = self.ref.child("users")
for contactPhone in phoneNumbers {
let queryRef = myQueryRef.queryOrdered(byChild: "contact_phone").queryEqual(toValue: contactPhone)
queryRef.observeSingleEvent(of: .childAdded, with: { snapshot in
if snapshot.exists() {
print("found \(contactPhone)") //or add to array etc
}
})
}
Having queries in a tight loop like this is generally not recommended but it usually works fine for me with low iterations. However, queries have a lot more overhead than .observers.
IMO, a better and considerably faster option is to keep a node of just phone numbers. Then iterate over the ones you are looking for and use .observe to see if that node exists.
phone_numbers
111-222-3333: true
444-555-6666: true
and then the code to see if the ones from the array exist
let phoneNumbers = ["111-222-3333","444-555-6666"] //an array of numbers to look for
let phoneNumberRef = self.ref.child("phone_numbers")
for contactPhone in phoneNumbers {
let ref = phoneNumberRef.child(contactPhone)
ref.observeSingleEvent(of: .value, with: { snapshot in
if snapshot.exists() {
print("found \(contactPhone)")
}
})
}
In testing, this second solution is must faster than the first solution.
I'd like to know the proper decision for this Meteor scenario:
User click button
Client side code calls a server Method.
Method code calls mongoDB and generates a random array with some content from collection A.
Server save the given array as a new doc into collection B.
Generated array data must get printed to the DOM.
What is better for performance maters? Return the array directly from method.call or publish/subscribe B collection?
NOTE THAT:
a) Array content are 20(tops) string elements of less than 120 characters each.
b) Array data is not reactive.
c) User won't need new data until he clicks '0 button' again.
I would go with the client->method->server and server->pub/sub->client way.
This way you get queries in minimongo. Also about reactivity you can throttle it so it's not a huge issue. Returning data via method would make you need promises etc cause it will take time to get back to you. Unless you will execute some function in return function from the call.
about the C), client won't get new data the pub/sub will handle this. Client will only get new data if there is new data/update.
In Meteor, I got a collection that the client subscribes to. In some cases, instead of publishing the documents that exists in the collection on the server, I want to send down some bogus data. Now that's fine using the this.added function in the publish.
My problem is that I want to treat the bogus doc as if it were a real document, specifically this gets troublesome when I want to update it. For the real docs I run a RealDocs.update but when doing that on the bogus doc it fails since there is no representation of it on the server (and I'd like to keep it that way).
A collection API that allowed me to pass something like local = true this would be fantastic but I have no idea how difficult that would be to implement and I'm not to fond of modifying the core code.
Right now I'm stuck at either creating a BogusDocs = new Meteor.Collection(null) but that makes populating the Collection more difficult since I have to either hard code fixtures in the client code or use a method to get the data from the server and I have to make sure I call BogusDocs.update instead of RealDocs.update as soon as I'm dealing with bogus data.
Maybe I could actually insert the data on the server and make sure it's removed later, but the data really has nothing to do with the server side collection so I'd rather avoid that.
Any thoughts on how to approach this problem?
After some further investigation (the evented mind site) it turns out that one can modify the local collection without making calls to the server. This is done by running the same methods as you usually would, but on MyCollection._collection instead of just on Collection. MyCollection.update() would thus become MyCollection._collection.update(). So, using a simple wrapper one can pass in the usual arguments to a update call to update the collection as usual (which will try to call the server which in turn will trigger your allow/deny rules) or we can add 'local' as the last argument to only perform the update in the client collection. Something like this should do it.
DocsUpdateWrapper = function() {
var lastIndex = arguments.length -1;
if (arguments[lastIndex] === 'local') {
Docs._collection.update(arguments.slice(0, lastIndex);
} else {
Docs.update(arguments)
}
}
(This could of course be extended to a DocsWrapper that allows for insertion and removals too.)(Didnt try this function yet but it should serve well as an example.)
The biggest benefit of this is imo that we can use the exact same calls to retrieve documents from the local collection, regardless of if they are local or living on the server too. By adding a simple boolean to the doc we can keep track of which documents are only local and which are not (An improved DocsWrapper could check for that bool so we could even omit passing the 'local' argument.) so we know how to update them.
There are some people working on local storage in the browser
https://github.com/awwx/meteor-browser-store
You might be able to adapt some of their ideas to provide "fake" documents.
I would use the transform feature on the collection to make an object that knows what to do with itself (on client). Give it the corruct update method (real/bogus), then call .update rather than a general one.
You can put the code from this.added into the transform process.
You can also set up a local minimongo collection. Insert on callback
#FoundAgents = new Meteor.Collection(null, Agent.transformData )
FoundAgents.remove({})
Meteor.call 'Get_agentsCloseToOffer', me, ping, (err, data) ->
if err
console.log JSON.stringify err,null,2
else
_.each data, (item) ->
FoundAgents.insert item
Maybe this interesting for you as well, I created two examples with native Meteor Local Collections at meteorpad. The first pad shows an example with plain reactive recordset: Sample_Publish_to_Local-Collection. The second will use the collection .observe method to listen to data: Collection.observe().
On my meteor project users can post events and they have to choose (via an autocomplete) in which city it will take place. I have a full list of french cities and it will never be updated.
I want to use a collection and publish-subscribes based on the input of the autocomplete because I don't want the client to download the full database (5MB). Is there a way, for performance, to tell meteor that this collection is "static"? Or does it make no difference?
Could anyone suggest a different approach?
When you "want to tell the server that a collection is static", I am aware of two potential optimizations:
Don't observe the database using a live query because the data will never change
Don't store the results of this query in the merge box because it doesn't need to be tracked and compared with other data (saving memory and CPU)
(1) is something you can do rather easily by constructing your own publish cursor. However, if any client is observing the same query, I believe Meteor will (at least in the future) optimize for that so it's still just one live query for any number of clients. As for (2), I am not aware of any straightforward way to do this because it could potentially mess up the data merging over multiple publications and subscriptions.
To avoid using a live query, you can manually add data to the publish function instead of returning a cursor, which causes the .observe() function to be called to hook up data to the subscription. Here's a simple example:
Meteor.publish(function() {
var sub = this;
var args = {}; // what you're find()ing
Foo.find(args).forEach(function(document) {
sub.added("client_collection_name", document._id, document);
});
sub.ready();
});
This will cause the data to be added to client_collection_name on the client side, which could have the same name as the collection referenced by Foo, or something different. Be aware that you can do many other things with publications (also, see the link above.)
UPDATE: To resolve issues from (2), which can be potentially very problematic depending on the size of the collection, it's necessary to bypass Meteor altogether. See https://stackoverflow.com/a/21835534/586086 for one way to do it. Another way is to just return the collection fetch()ed as a method call, although this doesn't have the benefits of compression.
From Meteor doc :
"Any change to the collection that changes the documents in a cursor will trigger a recomputation. To disable this behavior, pass {reactive: false} as an option to find."
I think this simple option is the best answer
You don't need to publish your whole collection.
1.Show autocomplete options only after user has inputted first 3 letters - this will narrow your search significantly.
2.Provide no more than 5-10 cities as options - this will keep your recordset really small - thus no need to push 5mb of data to each user.
Your publication should look like this:
Meteor.publish('pub-name', function(userInput){
var firstLetters = new RegExp('^' + userInput);
return Cities.find({name:firstLetters},{limit:10,sort:{name:1}});
});
I have a batch apex class where i'm building collections of websites and emails, so that i can use those collections to filter other other queries which will be made into collections. With all collections set, i want to run through a final loop of the scope to perform business processes.
Mockup:
for(Object o : scope)
{
listEmails.add(o.Email);
listWebsites.add(o.Websites);
}
Map<String, Account> accounts = Gather all accounts where website not in :listWebsties; //Website is key
List<String, Contact> contacts = Gather all contacts where email not in :listEmails; //Email is key
for(Object o : scope)
{
Account = accounts.get(o.website);
Contact = contacts.get(o.Email);
Perform business logic here
}
The problem is when i run this batch it stays processing for hours. When working with a rather small database this works fine. But in working in a larger environment perhaps this is not the best solution.
Can anyone help me speed up the batch process with a more effective approach?
Is there anyway to post the entire batch apex class? Or help understand the data more?
It looks like from your map that all of your accounts (in theory) have unique websites and all of your contacts have unique emails?
I assume you build those maps by hand? That is you loop over the accounts and do a
map.put(account.website,account)?
Do you have any system debug statements to confirm your map sizes?
What happens if there is no account or no contact when you call accounts.get()?
And the business logic - is it more looping?
And are you using Batch variables in a static manner - i.e. you can have a counter to count the total number of records processed. If so, is your variable a list? that can be dangerous of course.
Also what object is your scope object? Not that it matters, but I'd think you'd want to have your scope be the Accounts themselves or the Contacts themselves.
I'd try adding system.debug statements to your batch to verify it's running and to see where the infinite loop may be occurring.