I have a users collection and I want to compute a score for each user every 15 minutes
for this I'm planned to write a scheduled function to trigger this every 15 minutes
Loop on all users
for each user compute new score based on some criteria ... ( very simple mathematical operations ms )
update the user score field
Question : suppose we have 1 million users so approximately how many time this function can take to process all users ( seconds / minutes ) ? or if there is another way to achieve this to have result ASAP
NOTE : I have Blaze Planad and I'm expecting result should be available ASAP because the goal is to sort users by score every 15 minutes
user collection
- String : id
- String : name
- int : score
- String : email
- List<String> : data
any cloud function example to achieve efficiently this will be appreciated
Thanks
Related
It's probably the simplest thing in the world but I can't seem to add/sum/calculate tables in App Maker. I have attached an image of the App preview. I will try to keep this simple. The page has 3 data sources as follows:
Apex_customer_po - This is where our employee's input customer purchase orders including the PO number (string), the overall value of the PO (number), and the details (string)
Apex_customer_po_wo_details - Purchase orders are often broken up by our customer in to multiple work orders (WO), a task break down each with it's own values including Date (date), WO number (string), WO amount (number), details (string)
Apex_customer_po_wo_costing - This is where we keep track of what work we've done and how much of each work order's money we've used. The fields include a reference number(string), parts price (number), Part details (string), Labour hrs (number), Labour rate (number), Total labour (number), and Invoice total (number), Overall total of all invoices for the work order (number)
My data relations are as follows:
Apex_customer_po
Apex_customer_po_wo_details (One Apex_customer_po to Many Apex_customer_po_wo_details)
Apex_customer_po_wo_costing (One Apex_customer_po_wo_details to Many Apex_customer_po_wo_costing)
Shows the general configuration of the PO app preview mode. I includes 2 work orders each with their associated invoices
Shows the general configuration of the Layout view of the app page
I have figured out how to calculate the cost of my labour:
#datasource.item.Labour_total = #datasource.item.Labour_time * #datasource.item.Labour_rate
I have figured out how to calculate the total of each invoice:
#datasource.item.Invoice_total = #datasource.item.Parts_amount + (#datasource.item.Labour_time * #datasource.item.Labour_rate)
But I can't for the life of me figure out how to 'Add / Sum / Total / Calculate' the individual invoices from each Work Order to get a total value.
I've tried unsuccessfully to use reduce and a bunch of other methods, but I can't ever seem to get the syntax correct even on basic 1+1 type calculations. I don't have much experience in app maker but I've been doing fine and loving it... until now.
Update 001:
I have managed to get a reduce function to perform a simple table addition for me based off the following blog post. blog.supportgroup.com/google-app-maker-cold-calculations - And the ball is rolling again. Now I need to rework how my data sources are set-up changing the numbers to strings –
Update 002:
I managed to get the reduce function to work in my app. However it totals the values from the selected cells across the data source. For example I have a PO, lets call it PO#1 and there are 5 items that total $5. When I start a new PO#2, it carries over the total of PO#1. So it calculates the entire data source regardless of the fact that it's 2 different PO#'s and I don't know how to make it stop. The code I have used is as follows
#datasource.item.Total = getFormatMoney((#datasources.Apex_customer_po_wo_costing.items).reduce((b,a) => Number(a.Parts_amount) + (Number(a.Labour_rate) * Number(a.Labour_time))+ Number(b) , 0 ),2,",",".")
The getFormatMoney(),2,",",".") is a client side java script that formats the currency
Thanks in advance for the assistance.
UPDATE 003:
I am now trying to do this in a different way with a calculated Data source and I am following this tutorial:
YouTube 7.33min long - Totals in Drive Tables tutorial
I have re-created the app step by step but I am getting the following error regarding my SQL query code:
E Mon Nov 18 13:38:49 GMT-700 2019 Exception: Malformed SQL. More
information: Error with SQL statement: You have an error in your SQL
syntax; check the manual that corresponds to your MySQL server version
for the right syntax to use near var totals = (wholesale:0, retail:0, profit:0); var records = app.models.autos. at line 1.
E Mon Nov 18 13:38:49 GMT-700 2019 Executing query for datasource
totals: (Error) : Malformed SQL. More information: Error with SQL
statement: You have an error in your SQL syntax; check the manual that
corresponds to your MySQL server version for the right syntax to use
near var totals = (wholesale:0, retail:0, profit:0); var records =
app.models.autos. at line 1.
E Mon Nov 18 13:38:49 GMT-700 2019 Executing query for datasource
totals failed.
My code in my data source is as follows:
var totals = {wholesale:0, retail:0, profit:0};
var records = app.models.autos.newQuery().run();
records.forEach(function( item ){
totals.wholesale += item.wholesale;
totals.retail += item.retail;
totals.profit += (item.retail - item.wholesale);
});
var record = app.models.totals.newRecord();
record.wholesale = totals.wholesale;
record.retail = totals.retail;
record.profit = totals.profit;
return [record];
Anyone have any thoughts on what the mistake is?
Update 004
Success - The data source type needed to be a "Calculated" type not a "Calculated SQL". Now I am going to see if I can apply this new type of calculation to my existing problem.
It seems like the way to go on this is the calculated model route the tutorial I linked in Update 003 is a solid path to go down.
What is the procedure to do if I have 2 tables: From table-1 I get all the users I want, and then, after I have the list of user ids, I want to compare each user with current user with a field found on table-2. The first task is easy, I just have an onDataChange that populates the users list with their ids. But now that I have this list, how to iterate each user, and compare it with the current user based on a specific field from table-2.
What I currently try is to use a for loop to iterate each user on the list with each having onDataChange call to table-2, and then I populate the necessary dataset. But when this for loop ends, this dataset is no longer visible.
I hope what I try to achieve in this post is understandable.
I'll try to demonsrate with tables:
Assuming I get user list from table-1 based on data1:
table-1
|
|_____data1
|____uid20
|____uid30
|____uid44
Now I have list of 3 users: uid20, uid30, uid44. Then, I need to compare the list of users, with current user, call it user1, from table-1, based on another field (timestamp). What I mean is, after I have list of users, I want to filter these to have a timestamp that's close to the current user, for up to certain amount of time. So in my example, I want to have only users that are within 2 minutes of the current user timestamp.
table-2
|
|______uid1
| |____timestamp: <some_timestamp>
|
|______uid20
| |____timestamp: <some_timestamp>
|
|______uid30
| |____timestamp: <some_timestamp>
|
|______uid44
|____timestamp: <some_timestamp>
But every time there is something that's out of scope of the new listener, and also it looks like it's not the right procedure. Maybe I first need to save what's found on table-1 locally ? Or, it can be done somehow purely with Firebase calls?
**This is some code:
Getting the current user, is easy:
final FirebaseUser user = FirebaseAuth.getInstance().getCurrentUser();
final String userId = user.getUid();
So I always have it visible at any scope
First, as I understand, you can use the startAt and endAt to get a range of the values within two minutes of difference. What I mean is that before getting each value from your table-2 you can just get the values that matches your use case, in this case, values that are 2 minutes of the current timestamp.
For example, in your table 2, I would query like this:
ref.orderByChild('timestamp').startAt(yourCurrentTimeStamp).endAt(yourCurrentTimeStamp+120000);
where 120000 is 2 seconds in miliseconds
and then when looping through this elements I would use getKey to get each key of the values filtered by this query, so I would get only the users with 2 minutes of difference, and then compare them with the first for loop you did in order to see if they match.
to compare 2 users ID you can use equals, since it's a String:
if(snapshotUserTable1.getKey().equals(snapshotUserTable2.getKey())){
/...
}
I have a problem doing complex queries in Firestore database. I have read the documentation countless times so I know where the limitations are, but I wonder whether there is a way to structure the data so it supports my use cases. Let me explain what the use cases first:
I have a list of jobs, and users and I want to able to list/filter jobs according to some criteria and to list/filter users according to some criteria.
Job
JOB ID
- job type (1 of predefined values)
- salary (any number value)
- location (any value)
- long
- lat
- rating (1 - 5)
- views (any number value)
- timeAdded (any timestamp value)
- etc.
User
User ID
- experiences (0, 1 or more of predefined values)
- experience1
- jobCategory
- jobName
- timeEmployed
- experience2
- etc
- languages (0, 1 or more of predefined values)
- language1
- languageName
- proficency
- language2
- etc.
- location (any value)
- long
- lat
- rating (1 - 5)
- views (any number value)
- timeLastActive (any timestamp value)
- etc.
Filtering by field which can only have one value is fine. Even when I add sorting by "timeAdded" or a ragne filter.
1) The problem is when I introduce second range filter, such as jobs with "salary" higher then 15 bucks and at the same time "rating" higher then 4. Is there a way to get around this problem when I have N range filters?
2) Second problem is that I cannot use logical OR. Lets say, filter jobs, where "jobCategory" is Bartender or Server.
3) Another problem is to filter by fields, which can have more then 1 value, e.g. User can speak more than one language. Then if I want to filter users which speak English, it is not possible. Not to mention to filter users who speak e.g. English OR French.
I know I can model the data the way that I use the language as the name of the field, like -english = true, but when I introduce range filter to this, I need to create a Firestore index, which is very inconvenient since I can have around 20 languages and around 50 job types at the same time, and I would have to create indexes all the combinations together with different range filters.. is this assumption correct?
4) How would I filter jobs which up to 20 km from certain position? Radius and the position is on the user to choose.
5) What if I want to filter by all those fields at the same time? E.g. filter certain "jobCategory", location and radius, "salary" higher then something and "rating" higher then something, and sort it all by "timeAdded".
Is this possible with Firestore / Realtime database, can I model the data in some way to support this, or do I have to look for an alternative DB solution? I really like the real-time aspect of it. It will come handy when it is time to implement chat feature to the app. Is it solvable with Cloud functions? I am trying to avoid doing multiple requests, merging them together and sending that to client, since there can be any combination of filters.
If not doable with Firebase, do you know of any alternatives similar to Firestore with better querying options? I really hope I am just missing something :)
Thank you!
Let's say I have three basic models: a User, a Company, and a Visit. Every time a User goes to a Company, a Visit is recorded in this format (user_id, company_id, visit_date).
I'd like to be able to calculate the average time between visits for a company. Not visits overall, but specifically how long on average one of their customers waits before returning to the store.
For example, if one user visited on Tuesday, Wednesday, and Friday that gives one "gap" of one day, and one "gap" of two days => (1, 2). If another user visited on Monday and Friday, that gives one gap of 4 days => (4). If a third user visited only once, he should not be considered. The average time between user visits for the company is (1 + 2 + 4) / 3 = 2.333 days.
If I have thousands of users, taps, and companies and I want to calculate a single figure for each company, how should I go about this? I've only done basic MapReduce applications before and I can't figure out what my Map and Reduce steps would be to get this done. Can anyone help me figure out a MapReduce in pseudocode? Or is there some other method of distributed calculation I can reasonably perform? For the record, I'd like to perform this operation on my database every night.
The overly simplistic approach would be to have two job steps.
First job step has a mapper to write key values in the form "company:user" and "visit_date". In the example above, the mapper would write something like:
"user1:companyA" -> "2012/07/16"
"user1:comapnyA" -> "2012/07/17"
"user1:comapnyA" -> "2012/07/19"
"user2:comapnyA" -> "2012/07/15"
"user2:comapnyA" -> "2012/07/19"
...
This means that each call to the reducer will pass all of the visits by a single user to a single company. That means that one call to the reducer will pass in:
"user1:companyA" -> {2012/07/16, 2012/07/17, 2012/07/19}
and another call will pass in:
"user2:companyA" -> {2012/07/15, 2012/07/19}
I'm assuming the set of dates (passed in as an Iterable value) is easily managed as you sort it, figure out the gaps and write a record for each gap as a key value pair in the form "company" and "gap". For example, when passed:
"user1:companyA" -> {2012/07/16, 2012/07/17, 2012/07/19}
The first job's reducer will write to the context:
"companyA" -> 1
"compnayA" -> 2
The second job has a pass-through mapper that just passes the company/gap info on to the reducer. Each call to the reducer gives an Iterable value of gaps for a specific company. Iterate through the data to produce an average and write the key value pair in the form "company" and "average_gap".
If the original set of visits gets too big, we can talk about getting hadoop to do the sorting for you with some custom comparators.
I m on a problem that I can t figure out. I m building an application in c++ builder 2009 and oracle 11g. I have some calculated data that depend on users age. What I want to do is to re-calculate these data every new year. I thought I could have a trigger to do this, but I don t know which event I should catch and I didn t find something in internet.
My table is :
ATHLETE (name, ......, birthdate, Max_heart_frequency)
Max_heart_frequency is the field that depends on age. In insertion I calculate athlete's age, but what about next year??????
Can anyone help????
How is the max_heart_frequence calculated?
If this is a simply formula, I would create a view that returns that information. No need to store values that can easily be calculated:
CREATE VIEW v_athlete
AS
select name,
case
-- younger than 20 years
when (MONTHS_BETWEEN(sysdate, birthday) / 12) < 20 then 180
-- younger than 40 years
when (MONTHS_BETWEEN(sysdate, birthday) / 12) < 40 then 160
-- younger than 60 years
when (MONTHS_BETWEEN(sysdate, birthday) / 12) < 60 then 140
-- everyone else
else 120
end as max_heart_frequency
from athlete
Then you only need to select from the view and it will always be accurate.
You can use oracle scheduler to run a procedure at specific intervals (can be minutes hours, daily, yearly etc .. any time span).
Check this linke: http://download.oracle.com/docs/cd/B19306_01/server.102/b14231/schedover.htm
You have two options:
Have a stored procedure that calculates and updates the Max_Heart_Frequency of all the athletes every 01st Jan (using the yearly scheduling of a procedure)
Have a stored procedure that runs daily and calculates and updates the Max_Heart_Frequency of all the athletes every day (using the daily scheduling of a procedure)
If Max_Heart_Frequency changes over time because the user is getting older, why are you storing it in the table in the first place? Why not just call the function that computes the maximum heart rate at runtime when you need the value? Potentially, it may make sense to have a view on top of the Athlete table that adds the computed Max_Heart_Frequency column to hide from the callers that this is a computed column.