Our web application manages Employee entries on their own activities, and will show dashboards with isCube.
During the day, all the Employee will have to insert data on web application, and the dashboards will have to show real data, every time that incremental load will be scheduled.
All Employee will insert details of their own activity during the day, what they do, the projects on which they worked, the output they produced (for example how many insurance policy they have done) and so on.
During the day it is possible that Employee will have to update and delete their own datas.
So this is the reason why we would like to have incremental rows Inserted/Updated/Deleted.
Is there a way to do this or we have to schedule Full Load during the night?
Thank you
BR
Maria
This is not really a development questions, better to handle this in the standard icCube forum
Related
I am making a leaderboard by using firebase realtime database, unity and facebook ( for profile pic and name )
But I am struggling, I am not sure about the way of doing that. I want to display multiple leaderboards -> All time WORLD / REGION / FRIENDS score, monthly WORLD / REGION / FRIENDS and weekly one.
I know I can use cloud functions to reset weekly and monthly leaderboards.
But how can I store world and region scores ?
Now I just have this ->
https://cdn.discordapp.com/attachments/440873502535450627/821361474353889320/ld.JPG
Thanks to that, I can easily get region leaderboards getref.getchild users . getchild france and then orderbyvalue . limittolast ! this is good !
The problem is how could I get world leaderboard ? ( I have other countries ) I am so lost...Do I need to make another structure for my leaderboard ?
There are a number of considerations that you might put into this including pricing, how much you trust the client, &c. Generally, with a NoSQL database like Realtime Database, you might have a lot of redundant data to make up for the fact that your ability to query is limited. It's also really easy to pull in a lot of data by mistake, so you'll see a number of best practices around keeping your database shallow.
With that said, I think I might recommend reorganizing your data a little bit. For example:
Have one node named "users" with all of your users in there. Each "user" in users should be placed in a node that's simply the userid (which makes security rules easier to write), and here you can place the all time score, monthly score, and weekly score. I'd also recommend storing the time you got that score (using ServerValue.Timestamp) so you don't have to worry about going through your users and deleting all the old scores. If your weekly timestamp isn't this week, you know to ignore/overwrite it (obviously, time is hard, so you'll have to work out what a "week" means to players of your game wrt time zones &c).
I'd also put an array of all a user's "friends" under this user node by their user id. That way, when you go to look up friends, you just just ask for "users/" explicitly there.
Now for regional and world leaderboards on monthly and weekly cadences, I'd just copy everything you want to display into that leaderboard node (say username and score) and add the uid if you need to attribute it back to a user (say if you want to click on that score and see their all time record). So if a user lives in France and they get a weekly high score for them, I'd first write that user's "users/" node with the weekly score and timestamp, then I'd go out to the weekly leaderboard for France and add the new score if it qualifies (ie: don't add it if you're only tracking the top 10), then go out to the world leaderboard and add it there.
How you do this copying is up to you. You could make it the client's responsibility with security rules just making sure they're well behaved - which would probably be fast and cheap but might get weird if they go offline partway through updating. You could use a Firebase trigger that would listen for a user updating their node and copy the data out to the respective leaderboards (this would be more expensive since you're paying for Cloud Functions time and a bit slower but will always work once the user node is updated).
A final note is that for, say, weekly leaderboards. I'd have a node that says "this is France's weekly leaderboard right now" and have all clients read that first before figuring out where to write. That way, at a time you decide for when the week turns over, you can change that node and just have people start writing somewhere else. You can then keep the old leaderboard up for some time (ex: maybe you want to see who changed between this week and last), and you can delete it at your leisure after you're sure that all players have their score in (ex: over slow internets, disconnects, &c).
I'm creating a medical booking app in react native I almost done my UI and now I stuck in data structure in firebase. so the way app works is simple: user register/sign in select the doctor or the therapist and that therapist has some dates and times which user should choose from to set an appointment.
so the way I consider to structure my data is the following:
but I need a simpler approach so I can get all my dates as an array and feed my calendar with it (so user could see the available dates)
and also get all the timings for specific day so when a user select a time other users can't select that too.
how can I structure this data in firebase? sorry I'm new to this concepts if you feel this question is dumb and I'm thankful for any help.
if you need any extra information just let me know in comment section.
You might want to consider extractimg meetings to its own collection and using the dates as the id of each document.. So whenever someone wants to book a meeting, you check if there is a document with the ID of the date, and if not, create the meeting
This is my first question ever on SO and I just want you to know that this community have been helping me for quite some time, and I really got this far thanks to all of you. So thank you. 😀
Now, let me tell you a bit about the question:
I have been scratching my head for days trying to figure this database model for a quite complex (for me, at least) appointment app.
Patients choose a day and it's respective hour slot (15 hourly slots - from 7AM to 9PM) to finish booking a appointment with a therapist. The problem is:
I don't want to show unavailable slots so I must do some real-time querying everytime a date is picked.
I found a way to do this by performing a isEqual query to check if the picked date existed on the therapists agenda collection. If false, the date is available.
The thing is: each therapist has their own subcollection called Agenda but I also need to check if the same date is available in a root collection called Appointments, which is also used by other therapists. This Appointments collection is related to the rooms available for booking. There are 3 rooms available and any room can be automatically booked if it's available. It's a quite complex operation so this is driving me nuts.
Root collection
what I got so far
The therapist agenda
I'm pretty sure I lack a few logical skills to solve this and the answer is probably right in front of my face but I'm failing to see it.
Could anyone please shed me a light into solving this puzzle? Is it possible to achieve all of this in one query function (or performing multiple queries in the same function)? Is there a better way or database structure to do this?
Thanks again!
I would suggest that you store the documents with a random ID and store inside of them a new field called appointment_date which will hold the date, after that, you can query the whole appointment collection with a query like whereEqualTo("appointment_date", yourDate), that query will just fetch the appointments of any specialist.
Also you can store an aditional field inside the appointments documents called specialist_id in which will help you also to get all the appointments for each specialist with a query like
FirebaseFirestore.getInstance().collection("appointments").whereEqualTo("appointment_date",your_date).whereEqualTo("speciallist_id",your_speccialist_id)
This is a kotlin code example, but, with this structure you have more flexibility to query for dates in the agenda, and also query for any speciallist dates
Then, you don't need a subcollection inside your specialist called Agenda, you can just do the query above and you can get each specialist appointment
Check: https://firebase.google.com/docs/firestore/query-data/queries
I am looking for options to ingest Google Analytics data(historical data as well) into Redshift. Any suggestions regarding tools, API's are welcomed. I searched online and found out Stitch as one of the ETL tools, help me know better about this option and other options if you have.
Google Analytics has an API (Core Reporting API). This is good for getting the occasional KPIs, but due to API limits it's not great for exporting great amounts of historical data.
For big data dumps it's better to use the Link to BigQuery ("Link" because I want to avoid the word "integration" which implies a larger level of control than you actually have).
Setting up the link to BigQuery is fairly easy - you create a project in the Google Cloud Console, enable billing (BigQuery comes with a fee, it's not part of the GA360 contract), add your email address as BigQuery Owner in the "IAM&Admin" section, go to your GA account and enter the BigQuery Project ID in the GA Admin section, "Property Settings/Product Linking/All Products/BigQuery Link". The process is described here: https://support.google.com/analytics/answer/3416092
You can select between standard updates and streaming updated - the latter comes with an extra fee, but gives you near realtime data. The former updates data in BigQuery three times a day every eight hours.
The exported data is not raw data, this is already sessionized (i.e. while you will get one row per hit things like the traffic attribution for that hit will be session based).
You will pay three different kinds of fees - one for the export to BigQuery, one for storage, and one for the actual querying. Pricing is documented here: https://cloud.google.com/bigquery/pricing.
Pricing depends on region, among other things. The region where the data is stored might also important be important when it comes to legal matters - e.g. if you have to comply with the GDPR your data should be stored in the EU. Make sure you get the region right, because moving data between regions is cumbersome (you need to export the tables to Google Cloud storage and re-import them in the proper region) and kind of expensive.
You cannot just delete data and do a new export - on your first export BigQuery will backfill the data for the last 13 months, however it will do this only once per view. So if you need historical data better get this right, because if you delete data in BQ you won't get it back.
I don't actually know much about Redshift, but as per your comment you want to display data in Tableau, and Tableau directly connects to BigQuery.
We use custom SQL queries to get the data into Tableau (Google Analytics data is stored in daily tables, and custom SQL seems the easiest way to query data over many tables). BigQuery has a user-based cache that lasts 24 hours as long as the query does not change, so you won't pay for the query every time the report is opened. It still is a good idea to keep an eye on the cost - cost is not based on the result size, but on the amount of data that has to be searched to produce the wanted result, so if you query over a long timeframe and maybe do a few joins a single query can run into the dozens of euros (multiplied by the number of users who use the query).
scitylana.com has a service that can deliver Google Analytics Free data to S3.
You can get 3 years or more.
The extraction is done through the API. The schema is hit level and has 100+ dimensions/metrics.
Depending on the amount of data in your view, I think this could be done with GA360 too.
Another option is to use Stitch's own specfication singer.io and related open source packages:
https://github.com/singer-io/tap-google-analytics
https://github.com/transferwise/pipelinewise-target-redshift
The way you'd use them is piping data from into the other:
tap-google-analytics -c ga.json | target-redshift -c redshift.json
I like Skyvia tool: https://skyvia.com/data-integration/integrate-google-analytics-redshift. It doesn't require coding. With Skyvia, I can create a copy of Google Analytics report data in Amazon Redshift and keep it up-to-date with little to no configuration efforts. I don't even need to prepare the schema — Skyvia can automatically create a table for report data. You can load 10000 records per month for free — this is enough for me.
I am investigating chances to use Firebase for my next project.I spent several days reading and building a "prove of concepts" project. In the demo project i build a shopping cart.In the admin section i can create products, and the client can buy it.When the client checks out i push to the closed-orders node a complex object which stores all data for the deal like this simplified version:
closed-orders
-order_id
-date
-client_id
-products
-product_id
-sale_price
-delivery_price
-qùantity
-product_id
-sale_price
-delivery_price
-quantity
....more products sold in this order
next order....
It is easy to do it that way and i can acces every different order and show it in the admin, but i want to make queries about the total sales, sales by product and a query about the profit.
Example question asked
1.What is the total quantites sold for every product from date1 to date2
2.What is the total turnover from date1 to date2
3.What is the profit for date1 to date2.
I want to answer this questions without downloading the whole dataset in the brwoser of course, because i do not think i can afford to pay for such bandwidth.Orders for one year could be tens of thousands:)
I wrote about Elastic search, keen.io but i am not sure exactly what functionality they offer and if it will answer my questions in a bandwidth friendly way.