We're currently pushing transactions using Standard Ecommerce, and while the majority of our transactions appear in Google Analytics, a few do not. This has been confirmed by comparing what appears in Google Analytics vs the transactions records we have on our end. We tried adding "logging" immediately after the transaction data object was pushed to GTM by immediately sending that same JSON/object to our servers and then just logging that as-is, so that there was something to verify that we at least pushed the data object successfully.
Having just checked the logs after noticing a couple new missing transactions in Google Analytics, we noticed the transaction JSON/object was missing the uniqueEventId property whenever a transaction went missing. It would seem that the property is automatically added by GTM when an event is sent out, but it's unclear what would cause it to not be added.
Here's the data we're currently sending to GTM:
{
event: "deposit",
transactionId: <uuid string>,
transactionTotal: (<cents> / 100),
user_id: <uuid string>
}
we send this to GTM using window.dataLayer.push. That's also what we send to our servers to log after the JSON has been pushed to GTM's object. All of the logs show the above JSON with an additional property uniqueEventId, except for the transactions that don't appear in GA which all are missing that property in the logs.
Ultimately we just want to address the missing transactions in GA.
Was it a significant difference? If not, you might be able to explain the difference with ad-blockers and the like. They'd prevent the requests from showing up in Google Analytics, reflecting exactly what you're suggesting.
Related
A customer of ours is very persistent that they expect this "from any API" (meaning they don't want to pay for the changes). I seem to have trouble finding clear information on this though.
Say we have an API that creates an appointment for a calendar. Server-side everything was successful, data is committed to the database. API tries to send the HTTP 201 (Created) response, but something goes wrong there. Client ignores the response, or connection dropped, ...
They want our API to undo the database changes in that particular situation.
The question is not how to do this, but rather if this is something most APIs do? Is this standard behavior? Or something similar like refusing duplicate create requests?
The difficult part of course is to actually know if an API has failed to send the response, and as far as I am concerned with respect to the crux of the question, it is not a usual behavior implemented. If the user willingly inputs the data, you can go ahead and store it. If the response doesn't return properly due to timeouts (you are not responsible for user "ignoring" the response), then the client side code can refresh on failure and load fresh data. And the user can delete inputted data themselves(given you provide an endpoint for that)
Depending on the database, it is possible to make all database changes of an API reversible. For example, with SQL, you use [SQL transactions][1] using commit, rollback and savepoints. There is most likely a similar mechanism available for noSQL.
Using google analytics and it's measurement protocol, I am trying to track eCommerce transactions based on my customers (who aren't end-consumers meaning not sparse unique userid's, locations, etc...) which have a semantic idea of a "sale" with revenue.
The problem is that not all of my logged requests to the ga mp API are resulting in "rows" of transactions when looking at conversions->ecommerce->transactions. And additionally, the revenue reported is respectively missing too. An example of the discrepancy is listing all my non-zero transaction revenue API calls, I should see 321 transactions in the analytics dashboard. However, I see only 106... 30%!!! This is about the same every day even tweaking some attributes which I would think would force uniqueness of a session or transaction.
A semantic difference is that a unique consumer (cid or uid) can send a "t=transaction" with a unique "ti" (transaction id) which overlap and are not serial. I say this to suggest that maybe there is some session related deduplication happening even though my "ti" attribute is definitely unique across my notion of a "transaction". In other words, a particular cid/uid maybe have many different ti's in the same minute.
I have no google analytics javascript or client-side components in use and are simply not applicable to how I need to use google analytics which takes me to using the measurement protocol.
Using the hit-builder, /debug/collect, and logging of any http non-200 responses, I see absolutely no indication that all of my "t=transaction" messages would not be received and processed. Some of the typical debugging points I think are eliminated with this list of what I have tried
sent message via /collect
sent multiple message via /batch (t=transaction and t=item)
sent my UUID of my consumer as cid=, uid= and both
tried with and without "sc=start" to ensure there was no session deduplication of a transaction
tried with and without ua (user-agent) and uip (ip override) since it's server side but hits from consumers do come from different originations sometimes
took into consideration my timezone (UTC-8) and how my server logs these requests (UTC)
waited 24 to 48 hours to ensure data
ecommerce is turned on for my view
amount of calls to measurement protocol are < 10000 per day so I don't think I am hitting any limits
I have t=event messages too although I am taking a step back from using them for now until I can see that data is represented at least to 90%+.
Here is an example t=transaction call.
curl \
--verbose \
--request POST \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data 'ta=customer1&t=transaction&sc=start&v=1&cid=4b014cff-ccce-44c2-bfb8-e0f05fc7827c&tr=0.0&uid=4b014cff-ccce-44c2-bfb8-e0f05fc7827c&tid=UA-xxxxxxxxx-1&ti=5ef618370b01009807f780c5' \
'https://www.google-analytics.com/collect'
You've done a very good job debugging so unfortunately there isn't much left to do, a few things left to check:
View bot/spider filter: disable this option to be on the safe sife
500 hits / session: if you're sending lots of hits for the same cid/uid within 30 minutes (whatever your session-timeout is), then these would be recorded as per of the same session and thus you could reach quota limit.
10M hits / property / month: you didn't mention overall properly volume so I'm mentioning this in case
Paylod limit of 1KB = 8192 bytes: I've seen people running into that issue when tracking transactions with a crazy amount of products attached to it
Other view filters: same thing, you didn't mention so I'm mentioning just in case
Further debugging could include:
Using events instead of transactions: the problem with transactions is that they're a black box, if they don't show up you don't have any debug. I personally always track my transactions via events and I set a copy of the Ecommerce payload as event label (JSON string) for debugging, so if the event is present I know it's not a data ingestion issue but most likely my ecommerce payload which is malformed (and I have the event label to debug it), and if the event is missing then it's a data ingestion problem. See below example, replace UA-XXXXXXX-1 with your own:
v=1&t=event&tid=UA-XXXXXXX-1&cid=1373730658.1593409595&ec=Ecommerce&ea=Purchase&ti=T12345&ta=Online%20Store&tr=15.25&tt=0.00&ts=0.00&tcc=SUMMER_SALE&pa=purchase&pr1nm=Triblend%20Android%20T-Shirt&pr1id=12345&pr1pr=15.25&pr1br=Google&pr1ca=Apparel&pr1va=Gray&pr1qt=1&pr1cc=&el=%7B%22purchase%22%3A%7B%22actionField%22%3A%7B%22id%22%3A%22T12345%22%2C%22affiliation%22%3A%22Online%20Store%22%2C%22revenue%22%3A%2215.25%22%2C%22tax%22%3A%220.00%22%2C%22shipping%22%3A%220.00%22%2C%22coupon%22%3A%22SUMMER_SALE%22%7D%2C%22products%22%3A%5B%7B%22name%22%3A%22Triblend%20Android%20T-Shirt%22%2C%22id%22%3A%2212345%22%2C%22price%22%3A%2215.25%22%2C%22brand%22%3A%22Google%22%2C%22category%22%3A%22Apparel%22%2C%22variant%22%3A%22Gray%22%2C%22quantity%22%3A1%2C%22coupon%22%3A%22%22%7D%5D%7D%7D
Using a data collection platform like Segment: which will give you an extra level of debugging via their debugger (although their payload syntax is != than GA so that introduces another level of complexity, it does help me from time to time to spot issues with the underlying data though as I'm familiar with its syntax).
Currently, I am using Measurement Protocol to push the data to GA. The problem is I didn't get any response back for Success or Error on Production, If yes Please suggest?
Due to this, I am looking if there is any other options available for the same like can we achieve it using analytics 360?
The google analytics production data collection endpoint does not return a request status back (always 200 OK) by design to ensure ultra-light processing speed.
What I usually recommend to clients using Measurement Protocol server-side is to
To log a reasonable amount (or all of them) of requests somewhere. Storage is extremely cheap nowadays and knowing the data format if an emergency happens you will be able to manually extract the data
Every once in a while (one on thousand or one on a million or even more oftne depending on the importance of the data randomly) validate request on GA debug endpoint and parse the returned json. If there are any warnings or error send a notification for further investigation. This way, if anything went wrong you will be on top of the problem by the time BI & Marketing Team would be affected
I use google analytics to track user behaviour in an application. What I do is this:
Send a start session
Send a custom event "Lifetime start"
If there is an error: send a fatal exception
Send a custom event "Lifetime stop"
Send a stop session
Now when I look at the statistics, I see that a certain percentage of users had an exception over tha last 30 days. However, all users have had sessions without exception!. This is almost impossible, since I know there are users where the application crashes every time.
Is it possible that the fatal exception I submit terminates the session? So even users where the app crashes everytime get a second (short) session, containing only the "Lifetime stop" custom event? (That would explain my statistics)
That being said, I ran a test using Universal Analytics via the web (so this was not done in an app), but the results should be consistent with your setup.
I started a session and sent a "Before Exception" event which showed up in my real-time event report. I then waited a few seconds and successfully sent a fatal exception (which there is not a real-time report for). Without refreshing I then sent an "After Exception" event which came through fine in my real-time report.
From the following screenshot (User Explorer), you can see the two "Exception Test" events that I described both in the same session.
I would think that whatever fatal crash you're seeing is what is preventing additional data from appearing in Google Analytics- not that Google Analytics is ending the session at the time of the fatal exception. If it was ending the session, you'd still see events for "Lifetime stop", but it sounds like you aren't seeing those events at all.
The only things that end a GA session:
Session timeout (default: 30 minutes)
End of day
UTM/AdWords/Referral
Manually ending the session as you described
You may need to come up with some context clues to actually get to the bottom of this (maybe a remote server log?), but from the information provided (and if I'm understanding it correctly) I'm leaning towards the crash preventing the rest of the code from running.
I'm using firebase's .on() event listener to listen for entries inserted into my database in real time. I've noticed that even when a data insertion is denied, a user subscribed to the ref the data was supposed to be inserted into still gets that piece of data, even though in the database the data is never inserted.
I noticed this while developing the chat module of my web app. Here is a gif of the bug: https://gfycat.com/VariableFrailBasenji
I've set a validation rule on new messages that their length has to be under 200:
"$msg": { ".validate": "newData.val().length < 200"}
So when you see me paste in a bunch of letters, the console says the write gets denied, but the user who had the .on() subscription to that part of the database still got the message, even though it didn't get added to the database.
Anyways, this isn't really a post with a question, just wanted to share this strange bug that could potentially lead to data leakage.
The Firebase SDK uses advanced techniques like latency compensation when you write to the database. This means that before the server has acknowledged a write it goes into an in-memory database cache and optimistically assumes the write will be allowed.
When the server denies the write, it will remove the incorrectly cached data and you should see a corresponding child_removed event.