We have been testing the measurement protocol for a while now to pass Google Analytics information from our servers. We are passing our internal unique user id to tid/cid parameters and not the ga's.
Right now it looks like that in our current setup we are not able to generate any Audience reports and I would like to verify my concern that this is due the fact we are passing our own unique userId to the cid parameter.
Is it possible to generate Audience reports using the measurement protocol without ga's cid?
The way Google Analytics handles Audiences is tying the CID (or UID) with a third party cookie in the doubleclick domain, that is sent using a non-documented parameter of the measurement protocol.
This is only enabled once you turn on advertiser features in the Google Analytics interface. Because you can't access the doubleclick cookies from your server you won't be able to include that in your request.
In other words if your implementation relies solely on the measurement protocol from your server side it won't be possible to have audiences populated. If your implementation includes a mix of web tracking (using the supported client libraries such as gtag.js) and offline using measurement protocol it might be possible to make it work if you are able to use the same Ids on the offline and online tracking methods.
Related
Is it possible to use GA4 Measurement Protocol to send events to Google Analytics and view and analyze them in the GA dashboard without using gtag.js or any other front-end script? The use case would be that some events are being sent to my server and I will just push these events to GA through the API.
One thing that makes me think is that the official Measurement Protocol API say:
In order for an event to be valid, it must have a client_id that has already been used to send an event from gtag.js. You will need to capture this ID client-side and include it in your call to the measurement protocol. In send an event to your property, we use "client_id" as the client_id. You will need to replace this with a real client_id that comes from gtag.js.
(https://developers.google.com/analytics/devguides/collection/protocol/ga4/verify-implementation?client_type=gtag)
That suggests that only events that have a valid client_id that originate from gtag.js will be counted.
I did some experimenting with randomly generated client_ids and what I discovered was that I was able to see my events in the Realtime section of the GA4 console (the Event count by Event name section), but all the other sections would be empty and the Users in last 30 min section would always show 0.
Can someone please explain to me why it's zero and if such a use case is valid at all? Thanks
tl;dr
You can use any value in client_id, as long as it uniquely identifies the user (we use a GUID/UUID), but it seems like you also need to send a value in user_id. We use the same value for both.
Also, you need to add the 'engagement_time_msec' parameter to get any user metrics to register.
Longer answer:
We're trying to do the same, i.e. send all events to the GA4 Measurement Protocol from the server, so that it is not dependent on the current user's GDPR cookie settings.
We currently do this for a Universal Analytics property with no issues, but it seems that Google is trying to prevent this in future, by restricting the scope of the Measurement Protocol in GA4, whilst forcing everyone to move to it by July 1st 2023. See the documentation at https://developers.google.com/analytics/devguides/collection/protocol/ga4#full_server-to-server, where it states:
While it is possible to send events to Google Analytics solely with
measurement protocol, only partial reporting may be available. The
purpose of measurement protocol is to augment existing events
collected via gtag, GTM, or Firebase.
We have something working with GA4, in that the events are being registered on the GA4 property correctly, using a client id that is just a GUID/UUID that we define in our own site cookies. So, any value can be used in the client id, as long as it uniquely identifies the user. The same value is used to populate the user_id parameter.
When sending events, the realtime event details were showing on the GA4 dashboard, but user metrics were not until we also populated the 'engagement_time_msec' parameter, as described in https://stackoverflow.com/a/71482548/7205473
We still have issues with things like getting the user location and the platform details, which previously were automatically populated by passing the IP address and the User Agent, but which seem to no longer work in GA4.
We were also passing page load timing events through the Measurement Protocol, but again, these features seem to have been removed in GA4.
It is possible to use GA4 directly without gtag.ja or the Firebase SDK. Its not supported, so it takes some work. We have this working in a desktop app reasonably well. There a couple things that need to be done.
As stated elsewhere the "engagement_time_msec" param must be set using the "_et" parameter. This is the number of milliseconds between now and the previous event.
The client id "cid" has a specific format; it should be:
"randomNumbers(10).unixTimeStamp()"
The session id "sid" format is:
"randomNumbers(10)"
The "_z" parameter needs to be set. I think this is a cache buster. Looking deep into the gtag.js code it is a url safe base64 encoding of "CCD", which always results in the value "ccd.v9b"
The page hash parameter "_p" can be set to this; not totally sure its correct but it works.
"randomString(3).randomString(3)"
Set the "User-Agent" HTTP request header in whatever framework/lib you are using. GA4 uses this to determine many things including Operating System. You will need to create a fake user agent based on the local device information. This is what we use for a Windows 11 x64:
"myco.testapp/4.0.0 (Windows NT 10.0; Win64; x64)"
The IP will be taken fromn the web request which is where the geolocation data comes from.
Since a full working example is worth 1,000 words of documentation; here is a "test" event with a parameter "animal=dog":
https://www.google-analytics.com/g/collect?cid=0078745494.1659679529&_et=364&_p=pfJ.Aev&seg=1&sid=2678664821&tid=G-???&ul=en&v=2&_z=ccd.v9b&en=test&ep.animal=dog
It's possible to extract outgoing GA4 request from a GTM container debug/preview view and map any GA4 (automatically collected and custom) event.
Example page_view request URL:
https://www.google-analytics.com/g/collect?v=2&tid=G-XXXXXXXXXX>m=3oes1i1&_p=1545013558&_dbg=1&cid=P%2FdJWyULMwcT21TMrzn7pZdlNt%2FxtttGVqGUmqNYbhc%3D.1669722847&ul=nl-nl&sr=2560x1440&uaa=x86&uab=64&uafvl=Not_A%2520Brand%3B99.0.0.0%7CGoogle%2520Chrome%3B109.0.5414.75%7CChromium%3B109.0.5414.75&uamb=0&uam=&uap=Windows&uapv=10.0.0&uaw=0&_s=1&_uip=XXX.XXX.XXX.X&sid=1674235261&sct=1&dl=https%3A%2F%2FXXXXXXXXXX.com%2F%3Fgtm_debug%3D1674235654105&dr=https%3A%2F%2Ftagassistant.google.com%2F&dt=OM%20test&jscid=XXXXXXXXXX.1669722847&seg=1&en=page_view
Tip: use Postman to analyse and experiment with parameters
regardless of the platform used to make a call the Measurement Protocol, you should use a client id generated by gtag.js, or the app ID if using Firebase.
From backend java spring application, I need to send the following record per a specific api call to google analysitcs:
user_name,IP,time,...etc
Then I need to analyse data using one of these data (user_name,IP,time,...etc).
Can I utilize google analytics in that? or even is there any other cloud services to achieve that?
You could but you can't, that's because Analytics customers are prohibited from sending personal information to Google:
The Analytics terms of service, which all Analytics customers must adhere to, prohibits sending personally identifiable information (PII) to Analytics (such as names, social security numbers, email addresses, or any similar data), or data that permanently identifies a particular device (such as a mobile phone’s unique device identifier if such an identifier cannot be reset). Learn more about how to avoid sending PII. Your Analytics account could be terminated and your data destroyed if you use any of this information.
https://support.google.com/analytics/answer/2795983?hl=en
I was checking GA Measurement Protocol to send data to GA from backend.Which is working fine.Url i am using is:
https://www.google-analytics.com/collect?tid=UA-XXXXXXX-1&v=1&cid=9350&dp=home&t=pageview
(Please replace UA-XXXXXXX-1 with your own tracking id.)
Now i have a website which is GA enabled using javascript way.I checked in Chrome inspect that it sends information to google through url:
https://stats.g.doubleclick.net/__utm.gif?utmwv=5.6.5dc&utms=4&utmn=1588741400&............
I can use this url too to send information to google from backend. What's special and new in Measurement Protocol because using this url too i can send data to google for analytics purposes?
I need to send data from backend to GA so please guide.
The measurement protocol is now the "backbone" for data collection for Universal Analytics for all versions (web, mobile etc). Unlike the gif-method it is well documented and it can be called from every device/programming language that can send http requests (it will still return a transparent gif, though).
The main reason for using the measurement protocol is that everything else is deprecated and Google has announced that support for older tracking methods will be dropped. It looks like your second example uses an older version of the Analytics tracking code, current versions send their calls to the https://www.google-analytics.com/collect endpoint. You should not invest a lot of effort in developing solutions based on deprecated technology, especially if it offers no advantages over the current versions.
Google measurement protocol is very useful when implementing distributed solution of a service like restful APIs.
The client is not known by the server, each client world wide can call the rest API so no javascript code is possible to be injected at the client.
Measurement protocol enabled us to track which kind of devices are calling our API, how many requests per endpoint, etc...
Is there any code for using Google Analytics on a TCL-Tk program?
I've tried to implement it "handmade" using a webserver, and sending messages via TclWs but I'm sure that Google Analytics can be a better (or best) solution for this problem.
You might want to have a look at the Measurement Protocol API:
The Google Analytics Measurement Protocol allows developers to make
HTTP requests to send raw user interaction data directly to Google
Analytics servers. This allows developers to measure how users
interact with their business from almost any environment. Developers
can then use the Measurement Protocol to:
Measure user activity in new environments.
Tie online to offline
behavior.
Send data from both the web and server.
Note: The
Measurement Protocol only allows developers to collect
user-interaction (event / hit) data. It does not allow developers to
upload aggregated data like tables.
How does google analytics measure active users vs. new users in measurement protocol .
How does it measure new users? , Is it based on ipaddress ? I send the same cid=555 for all requests.
In website I see that it uses cookies though
Thank you
CID actually stands for client-id = random number generated by Google Analytics when you come to a website for the first time. You can easily grab it (even with GTM) and use it with Measurement Protocol.
If you use client-id that GA can recognize, it will stitch the data together (campaign source/medium) etc. If you use a new one (like '555' you mentioned), it will then see the incoming measurement protocol request as a request from new user.
Be smart though what you use as CID -- it the key that's vital to pairing any information you might already have about your visitors.