I'm pretty sure the answer to this question is "no", but I would like to get a definitive answer from an official source, and also understand what my alternative options might be.
Long story short, my app has old data in it that used to include user email addresses as a GET parameter. Those URLs are showing up as unique page view URLs in Google analytics, like this:
I don't want to be recording email addresses in my Google Analytics account (for privacy reasons), and I have fixed the code that was causing this in the first place, but I also want to delete or scrub the old data that currently exists in Google Analytics.
From everything I've read, it doesn't sound like this is possible without completely deleting the property, maybe even the account?
To be clear, I am NOT interested in creating new views that don't include URLs with email parameters in them, or otherwise change the view and not the data. The data needs to be gone and be completely inaccessible to anyone with access to this Google Analytics account.
Here are the options I've come up with:
Delete the property and start over. I'm pretty sure this will
actually delete the collected data, but it's not clear to me if I
would have to actually delete the account itself to achieve that.
Set the data retention time to the lowest possible value (looks like 14 months right now) and wait 14 months for it to go away https://support.google.com/analytics/answer/7667196?hl=en
Perform some kind of magic to get in contact with an actual human at Google who could help me scrub or remove this data.
Does this sound right? Are there options I'm missing? If there's a way to do this through a Google API that would not be a problem.
If this is still a relevant issue. GoogleAnalytics provides a way to delete some data. Universal Analytics https://support.google.com/analytics/answer/9450800?hl=en and GA4 https://support.google.com/analytics/answer/9940393?hl=en&ref_topic=2919631
You are right: changing the data, that you have collected, and Google Analytics have already processed, is not possible. You have the option to make changes during processing with various filters, e.g. Search-and-replace filters, but as it is written in this official support article:
Like all filters, search-and-replace filters only apply to hits
collected after you've applied the filter to the view (filters cannot
change historical data).
Regarding you suggested options:
Deleting a view or property will result in a permanent loss of data after a 35 days period of waiting time. (While this could be undone.) So unless the requirement of scrubbing the collected PII is more important than having your historical data, this should not be a way to go. The same applies to deleting the whole account, so it would be enough to delete affected properties or views.
From the article you have linked as well, you can see, that data retention is about removing user and event level data, and it will not affect the data in aggregated reports. My understanding is, that an already created, page level report will keep showing the page with an email address:
Keep in mind that standard aggregated Google Analytics reporting is
not affected.
I hope these references help you to evaluate your options. Sorry for not being able to come up with a solution, but the basic concept is, as highlighted in this Google article:
Once Analytics processes the data, it’s stored in a database where it can’t be changed
I've just set up a tool on a client site that users can use to request a quote from our client. To do this the user lands on a form page, fills in their details, submits and then lands on a thank-you page. Pretty basic.
I set this process up as a goal in Google Analytics, using the destination type goal: "begins with /thank-you" and shared that goal as a conversion in Google AdWords.
I decided to run a few Google AdWords ads to promote the tool. I also wanted to double-check the conversion data that AdWords gives you so I set the destination URL in Adwords to www.example.com/form-page?adsrc=adwords1 (2, 3, 4 etc. for each ad) and I configured the DB so that there was a column that tracked which URL the user was on when filling in the form (this would be the column I counted to get the number of conversions that came from AdWords so I could compare)
Further to this, I made sure that the initial URL parameters that the user landed on were stored in the session so that if the user browsed to other pages and came back to fill in the form later, it would still attribute the conversion to AdWords.
I tested this thoroughly on a staging and production environment and everything was working correctly.
I ran the campaign for a week and when I checked, the conversion results in the Data Base vs the ones coming from AdWords are wildly different. The DB tells me I've had 5 conversions while AdWords gives me 21.
Is there anything in the way Google uses its gclid that may be causing this issue? Or is there a problem with the way I've set up the measurement structure?
This can be caused by few things, but I think this is the GA/AdWords issue, more than your DB/session set-up.
Gclid shouldn't influence your goal, since it is used only for AdWords/Analytics interactions, Goals should not be affected in your set-up.
https://support.google.com/analytics/answer/2938246?hl=en
Probable cause: If your goal set-up only contains "begins with /thank-you", isn't it possible, that you are counting all the sessions which reach thanks-you page? Not just AdWords?
Solution: if you need to count conversions in AdWords (for performance improvements), use AdWords conversion code at the same page, this counts only those users, who clicks an ad and reach your thank-you page in x (default 30) days. Be sure to count only unique conversions (users by cookie).
Differences between GA/AdWords conversion count:
https://support.google.com/analytics/answer/2679221?hl=en
Google attributes conversions to the last marketing channel, where direct visits do not count as a marketing channel (if you look at their attribution flow visualization you see that the penultimate step is to check for existing campaign information for the user). So GA might overcount Adwords visits (or other campaigns) and conversely shows fewer conversions for direct visits.
On contrast your database probably records the last traffic channel without an elaborate attribution model, so it will show less campaign traffic.
Also IIRC the adwords interface records the conversion for the time of the ad click, not the actual goal conversion, so the timeframes for the conversions differ.
In my app, there's a requirement to track search queries, which individuals to the app. The point here is to track specifically individuals/search_queries, i.e. I want to be able to say:
User 10.20.30.40 (IP is derived from client, ok) came to my website
from Google's search results page of 'hello world' query
Is that even possible?
I investigated the problem a bit. It turned out, that when the user on Google Search result page clicks a link and gets to the underlying page, the referer doesn't contain the search query. Say, for example, I entered a "Hello world" query...
I open google and enter a "Hello world" query;
I see a serach result page with a link to wikipedia on top; I follow it
I enter "document.referrer" in consonle to see:
http://www.google.com.ua/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0CGkQFjAA&url=http%3A%2F%2Fru.wikipedia.org%2Fwiki%2FHello%2C_world!&ei=HZ0fUIXTIordtAau54GwAQ&usg=AFQjCNHSAHv8NwVNdaxMvh0OhIxs6Zb1rQ&sig2=a8tEGTBNcFWOPykloXj_Xg
There's a &q= param with a blank value, preventing me to figure out the query the user entered.
I believe Google internally uses some of the query params (i.e. ved, ei, usg or sig2) from the URL above to keep track of query used to get to the site, and thus gatehr the statistics about which queries are used mostly to the get to the website. I couldn't find any information regarding how to use them.
In the meantime, Google Analytics would only gather a general statistics, not for individuals. Yet I noticed that there's an ability to use Custom Variables. Is it possible to use them somehow in order to track query the user came with? If so, wouldn't it be a violation of Google Analytics Terms Of Service?
So again: is there a way to know a search query one came to the site with?
This is not possible. Google erases the value of q parameter, so instead of
`?q=search+query`
one always observs.
`?q=`
Though, Google provides another way (the so called convinient one) to know which queries were used to come to your website: Google Analytics.
Also, it's impossible to track user_ip/search_query pair due to Google Analytics ToS.
We are currently using Google Conversion tracking as well as Google Analytics, but would like to implement a way to track actual sales from known conversions.
My original idea was to check for a specific cookie when the user finalizes their payment, and if it's found, save the OrderID and ConversionCode to a database. This idea went up in flames, of course, when I found out that the web server cannot actually access the cookies for conversions.
My next idea is to save my own cookie whenever the google conversion cookie is written. The cookie would contain a unique ID, which would make my first idea usable again.
Does anyone see any significant issues with this idea? Is there a more effective way of doing this that I don't know about? I'm still very new to Google tools (and cookies in general), so I could be missing something.
You can set the order value via the google_conversion_value var in the tracking code. This obviously doesn't help with OrderID tracking, but should help you with your attribution model.
Further details here: http://adwords.google.com/support/aw/bin/answer.py?hl=en&answer=86277
Now, I realise the initial response to this is likely to be "you can't" or "use analytics", but I'll continue in the hope that someone has more insight than that.
Google adwords with "autotagging" appends a "gclid" (presumably "google click id") to link that sends you to the advertised site. It appears in the web log since it's a query parameter, and it's used by analytics to tie that visit to the ad/campaign.
What I would like to do is to extract any useful information from the gclid in order to do our own analysis on our traffic. The reasons for this are:
Stats are imperfect, but if we are collating them, we know exactly what assumptions we have made, and how they were calculated.
We can tie the data to the rest of our data and produce far more accurate stats wrt conversion rate.
We don't have to rely on javascript for conversions.
Now it is clear that the gclid is base64 encoded (or some close variant), and some parts of it vary more than others. Beyond that, I haven't been able to determine what any of it relates to.
Does anybody have any insight into how I might approach decoding this, or has anybody already related gclids back to compaigns or even accounts?
I have spoken to a couple of people at google, and despite their "don't be evil" motto, they were completely unwilling to discuss the possibility of divulging this information, even under an NDA. It seems they like the monopoly they have over our web stats.
By far the easiest solution is to manually tag your links with Google Analytics campaign tracking parameters (utm_source, utm_campaign, utm_medium, etc.) and then pull out that data.
The gclid is dependent on more than just the adwords account/campaign/etc. If you click on the same adwords ad twice, it could give you different gclids, because there's all sorts of session and cost data associated with that particular click as well.
Gclid is probably not 100% random, true, but I'd be very surprised and concerned if it were possible to extract all your Adwords data from that number. That would be a HUGE security flaw (i.e. an arbitrary user could view your Adwords data). More likely, a pseudo-random gclid is generated with every impression, and if that ad is clicked on, the gclid is logged in Adwords (otherwise it's thrown out). Analytics then uses that number to reconcile the data with Adwords after the fact. Other than that, there's no intrinsic value in the gclid number itself.
In regards to your last point, attempting to crack or reverse-engineer this information is explicitly forbidden in both the Google Analytics and Google Adwords Terms of Service, and is grounds for a permanent ban. Additionally, the TOS that you agreed to when signing up for these services says that it is not your data to use in any way you feel like. Google is providing a free service, so there are strings attached. If you don't like not having complete control over your data, then there are plenty of other solutions out there. However, you will pay a premium for that kind of control.
Google makes nearly all their money from selling ads. Adwords is their biggest money-making product. They're not going to give you confidential information about how it works. They don't know who you are, or what you're going to do with that information. It doesn't matter if you sign an NDA and they have legal recourse to sue you; if you give away that information to a competitor, your life isn't worth enough to pay back the money you will have lost them.
Sorry to break it to you, but "Don't be Evil" or not, Google is a business, not a charity. They didn't become one of the most successful companies in the world by giving away their search algorithm to the first guy who asked for it.
The gclid parameter is encoded in Protocol Buffers, and then in a variant of Base64.
See this guide to decoding the gclid and interpreting it, including an (Apache-licensed) PHP function you can use.
There are basically 3 parameters encoded inside it, one of which is a timestamp. The other 2 as yet are not known.
As far as understanding what these other parameters mean—it may be helpful to compare it to the ei parameter, which is encoded in an extremely similar way (basically Protocol Buffers with the keys stripped out). The ei parameter also has a timestamp, with what seem to be microseconds, and 2 other integers.
FYI, I just posted a quick analysis of some glcid data from my sites on this post. There definitely is some structure to the gclid, but it is difficult to decipher.
I think you can get all the goodies linked to the gclid via google's adword api. Specifically, you can query the click performance report.
https://developers.google.com/adwords/api/docs/appendix/reports#click
I've been working on this problem at our company as well. We'd like to be able to get a better sense of what our AdWords are doing but we're frustrated with limitations in Analytics.
Our current solution is to look in the Apache access logs for GET requests using the regex:
.*[?&]gclid=([^$&]*)
If that exists, then we look at the referer string to get the keyword:
.*[?&]q=([^$&]*).*
An alternative option is to change your Apache web log to start logging the __utmz cookie that google sets, which should have a piece for the keyword in utmctr. Google __utmz cookie and you should be able to find plenty of information.
How accurate is the referer string? Not 100%. Firewalls and security appliances will strip it out. But parsing it out yourself does give you more flexibility than Google Analytics. It would be a great feature to send the gclid to AdWords and get data back, but that feature does not look like it's available.
EDIT: Since I wrote this we've also created our own tags that are appended to each destination url as a request parameter. Each tag is just an md5 hash of the text, ad group, and campaign name. We grab it using regex from the access log and look it up in a SQL database.
This is a non-programmatic way to decode the GCLID parameter. Chances are you are simply trying to figure out the campaign, ad group, keyword, placement, ad that drove the click and conversion. To do this, you can upload the GCLID into AdWords as a separate conversion type and then segment by conversion type to drill down to the criteria that triggered the conversion. These steps:
In AdWords UI, go to Tools->Conversions->Add conversion with source "Import from clicks"
Visit the AdWords help topic about importing conversions https://support.google.com/adwords/answer/7014069 and create a bulk load file with your GCLID values, assigning the conversions to you new "Import from clicks" conversion type
Upload the conversions into AdWords in Tools->Conversions->Conversion actions (Uploads) on left navigation
Go to campaigns tab, Segment->Conversions->Conversion name
Find your new conversion name in the segment list, this is where the conversion came from. Continue this same process on the ad groups and keywords tab until you know the GCLID originating criteria
Well, this is no answer, but the approach is similar to how you'd tackle any cryptography problem.
Possibility 1: They're just random, in which case, you're screwed. This is analogous to a one-time pad.
Possibility 2: They "mean" something. In that case, you have to control the environment.
Get a good database of them. Find gclids for your site, and others. Record all times that all clicks occur, and any other potentially useful data
Get cracking! As you have started already, start regressing your collected data against your known, and see if you can find patterns used decrypting techniques
Start scraping random gclid's, and see where they take you.
I wouldn't hold high hope for this to be successful though, but I do wish you luck!
Looks like my rep is weak, so I'll just post another answer rather than a comment.
This is not an answer, clearly. Just voicing some thoughts.
When you enable auto tagging in Adwords, the gclid params are not added to the destination URLs. Rather they are appended to the destination URLs at run time by the Google click tracking servers. So, one of two things is happening:
The click servers are storing the gclid along with Adwords entity identifiers so that Analytics can later look them up.
The gclid has the entity identifiers encoded in some way so that Analytics can decode them.
From a performance perspective it seems unlikely that Google would implement anything like option 1. Forcing Analytics to "join" the gclid to Adwords IDs seems exceptionally inefficient at scale.
A different approach is to simply look at the referrer data which will at least provide the keyword which was searched.
Here's a thought: Is there a chance the gclid is simply a crytographic hash, a la bit.ly or some other URL shortener?
In which case the contents of the hashed text would be written to a database, and replaced with a unique id.
Afterall, the gclid is shortening a bunch of otherwise long text.
Takes this example:
www.example.com?utm_source=google&utm_medium=cpc
Is converted to this:
www.example.com?gclid=XDF
just like a URL shortener.
One would need a substitution cipher in order to reverse engineer the cryptographic hash... not as easy task: https://crypto.stackexchange.com/questions/300/reverse-engineering-a-hash
Maybe some deep digging into logs, looking for patterns, etc...
I agree with Ophir and Chris. My feeling is that it is purely a serial number / unique click ID, which only opens up its secrets when the Analytics and Adwords systems talk to each other behind the scenes.
Knowing this, I'd recommend looking at the referring URL and pulling as much as possible from this to use in your back end click tracking setup.
For example, I live in NZ, and am using Firefox. This is a search from the Firefox Google toolbar for "stack overflow":
http://www.google.co.nz/search?q=stack+overflow&ie=utf-8&oe=utf-8&aq=t&client=firefox-a&rlz=1R1GGLL_en-GB
You can see that: a) im using .NZ domain, b) my keyword "stack+overflow", c) im running firefox.
Finally, if you also stash the full landing page URL, you can store the GCLID, which will tell you the visitor came from paid, whereas if it doesn't have a GCLID, then the user must have come from natural search (if URL tagging is enabled of course).
This would theoretically allow you to then search for the keyword in your campaign, and figure out which adgroup them came from. Knowing the creative would probably be impossible though, unless you split test your landing URLs or tag them somehow.