GA to BigQuery export. Can't change a dataset localization - google-analytics

I faced with the problem regarding dataset localization.
We configured the default GA360 to BigQuery export. It stored the data in dataset created in US (by default).
Then we decided to move this dataset to EU. We used this document for changing data localization: https://support.google.com/analytics/answer/7584198?hl=en&ref_topic=3416089#zippy=%2Cin-this-section%2Clocalizing-existing-exports
So I backed up the existing data, dropped the US dataset and created the dataset with the same id in EU.
Then we linked the GA view back to BigQuery project.
But during data export we started getting the error message like:
{
"protoPayload": {
"#type": "type.googleapis.com/google.cloud.audit.AuditLog",
"status": {
"code": 7,
"message": "IAM setPolicy failed for Dataset <PROJECT_NAME>:<VIEW_ID>: One or more users named in the policy do not belong to a permitted customer."
},
"authenticationInfo": {
"principalEmail": "analytics-processing-dev#system.gserviceaccount.com"
},
"requestMetadata": {
"requestAttributes": {},
"destinationAttributes": {}
},
"serviceName": "bigquery.googleapis.com",
"methodName": "datasetservice.insert",
"authorizationInfo": [
{
"resource": "projects/<PROJECT_NAME>",
"permission": "bigquery.datasets.create",
"granted": true,
"resourceAttributes": {}
}
],
"resourceName": "projects/<PROJECT_ID>/datasets",
"serviceData": {
"#type": "type.googleapis.com/google.cloud.bigquery.logging.v1.AuditData",
"datasetInsertRequest": {
"resource": {
"datasetName": {
"projectId": "<PROJECT_ID>",
"datasetId": "<VIEW_ID>"
},
"info": {},
"acl": {}
}
},
"datasetInsertResponse": {
"resource": {
"datasetName": {},
"info": {},
"acl": {}
}
}
},
"resourceLocation": {
"currentLocations": [
"US"
]
}
},
"insertId": "<INSERT_ID>",
"resource": {
"type": "bigquery_resource",
"labels": {
"project_id": "<PROJECT_NAME>"
}
},
"timestamp": "2022-02-10T10:44:23.722901Z",
"severity": "ERROR",
"logName": "projects/<PROJECT_NAME>/logs/cloudaudit.googleapis.com%2Factivity",
"receiveTimestamp": "2022-02-10T10:44:24.043846681Z"
}
The IAM setPolicy failed for Dataset <PROJECT_NAME>:<VIEW_ID>: One or more users named in the policy do not belong to a permitted customer. message basically means that our service account doesn't have permission to act on this dataset.
But if I drop this dataset from EU and recreate it back in US, the relinking succeeds.
I was looking into the constraints/iam.allowedPolicyMemberDomains, but if it were the case it wouldn't allow actions on both US and EU region datasets (not only on EU one).
I would appreciate any help,
Thanks!

So the case was precisely in constraints/iam.allowedPolicyMemberDomains policy.
We tried suggestions from from doc, but eventually, we had to disable this policy.

Related

Discrepancy between UI and API for transactions/purchase events for GA4

My goal is to pull in purchase transaction count and transaction revenue from a client, segmented by Google Ads campaign ID. My current query looks like the following:
{
"propertyId": "*********",
"query": {
"dateRanges": [
{
"startDate": "2022-09-30",
"endDate": "2022-10-06"
}
],
"dimensions": [
{
"name": "googleAdsCampaignId"
},
{
"name": "googleAdsCampaignName"
}
],
"metrics": [
{
"name": "advertiserAdClicks"
},
{
"name": "advertiserAdCost"
},
{
"name": "transactions"
},
{
"name": "purchaseRevenue"
}
],
"dimensionFilter": {
"andGroup": {
"expressions": [
{
"filter": {
"fieldName": "googleAdsCustomerId",
"stringFilter": {
"matchType": "EXACT",
"value": "*********",
"caseSesnsitive": false
}
}
}
]
}
}
}
}
What I expect when querying the 'transactions' dimension is, as the API schema describes, "The count of transaction events with purchase revenue. Transaction events are in_app_purchase, ecommerce_purchase, purchase, app_store_subscription_renew, app_store_subscription_convert, and refund."
The response from my query comes back with these numbers, for an example campaign:
{
"dimensionValues": [
{
"value": "***********",
"oneValue": "value"
},
{
"value": "Example Campaign",
"oneValue": "value"
}
],
"metricValues": [
{
"value": "2482480",
"oneValue": "value"
},
{
"value": "6492393600000",
"oneValue": "value"
},
{
"value": "331",
"oneValue": "value"
},
{
"value": "31374.205645000002",
"oneValue": "value"
}
]
}
However, if, in the GA4 dashboard, I attempt to view a report of purchase conversions by campaign over the same date range, this is what is displayed for the 'example campaign':
6 Example Campaign 239.47 31,981.63
Where 239.47 is the number of transactions, and 31,981.63 is the event value (transaction revenue). Notably, the transactions are off by over 25%. The revenue/event value is similar but also off by a slight amount. This is consistent for all campaigns under the client, with the API response being significantly (but by varying percentages) higher. The dashboard value is always lower. These numbers don't change if the report is run on different dates.
Additionally, I suspected that there may be some additional event being tracked under the transactions api field that was not displaying in the dashboard, so I also tried adding this filter to my query:
{
"filter": {
"fieldName": "eventName",
"stringFilter": {
"matchType": "EXACT",
"value": "purchase",
"caseSesnsitive": false
}
}
}
the transactions field still came back as 331.
What I want to figure out is if I'm querying the wrong field, the frontend is under reporting data, or the API is over reporting data. I found that I was not able to post on the official GA4 Issue tracker so I've come here.
Someone better versed in GA4 at my company explained that the issue here had to do with dimension scopes. The transactions field is session scoped, but the dimensions I was pulling in were event scoped. This means that the numbers being output were basically meaningless. As a matter of fact through the dashboard, it won't let you create a free-form explore report using the combination of fields that I did in my query, explaining that the metrics are incompatible. It appears to just be a bug that the API would allow that to happen.
As an alternative, I'm now pulling the conversions field, with a filter on eventName as listed above, including only purchase events.

Landing Zone Automation - Disable Owner requirement from ESLZ ARM template

I am trying to deploy ESLZ Arm template in this link "https://github.com/Azure/Enterprise-Scale/blob/main/docs/reference/adventureworks/README.md" and it requires owner permission to do that. Is it possible to remove the Global Admin and/or Owner requirement and run the template using contributor role
I created a management group under tenant root and assigned contributor role. Now I'm trying to create additional management groups using below ARM template
{
"$schema": "https://schema.management.azure.com/schemas/2019-08-01/tenantDeploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"topLevelManagementGroupPrefix": {
"type": "string",
"metadata": {
"description": "Provide prefix for the management group structure."
}
},
"platformMgs": {
"type": "array",
"defaultValue": [
"management",
"connectivity",
"identity"
],
"metadata": {
"description": "Management groups for platform specific purposes, such as management, networking, identity etc."
}
},
"landingZoneMgs": {
"type": "array",
"defaultValue": [
"online",
"corp"
],
"metadata": {
"description": "These are the landing zone management groups."
}
}
},
"variables": {
"enterpriseScaleManagementGroups": {
"ESLZ": "[concat(parameters('topLevelManagementGroupPrefix'))]",
"platform": "[concat(parameters('topLevelManagementGroupPrefix'), '-', 'platform')]"
}
},
"resources": [
{
// Create management group for platform management groups
"type": "Microsoft.Management/managementGroups",
"apiVersion": "2020-05-01",
"scope": "/",
"name": "[variables('enterpriseScaleManagementGroups').platform]",
"properties": {
"displayName": "[variables('enterpriseScaleManagementGroups').platform]",
"details": {
"parent": {
"id": "[tenantResourceId('Microsoft.Management/managementGroups/', parameters('topLevelManagementGroupPrefix'))]"
}
}
}
}
],
"outputs": {}
}
While deploying the template i'm getting permission error, however able to create management group manually. Am I missing something in this template. Any help is really appreciated
You should be able to be able to deploy the template with contributor permissions at the tenant level... You need an owner/userAccessAdmin/global admin to grant those permissions.

Exception in function does not return exceptions on functions in function monitor

The azure function is a .net core class library that will receive the message based on the namespace of the model being sent (in the filter as eventType) as an . All deployments are being done using arm templates, which is where this struggle is originating from. The function and eventgrid are deployed fine, but I don't know what i'm doing wrong with the subscription. If I create the subscription in the portal then the handler receives the message and displays traffic on the monitor. If I create the subscription as below then it appears exactly the same in the portal as the portal created one but nothing shows up in the monitor. Am I missing a resource or connection that still needs to be created? I read about system topics and how they're made implicitly in some instances but can be made explicitly, is that what I'm missing? This would be easier to debug if there was a place to export the template for those subscriptions but I don't see them.
Function handler
[FunctionName("FunctionName")]
public async Task Run([EventGridTrigger]EventGridEvent eventGridEvent)
{
...
}
}
eventgrid creation
{
"type": "Microsoft.EventGrid/topics",
"apiVersion": "2020-06-01",
"name": "[variables('EventGridName')]",
"location": "[resourceGroup().location]"
}
subscription creations
{
"name": "[concat(variables('eventSubscriptions')[copyIndex()].eventGridName, '/Microsoft.EventGrid/', variables('eventSubscriptions')[copyIndex()].name)]",
"type": "Microsoft.EventGrid/topics/providers/eventSubscriptions",
"apiVersion": "2020-01-01-preview",
"location": "[resourceGroup().location]",
"copy": {
"name": "subscriptionCopy",
"count": "[length(variables('eventSubscriptions'))]"
},
"properties": {
"topic": "[concat('/subscriptions/', subscription().subscriptionId,'/resourcegroups/', resourceGroup().name, '/providers/Microsoft.EventGrid/topics/', variables('eventSubscriptions')[copyIndex()].eventGridName)]",
"destination": {
"endpointType": "AzureFunction",
"properties": {
"resourceId": "[concat('/subscriptions/', subscription().subscriptionId,'/resourcegroups/', resourceGroup().name, '/providers/Microsoft.Web/sites/', variables('eventSubscriptions')[copyIndex()].functionApp, '/functions/' , variables('eventSubscriptions')[copyIndex()].functionName)]",
"maxEventsPerBatch": 1,
"preferredBatchSizeInKilobytes": 64
}
},
"filter": {
"includedEventTypes": [
"[variables('eventSubscriptions')[copyIndex()].eventType]"
]
},
"labels": [],
"eventDeliverySchema": "EventGridSchema"
},
"dependsOn": [
]
}

Logic App workflow creating or updating documents in CosmosDB fails with conflict

I'm in the process of creating a Logic App which should copy all documents from specific containers in the database to a different database and its corresponding collections.
For this, I've created a workflow which retrieves all documents, loops through them and tries to do a Create or Update document on the target location.
As you can see on the image, this looks fairly straightforward.
I've specified the PartitionKey in the header and Upsert is set to true, so if a document already exists it'll be updated, otherwise created.
Originally, the body was filled with #items('[blurred]_-_For_each_document').
However, I got an error stating there was a conflict on the id.
I've also tried to remove all 'system' properties like so:
#removeProperty(removeProperty(removeProperty(removeProperty(removeProperty(items('[blurred]_-_For_each_document'), 'id'), '_rid'), '_self'), '_ts'), '_etag')
but it appears not having an id in place isn't valid, so now I've got this as I'm only interested in having the contents of the 'actual' document:
#removeProperty(removeProperty(removeProperty(removeProperty(items('[blurred]_-_For_each_document'), '_rid'), '_self'), '_ts'), '_etag')
This still fails.
The raw input looks a bit like this:
{
"method": "post",
"headers": {
"x-ms-documentdb-raw-partitionkey": "1050",
"x-ms-documentdb-is-upsert": "True"
},
"path": "/dbs/[myDatabase]/colls/[myCollection]/docs",
"host": {
"connection": {
"name": "/subscriptions/[subscriptionGuid]/resourceGroups/[resourcegroup]/providers/Microsoft.Web/connections/[connectionName]"
}
},
"body": {
"id": "2faee185-4a51-4797-bff9-3ce23a603690",
"MyPartitionKeyNumber": 1050,
"SomeValue": false,
}
}
With the following response:
{
"code": "Conflict",
"message": "Entity with the specified id already exists in the system., ... ResourceType: Document, OperationType: Upsert ..."
}
I know for a fact, the id doesn't exist, but changed the step to do a PUT anyway.
{
"method": "put",
"headers": {
"x-ms-documentdb-raw-partitionkey": "1050",
"x-ms-documentdb-is-upsert": "True"
},
"path": "/dbs/[myDatabase]/colls/[myCollection]/docs",
"host": {
"connection": {
"name": "/subscriptions/[subscriptionGuid]/resourceGroups/[resourcegroup]/providers/Microsoft.Web/connections/[connectionName]"
}
},
"body": {
"id": "2faee185-4a51-4797-bff9-3ce23a603690",
"MyPartitionKeyNumber": 1050,
"SomeValue": false,
}
}
With a response I'm expecting to see:
{
"statusCode": 404,
"message": "Resource not found"
}
I've also tried to do a Delete document before running the Create or Update document step, but got the same error(s).
There's some post here on Stack Overflow stating the x-ms-documentdb-raw-partitionkey should be x-ms-documentdb-partitionkey (without the raw-part), but doing so results in the following error message:
PartitionKey extracted from document doesn't match the one specified in the header
For completeness sake, these are the relevant steps of my workflow:
"[blurred]_-_Get_all_documents": {
"type": "ApiConnection",
"inputs": {
"host": {
"connection": {
"name": "#parameters('$connections')['documentdb_3']['connectionId']"
}
},
"method": "get",
"path": "/v2/dbs/#{encodeURIComponent('myDatabase')}/colls/#{encodeURIComponent(items('[blurred]_-_For_each_collection'))}/docs"
},
"runAfter": {}
},
"[blurred]_-_For_each_document": {
"type": "Foreach",
"foreach": "#body('[blurred]_-_Get_all_documents')?['value']",
"actions": {
"[blurred]_-_Create_or_Update_document_on_blurred2": {
"type": "ApiConnection",
"inputs": {
"host": {
"connection": {
"name": "#parameters('$connections')['documentdb_4']['connectionId']"
}
},
"method": "post",
"body": "#removeProperty(removeProperty(removeProperty(removeProperty(items('[blurred]_-_For_each_document'), '_rid'), '_self'), '_ts'), '_etag')",
"headers": {
"x-ms-documentdb-raw-partitionkey": "#{items('[blurred]_-_For_each_document')['MyPartitionKeyNumber']}",
"x-ms-documentdb-is-upsert": true
},
"path": "/dbs/#{encodeURIComponent('myDatabase')}/colls/#{encodeURIComponent(items('[blurred]_-_For_each_collection'))}/docs"
},
"runAfter": {}
}
},
"runAfter": {
"[blurred]_-_Get_all_documents": [
"Succeeded"
]
}
}

JAGQL - Why do I need an id for a post call?

I'm using JAGQL to build a JSON API compatible express server. My database behind it is MongoDB (jsonapi-store-mongodb). I posted my question here as well: https://github.com/holidayextras/jsonapi-store-mongodb/issues/59
According to the JAGQL documentation, https://jagql.github.io/pages/project_setup/resources.html#generateid,
I am told that
generateId
By default, the server autogenerates a UUID for resources which are created without specifying an ID. To disable this behavior (for example, if the database generates an ID by auto-incrementing), set generateId to false. If the resource's ID is not a UUID, it is also necessary to specify an id attribute with the correct type. See /examples/resorces/autoincrement.js for an example of such a resource.
But when I send a POST request to one of my resources, I get this:
"jsonapi": {
"version": "1.0"
},
"meta": {},
"links": {
"self": "/myresource"
},
"errors": [
{
"status": "403",
"code": "EFORBIDDEN",
"title": "Param validation failed",
"detail": [
{
"message": "\"id\" is required",
"path": [
"id"
],
"type": "any.required",
"context": {
"key": "id",
"label": "id"
}
}
]
}
]
What am I missing?
See here for more details: https://github.com/jagql/framework/issues/106
In your resource definition, you want to add primaryKey: 'uuid':
{
resource: 'things',
handlers,
primaryKey: 'uuid',
attributes: {
...
}
}

Resources