Have been looking for cluster configs in JSON format to create a dataproc cluster(GCE) with Dataproc Metastore service and Spark-BQ dependency jars, unable to find any reference document that specifies how to use those JSON configs.
I have looked through below links :
https://airflow.apache.org/docs/apache-airflow/1.10.6/_api/airflow/contrib/operators/dataproc_operator/index.html
https://cloud.google.com/dataproc/docs/reference/rest/v1/projects.regions.clusters
https://cloud.google.com/dataproc/docs/reference/rest/v1/MetastoreConfig
but it does not specify GCE cluster configs, its REST API and GKE cluster configs
Please see below configs that I am trying out to create a dataproc cluster :
CLUSTER_CONFIG = {
"gce_cluster_config": {
"internal_ip_only": True,
"metadata": {
"spark-bigquery-connector-version": spark_bq_connector_version
},
"service_account_scopes": [
service_account_scopes
],
"subnetwork_uri": subnetwork_uri,
"zone_uri": zone_uri
},
"initialization_actions": [
{
"executable_file": initialization_actions,
"execution_timeout": execution_timeout
}
],
"master_config": {
"disk_config": {
"boot_disk_size_gb": master_boot_disk_size_gb
},
"machine_type_uri": master_machine_type_uri
},
"metastore_config": {
"dataproc_metastore_service": dataproc_metastore
},
"software_config": {
"image_version": software_image_version
},
"worker_config": {
"disk_config": {
"boot_disk_size_gb": worker_boot_disk_size_gb
},
"machine_type_uri": worker_machine_type_uri,
"num_instances": worker_num_instances
}
}
Any lead would be really appreciated, please attach links to refer full config examples
Thanks !
As mentioned in this doc, external Hive metastore (non Dataproc Metastore service) needs to be specified through the hive:hive.metastore.uris property. Note the hive: prefix.
When creating the cluster with gcloud, if you add --log-http:
$ gcloud dataproc clusters create ... \
--properties hive:hive.metastore.uris=thrift://my-metastore:9083 \
--log-http
it will show you the actual HTTP request:
{
"clusterName":"...",
"config":{
"endpointConfig":{
"enableHttpPortAccess":true
},
"gceClusterConfig":{
"internalIpOnly":false,
"serviceAccountScopes":[
"https://www.googleapis.com/auth/cloud-platform"
],
"zoneUri":"us-west1-a"
},
"masterConfig":{
"diskConfig":{
},
"machineTypeUri":"e2-standard-2"
},
"softwareConfig":{
"imageVersion":"1.5",
"properties":{
"hive:hive.metastore.uris":"thrift://my-metastore:9083"
}
},
"workerConfig":{
"diskConfig":{
},
"machineTypeUri":"e2-standard-2"
}
},
"projectId":"..."
}
You can also find the request spec in the Dataproc REST API doc.
Related
I have a different google-service.json file for every releaseChannel (a file for prod, stage, QA, and dev)
for more details I want to implement push notifications but in a different environment.
cuz I don't want to send a test notification in the QA and the notification sent to the prod users!
this is the android config in app.json
"android": {
"googleServicesFile": "./google-services.json",
"adaptiveIcon": {
"foregroundImage": "./src/assets/adaptive-icon.png"
},
"permissions": [
"android.permission.CAMERA",
"android.permission.ACCESS_FINE_LOCATION",
"android.permission.WRITE_EXTERNAL_STORAGE",
"android.permission.READ_EXTERNAL_STORAGE",
"android.permission.ACTION_BOOT_COMPLETED",
"android.permission.RECORD_AUDIO",
"com.google.android.gms.permission.AD_ID"
],
"package": "com.beyondbelievers.awal",
"versionCode": 1
},Ï
is there any way I can load the different files with every environment?
this answer inspired from an old question you can check it here
after I searched the whole internet I found a way but using app.config.js
I will try to explain how I archive my goal in the following steps:
convert app.json to app.config.js: here's expo docs explain how to migrate from app.json.
then in my eas.json I added the following line to all profiles "APP_VARIANT": "qa":
{
"qa": {
...
"releaseChannel": "qa",
"env": {
"APP_VARIANT": "qa"
}
...
},
"stage": {
...
"releaseChannel": "stage",
"env": {
"APP_VARIANT": "stage"
}
...
},
"prod": {
...
"releaseChannel": "prod",
"env": {
"APP_VARIANT": "prod"
}
...
}
}
}
the "APP_VARIANT" variable contains the type of env and based on it I can later check which file to us.
now back to the app.config.js add the following lines:
// this will return the value of APP_VARIANT defined in eas.json
const APP_RELEASE_CHANNEL = process.env.APP_VARIANT;
// define the AndroidGoogleServicesFile variable to use instead of the normal string and assign a default value to it
let AndroidGoogleServicesFile = "./google-services-dev.json";
// now check the value of the APP_VARIANT and based on it assign the path of the google-services you wanna use
if (APP_RELEASE_CHANNEL === "qa") {
AndroidGoogleServicesFile = "./google-services-qa.json";
} else if (APP_RELEASE_CHANNEL === "stage") {
AndroidGoogleServicesFile = "./google-services-stage.json";
} else if (APP_RELEASE_CHANNEL === "prod") {
AndroidGoogleServicesFile = "./google-services-prod.json";
}
now you know which file you need to use one step left is to assign that file so your app can use it
android: {
...
googleServicesFile: AndroidGoogleServicesFile,
...
},
this is how I solve my problem I hope this solves your problem too.
If you didn't understand something I welcomed to explain in more details
I'm using GithubAuthProvider with the added scope repo to get the user's access token which is later used to access the GitHub GraphQL API (the GitHub App has the permissions for Contents and Metadata set to Read-only).
The problem is when I'm trying to list private repos. The API returns an empty array as if I don't have the required permissions. Moreover, when I try to list branches of a user's repo it returns an error of type FORBIDDEN.
Query:
query {
viewer {
repository(name: "some-repo") {
refs(refPrefix: "refs/heads/", first: 10) {
nodes {
name
}
}
}
}
}
Response:
{
"data": {
"viewer": {
"repository": {
"refs": null
}
}
},
"errors": [
{
"type": "FORBIDDEN",
"path": [
"viewer",
"repository",
"refs"
],
"extensions": {
"saml_failure": false
},
"locations": [
{
"line": 7,
"column": 7
}
],
"message": "Resource not accessible by integration"
}
]
}
What am I missing?
For GitHub GraphQL API, only scope repo may not be enough.
Following are scopes requested for reference. Authenticating with GraphQL
user
public_repo
repo
repo_deployment
repo:status
read:repo_hook
read:org
read:public_key
read:gpg_key
The API returns an empty array as if I don't have the required permissions.
I wonder whether you are query under the viewer. viewer can get the private repos he/she owned. For repos owned by other people, you can try
repository(name: "repo-name", owner: "login") {
name
}
It will return NOT_FOUND error if you don't have the required permissions.
Your query works fine for me :)
It turns out I read through the Firebase instructions too fast and created a Github App instead of an OAuth App.
It's now working as it should.
I am deploying an ARM template with Terraform.
We deploy all our Azure infra with Terraform but for AKS there are some preview features which are not in terraform yet so we want to deploy an AKS cluster with an ARM template.
If I create a Log Analytics workspace with TF, how can I pass the workspace id to ARM.
resource "azurerm_resource_group" "test" {
name = "k8s-test-bram"
location = "westeurope"
}
resource "azurerm_log_analytics_workspace" "test" {
name = "lawtest"
location = "${azurerm_resource_group.test.location}"
resource_group_name = "${azurerm_resource_group.test.name}"
sku = "PerGB2018"
retention_in_days = 30
}
So here is a snippet of the AKS ARM where I want to enable monitoring and I refer to the workspaceresourceId. But how do I define/declare the parameter to get the id from the workspace that I created with TF
"properties": {
"kubernetesVersion": "[parameters('kubernetesVersion')]",
"enableRBAC": "[parameters('EnableRBAC')]",
"dnsPrefix": "[parameters('DnsPrefix')]",
"addonProfiles": {
"httpApplicationRouting": {
"enabled": false
},
omsagent": {
"enabled": true,
"config": {
"logAnalyticsWorkspaceResourceID": "[parameters('workspaceResourceId')]"
}
}
},
you could use the parameters property of the azurerm_template_deployment deployment to pass in parameters:
parameters = {
"workspaceResourceId" = "${azurerm_log_analytics_workspace.test.id}"
}
I think it should look more or less like that, here's the official doc on this.
I'm referring to use Openstack API to get quota usage from Openstack Cloud. I did exactly as document at https://developer.openstack.org/api-ref/compute/.
But It didn't work, with api:
<host>/v2/{tenant_id}/os-quota-sets/{tenant_id}/detail
or
<host>/v2/{tenant_id}/os-quota-sets/detail
It worked with api:
<host>/v2/{tenant_id}/os-quota-sets/{tenant_id}
But, I want to get details. Did I do anything wrong?
OpenStack client can be used... And you can use command line tools.... Below is the link which can help you.
https://docs.openstack.org/nova/pike/admin/quotas.html
You can install OpenStack SDK, you can go through API documentation section for networking.
Below is the link:
https://docs.openstack.org/openstacksdk/latest/user/proxies/network.html#openstack.network.v2._proxy.Proxy.update_quota
You may find methods like:
delete_quota(quota, ignore_missing=True)
get_quota(quota, details=False)
API for getting project quota can be called as,
requests.get('http://'+url+':8774/v2.1/os-quota-sets/'+projectid+'/detail',headers={'content-type': 'application/json', 'X-Auth-Token': token})
You will have to pass your project id in path and 'X-Auth-Token' parameter in headers which can be extracted as,
url = [Your Cloud Machine IP Goes here]
def gettokenForUser(username,password):
payload1 = {
"auth": {
"identity": {
"methods": [
"password"
],
"password": {
"user": {
"name": username,
"domain": {
"name": "Default"
},
"password": password
}
}
}
}
}
response = requests.post('http://'+url+':5000/v3/auth/tokens',
headers={'content-type': 'application/json'},
data=json.dumps(payload1))
return response.headers['X-Subject-Token']
I am following campaign measurement guide to do
https://developers.google.com/analytics/devguides/collection/android/v4/campaigns
Google Play Campaign Attribution
General Campaign & Traffic Source Attribution
For the Google Play Campaign Attribution, after adding the campaign service and receiver to the androidManifest.xml, i followed the testing guide.
https://developers.google.com/analytics/solutions/testing-play-campaigns
In the log successfully received the install referrer intent.
07-29 17:05:32.968 8333-8363/? D/GAv4: Received installation campaign: content=test_content, keyword=test_term, medium=test_medium, name=test_name, source=test_source
But in my google analytic account, Acquisition -> Sources -> All, nothing shows up. Can anyone have any idea what is the root cause?
I also implemented the hit screen analytic, this is working.
My google-services.json is below:
{
"project_info": {
"project_number": "123456788680",
"project_id": "test-app-project"
},
"client": [
{
"client_info": {
"mobilesdk_app_id": "1:12345678680:android:488edac1c6c2df62",
"android_client_info": {
"package_name": "com.example.test.analytic"
}
},
"oauth_client": [],
"api_key": [
{
"current_key": "xxxxxxxx_xxxxxxxxxxxxxxKoT0XEkMgOCUe0c"
}
],
"services": {
"analytics_service": {
"status": 2,
"analytics_property": {
"tracking_id": "UA-xxxxxxxx-2"
}
},
"appinvite_service": {
"status": 1,
"other_platform_oauth_client": []
},
"ads_service": {
"status": 1
}
}
},
{
"client_info": {
"mobilesdk_app_id": "1:12345678680:android:6092a3d09b6b18d2",
"android_client_info": {
"package_name": "com.example.test"
}
},
"oauth_client": [],
"api_key": [
{
"current_key": "xxxxxxxx_xxxxxxxxxxxxxxKoT0XEkMgOCUe0c"
}
],
"services": {
"analytics_service": {
"status": 2,
"analytics_property": {
"tracking_id": "UA-xxxxxxxx-2"
}
},
"appinvite_service": {
"status": 1,
"other_platform_oauth_client": []
},
"ads_service": {
"status": 1
}
}
}
],
"configuration_version": "1"
}
I modified the project name and the key to post here. In my actual project, i didn't modify anything. There are 2 clients with different package name, because i have 2 projects, one is the main entry app and another is the library. All both needs to send info the google analytic. But both in one application, so I set it to use the same analytic property when request configuration form google. Is that impact? I think it shouldn't.
Can anyone help me?
For me, the information was not sent before building the tracker for the first time.
After sending the broadcast I also only get:
D/GAv4: Received installation campaign: content=test_content, keyword=test_term, medium=test_medium, name=test_name, source=test_source
Then after sending my first own tracking information to Google Analytis, I get:
D/GAv4: Sending first hit to property: UA-63098830-1
D/GAv4: Found relevant installation campaign: content=test_content, keyword=test_term, medium=test_medium, name=test_name, source=test_source
D/GAv4: Sending installation campaign to: UA-63098830-1, content=test_content, keyword=test_term, medium=test_medium, name=test_name, source=test_source