I am deploying an ARM template with Terraform.
We deploy all our Azure infra with Terraform but for AKS there are some preview features which are not in terraform yet so we want to deploy an AKS cluster with an ARM template.
If I create a Log Analytics workspace with TF, how can I pass the workspace id to ARM.
resource "azurerm_resource_group" "test" {
name = "k8s-test-bram"
location = "westeurope"
}
resource "azurerm_log_analytics_workspace" "test" {
name = "lawtest"
location = "${azurerm_resource_group.test.location}"
resource_group_name = "${azurerm_resource_group.test.name}"
sku = "PerGB2018"
retention_in_days = 30
}
So here is a snippet of the AKS ARM where I want to enable monitoring and I refer to the workspaceresourceId. But how do I define/declare the parameter to get the id from the workspace that I created with TF
"properties": {
"kubernetesVersion": "[parameters('kubernetesVersion')]",
"enableRBAC": "[parameters('EnableRBAC')]",
"dnsPrefix": "[parameters('DnsPrefix')]",
"addonProfiles": {
"httpApplicationRouting": {
"enabled": false
},
omsagent": {
"enabled": true,
"config": {
"logAnalyticsWorkspaceResourceID": "[parameters('workspaceResourceId')]"
}
}
},
you could use the parameters property of the azurerm_template_deployment deployment to pass in parameters:
parameters = {
"workspaceResourceId" = "${azurerm_log_analytics_workspace.test.id}"
}
I think it should look more or less like that, here's the official doc on this.
Related
I have a different google-service.json file for every releaseChannel (a file for prod, stage, QA, and dev)
for more details I want to implement push notifications but in a different environment.
cuz I don't want to send a test notification in the QA and the notification sent to the prod users!
this is the android config in app.json
"android": {
"googleServicesFile": "./google-services.json",
"adaptiveIcon": {
"foregroundImage": "./src/assets/adaptive-icon.png"
},
"permissions": [
"android.permission.CAMERA",
"android.permission.ACCESS_FINE_LOCATION",
"android.permission.WRITE_EXTERNAL_STORAGE",
"android.permission.READ_EXTERNAL_STORAGE",
"android.permission.ACTION_BOOT_COMPLETED",
"android.permission.RECORD_AUDIO",
"com.google.android.gms.permission.AD_ID"
],
"package": "com.beyondbelievers.awal",
"versionCode": 1
},Ï
is there any way I can load the different files with every environment?
this answer inspired from an old question you can check it here
after I searched the whole internet I found a way but using app.config.js
I will try to explain how I archive my goal in the following steps:
convert app.json to app.config.js: here's expo docs explain how to migrate from app.json.
then in my eas.json I added the following line to all profiles "APP_VARIANT": "qa":
{
"qa": {
...
"releaseChannel": "qa",
"env": {
"APP_VARIANT": "qa"
}
...
},
"stage": {
...
"releaseChannel": "stage",
"env": {
"APP_VARIANT": "stage"
}
...
},
"prod": {
...
"releaseChannel": "prod",
"env": {
"APP_VARIANT": "prod"
}
...
}
}
}
the "APP_VARIANT" variable contains the type of env and based on it I can later check which file to us.
now back to the app.config.js add the following lines:
// this will return the value of APP_VARIANT defined in eas.json
const APP_RELEASE_CHANNEL = process.env.APP_VARIANT;
// define the AndroidGoogleServicesFile variable to use instead of the normal string and assign a default value to it
let AndroidGoogleServicesFile = "./google-services-dev.json";
// now check the value of the APP_VARIANT and based on it assign the path of the google-services you wanna use
if (APP_RELEASE_CHANNEL === "qa") {
AndroidGoogleServicesFile = "./google-services-qa.json";
} else if (APP_RELEASE_CHANNEL === "stage") {
AndroidGoogleServicesFile = "./google-services-stage.json";
} else if (APP_RELEASE_CHANNEL === "prod") {
AndroidGoogleServicesFile = "./google-services-prod.json";
}
now you know which file you need to use one step left is to assign that file so your app can use it
android: {
...
googleServicesFile: AndroidGoogleServicesFile,
...
},
this is how I solve my problem I hope this solves your problem too.
If you didn't understand something I welcomed to explain in more details
I am using below Bicep file for Azure role assignments . So here I have a Azuredevops pipeline which wil build the bicepfile to arm template and from pipeline variables, the paramaters.json file will be getting updated.
main.bicep
targetScope = 'resourceGroup'
#description('Principal type of the assignee.')
#allowed([
'Device'
'ForeignGroup'
'Group'
'ServicePrincipal'
'User'
])
param principalType string
#description('the id for the role defintion, to define what permission should be assigned')
param RoleDefinitionId string
#description('the id of the principal that would get the permission')
param principalId string
#description('the role deffinition is collected')
resource roleDefinition 'Microsoft.Authorization/roleDefinitions#2018-01-01-preview' existing = {
scope: resourceGroup()
name: RoleDefinitionId
}
resource RoleAssignment 'Microsoft.Authorization/roleAssignments#2020-10-01-preview' = {
name: guid(resourceGroup().id, RoleDefinitionId, principalId)
properties: {
roleDefinitionId: roleDefinition.id
principalId: principalId
principalType: principalType
}
}
paramters.json
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"principalType": {
"value": "#{principalType}#"
},
"RoleDefinitionId": {
"value": "#{RoleDefinitionId}#"
},
"principalId": {
"value": "#{principalId}#"
}
}
}
pipeline build task for creation deployment.
'az deployment group create --resource-group $(resourceGroup) --template-file $(System.DefaultWorkingDirectory)/template/main.json --parameters $(System.DefaultWorkingDirectory)/template/parameters.json'
When I triggered the pipeline firstime, i got output summary as below.
The deployment will update the following scope:
Scope: /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/XXXXXXXXXXXXXXXXXXX-rg
+ Microsoft.Authorization/roleAssignments/xxxxxxxxxxxxxxxx [2020-10-01-preview]
apiVersion: "2020-10-01-preview"
id: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/XXXXXXXXXXXXXXXXXXX-rg/providers/Microsoft.Authorization/roleAssignments/xxxxxxxxxxxxxxxxx"
name: "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
properties.principalId: "xxxxxxxxxxxxx"
properties.roleDefinitionId: "/subscriptions/XXXXXXXXXXXXXXXXXXXXX/resourceGroups/XXXXXXXXXXXXXXXXXXX-rg/providers/Microsoft.Authorization/roleDefinitions/xxxxxxxxxxxxxxxxxxxxxxx"
type: "Microsoft.Authorization/roleAssignments"
And after that, if I retrigger the pipeline again without any change to the templates. Its showing as 1 to modify, but expected that the output will show as "no change". Because we havenet made any changes to the resource either from pipeline side or manually.
Scope: /subscriptions/xxxxxxxxxxxxxxxxxx/resourceGroups/xxxxxxxxxxxxxxxx-rg
~ Microsoft.Authorization/roleAssignments/xxxxxxxxxxxxxxxxxxxxxxx [2020-10-01-preview]
~ properties.roleDefinitionId: "/subscriptions/xxxxxxxxxxxxxxxxxxx/providers/Microsoft.Authorization/roleDefinitions/xxxxxxxxxxxxxxxxxxxxxxxxx" => "/subscriptions/xxxxxxxxxxxxxxxxxxxx/resourceGroups/xxxxxxxxxxxxxxx-rg/providers/Microsoft.Authorization/roleDefinitions/xxxxxxxxxxxxxxxxxxx"
x properties.principalType: "Group"
Resource changes: 1 to modify
iF i again deploy also, the next time again will show the same output as 1 to modify
What is the issue here, Why ARM deployment is showing changes eventhough there are no changes.
Azure built-in role definitions are defined at the subscription level unless it is a custom role that you've created at the another scope.
In your bicep file, you can change the scope of the roleDefinition resource:
#description('the role deffinition is collected')
resource roleDefinition 'Microsoft.Authorization/roleDefinitions#2018-01-01-preview' existing = {
scope: subscription()
name: RoleDefinitionId
}
or you could also use subscriptionResourceId:
resource RoleAssignment 'Microsoft.Authorization/roleAssignments#2020-10-01-preview' = {
name: guid(resourceGroup().id, RoleDefinitionId, principalId)
properties: {
roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', RoleDefinitionId)
principalId: principalId
principalType: principalType
}
}
Have been looking for cluster configs in JSON format to create a dataproc cluster(GCE) with Dataproc Metastore service and Spark-BQ dependency jars, unable to find any reference document that specifies how to use those JSON configs.
I have looked through below links :
https://airflow.apache.org/docs/apache-airflow/1.10.6/_api/airflow/contrib/operators/dataproc_operator/index.html
https://cloud.google.com/dataproc/docs/reference/rest/v1/projects.regions.clusters
https://cloud.google.com/dataproc/docs/reference/rest/v1/MetastoreConfig
but it does not specify GCE cluster configs, its REST API and GKE cluster configs
Please see below configs that I am trying out to create a dataproc cluster :
CLUSTER_CONFIG = {
"gce_cluster_config": {
"internal_ip_only": True,
"metadata": {
"spark-bigquery-connector-version": spark_bq_connector_version
},
"service_account_scopes": [
service_account_scopes
],
"subnetwork_uri": subnetwork_uri,
"zone_uri": zone_uri
},
"initialization_actions": [
{
"executable_file": initialization_actions,
"execution_timeout": execution_timeout
}
],
"master_config": {
"disk_config": {
"boot_disk_size_gb": master_boot_disk_size_gb
},
"machine_type_uri": master_machine_type_uri
},
"metastore_config": {
"dataproc_metastore_service": dataproc_metastore
},
"software_config": {
"image_version": software_image_version
},
"worker_config": {
"disk_config": {
"boot_disk_size_gb": worker_boot_disk_size_gb
},
"machine_type_uri": worker_machine_type_uri,
"num_instances": worker_num_instances
}
}
Any lead would be really appreciated, please attach links to refer full config examples
Thanks !
As mentioned in this doc, external Hive metastore (non Dataproc Metastore service) needs to be specified through the hive:hive.metastore.uris property. Note the hive: prefix.
When creating the cluster with gcloud, if you add --log-http:
$ gcloud dataproc clusters create ... \
--properties hive:hive.metastore.uris=thrift://my-metastore:9083 \
--log-http
it will show you the actual HTTP request:
{
"clusterName":"...",
"config":{
"endpointConfig":{
"enableHttpPortAccess":true
},
"gceClusterConfig":{
"internalIpOnly":false,
"serviceAccountScopes":[
"https://www.googleapis.com/auth/cloud-platform"
],
"zoneUri":"us-west1-a"
},
"masterConfig":{
"diskConfig":{
},
"machineTypeUri":"e2-standard-2"
},
"softwareConfig":{
"imageVersion":"1.5",
"properties":{
"hive:hive.metastore.uris":"thrift://my-metastore:9083"
}
},
"workerConfig":{
"diskConfig":{
},
"machineTypeUri":"e2-standard-2"
}
},
"projectId":"..."
}
You can also find the request spec in the Dataproc REST API doc.
After using terraform for a very long time, I decided to start learning azure bicep. So far I am trying to have a grip on the logic. So far I have playing around on deployment of a storage account and keyvault. What I am doing here is the following.
create a storage account
use existing key vault to store storage account connection string as secret
create a key based on the storage account name
And this works as I am expected.
So I wanted to take one step forward. and here is where I am a bit confused.
What I wanted to do, is to use the same bicep template, to create a new secret but in a different resource group into a different key vault.
Now according to my understand of azure documentation, the template comes with a default scope which in my specific case target my default subscription and to run my bicep template from the terminal I use the command
az deployment group create -f ./template.bicep -g <resource-group-name>
and this is my template:
// Default values I'm using to test
param keyVaultName string = '<keyvault-name>'
param managedIdentityName string = 'test-managed-identity'
param tenantCodes array = [
'elhm'
'feor'
]
// I'm using prefix so I dont need to create additional arrays
var keyVaultKeyPrefix = 'Client-Key-'
var storagePrefix = 'sthrideveur'
// Get a reference to key vault
resource keyVault 'Microsoft.KeyVault/vaults#2021-06-01-preview' existing = {
name: keyVaultName
}
// Create a managed identity
resource managedIdentity 'Microsoft.ManagedIdentity/userAssignedIdentities#2018-11-30' = {
name: managedIdentityName
location: resourceGroup().location
}
// Grant permissions to key vault
resource accessPolicy 'Microsoft.KeyVault/vaults/accessPolicies#2019-09-01' = {
name: '${keyVault.name}/add'
properties: {
accessPolicies: [
{
tenantId: subscription().tenantId
objectId: managedIdentity.properties.principalId
permissions: {
// minimum required permissions
keys: [
'get'
'unwrapKey'
'wrapKey'
]
}
}
]
}
}
// Create key vault keys
resource keyVaultKeys 'Microsoft.KeyVault/vaults/keys#2021-06-01-preview' = [for tenantCode in tenantCodes: {
name: '${keyVault.name}/${keyVaultKeyPrefix}${tenantCode}'
properties: {
keySize: 2048
kty: 'RSA'
// storage key should only needs these operations
keyOps: [
'unwrapKey'
'wrapKey'
]
}
}]
// Create storage accounts
resource storageAccount 'Microsoft.Storage/storageAccounts#2021-04-01' = [for tenantCode in tenantCodes: {
name: '${storagePrefix}${tenantCode}'
location: resourceGroup().location
kind: 'StorageV2'
sku: {
name: 'Standard_RAGRS'
}
// Assign the identity
identity: {
type: 'UserAssigned'
userAssignedIdentities: {
'${managedIdentity.id}': {}
}
}
properties: {
allowCrossTenantReplication: true
minimumTlsVersion: 'TLS1_2'
allowBlobPublicAccess: false
allowSharedKeyAccess: true
networkAcls: {
bypass: 'AzureServices'
virtualNetworkRules: []
ipRules: []
defaultAction: 'Allow'
}
supportsHttpsTrafficOnly: true
encryption: {
identity: {
// specify which identity to use
userAssignedIdentity: managedIdentity.id
}
keySource: 'Microsoft.Keyvault'
keyvaultproperties: {
keyname: '${keyVaultKeyPrefix}${tenantCode}'
keyvaulturi: keyVault.properties.vaultUri
}
services: {
file: {
keyType: 'Account'
enabled: true
}
blob: {
keyType: 'Account'
enabled: true
}
}
}
accessTier: 'Cool'
}
}]
// Store the connectionstrings in KV if specified
resource storageAccountConnectionStrings 'Microsoft.KeyVault/vaults/secrets#2019-09-01' = [ for (name, i) in tenantCodes :{
name: '${keyVault.name}/${storagePrefix}${name}'
properties: {
value: 'DefaultEndpointsProtocol=https;AccountName=${storageAccount[i].name};AccountKey=${listKeys(storageAccount[i].id, storageAccount[i].apiVersion).keys[0].value};EndpointSuffix=${environment().suffixes.storage}'
}
}]
according to the documentation here https://learn.microsoft.com/en-us/azure/azure-resource-manager/bicep/deploy-to-resource-group?tabs=azure-cli
When I need to target a specific resource group, I can use the scope in the resource, so I create this:
resource keyvaultApi 'Microsoft.KeyVault/vaults#2021-06-01-preview' existing = {
name: keyVaultApiName
scope: resourceGroup('secondresourcegroup')
}
So far no errors, but the problem happens when I had to create a managed identity resource.
resource keyvaultApi 'Microsoft.KeyVault/vaults#2021-06-01-preview' existing = {
name: keyVaultApiName
scope: resourceGroup('secondresourcegroup')
}
resource managedIdentityTwo 'Microsoft.ManagedIdentity/userAssignedIdentities#2018-11-30' = {
name: managedIdentityNameTwo
location: resourceGroup().location
}
resource accessPolicyApi 'Microsoft.Media/videoAnalyzers/accessPolicies#2021-11-01-preview' = {
name: '${keyvaultApi.name}/add'
properties: {
accessPolicies: [
{
tenantId: subscription().tenantId
objectId: managedIdentityTwo.properties.principalId
permissions: {
// minimum required permissions
keys: [
'get'
'unwrapKey'
'wrapKey'
]
}
}
]
}
}
In the key vault I could declare the scope, but to the underlying resources, such as access policy etc, I cannot declare the scope. So how can bicep understand that those resources needs to target a specific resource group and specific key vault?
Because when I run the terminal command, I am targeting a specific resource group, so I don't really understand how I can use one template to target different resource groups and resources accordingly.
I hope I made my point clear, and please if I didn't, just feel free to ask me more informations.
Thank you so much for your time and help
UPDATE:
When I try to run the code as it is, I get the following error:
{"status":"Failed","error":{"code":"DeploymentFailed","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.","details":[{"code":"NotFound","message":"{\r\n \"error\": {\r\n \"code\": \"ParentResourceNotFound\",\r\n \"message\": \"Can not perform requested operation on nested resource. Parent resource 'secondkeyvault' not found.\"\r\n }\r\n}"}]}}
UPDATE:
So I followed the Daniel lead and in a second template I deployed the code I needed for the second template as follow:
template2.bicep
param deploymentIdOne string = newGuid()
param deploymentIdTwo string = newGuid()
output deploymentIdOne string = '${deploymentIdOne}-${deploymentIdTwo}'
output deploymentIdTwo string = deploymentIdTwo
// Default values I'm using to test
param keyVaultApiName string = 'secondkeyvaultapi'
param managedIdentityNameTwo string = 'second-second-identity'
var keyVaultKeyPrefixTw = 'Client-Key-'
param tenantCodes array = [
'tgrf'
]
resource keyvaultApi 'Microsoft.KeyVault/vaults#2021-06-01-preview' existing = {
name: keyVaultApiName
}
resource managedIdentityTwo 'Microsoft.ManagedIdentity/userAssignedIdentities#2018-11-30' = {
name: managedIdentityNameTwo
location: resourceGroup().location
}
resource accessPolicyApi 'Microsoft.KeyVault/vaults/accessPolicies#2019-09-01' = {
name: '${keyvaultApi.name}/add'
properties: {
accessPolicies: [
{
tenantId: subscription().tenantId
objectId: managedIdentityTwo.properties.principalId
permissions: {
// minimum required permissions
keys: [
'get'
'unwrapKey'
'wrapKey'
]
}
}
]
}
}
// Store the connectionstrings in KV if specified
resource clientApiKeys 'Microsoft.KeyVault/vaults/secrets#2019-09-01' = [ for name in tenantCodes :{
name: '${keyvaultApi.name}/${keyVaultKeyPrefixTw}${name}'
properties: {
value: '${deploymentIdOne}-${deploymentIdTwo}'
}
}]
and in my main template I added the module:
module clientKeyApi 'template2.bicep' = {
name: 'testfrgs'
scope: 'secondresourcegroup'
}
But there is something that is not clear 100% to me.
How does it work to override all the for loop and parameters name I have declared in my template2.bicep , and yet the module require a scope subscription, if I declare the scope, wouldn't this override the default value?
Sorry guys for the newbie questions, I am trying to break my mindset from terraform and understand better how bicep work.
Any explanation would be amazing and helpful
You can't specify scope on the resource, but you can specify it on a module. You'll need to turn the resource that adds the access policy to the keyvault into a separate module, then specify scope on the module. You can also make the scope for your deployment subscription, but then you'll need to break everything that targets a specific resource group into modules, as well.
This is due to how ARM deployments work. The default scope for an ARM deployment is at the resource group level. You can't point a resource at a different resource group because it's outside the scope of the deployment.
Modules, however, run as sub-deployments and therefore can have a different scope set.
This is a case where Terraform is more straightforward, since it calls the Azure APIs directly instead of using the ARM deployment model. Terraform doesn't care about deployment scopes because it doesn't use them.
I would like to deploy a minimal k8s cluster on AWS with Terraform and install a Nginx Ingress Controller with Helm.
The terraform code:
provider "aws" {
region = "us-east-1"
}
data "aws_eks_cluster" "cluster" {
name = module.eks.cluster_id
}
data "aws_eks_cluster_auth" "cluster" {
name = module.eks.cluster_id
}
variable "cluster_name" {
default = "my-cluster"
}
variable "instance_type" {
default = "t2.large"
}
provider "kubernetes" {
host = data.aws_eks_cluster.cluster.endpoint
cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
token = data.aws_eks_cluster_auth.cluster.token
load_config_file = false
version = "~> 1.11"
}
data "aws_availability_zones" "available" {
}
module "vpc" {
source = "terraform-aws-modules/vpc/aws"
version = "3.0.0"
name = "k8s-${var.cluster_name}-vpc"
cidr = "172.16.0.0/16"
azs = data.aws_availability_zones.available.names
private_subnets = ["172.16.1.0/24", "172.16.2.0/24", "172.16.3.0/24"]
public_subnets = ["172.16.4.0/24", "172.16.5.0/24", "172.16.6.0/24"]
enable_nat_gateway = true
single_nat_gateway = true
enable_dns_hostnames = true
public_subnet_tags = {
"kubernetes.io/cluster/${var.cluster_name}" = "shared"
"kubernetes.io/role/elb" = "1"
}
private_subnet_tags = {
"kubernetes.io/cluster/${var.cluster_name}" = "shared"
"kubernetes.io/role/internal-elb" = "1"
}
}
module "eks" {
source = "terraform-aws-modules/eks/aws"
version = "12.2.0"
cluster_name = "eks-${var.cluster_name}"
cluster_version = "1.18"
subnets = module.vpc.private_subnets
vpc_id = module.vpc.vpc_id
worker_groups = [
{
name = "worker-group-1"
instance_type = "t3.small"
additional_userdata = "echo foo bar"
asg_desired_capacity = 2
},
{
name = "worker-group-2"
instance_type = "t3.small"
additional_userdata = "echo foo bar"
asg_desired_capacity = 1
},
]
write_kubeconfig = true
config_output_path = "./"
workers_additional_policies = [aws_iam_policy.worker_policy.arn]
}
resource "aws_iam_policy" "worker_policy" {
name = "worker-policy-${var.cluster_name}"
description = "Worker policy for the ALB Ingress"
policy = file("iam-policy.json")
}
The installation performs correctly:
helm install my-release nginx-stable/nginx-ingress
NAME: my-release
LAST DEPLOYED: Sat Jun 26 22:17:28 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The NGINX Ingress Controller has been installed.
The kubectl describe service my-release-nginx-ingress returns:
Error syncing load balancer: failed to ensure load balancer: could not find any suitable subnets for creating the ELB
The VPC is created and the public subnet seems to be correctly tagged, what is lacking to make the Ingress aware of the public subnet ?
In the eks modules you are prefixing the cluster name with eks-:
cluster_name = "eks-${var.cluster_name}"
However you do not use the prefix in your subnet tags:
"kubernetes.io/cluster/${var.cluster_name}" = "shared"
Drop the prefix from cluster_name and add it to the cluster name variable (assuming you want the prefix at all). Alternatively, you could add the prefix to your tags to fix the issue, but that approach makes it easier to introduce inconsistencies.