Grant permission to use the key in GKE - encryption

Question

I'm trying to provision an encrypted disk for GKE dynamically.
But I really don't understand below part.
Grant permission to use the key
You must assign the Compute Engine service account used by nodes in your cluster the Cloud KMS CryptoKey Encrypter/Decrypter role. This is required for GKE Persistent Disks to access and use your encryption key.
The Compute Engine service account's name has the following format:
service-[PROJECT_NUMBER]#compute-system.iam.gserviceaccount.com
Does it really necessary to grant "Cloud KMS CryptoKey Encrypter/Decryter" to Compute Engine Service account? Can I create a new SA and grant this role to it? The description said, the SA used by nodes. So I'm wondering if I can create a new SA and grant Cloud KMS role then use this SA to spin up GKE cluster. Then I think it should be available to provision encrypted disks for GKE.
official document below:
dynamically_provision_an_encrypted

Answer 1

I tried to follow this documentation step by step:
create gke cluster (check Kubernetes Compatibility compatibility, I decided to stick with 1.14 this time), key-ring and key
deploy CSI driver to the cluster
2.1. download driver $git clone https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver /PATH/gcp-compute-persistent-disk-csi-driver
2.2. configure variables for your project in /PATH/gcp-compute-persistent-disk-csi-driver/deploy/setup-project.sh
2.3. create service account with /PATH/gcp-compute-persistent-disk-csi-driver/deploy/setup-project.sh
2.4. configure variables for driver deployment in /PATH/gcp-compute-persistent-disk-csi-driver/deploy/kubernetes/deploy-driver.sh and /PATH/gcp-compute-persistent-disk-csi-driver/deploy/kubernetes/deploy-driver.shinstall-kustomize.sh
2.5. deploy CSI driver (I stick with stable version)
$./deploy-driver.sh
enable the Cloud KMS API
assign the Cloud KMS CryptoKey Encrypter/Decrypter role (roles/cloudkms.cryptoKeyEncrypterDecrypter) to the Compute Engine Service Agent (service-[PROJECT_NUMBER]#compute-system.iam.gserviceaccount.com)
create StorageClass
$cat storage.yaml
apiVersion: storage.k8s.io/v1beta1
kind: StorageClass
metadata:
name: csi-gce-pd
provisioner: pd.csi.storage.gke.io
parameters:
type: pd-standard
disk-encryption-kms-key: projects/test-prj/locations/europe-west3/keyRings/TEST-KEY-RING/cryptoKeys/TEST-KEY
$kubectl describe storageclass csi-gce-pd
Name: csi-gce-pd
IsDefaultClass: No
Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"storage.k8s.io/v1beta1","kind":"StorageClass","metadata":{"annotations":{},"name":"csi-gce-pd"},"parameters":{"disk-encryption-kms-key":"projects/test-prj/locations/europe-west3/keyRings/TEST-KEY-RING/cryptoKeys/TEST-KEY","type":"pd-standard"},"provisioner":"pd.csi.storage.gke.io"}
Provisioner: pd.csi.storage.gke.io
Parameters: disk-encryption-kms-key=projects/test-prj/locations/europe-west3/keyRings/TEST-KEY-RING/cryptoKeys/TEST-KEY,type=pd-standard
AllowVolumeExpansion: <unset>
MountOptions: <none>
ReclaimPolicy: Delete
VolumeBindingMode: Immediate
Events: <none>
create persistent volume
$kubectl apply -f pvc.yaml
persistentvolumeclaim/podpvc created
$kubectl describe pvc podpvc
Name: podpvc
Namespace: default
StorageClass: csi-gce-pd
Status: Bound
Volume: pvc-b383584a-32c5-11ea-ad6e-42010a9c007d
Labels:
Annotations:
kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"podpvc","namespace":"default"},"spec":{"accessModes...
pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
volume.beta.kubernetes.io/storage-provisioner: pd.csi.storage.gke.io
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 6Gi
Access Modes: RWO
VolumeMode: Filesystem
Mounted By: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Provisioning 31m pd.csi.storage.gke.io_gke-test-cluster-default-pool-cd22e088-t1h0_c158f4fc-07ba-411e-8a94-74595f2b2f1d External provisioner is provisioning volume for claim "default/podpvc"
Normal ExternalProvisioning 31m (x2 over 31m) persistentvolume-controller waiting for a volume to be created, either by external provisioner "pd.csi.storage.gke.io" or manually created by system administrator
Normal ProvisioningSucceeded 31m pd.csi.storage.gke.io_gke-test-cluster-default-pool-cd22e088-t1h0_c158f4fc-07ba-411e-8a94-74595f2b2f1d Successfully provisioned volume pvc-b383584a-32c5-11ea-ad6e-42010a9c007d
And it's successfully provisioned.
Then I removed Cloud KMS CryptoKey Encrypter/Decrypter role from the Compute Engine Service Agent and persistent volume created at step 6 and tried again:
$kubectl apply -f pvc.yaml
persistentvolumeclaim/podpvc created
$kubectl describe pvc podpvc
Name: podpvc
Namespace: default
StorageClass: csi-gce-pd
Status: Pending
Volume:
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"podpvc","namespace":"default"},"spec":{"accessModes...
volume.beta.kubernetes.io/storage-provisioner: pd.csi.storage.gke.io
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Mounted By: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Provisioning 2m15s (x10 over 11m) pd.csi.storage.gke.io_gke-serhii-test-cluster-default-pool-cd22e088-t1h0_c158f4fc-07ba-411e-8a94-74595f2b2f1d External provisioner is provisioning volume for claim "default/podpvc"
Warning ProvisioningFailed 2m11s (x10 over 11m) pd.csi.storage.gke.io_gke-serhii-test-cluster-default-pool-cd22e088-t1h0_c158f4fc-07ba-411e-8a94-74595f2b2f1d failed to provision volume with StorageClass "csi-gce-pd": rpc error: code = Internal desc = CreateVolume failed to create single zonal disk "pvc-b1a238b5-35fa-11ea-bec8-42010a9c01e6": failed to insert zonal disk: unkown Insert disk error: googleapi: Error 400: Cloud KMS error when using key projects/serhii-test-prj/locations/europe-west3/keyRings/SERHII-TEST-KEY-RING/cryptoKeys/SERHII-TEST-KEY: Permission 'cloudkms.cryptoKeyVersions.useToEncrypt' denied on resource 'projects/serhii-test-prj/locations/europe-west3/keyRings/SERHII-TEST-KEY-RING/cryptoKeys/SERHII-TEST-KEY' (or it may not exist)., kmsPermissionDenied
Normal ExternalProvisioning 78s (x43 over 11m) persistentvolume-controller waiting for a volume to be created, either by external provisioner "pd.csi.storage.gke.io" or manually created by system administrator
and persistent volume stayed in pending status.
And, as you can see, in the documentation it's necessary:
Grant permission to use the key
You must assign the Compute Engine service account used by nodes in
your cluster the Cloud KMS CryptoKey Encrypter/Decrypter role. This is
required for GKE Persistent Disks to access and use your encryption
key.
and it's not enough to create service account with /PATH/gcp-compute-persistent-disk-csi-driver/deploy/setup-project.sh provided by CSI driver.
EDIT Please notice that:
For CMEK-protected node boot disks, this Compute Engine service
account is the account which requires permissions to do encryption
using your Cloud KMS key. This is true even if you are using a custom
service account on your nodes.
So, there's no way to use only service account in this case without Compute Engine service account, because CMEK-protected persistent volumes are managed by GCE, not by GKE. Meanwhile, you can provide only necessary perdition to your custom service account to improve security of your project.