Ensure that Dataproc cluster is encrypted using customer-managed encryption key

CIS GCP 8.1

GCP

October 5, 2026

GCP

October 5, 2026

A Dataproc cluster writes cluster and job data to the Persistent Disks attached to its Compute Engine VMs and to a Cloud Storage staging bucket, and by default both are encrypted under keys that Google generates and holds. The control requires each cluster to be created with a customer-managed encryption key (CMEK) in Cloud KMS, which wraps the keys Google uses to encrypt that data.

What the benchmark asks for

Every Dataproc cluster must have a Cloud KMS key set on its encryption configuration. The field that carries it is config.encryptionConfig.gcePdKmsKeyName, and a cluster created without the --gce-pd-kms-key flag has no value there. In the Google Cloud console the same state shows on the cluster's Configurations tab as an encryption type of Google-managed key.

The setting is fixed at cluster creation, so remediating a cluster that fails means creating a replacement cluster with the key attached, moving the jobs to it, and deleting the original.

How it works

CMEK does not change which key encrypts the bytes on the disk. Google's infrastructure still generates a data encryption key (DEK) and uses it for the data itself. The customer-managed key is a key encryption key. It wraps the data encryption key that Google generates and uses. Holding the CMEK means holding the wrapping key, not the key that touches the data.

Two properties follow from that design. The first is control over availability. If the key's primary version is disabled or destroyed without a replacement, the resource cannot perform new encryption operations with that key, which blocks new writes while existing data stays as it is. The second is auditability. CMEK is the only option that lets you log administrative and data access to the encryption key and apply CMEK-specific organization policy constraints. Google-managed default keys give neither, because the customer never holds or audits those keys.

The key must be symmetric. Only symmetric encryption keys are supported as CMEKs, so an asymmetric key cannot be selected. The key ring must also sit in the same location as the cluster it protects. A key ring in the wrong region cannot be selected at all, which means the key ring has to exist in the matching region before cluster creation rather than being created as part of it.

The service agent needs permission on the key. Encryption and decryption run through a project-level service agent, and that agent starts with no access to your key at all, so bind it to the key with roles/cloudkms.cryptoKeyEncrypterDecrypter. Which agent needs the binding follows the resource the key protects. The key passed to --gce-pd-kms-key protects Persistent Disks attached to the cluster's VMs, so the Compute Engine service agent service-<project_number>@compute-system.iam.gserviceaccount.com needs it, while a key set on a Cloud Storage bucket is used by the Cloud Storage service agent instead. Without the grant, the account performing encryption on your behalf is not authorized to use the key and cluster creation fails.

What the control does not cover

The control checks that a key is attached. It does not check who can use that key, and the IAM policy on the key is what decides whether CMEK is a meaningful boundary. A key whose Encrypter/Decrypter role is granted at the project level to a broad group gives the same cryptographic protection as a tightly scoped one and a much weaker access boundary.

Rotation is not covered either, and the way rotation behaves is a reason to be careful about what you expect from it. Rotating a key only changes which version is used for new encryption operations going forward. Older versions are not disabled or destroyed, so data already wrapped under a previous version still needs that version to be readable. Rotation on its own does not retire a compromised key version.

Revocation is not immediate. Disabling or destroying a key version takes effect after a propagation delay, so a change made to cut off access right now may still allow operations to succeed for a short window afterward. Treat key disablement as a control that converges rather than one that fires instantly.

What it costs to comply

The ongoing cost is key administration. The key ring, the key, and the grant to the service agent all have to be created and then maintained by someone, and that work does not exist with Google-managed keys.

The cost of getting it wrong is an outage or data loss, and both failure modes are specific. A key version disabled by mistake stops new writes to the cluster. A key version destroyed by mistake takes the data with it. Neither is recoverable by a support ticket.

Remediating existing clusters costs a migration. Because the encryption configuration is fixed at creation, every failing cluster needs a replacement built with the key, every job and workflow template repointed at the new cluster name, and the old cluster deleted.

Software-backed keys do not consume Cloud KMS cryptographic-operation quota, but hardware-backed Cloud HSM keys and external Cloud EKM keys draw against their own separate quotas in the project that owns the key.

This is a control for clusters that process data you must be able to prove you control.

How to check your environment

List the clusters in a region, then read the encryption configuration of each one:

gcloud dataproc clusters list --region='us-central1'

gcloud dataproc clusters describe <cluster_name> --region=us-central1 --flatten=config.encryptionConfig.gcePdKmsKeyName

Output of null means the cluster has no customer-managed key and uses Google-managed encryption. A key resource name means the cluster passes.

Both commands are per region, so the check has to run across every region where clusters exist, and then across every project:

gcloud config set project <project_ID>

The console reports the same state. Each cluster has an Encryption type attribute on its Configurations tab, and a value of Google-managed key there means the cluster does not meet the control.

How to implement it

Create the key in a key ring whose location matches the cluster's region, since Cloud KMS will not let you select a key from anywhere else. Before the cluster exists, give the service agent that will use the key the encrypt and decrypt binding on it: the Compute Engine service agent for the Persistent Disk key passed to --gce-pd-kms-key, and the Cloud Storage service agent for any key you set on the staging bucket. Then create the cluster with the key attached:

gcloud dataproc clusters create <cluster_name> --region=us-central1 --gce-pd-kms-key=<key_resource_name>

Migrate workloads from the old cluster, then remove it:

gcloud dataproc clusters delete <cluster_name> --region=us-central1

Where it is available, Cloud KMS Autokey removes most of the setup. Autokey generates the key ring, key, and service-agent IAM bindings at resource-creation time, so an administrator does not pre-create the key ring and grant access by hand.

Stopping new clusters from failing is a separate job from fixing the ones that fail today. Two mechanisms do it. The organization policy constraint constraints/gcp.restrictNonCmekServices rejects resource creation for a named service unless a CMEK is supplied, and constraints/gcp.restrictCmekCryptoKeyProjects limits which projects a key may come from. Set both at the folder or organization level and a cluster created without a key is refused at the API rather than found by the next audit.

Build the key ring into whatever provisions projects. Because the key ring is per region and must exist before the cluster, a team that creates a cluster in a new region without that step cannot comply at creation time and will create the cluster without a key instead.