Evaluating GCP's Cloud Key Management Service

Intro to Cloud KMS

GCP’s Key Management Service is one of their main offerings with respect to security and encryption.

But how does it compare to what else they have?

portfolio

As we increase from left to right on the above spectrum, we go from the least amount of control and responsibility to the most.

Default encryption - Yep, Google encrypts data at rest by default. If you store data in Cloud Storage, it’s encrypted.
Customer Managed Encryption Keys (CMEK) - You’re storing sensitive data and you need even further levels of encryption.
Cloud Hardware Security Module (HSM) - We would use this in tandem with Cloud KMS, we would just get the benefit that our keys are stored in a Hardware Security Module.
Migrate existing keys - This is less about more control and responsibility, and more a feature to help people migrating to GCP with a brownfield system. Which is a ton of people.
Customer Supplied Encryption Keys (CSEK) - Now this is a feature about more control. You’re actually providing the key that Google will use for its encryption by default. It’s only supported for Cloud Storage and Compute Engine.
External Key Manager - This is like Cloud KMS except you want to use some external provider in order to encrypt your data. This can give solid access control and can help meet stringent compliance requirements.

So Cloud KMS sits somewhere in the middle of all of these. It’s great for use cases like

Storing passwords
Storing API keys
Storing credit card details
You want a clear audit trail of access to some data.

With that said, let’s dive in.

Symmetric Encryption

With symmetric encryption, we use the key to encrypt and decrypt the data.

Pros

Can encrypt much more data than asymmetric encryption. The size limit is 64KiB. With HSM, it’s lower — in that case the keys plus the additionalAuthenticatedData fields have a max size of 8KiB.
“Automatic key rotation” - note that they cannot automatically re-encrypt data for us (Obviously — They can’t build our application for us either 😅).

That being said, key rotation is easier with symmetric encryption. - Generally simpler.

Cons

Less secure — we’ve coupled encryption and decryption.

If our use case is simply we need to encrypt some passwords or API keys before we put them in our relational DB, symmetric encryption is probably what I would use in practice.

I would look in more detail at what compliance requirements we want to hit, but barring some compliance requirement stopping me, that’s the path I would take.

Example

Finally let’s get to some real code! And by that I mean some HCL, of course.

Note that Cloud KMS needs to be enabled in your GCP project for any of this to work.

The first thing we need to do is create a key ring. A “key ring” is just a grouping of keys, like it is in real life. We do this for access management — we can grant access to a key, or to a key ring, depending on our requirements.

All keys in this example will live on this one key ring.

resource "google_kms_key_ring" "example_key_ring" {
  project  = var.project
  name     = "example_key_ring"
  location = var.location
}

Pretty simple. Now we can create our symmetric key.

resource "google_kms_crypto_key" "symmetric_key_1" {
  name     = "symmetric_key_1"
  key_ring = google_kms_key_ring.example_key_ring.self_link
  # purpose = ENCRYPT_DECRYPT # this is the default, no need to specify
}

Then we can use it.

# encrypt!
$ gcloud kms encrypt --project=$GCP_PROJECT \
	  --location=$(terraform -chdir=$TF_DIR output -raw location) \
	  --keyring=$(terraform -chdir=$TF_DIR output -raw key_ring) \
	  --key=$(terraform -chdir=$TF_DIR output -raw symmetric_key_name) \
	  --plaintext-file=$INPUT_FILE \
	  --ciphertext-file=$CIPHERTEXT_FILE

# decrypt!
$ gcloud kms decrypt --project=$GCP_PROJECT \
	  --location=$(terraform -chdir=$TF_DIR output -raw location) \
	  --keyring=$(terraform -chdir=$TF_DIR output -raw key_ring) \
	  --key=$(terraform -chdir=$TF_DIR output -raw symmetric_key_name) \
	  --ciphertext-file=$CIPHERTEXT_FILE \
	  --plaintext-file=$PLAINTEXT_FILE

$ # some more bash goodness
input file:
On August 19th, 1981, two Russian-made Libyan Su-22 Fitters fired upon and were subsequently shot down by two U.S. F-14 Tomcats off the Libyan coast.
Libya, under Colonel Gaddafi, had claimed that the entire Gulf was their territory...

encrypted file:

$�'uK���y�H�?l�����sTC�j���S�C�@�P�!5

                                      �O�Iܷ�/�~�y߻#���t�?��m�����iq�
                                                                   ��K��
                                                                        j'��($�C�
 ��Ü�;	bbh-���ᙚ%�B�4.!�K�6R�
�NY9@_C�WL����               ��mf+i� (�p=��k��/�vo��az��n�&��Ҡ�0�
              t
�$����Ƣ/M=��?���|��z�b�>�%~��5�F��5���8f�|S����
                                               )��b�5���ږ�:�t

�.��X6�q�%��d�!T�TM�����ǭt�	�1�

decrypted file:
On August 19th, 1981, two Russian-made Libyan Su-22 Fitters fired upon and were subsequently shot down by two U.S. F-14 Tomcats off the Libyan coast.
Libya, under Colonel Gaddafi, had claimed that the entire Gulf was their territory...

See full script here.

This text, clearly a sensitive military document, and not an excerpt from the Wikipidia page on the Gulf of Sidra incident, is too long to be asymmetrically encrypted.

There you go, a pro of symmetric encryption in action. 😃

Asymmetric Encryption

As a quick review, asymmetric encryption is where we generate a public / private key pair. We encrypt with the public key, and decrypt with the private key.

Let me say that again, it’s important — we encrypt with the public key, and decrypt with the private key.

There is no way to export the private key — you have to go through GCP’s APIs. This is a good thing; it leaves an audit trail of anytime someone decrypts.

Pros

We’ve separated encryption and decryption! Now anybody can encrypt, and we have a chokepoint at decryption.

This can lead to much greater security in some circumstances.

Cons

Asymmetric encryption can only handle small plaintext sizes. Much smaller than symmetric encryption. That’s why we normally use it to encrypt encryption keys (more on this in a second…).

This is because RSA can only encrypt data maximum amount equal to your key size, minus any padding and header data.

So in practice, in GCP, we’re limited to encrypting about 2048 - 4096 bits with asymmetric encryption. In comparison to symmetric encryption’s 64KiB, mucho menos.

You have to distribute a public key. That’s work. Also makes it more secure….but it is another thing to do. This makes key rotation more challenging.

Due to the above cons, we usually use asymmetric encryption in a pattern called envelope encryption.

Envelope Encryption

In Cloud Storage’s default encryption, some blob of data is split into chunks. A chunk is then encrypted with a symmetric key, called a data encryption key, a DEK.

That data encryption key (DEK) is then encrypted with another key, called a key encryption key, a KEK.

The encrypted data encryption key (DEK) is stored with the data. The key encryption key (KEK) is stored in Google’s internal key management service.

The encrypted chunk, with the encrypted DEK, is then stored.

Why do this? It seems so unnecessary!

Well, we get the benefits of both symmetric and asymmetric encryption this way.

The DEK is a symmetric key. It can encrypt a “large” amount (64KiB) of data.
The KEK is an asymmetric key pair. We can separate encryption and decryption. When we want to decrypt the chunk, we have to decrypt the DEK, using the private key of the KEK. We still have a chokepoint on decryption.

Key Management Service is really made for storing KEKs.

Example

Now, we’ll create an asymmetric key pair.

resource "google_kms_crypto_key" "asymmetric_key_1" {
  name     = "asymmetric_key_1"
  key_ring = google_kms_key_ring.example_key_ring.self_link
  purpose  = "ASYMMETRIC_DECRYPT"

  version_template {
    algorithm = "RSA_DECRYPT_OAEP_2048_SHA256"
  }
}

The usage pattern is a little different with asymmetric keys. There’s no GCP method to encrypt.

We have to export the public key, and encrypt it with some other tool — I’m using openssl (not the one you have by default on your Mac! It won’t work. You must brew install openssl).

# Get the public key
$ gcloud kms keys versions get-public-key $(terraform -chdir=$TF_DIR output -raw asymmetric_key_version) \
	  --project=$GCP_PROJECT \
	  --key=$(terraform -chdir=$TF_DIR output -raw asymmetric_key_name) \
	  --keyring=$(terraform -chdir=$TF_DIR output -raw key_ring) \
	  --location=$(terraform -chdir=$TF_DIR output -raw location) >> $PUBLIC_KEY_FILE

# encrypt
$ openssl pkeyutl -in dek.key \
  -encrypt -pubin -inkey $PUBLIC_KEY_FILE \
  # carefully match these options to version_template.algorithm
  -pkeyopt rsa_padding_mode:oaep \  
  -pkeyopt rsa_oaep_md:sha256 \
  -pkeyopt rsa_mgf1_md:sha256 > $CIPHERTEXT_FILE

# decrypt
$ gcloud kms asymmetric-decrypt \
	  --project=$GCP_PROJECT \
	  --version=$(terraform -chdir=$TF_DIR output -raw asymmetric_key_version) \
	  --key=$(terraform -chdir=$TF_DIR output -raw asymmetric_key_name) \
	  --keyring=$(terraform -chdir=$TF_DIR output -raw key_ring) \
	  --location=$(terraform -chdir=$TF_DIR output -raw location)  \
	  --ciphertext-file=$CIPHERTEXT_FILE \
	  --plaintext-file=$PLAINTEXT_FILE

# some more bash goodness
original DEK:
487ab0d9-eb31-4740-b677-f2c5fa65fffa

encrypted ("wrapped") DEK:
&�ɉ�ݦg�lc�?�֥9/5	^"�yq`tO�`{U����N�9a\�i�E�������}X���U�t�����pB��
                                                                         �m1�
ac��]s	q���W׈��Ɋ�l%�ӯ��gR��y/��2Jz:Й�(��%$K���ι��Tg��*�C��>	�oF
�M
;�����>��/+�
            �~N۩BiK���d�rL�t�4k��5��n��SOv�{o{,2�3
                                                   SQ��[��Xp��YH

decrypted DEK:
487ab0d9-eb31-4740-b677-f2c5fa65fffa

Again, asymmetric encryption in KMS is really for envelope encryption.

See full script here.

Digital Signatures

Digital signing, like we do with JWTs, is the reverse of asymmetric encryption.

We still use a public / private key pair, but we sign with the private key, and validate with the public key.

In asymmetric encryption, anyone can write and only you can read.

In asymmetric signing, only you can write and anyone can read.

The use case for this is simple — to verify that some data is valid. Just as with asymmetric encryption, you cannot export the private key.

Example

resource "google_kms_crypto_key" "asymmetric_signing_key" {
  name     = "asymmetric_signing_key"
  key_ring = google_kms_key_ring.example_key_ring.self_link
  purpose  = "ASYMMETRIC_SIGN"

  version_template {
    algorithm = "EC_SIGN_P256_SHA256" # The recommended one https://cloud.google.com/kms/docs/algorithms#elliptic_curve_signing_algorithms
  }
}

Usage:

# sign
$ gcloud kms asymmetric-sign \
	  --project=$GCP_PROJECT \
	  --version=$(terraform -chdir=$TF_DIR output -raw asymmetric_signing_key_version) \
	  --key=$(terraform -chdir=$TF_DIR output -raw asymmetric_signing_key_name) \
	  --keyring=$(terraform -chdir=$TF_DIR output -raw key_ring) \
	  --location=$(terraform -chdir=$TF_DIR output -raw location)  \
	  --input-file=$MESSAGE_FILE \
	  --signature-file=$SIGNATURE_FILE \
	  --digest-algorithm=sha256

# distribute public key
$ gcloud kms keys versions get-public-key $(terraform -chdir=$TF_DIR output -raw asymmetric_signing_key_version) \
	  --project=$GCP_PROJECT \
	  --key=$(terraform -chdir=$TF_DIR output -raw asymmetric_signing_key_name) \
	  --keyring=$(terraform -chdir=$TF_DIR output -raw key_ring) \
	  --location=$(terraform -chdir=$TF_DIR output -raw location) >> $PUBLIC_KEY_FILE

# verify
$ openssl dgst \
	  -sha256 \
	  -verify $PUBLIC_KEY_FILE \
	  -signature $SIGNATURE_FILE \
	  $MESSAGE_FILE

# some more bash goodness
original message:
hello mr. president, this is from me.

message signature:
0F!����Ⓩ��11��	A�?T�Rr^�H+�!Ϡ

                              `0��p42�;�(J�T�nUeT��8

verification:
Verified OK

Full script here.

Managing Access To KMS

It goes without saying that this would be one of the more useful things to compromise for a bad actor in or outside of your organization. So how do we limit that risk?

The Resource Hierarchy

First let’s review a concept that comes up when we’re talking about access—GCP’s resource hierarchy. Generally it’s this (read -> as “live in”).

resources -> projects -> folders (optional) -> organization

With KMS, we can be more specific.

key version -> keys -> key rings -> projects -> folders (optional) -> organization

The lower on this we can grant access, the better. And for KMS, we can have policies at the key level.

Other Best Practices

Here are some other best practices we can follow, in accordance with separation of duties and principle of least privilege.

Put KMS in its own project, so that anyone with the owner role in your main project does not gain encryption / decryption capabilities.
Give the roles/cloudkms.admin role to whichever service account needs to manage keys for that project, not the owner role (which includes encryption / decryption capabilities)
If we’re using asymmetric encryption, and some service account only needs to encrypt, they only need roles/cloudkms.publicKeyViewer. (This is the whole benefit of asymmetric encryption!)

In short, we want to be very careful with who can decrypt data (or sign). That’s what we’re trying to protect here.

Key Rotation

Why do we have to put up with rotating keys?

If someone gets ahold of a key, it limits the damage
If we don’t do it regularly, it will be a huge pain in the ass in the event a key is actually compromised. Let’s not make a bad day worse.
We might ~~want to~~ be forced to upgrade to a better security algorithm
Some compliance standards require it

In reality it depends how secure we want to be, and regular key rotation is definitely on the “more secure” side. So how do we do it?

Symmetric Keys

Symmetric keys do support automatic key rotation, but I can’t say I’m a huge fan — automatically rotating the keys does not re-encrypt the data for you.

So if you’re storing encrypted data, I would recommend manual key rotation.

KMS does automatically infers the correct key version to decrypt data with for symmetric keys, as long as that key is enabled.

So re-encryption for symmetric keys is simple — just rotate keys (create a new version, and it should be the primary), then decrypt and re-encrypt the data, without specifying the version.

Once we’ve re-encrypted our data, we can disable our old key version.

Asymmetric Keys

For this I would recommend having a separate migration script, that uses both the old private key and the new public key.

Some additional notes

You have to re-distribute the public key
When decrypting with asymmetric keys in GCP, you do have to specify the key version

Conclusion

Key Management Service is a pretty cool technology. We’ve only really scratched the surface here, there’s some interesting other options available, like

Email me if you have any questions 😃.

Resources

GCP KMS Deep Dive