Reduce AWS key management service costs by up to 99% with bucket S3 keys

Customers in many industries face increasingly stringent auditing data security and privacy compliance requirements. Some compliance frameworks, such as FISMA, FEDRAMP, PCI DSS, and SOC 2, have specific regulatory standards for validating system security. A common requirement for these compliance frameworks is more stringent data-at-rest encryption standards, where organizations must encrypt data using an encryption key they control. To meet these requirements locally, customers must dedicate resources to maintaining cryptographic modules. In the cloud, customers benefit from encryption offerings priced based on their usage.

Amazon Simple Storage Service (Amazon S3) encrypts all new objects using Amazon S3 managed server-side encryption (SSE-S3), unless the customer chooses a different type of server-side encryption. SSE-S3 meets the security requirements of many Amazon S3 clients, but clients adhering to certain compliance frameworks may need more control over their encryption keys. These customers can rely on AWS Key Management Service (AWS KMS) and server-side encryption with AWS KMS (SSE-KMS). SSE-KMS involves an AWS KMS fee per request and a monthly storage fee per AWS KMS key. As customers scale their workloads to millions or billions of objects, their AWS KMS costs can increase. Customers experience this acutely in read/write-intensive applications where customer data is frequently accessed. To help customers reduce this cost, Amazon S3 has introduced S3 Bucket Keys. S3 Bucket Keys can reduce the cost of AWS KMS by up to 99% and save customers more than $80m since their launch in late 2020.

In this article, the authors will demonstrate how to require SSE-KMS for all objects stored in a bucket, enable S3 bucket keys, use S3 bucket keys with multiple tenants, and evaluate existing objects encrypted by SSE-KMS at the object level. Using S3 Bucket Keys helps you save on your AWS KMS bill while complying with security requirements for encrypting data with SSE-KMS.

Background

SSE-KMS gives customers control over encryption keys and can audit key usage. These additional controls are essential for corporate customers who must demonstrate to auditors that they own their encryption keys, control key rotation schedules and manage encryption key permissions. When an object is encrypted using SSE-KMS, AWS KMS receives an encryption request from Amazon S3 and sends back a unique data key to encrypt the object.

With S3 Bucket Keys, instead of individually calling AWS KMS to encrypt each object, AWS KMS generates a time-limited key at the bucket level. Amazon S3 uses this bucket-level key to create unique data keys for encrypting objects in the bucket, avoiding the need for additional AWS KMS requests to perform cryptographic operations. S3 Bucket Keys reduce request traffic from Amazon S3 to AWS KMS, allowing access to encrypted objects in Amazon S3 at a fraction of the previous cost.

Prerequisites

To follow the blog tutorial, your AWS Identity and Access Management (IAM) principal (user or role) needs permission to create buckets and use AWS KMS keys.

Minimum IAM permissions to use Bucket Keys
{
   "Version":"2012-10-17",
   "Statement":[
{
   "Sid": "Permissions to use Bucket Keys",
   "Effect":"Allow",
   "Principal": {"AWS": "arn:aws:iam::111122223333:user/willcavin"},
         "Action":[
         "s3:PutObject",
         "s3:GetObject",
         "s3:CreateBucket",
         "kms:Decrypt",
         "kms:GenerateDataKey"
         ],
         "Resource":"arn:aws:s3:::willcavinbucket/*"
      }
   ]
}

Require SSE-KMS for all objects stored in the bucket

Customers with compliance requirements often require encryption with SSE-KMS for all objects uploaded to their S3 buckets. This can be achieved using S3 Bucket Policies. The following bucket policies prevent writing to a bucket that is not encrypted using SSE-KMS. Using these policies, you can successfully transfer objects to a bucket by adding an encryption request header to the PutObject request that specifies SSE-KMS. Alternatively, you can set the default bucket encryption configuration in the bucket to automatically encrypt new objects using SSE-KMS, even if no encryption header is added to the clients' upload requests. This bucket rule, combined with including S3 bucket keys in the default bucket encryption configuration, verifies that uploaded objects use S3 bucket keys unless you explicitly disable S3 bucket keys in the PutObject request.

JSON

 {
 "Version": "2012-10-17",
 "Id": "PutObjPolicy",
 "Statement": [{
 		"Sid": "DenyObjectsThatAreNotSSEKMS",
 		"Effect": "Deny",
 		"Principal": "*",
 		"Action": "s3:PutObject",
 		"Resource": "arn:aws:s3:::example-bucket/*",
 		"Condition": {
 		"Null": {
 		"s3:x-amz-server-side-encryption-aws-kms-key-id": "true"
           }
      }
   }]
 }

Activation of S3 bucket keys

S3 Bucket Keys can be activated via the Amazon S3 console by editing the default encryption in the bucket. With the bucket selected, select Properties. In the Default encryption area, choose Edit.

In the Encryption key type area, select AWS Key Management Service key (SSE-KMS). In the AWS KMS Key area, choose your preferred AWS KMS key. In the Bucket Key area, select Enable. Then, choose Save Changes.

BASH

aws s3api put-bucket-encryption --bucket <bucket-name> --server-side-encryption-configuration '{
        "Rules": [
            {
                "ApplyServerSideEncryptionByDefault": {
                    "SSEAlgorithm": "aws:kms",
                    "KMSMasterKeyID": "<KMS-Key-ARN>"
                },
                "BucketKeyEnabled": true
            }
        ]
    }'

Using S3 Bucket Keys in a multi-tenant environment

Many customers use S3 buckets in multi-tenant scenarios where they may have multiple teams or end clients uploading objects to a shared bucket for analysis or processing. S3 Bucket Keys were designed to work in this model. You can simply activate S3 bucket keys in the default bucket encryption configuration and ask end clients to add their preferred, customer-managed AWS KMS key to Amazon S3 request headers. This allows end clients to encrypt objects with their managed AWS KMS key, and multi-tenant owners can enable S3 bucket keys for objects uploaded to their bucket.

In this design, each tenant provides you with their customer-managed AWS KMS key to provide your services (data analytics, data processing, etc.). If tenants no longer want you to access their data, they revoke access to their key. This model preserves the confidentiality of the critical material while providing access to the relevant parties.

Evaluation of buckets with a high number of reads using SSE-KMS objects

Since introducing S3 Bucket Keys, many customers have re-encrypted their existing data to use Bucket Keys and reduce AWS KMS costs. If you have a bucket with many read-with objects encrypted using SSE-KMS at the object level, you should carefully evaluate your access patterns to see if this makes sense for your applications. You can use S3 Storage Lens to identify AWS KMS requests across your organization and then zoom in on the accounts and buckets with the highest number of AWS KMS requests. If you typically only access your data briefly after it has been created, we recommend simply allowing objects to age out and using S3 Bucket Keys for all new objects. If your data has a heavy read load over an extended period of time, perhaps years, it may make sense to re-encrypt existing SSE-KMS objects at the object level using S3 segment keys.

To encrypt existing Amazon S3 objects with S3 Bucket Keys, you can use Amazon S3 Batch Operations. You provide S3 Batch Operations with a list of objects, and Batch Operations performs a specific operation on each object. You can use an S3 Batch Operations copy operation to copy existing SSE-KMS objects and write them back to the same bucket, thereby enabling S3 bucket keys on those objects. A single batch operation job can perform a specific operation on billions of objects. This post provides an excellent example of using the AWS Lambda function to retrieve the SSE-KMS key identifier for each object and enable the S3 bucket keys by copying the objects into place using the appropriate client-managed key.

Worth knowing

Many customers often want to quickly check the default encryption configuration of their buckets in their accounts for buckets without the S3 Bucket Key enabled. The article above details promptly identifying and correcting buckets without the cost-saving feature enabled.

Changes to keep in mind before activating S3 bucket key

Suppose your existing IAM or AWS KMS key policies use your Amazon Resource Name (ARN) object as contextual encryption to refine or restrict access to your AWS KMS key. In that case, these policies will not work with S3 bucket keys. S3 Bucket Keys use the bucket's ARN as the encryption context. Before you enable S3 Bucket Keys, update your IAM policy or AWS KMS key policy to use the ARN bucket as the encryption context.

When you enable S3 Bucket Keys, you will see fewer AWS KMS CloudTrail events in the logs for SSE-KMS objects because fewer AWS KMS calls are made. In addition, AWS KMS CloudTrail events log the ARN of your bucket instead of the ARN of the object.

Using the Copy API

You can use the Copy operation to re-encrypt an existing object. Note that the Copy operation changes the last modification date of the object, which resets the object's Lifecycle permissions if Lifecycle rules apply. Optionally, you can add a new object tag to identify the re-encrypted objects retroactively during the Copy operation. You can then create a new Lifecycle rule to move these tagged objects to colder storage or expire them at your discretion. In addition, any object larger than 5 GB must be copied using the Multipart Upload API, as the Copy API does not support objects larger than 5 GB.

Ordering

If you have created new resources to test the suggestions shared in this post, remove the client-managed AWS KMS keys and any objects added to the buckets.

Summary

Many customers have compliance requirements that place encryption essential control on the customer. These customers use SSE-KMS with customer-managed keys to comply with these regulatory standards. This article taught you how to enforce SSE-KMS encryption at the bucket level using S3 bucket policies. You also reviewed how to enable S3 bucket keys in the default bucket encryption configuration to take advantage of this cost-saving feature easily. The authors discussed how multi-tenant bucket owners can enable S3 bucket keys while allowing their clients to use their own client-managed KMS keys. Finally, the authors and I reviewed some critical considerations for access patterns before re-encrypting existing SSE-KMS objects to use S3 segment keys.

Activating S3 Bucket Keys is a recommended best practice for customers with data encryption requirements. This feature saves your AWS KMS bill while helping you remain compliant with data encryption standards. From now on, you will have the knowledge and tools to maximize your savings using S3 Bucket Keys.