How do I configure ongoing replication from an external secret manager to AWS Secrets Manager?

Secrets Managers are a great tool for securely storing confidential information and providing access to secret material to trusted individuals, applications, or systems. In different environments, multiple Secrets Managers may be hosted by other vendors, increasing the complexity of maintaining a consistent operating model for your entries. In these situations, centralizing secrets in one source of truth and replicating subsets in other Secrets Managers can simplify the operational model.

This article explains how you can use an external Secrets Manager as your source of truth while replicating a subset of this information into AWS Secrets Manager. In this way, you can use the data that originates and is managed by your external Secrets Manager in Amazon Web Services (AWS) applications or AWS services that use such a solution.

The authors will demonstrate this approach in the following article by configuring an example open-source HashiCorp Vault to create and maintain secrets and create a replication mechanism that allows these secrets to be used in AWS using AWS Secrets Manager. Although this text uses HashiCorp Vault as an example, the replication mechanism can also be modified to use Secrets Managers from other providers.

Important: This article provides guidance for planning and implementing a replication mechanism for secret entries. The examples in this post are not intended to run directly in a production environment, and you should consider your requirements for enhancing security before implementing this solution. For example, HashiCorp provides tutorials on enhancing production vaults.

When and why consider replicating secret entries?

The primary use case in this article is for customers who run applications on AWS and currently use an external Secrets Manager to manage their secret data, either hosted locally, in the AWS cloud, or with a third-party provider. These customers typically have existing deployment pipeline procedures and processes for managing secret data. Customers with this configuration may want to retain their existing external Secrets Manager and have a set of data accessible to workloads running outside AWS and workloads running within AWS using AWS Secrets Manager.

Another use case is for customers migrating workloads to the AWS cloud and wanting to maintain a (temporary) hybrid form of secret data management. By replicating them from the existing external Secrets Manager, customers can migrate their secrets to the AWS cloud individually, test that they work, integrate them into their intended applications and systems, and delete the external Secret Manager once the migration is complete.

In addition, some AWS services, such as Amazon Relational Database Service (Amazon RDS) Proxy, AWS Direct Connect MACsec, and AD Connector seamless join (Linux), only support secrets from AWS Secrets Manager. Customers can use secret replication if they have a third-party Secret Manager and want to use third-party secrets in services that require integration with AWS Secrets Manager. This way, customers no longer need to manage secrets in two places.

Two approaches to data replication

In this article, the authors will discuss the two main models for replicating secrets from a third-party Secret Manager to an AWS Secrets Manager: the Pull and Push models.

Pull model

In the Pull model, you can use AWS services such as Amazon EventBridge and AWS Lambda to periodically call the external Secret Manager to retrieve and update secrets. The main advantage of this model is that it does not require any significant configuration of the external Secret Manager. AWS resources and the mechanism used to 'pull' the secret entries must have appropriate permissions and network access to these secrets. However, there may be a delay between when a secret key is created and updated and when it is downloaded for replication, depending on the configured download interval from AWS to the external Secret Manager.

Push model

The main advantage of this solution is the minimal delay between creating or updating a secret key and making this data available in AWS Secrets Manager. The Push model also minimizes the network traffic required for replication, as it is a one-way flow. However, this model adds a layer of complexity to replication as it requires additional configuration in the external Secrets Manager. Specifically, the Push model depends on the external Secrets Manager's ability to run event-based 'push' integrations with AWS resources. This will require developing and managing a custom integration on the external Secrets Manager side.

This text focuses on the Pull model to provide an example integration that does not require additional configuration in a third-party Secrets Manager.

Replicate data to AWS Secrets Manager using the Pull model

This section will discuss an example of using the Pull model to replicate data from a third-party Secrets Manager to AWS Secrets Manager.

Overview of the solution

The architecture shown in Figure 1 consists of the following main steps, numbered as shown in the diagram:

1. A Cron statement in Amazon EventBridge calls the AWS Lambda function every 30 minutes.
2. To connect to the external Secrets Manager, the Lambda function written in NodeJS retrieves a set of user-defined API keys belonging to the data manager from AWS Secrets Manager. The scope of these API keys has been restricted to provide read-only access to the secret keys that should be replicated, according to the principle of lowest privilege. For more information, see Step 3: Updating the Vault connection secret key.
3. The third step has two variations depending on where your external secret manager is hosted:
4. Lambda is configured to retrieve data from a third-party Secrets Manager hosted outside AWS. This requires sufficient networking and routing to allow communication from the Lambda function.

Note: Depending on the location of the third-party Secrets Manager, you may need to consider different network topologies. For example, it may be necessary to configure hybrid connectivity between the external environment and the AWS cloud using AWS Site-to-Site VPN or AWS Direct Connect, or both.

1. Lambda is configured to retrieve third-party Secrets Manager secret keys running on Amazon Elastic Compute Cloud (Amazon EC2).

Important: To simplify the implementation of this sample integration, the authors will use Secrets Manager hosted on a publicly accessible Amazon EC2 instance within the same VPC as the Lambda function (3b). This minimizes the additional network components required to interact with the manager. More specifically, the EC2 instance supports the open-source HashiCorp Vault storage. In the remainder of this article, the authors will refer to the HashiCorp Vault API keys as Vault tokens.

1. The Lambda function compares the version of the secret key it retrieved from the external Secrets Manager with the version of the secret key in AWS Secrets Manager (by tag). The function will create a new secret key in AWS Secrets Manager if this one does not exist and update it if a new version is available. The Lambda function will only include secret entries from external Secrets Manager secret entries in the replication if they match the specified prefix. For example, hybrid-aws-secrets/.
2. In the event of a secret key synchronisation error, an email notification is sent to email addresses that are subscribed to the deployed Amazon Simple Notification Service (Amazon SNS) theme. This sample application uses email notifications from Amazon SNS as an example, but can also be integrated with services such as ServiceNow, Jira, Slack or PagerDuty. Learn more about how to use webhooks to publish Amazon SNS messages to external services.

Setting up the solution

This section will discuss implementing the Pull model solution shown in Figure 1, using the following steps:

Step 1: Deploy the solution using the AWS CDK toolkit.

For this article, an AWS Cloud Development Kit (AWS CDK) script has been created and can be found in this AWS GitHub repository. Using the AWS CDK, the infrastructure shown in Figure 1 was defined as Infrastructure as Code (IaC), written in TypeScript, ready to be deployed and tested. AWS CDK is an open-source development platform that allows you to write cloud application infrastructure as code using popular programming languages such as TypeScript, Python, Java, Go, and so on.

Prerequisites

To implement the solution, the following should be present in the system:

1. Git
2. Node (version 16 or later)
3. jq
4. AWS CDK Toolkit. Install using npm (included in the node configuration) by running npm install -g aws-cdk in your local terminal.
5. An AWS access key ID and secret access configured in this way will interact with your AWS account. For more information, see Configuration Basics in the AWS Command Line Interface User Guide.
6. Docker is installed and running on your machine.

To deploy the solution

1. Clone the CDK script for secret replication.

git clone https://github.com/aws-samples/aws-secrets-manager-hybrid-secret-replication-from-hashicorp-vault.git SecretReplication

1. Use the cloned project as the current directory.

cd SecretReplication

1. Install the required dependencies to deploy the application.

npm install

1. Customize the configuration values for your installation in the cdk.json file. For example, you can adjust the secretsPrefix value to change the prefix used by the Lambda function to determine the subset of secret entries that should be replicated from an external secret entry manager.
2. Prepare your AWS environments with some of the resources required to deploy the solution. With correctly configured AWS credentials, run the following command.

cdk bootstrap

The primary resources created by bootstrapping are the Amazon Elastic Container Registry (Amazon ECR) repository for the AWS Lambda Docker image, the Amazon Simple Storage Service (Amazon S3) tray for static resources, and AWS Identity and Access Management (IAM) roles with the relevant IAM policies. You can find the full list of resources by navigating to the CDKToolkit stack in AWS CloudFormation after completing the command.

1. Deploy the infrastructure
2. cdk deploy

This command deploys the infrastructure shown in Figure 1 using the AWS CloudFormation service. After the deployment is complete, you can view the SecretsManagerReplicationStack in AWS CloudFormation for a complete list of resources.

Note: If your local environment does not have a terminal that allows you to run these commands, consider using AWS Cloud9 or AWS CloudShell.

When the deployment completes in the terminal, you should see an output that looks like the one in Figure 2. If successful, the output will include the IP address of the sample HashiCorp Vault and its network interface.

Step 2: Initialise the HashiCrop Vault

Within the output of the deployment script, you will receive a URL to access the user interface of the open HashiCorp Vault storage. To simplify accessibility, the URL points to the publicly accessible Amazon EC2 instance on which the HashiCorp Vault user interface runs, as shown in step 3b in Figure 1.

It is time to look at the HashiCorp Vault you created with your own eyes. Navigate to the URL in your browser, and you should see the Raft Storage initialization page, as shown in Figure 3.

The vault requires initial configuration to prepare the storage and obtain an initial set of master keys. You can go through these steps manually in the HashiCorp Vault user interface, but it is recommended that you use the initialise_vault.sh script, which is part of the SecretsManagerReplication project.

Using the HashiCorp Vault API, the initialize script will automatically do the following:

1. Initialise Raft memory to allow Vault to store secrets locally in the instance.
2. Create an initial set of keys to unseal the Vault. Importantly, for demonstration purposes, the script uses a single key share. In production environments, it is recommended that multiple key shares are used so that in an emergency, various shares are needed to reconstruct the master key.
3. Save the unseal keys in init/vault_init_output.json in your project.
4. Open the HashiCorp vault with the previously generated keys.
5. Enable two key-value secret data engines:
6. An engine named after the prefix used for replication is defined in the cdk.json file. In this example, it is hybrid-aws-secrets. The developers intend to use the secrets in this engine for replication to AWS Secrets Manager.
7. You will use An engine called super-secret-engine to show that your replication engine cannot access secret data outside of the engine used for replication.
8. Create three sample secrets, two in hybrid-aws-secrets and one in super-secret-engine.
9. Create a read-only rule that can be seen in the init/replication-policy-payload.json file after the script has completed. This rule allows read-only access only to data that should be replicated.
10. Create a new vault token with a read-only policy attached so that the AWS Lambda function can later use it to retrieve secret information for replication.

Return to the terminal and run the following command to run the initialize script. ./initialise_vault.sh

The script will ask you to enter the IP address of your HashiCorp vault. Enter the IP address (without the port) and select enter. Enter y for the script to create some sample secret entries.

If all is successful, you should see an output containing tokens to access the HashiCorp vault, similar to that shown in Figure 4.

The installation script has generated two tokens: one master token, which will be used for administrative tasks, and a read-only token, which will be used to read secret information for replication. Ensure you can access these tokens when performing the steps described in this text.

Note: The master token is used in this article for demonstration purposes only. In production environments, root tokens are not used for normal administrative activities. Instead, limited roles should be used depending on the organization's needs. In this case, the root token highlights that data in the super-secret engine is not intended for replication. These secret entries cannot be seen or accessed using the read-only token.

Go back to your browser and refresh the HashiCorp Vault interface. You should now see the Sign in to Vault page. Sign in using the Token method and use your master token. If you do not already have a master token in your terminal, you can find it in the init/vault_init_output.json file.

Once logged in, you should see an overview page with the three secret entry engines enabled, as shown in Figure 5.

If you explore hybrid-aws-secrets and super-secret-engine, you can see the data automatically created by the init script. For example, the first replication secret key contains an example of a key-value secret key along with the hidden keys and value manager.

If you go to Policies on the top navigation bar, you will also see the aws-replication-read-only policy, as shown in Figure 6. This policy provides read-only access to the hybrid-aws-secrets path.

The read-only policy is attached to a read-only token that you will use to replicate Lambda functions secretly. This policy is crucial because it limits the access gained by the Lambda function using the token to the specific prefix intended for replication. For secret replication, we only need to perform read operations. This policy ensures that we can read but cannot add, change, or delete any secrets in the HashiCorp Vault using the token.

You can verify the permissions of the read-only token by logging into the HashiCorp Vault user interface using the read-only token rather than the master token. You should now only see hybrid-aws-secrets. You can no longer access the super-secret engine you saw in Figure 5. If you try to create or update a secret key, you will get an error indicating permission is denied.

Great! Your HashiCorp Vault is ready to replicate its secret data from hybrid-aws-secrets to AWS Secrets Manager. The following section describes the final configuration you need to perform to allow the replication mechanism in AWS to access your HashiCorp Vault data.

Step 3: Update the Vault secret connection key

To allow secret replication, you must grant the AWS Lambda function access to the HashiCorp Vault read-only token created by the initiator script. To do this, you must update the vault-connection-secret initialized in AWS Secrets Manager as part of the AWS CDK implementation.

The authors will demonstrate how to do this using the AWS Management Console, but you can also do it programmatically using the AWS Command Line Interface (AWS CLI) or AWS SDK via the update-secret command.

To update the Vault connection secret key (console):

1. In the AWS Management Console, navigate to AWS Secrets Manager > Secrets > hybrid-aws-secrets/vault-connection-secret.
2. Under Secret value, select Recover secret value and then choose Edit.
3. Update the vaultToken value to include the read-only token generated by the initialization script.

Step 4: (Optional) Configure email notifications for replication errors.

As presented in Figure 1, the Lambda function will send an email via Amazon SNS to a designated email address when one or more secret keys are not replicated. You will need to configure the solution to use the correct email address. To do this, navigate to the cdk.json file in the root directory of the SecretReplication folder and adjust the notice email parameter to the email address you own. When you have finished, implement the changes using the cdk deploy command. Within a few minutes, you will receive an email asking you to confirm your subscription. In the future, you will receive email notifications if replication of at least one secret key fails.

Test the secret replication

You can wait up to 30 minutes for the Lambda function to be automatically called to replicate secret entries, or call the function manually.

To test your secret replication:

1. Open the AWS Lambda console and find the Secret Replication function (the name starts with SecretsManagerReplication-SecretReplication).
2. Go to the Test tab.
3. For the text event action, select Create New Event, create the event using the default parameters and then select the Test button on the right, as shown in Figure 8.

This will trigger the function. At the same time, a success message should be displayed, as shown in Figure No. 9. If this is the first time the Lambda function has been called, you will see in the results that two secret entries have been created.

The relevant logs for the Lambda function call can be found in the AWS CloudWatch Log group called /aws/lambda/SecretsManagerReplication-SecretReplicationLambdaF-XXXX.

To check that the secret has been added, go to AWS Secrets Manager in the console. In addition to the vault-connection-secret you edited earlier, you should now see two new secret keys with the same hybrid-aws-secrets prefix, as shown in Figure 10.

For example, if you look at first-secret-for-replication, you can see the first version of the secret key, with the secret vital secrets and the secret key value manager, as shown in Figure No. 11.

Success! You now have access to the secret values from the HashiCorp Vault in AWS Secrets Manager. Also note that the secret key has a version tag attached. This element is required to update the key, which you will learn more about in the next two sections.

Update the key

The recommended security practice is to rotate secret entries frequently. The Lambda function in this solution not only replicates keys as they are created but also periodically checks whether existing secrets in AWS Secrets Manager should be updated when an external Secrets Manager (in this case, HashiCorp Vault) has a new version of the secret. To check if this works, you can manually update the key in your HashiCorp Vault and observe its replication in AWS Secrets Manager in the same way as described in the previous section. You will notice that the version tag of your secret key is updated automatically when there is a new replication of the secret key from the external Secrets Manager to the AWS Secrets Manager.

Secret replication logic

This section will explain the secret replication logic in more detail. It is time to look at the sequence diagram below, which explains the general logic implemented in the Lambda function.

This diagram highlights that the Lambda function will first retrieve a list of secret names from the HashiCorp Vault. The function will then get a list of keys from AWS Secrets Manager, matching the prefix configured for replication. AWS Secrets Manager will return a list of keys matching that prefix and also return their metadata and tags. Note that the function has not yet retrieved any secret material.

The function will then go through each secret name provided by HashiCorp Vault and check if the secret exists in AWS Secrets Manager:

• Suppose there is no critical matching of that name. In that case, the function will retrieve the secret material from the HashiCorp Vault, including the version number, and create a new secret in AWS Secrets Manager. It also adds a version tag to the hidden key to match the version.
• If a key already exists in AWS Secrets Manager that matches this name, the Lambda function will first retrieve the metadata from this key in HashiCorp Vault. This is required to obtain the version number of the secret key, as the version number was not revealed when the function initially retrieved the list of secret keys from HashiCorp Vault. If the secret version from HashiCorp Vault does not match the secret key version value in AWS Secrets Manager (for example, the version in HashiCorp Vault is two and the version in AWS Secrets Manager is 1), an update is required to get the values back in sync. Only now will the Lambda function retrieve the actual secret material from HashiCorp Vault and update the secret in AWS Secrets Manager, including the version number in the tag.

The Lambda function retrieves metadata about the keys rather than immediately retrieving the secret material from the HashiCorp Vault. Typically, secret entries are not updated very often. If the Lambda function is called every 30 minutes, it should not add or update any keys in most calls. Using metadata to determine if a secret is needed to create or update secrets, you minimize the number of secret downloads from HashiCorp Vault and AWS Secrets Manager.

Note: the AWS Lambda feature has permission to download specific secret keys from HashiCorp Vault. It is essential to carefully review the Lambda code and any subsequent changes to prevent data leakage. For example, ensure that the Lambda function is not updated with code that inadvertently records secret material outside of the Lambda function.

Use your key

Now that you have created and replicated your keys, you can use them in your AWS applications or AWS services integrated with Secrets Manager. For example, you can use your keys when setting up a connection for a proxy server in Amazon RDS as follows.

To use a secret key when setting up a proxy server in Amazon RDS:

1. Go to the Amazon RDS service in the console.
2. In the left navigation pane, select Proxy and then select Create Proxy.
3. On the Connectivity tab, you can now select the first replication secret key or the second one created by the Lambda function after replicating them from the HashiCorp Vault.

Note that consumers of replicated secret keys in AWS Secrets Manager will require restricted IAM permissions to use the hidden keys and AWS Key Management Service (AWS KMS) used to encrypt the secrets. For example, see Step 3: Create IAM roles and policies on the Configure shared database connections using the Amazon RDS proxy.

Manage permissions

Due to the confidential nature of the keys, it is essential to keep the scope of permissions as small as possible. This will prevent accidental access to secret information. The installer adopts a permissions strategy with the lowest level of permissions, in which only necessary actions on resources required for replication are explicitly allowed. However, permissions must be verified according to your security standards.

The architecture of this solution has two main places where you can control access to manage your secrets in Secrets Manager.

Lambda execution IAM role: The IAM role taken over by the Lambda function during execution contains the appropriate permissions for secret replication. There is an additional security measure that explicitly rejects any action on a resource that is not required for replication. For example, the Lambda function only has permission to publish to an Amazon SNS topic created for failed replication and will explicitly deny the action of publishing to any other topic. Even if someone accidentally adds permission to the policy for another topic, an explicit denial will still block that action.

AWS KMS key policy: when other services need access to the replicated secret key in AWS Secrets Manager, they also need permission to use the AWS KMS hybrid-aws-secrets-encryption-key. You must allow the principal these permissions via the AWS KMS key policy. In addition, you can manage the principal's AWS KMS key permissions via the identity policy. For example, this is required when accessing AWS KMS keys on AWS accounts. See Permissions for AWS services in key policies and Specifying KMS keys in IAM policy statements in the AWS KMS Developer Guide.

Customisation options for the example solution

The solution discussed in this post is an example of replicating keys from HashiCorp Vault to AWS Secrets Manager using the Pull model. This section provides additional customization options for you to consider when configuring the solution or your variation of it.

1. Depending on the solution you are using, you may have access to different metadata attached to keys that you can use to determine if a key should be updated. For example, if you can access data representing the last_updated_datetime property, you can use this to determine whether a secret key should be updated.
2. It is recommended that long-lived tokens are not used if possible. This example uses a static Vault token to give the Lambda function access to the HashiCorp Vault. Depending on your solution, you may want to implement better authentication and authorisation mechanisms. For example, HashiCorp Vault allows you to use IAM authentication using AWS IAM instead of a static token.
3. This post was about creating keys and updating them, but for a production setup you should also consider deleting keys. Depending on your requirements, you can choose the strategy that best suits your needs for handling secret data in AWS Secrets Manager after deleting the original key in HashiCorp Vault. In the Pull model, you may consider removing the key in AWS Secrets Manager if the corresponding key in the external Secrets Manager is no longer present.
4. In the example configuration, the same AWS KMS key is used to encrypt both Lambda function environment variables and keys in AWS Secrets Manager. You could add an additional AWS KMS key (which would incur additional costs) to have two separate keys for these tasks. This would allow you to apply more specific permissions to the two keys in the respective KMS key policies or IAM identity policies that use the keys.

Conclusions

With this article, you have learned how you can approach replicating your keys from an external Secret Manager to AWS Secrets Manager. This text focused on the Pull model, where the solution periodically retrieved keys from the external HashiCorp Vault and automatically created or updated the corresponding key in AWS Secrets Manager. Using this model, you can now use external keys in your AWS Cloud applications or services that are integrated with AWS Secrets Manager.