External Secrets Operator Setup for EKS using Secrets Manager

In this blog we will look at Kubernetes External secrets operator setup on AWS EKS and integrate with AWS secrets manager for fetching secrets.

What is an External Secrets Operator?

External Secrets Operator is an operator for Kubernetes that manages Kubernetes Secrets on external secrets managers like AWS Secrets Manager, Google Secrest Manager, Azure Key Vault, etc.

Here is how it works.

External Secret Operator fetches the secrets stored in External Secrets Managers like AWS Secrest Manager and creates encrypted Kubernetes Secrets on the cluster.

The SecretStore contains the authentication details for the External Secrets Managers, which helps the External Secrest Operator to access the secret.

The Kubernetes Secrets created by the External Secrets Operator get the required secrets for an application from the External Secrets Manager and refresh the Secrets with a time period to keep it up to date.

After the Kubernetes Secret is created you can use the secrets in the application by specifying the Secret name and Secret key in which the secret values are mapped.

External Secrets Operator Setup on EKS

I am going to use AWS Secrets Manager as an External Secret Manager and the prerequisites are given below.

Prerequisites

1. AWS account with access to IAM and EKS
2. AWS CLI
3. Helm
4. kubectl
5. eksctl

If you are ready with the prerequisites, follow the steps below to set up an External Secrets Operator on EKS.

Step 1: Install External Secrets Operator

We are going to install External Secrets Operator using Helm, add the repo for External Secrets Operator on your system using the command given below

helm repo add external-secrets https://charts.external-secrets.io

After adding the repo install External Secrets Operator using the command

helm install external-secrets \
  external-secrets/external-secrets \
    --namespace external-secrets \
    --create-namespace \
    --set installCRDs=true

This command creates a namespace external-secrets and installs the External Secrets Operator on the external-secrets namespace.

The CRDs are set to true so that the Helm can install custom resources for the External Secret Operator on the cluster.

Step 2: Create an IAM Role for the Service Account

Before creating the IAM role you need to associate an existing OIDC provider with your EKS cluster using the command

eksctl utils associate-iam-oidc-provider --cluster=eks-cluster --approve

If there is no existing OIDC provider, create a new OIDC provider in your AWS consul under IAM->Identity providers then run the above command.

Make sure to replace eks-cluster with your cluster name in the above command.

After associating the OIDC provider to the cluster, create an IAM Role and IAM Policy with the required permissions to retrieve the secret from the Secret Manager.

An example IAM Role trust policy and IAM Policy JSON template is given below.

IAM Role trust policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::{account-id}:oidc-provider/{OIDC-URL}"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "{OIDC-URL}:sub": "system:serviceaccount:{namespace}:{service-account-name}"
                }
            }
        }
    ]
}

In the above JSON template make sure to replace account-id, OIDC-URL, namespace, and service-account-name with your account ID and OIDC URL.

You can find the OIDC URL in your cluster under Overview->OpenID Connect provider URL, replace the OIDC URL without https://.

For example, if your OIDC URL is https://oidc.eks.us-west-2.amazonaws.com/id/H7YU89GH45KA6H, then replace it in the above template as oidc.eks.us-west-2.amazonaws.com/id/H7YU89GH45KA6H.

IAM Policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "secretsmanager:GetSecretValue",
                "secretsmanager:ListSecretVersionIds"
            ],
            "Resource": "arn:aws:secretsmanager:{region}:{account-id}:secret:*",
            "Effect": "Allow"
        }
    ]
}

Make sure to replace region and account-id in the above JSON template.

This policy gives access to a list and gets every secret from the secret manager.

Step 3: Create a Service Account

To create a Service Account, create a YML file sa.yml and copy the below content

apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-secrets-operator
  namespace: external-secrets
  annotations:
    eks.amazonaws.com/role-arn: {iam-role-arn}

Run this YML file to create a service account external-secrets-operator on the namespace external-secrets.

Make sure to replace iam-role-arn with the IAM Role ARN you created before.

Now, run the following command to create the service account.

kubectl apply -f sa.yml

Run the following command to check if the service account has been created successfully

kubectl get sa -n external-secrets

You will get the following output as shown below

Step 4: Create a SecretStore

To create a SecretStore, create a YML file ss.yml and copy the below content

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secrets-manager
  namespace: external-secrets
spec:
  provider:
    aws:
      service: SecretsManager
      region: {region}
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-operator

Make sure to replace region with the region you are using.

Run this YML file to create a SecretStore aws-secrets-manager on the namespace external-secrets.

According to the above file, the SecretStore tells the External Secrets controller to fetch and store the secrets from the Secret Manager in the EKS cluster.

It uses the role we attached to the Service Account external-secrets-operator for authentication.

Now, run the following command to create the secret store.

kubectl apply -f ss.yml

Run the following command to check if the secret store has been created successfully

kubectl get secretstore -n external-secrets

You will get the following output as shown below

You can see in the above image, the status of the secret store is valid and it is ready to use.

Step 5: Create a Secret

To create a Secret, create a YML file secret.yml and copy the below content

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: secret
  namespace: external-secrets
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: secrets-manager-secret
    creationPolicy: Owner
  data:
  - secretKey: aws-secretsmanager
    remoteRef:
      key: {secret-name}
      property: {secret-key}

Make sure to replace secret-name and secret-key with your Secrets Manager Secrets name and key of the value you want to fetch.

The above YML file stores the secret on the target secrets-manager-secret on the namespace external-secrets.

For every 1hr it refreshes and updates the secrets from the SecretStore.

It fetches the species secret from SecretStore aws-secrets-manager and stores it in the target secret secrets-manager-secret.

To use the secret add a block on the deployment file with the target name and secretKey.

For example, according to the above YML file, the target name is secrets-manager-secret and the secretKey is aws-secretsmanager, then the block to use the secret will be

- name: AWS_SECRET
  valueFrom:
     secretKeyRef:
        name: secrets-manager-secret
        key: aws-secretsmanager

Now, run the following command to create a Secret and set sync between SecretStore and Secret in the Cluster.

kubectl apply -f secret.yml

Run the following command to check if the secret has been created successfully

kubectl get secret secrets-manager-secret -n external-secrets

You will get the following output as shown below