External Secrets Operator Setup for EKS using Secrets Manager

In this blog we will look at Kubernetes External secrets operator setup on AWS EKS and integrate with AWS secrets manager for fetching secrets.

What is an External Secrets Operator?

External Secrets Operator is an operator for Kubernetes that manages Kubernetes Secrets on external secrets managers like AWS Secrets Manager, Google Secrest Manager, Azure Key Vault, etc.

Here is how it works.

External Secret Operator fetches the secrets stored in External Secrets Managers like AWS Secrest Manager and creates encrypted Kubernetes Secrets on the cluster.

The SecretStore contains the authentication details for the External Secrets Managers, which helps the External Secrest Operator to access the secret.

The Kubernetes Secrets created by the External Secrets Operator get the required secrets for an application from the External Secrets Manager and refresh the Secrets with a time period to keep it up to date.

After the Kubernetes Secret is created you can use the secrets in the application by specifying the Secret name and Secret key in which the secret values are mapped.

External Secrets Operator Setup on EKS

I am going to use AWS Secrets Manager as an External Secret Manager and the prerequisites are given below.

Prerequisites

1. AWS account with access to IAM and EKS
2. AWS CLI
3. Helm
4. kubectl
5. eksctl

If you are ready with the prerequisites, follow the steps below to set up an External Secrets Operator on EKS.

Step 1: Install External Secrets Operator

We are going to install External Secrets Operator using Helm, add the repo for External Secrets Operator on your system using the command given below

helm repo add external-secrets https://charts.external-secrets.io

After adding the repo install External Secrets Operator using the command

helm install external-secrets \
  external-secrets/external-secrets \
    --namespace external-secrets \
    --create-namespace \
    --set installCRDs=true

This command creates a namespace external-secrets and installs the External Secrets Operator on the external-secrets namespace.

The CRDs are set to true so that the Helm can install custom resources for the External Secret Operator on the cluster.

Step 2: Create an IAM Role for the Service Account

Before creating the IAM role you need to associate the EKS cluster with an OIDC provider using the command

eksctl utils associate-iam-oidc-provider --cluster=eks-cluster --approve

Make sure to replace eks-cluster with your cluster name in the above command.

After associating the OIDC provider to the cluster, create an IAM Role with the required permissions to retrieve the secret from the Secret Manager.

To create a role go to IAM -> Roles -> Create role and select the Trusted entity type as Web identity.

Since you have already have created an OIDC provider for our cluster using the previous command, you can see your cluster OIDC provider and audience as shown below

Click the Next button, on the next page you can select the policy that gives permission to the Secrets Manager

Once you have selected the policy go to the next page there you can specify the name of your Role and press the Create button to create the Role.

Step 3: Create a Service Account

To create a Service Account, create a YML file sa.yml and copy the below content

apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-secrets-operator
  namespace: external-secrets
  annotations:
    eks.amazonaws.com/role-arn: {iam-role-arn}

Run this YML file to create a service account external-secrets-operator on the namespace external-secrets.

Make sure to replace iam-role-arn with the IAM Role ARN you created before.

Now, run the following command to create the service account.

kubectl apply -f sa.yml

Run the following command to check if the service account has been created successfully

kubectl get sa -n external-secrets

You will get the following output as shown below

Step 4: Create a SecretStore

To create a SecretStore, create a YML file ss.yml and copy the below content

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secrets-manager
  namespace: external-secrets
spec:
  provider:
    aws:
      service: SecretsManager
      region: {region}
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-operator

Make sure to replace region with the region you are using.

Run this YML file to create a SecretStore aws-secrets-manager on the namespace external-secrets.

According to the above file, the SecretStore tells the External Secrets controller to fetch and store the secrets from the Secret Manager in the EKS cluster.

It uses the role we attached to the Service Account external-secrets-operator for authentication.

Now, run the following command to create the secret store.

kubectl apply -f ss.yml

Run the following command to check if the secret store has been created successfully

kubectl get secretstore -n external-secrets

You will get the following output as shown below

You can see in the above image, the status of the secret store is valid and it is ready to use.

Step 5: Create a Secret

To create a Secret, create a YML file secret.yml and copy the below content

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: secret
  namespace: external-secrets
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: secrets-manager-secret
    creationPolicy: Owner
  data:
  - secretKey: aws-secretsmanager
    remoteRef:
      key: {secret-name}
      property: {secret-key}

Make sure to replace secret-name and secret-key with your Secrets Manager Secrets name and key of the value you want to fetch.

The above YML file stores the secret on the target secrets-manager-secret on the namespace external-secrets.

For every 1hr it refreshes and updates the secrets from the SecretStore.

It fetches the species secret from SecretStore aws-secrets-manager and stores it in the target secret secrets-manager-secret.

To use the secret add a block on the deployment file with the target name and secretKey.

For example, according to the above YML file, the target name is secrets-manager-secret and the secretKey is aws-secretsmanager, then the block to use the secret will be

- name: AWS_SECRET
  valueFrom:
     secretKeyRef:
        name: secrets-manager-secret
        key: aws-secretsmanager

Now, run the following command to create a Secret and set sync between SecretStore and Secret in the Cluster.

kubectl apply -f secret.yml

Run the following command to check if the secret has been created successfully

kubectl get secret secrets-manager-secret -n external-secrets

You will get the following output as shown below