How To Configure Loki S3 For Log Storage

In this blog, we will look into setting up AWS S3 in Loki for storing its logs.

We will not just configure Loki S3, but we will also make use of S3's features like lifecycle policy and bucket policy.

Configuring an external object storage is important for data persistence, because if you don't configure an external object storage, Loki stores logs on its pod's filesystem, which will be deleted only when the pod restarts or fails.

With the help of external objects storage we can store large volume of log data and its scales according to the data size.

External object storage is the best option for storing logs for a long time, since the local file system storage is temporary.

By configuring an external object storage you don't have to worry about data loss at all. You can also refer to the official documentation for available storage options.

In the end, you will get a clear understanding of how to configure S3 for Loki.

How does Loki Integrate with S3?

The workflow diagram shows how Loki integrates with S3 for logs storage.

Explanation:

Promtail collects logs from /var/log/pods and sends them to Loki.
Once Loki gets the logs, it compresses and stores as chunks with its index in the local.
The chunks and indexes stored on the local is pushed to S3 based on certain conditions, when the specified push time is reached, when the chunks reach the specified time limit, etc.
Loki pushes the log chunks and indexes to S3 using the role attached to the service account for S3 permission, like get, put, delete.
The S3 has a bucket policy that only gives access to a specific role, and the lifecycle policy deletes the object once after the specified time, e.g., 30 days.

Setup Prerequisites

The following are the prerequisites for this setup.

Helm installed in your system
Kubernetes Cluster
OIDC enabled on your cluster
Kubectl
AWS CLI with access to create IAM roles and S3 bucket
Loki setup in your cluster

Steps to Configure Loki S3 for Log Storage

Below are the steps for configuring AWS S3 in Loki to store its logs.

Step 1: Create a S3 Bucket

Let's start with creating an S3 bucket, we can create an S3 bucket using a single command.

Run the following command to create the S3 bucket.

aws s3api create-bucket \
    --bucket <bucket-name> \
    --region us-west-2 \
    --create-bucket-configuration LocationConstraint=us-west-2

Then, run the following command to check if your bucket is created.

aws s3 ls

You will get your bucket name as shown below if it's created.

aws command to check the available s3 buckets

Step 2: Create IAM Policy for Loki

The next step is to create an IAM policy for Loki to push logs to S3.

Run the following command to create a file with the required policy.

cat << EOF | envsubst > loki-policy.json
{
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": [
                    "s3:ListBucket",
                    "s3:GetObject",
                    "s3:PutObject",
                    "s3:DeleteObject"
                ],
                "Resource": [
                    "arn:aws:s3:::<bucket-name>",
                    "arn:aws:s3:::<bucket-name>/*"
                ]
            }
        ]
    }
}

Give a bucket name in the above command before running it.

This policy will give Loki to list, get, put, and delete permission to all the objects inside the specified bucket.

Now, run the following command to create the policy.

aws iam create-policy \
    --policy-name loki_bucket_policy \
    --policy-document file://loki-policy.json

Then run the following command to get the ARN of the policy, which will be needed in upcoming steps.

export POLICY_ARN=$(aws iam list-policies --query "Policies[?PolicyName=='loki_bucket_policy'].Arn" --output text)

Step 3: Create IAM Role for Loki

Before creating the role, get the OIDC ID attached to the cluster using the following command.

aws eks describe-cluster --name <cluster-name> --query "cluster.identity.oidc.issuer" --output text

Update the cluster name before running the above command.

Now, run the following command to create a file with the roles trustpolicy.

cat << EOF | envsubst > trust-policy.json
{
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": "sts:AssumeRoleWithWebIdentity",
                "Principal": {
                    "Federated": "arn:aws:iam::<account-id>:oidc-provider/<oidc-id>"
                },
                "Condition": {
                    "StringEquals": {
                        "<oidc-id>:aud": [
                            "sts.amazonaws.com"
                        ]
                    }
                }
            }
        ]
    }
}

In the above command, update your account ID and OIDC ID. Add the OIDC ID without the https:// on it.

Now, run the following command to create the role.

aws iam create-role \
    --role-name loki_bucket_role \
    --assume-role-policy-document file://trust-policy.json

Then, run the following command to attach the policy to the role.

aws iam attach-role-policy \
    --role-name loki_bucket_role \
    --policy-arn $POLICY_ARN

Once it is created, run the following command to get the ARN of the role.

aws iam get-role --role-name loki_bucket_role --query "Role.Arn" --output text

Note the ARN of the role from the output, we need that while creating the bucket policy and updating the Loki configuration.

Step 4: Add Bucket Policy to your S3 Bucket

Adding bucket policy to the bucket is a security best practice to limit the access to the bucket.

In this setup, we will use the bucket policy to allow access only to a specific role attached to the service account.

Run the following command to create a policy file.

cat << EOF | envsubst > s3-bucket-policy.json
{
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Principal": {
                    "AWS": "<role-arn>"
                },
                "Action": [
                    "s3:GetObject",
                    "s3:PutObject",
                    "s3:DeleteObject"
                ],
                "Resource": "arn:aws:s3:::<buckt-name>/*"
            },
            {
                "Effect": "Allow",
                "Principal": {
                    "AWS": "<role-arn>"
                },
                "Action": "s3:ListBucket",
                "Resource": "arn:aws:s3:::<buckt-name>"
            }
        ]
    }
}

Update your role arn and bucket name in the above command.

Then, run the following command to add the bucket policy to the bucket.

aws s3api put-bucket-policy --bucket <bucket-name> --policy file://s3-bucket-policy.json

Add the bucket name before running the command.

Step 5: Add LifeCycle Policy to your S3 Bucket

The next step is to add a lifecycle policy to your bucket for automatically deleting the objects in the bucket after a certain time.

Run the following command to create a policy file.

cat << EOF | envsubst > lifecycle-policy.json
{
    {
        "Rules": [
            {
                "ID": "DeleteObjectsAfter30Days",
                "Status": "Enabled",
                "Prefix": "",
                "Expiration": {
                "Days": 30
                }
            }
        ]
    }
}

This policy will delete the object once it becomes 30 days old.

Now, run the following command to add the policy to the bucket.

aws s3api put-bucket-lifecycle-configuration --bucket <bucket-name> --lifecycle-configuration file://lifecycle-policy.json

Update the bucket name in the above command before running it.

Step 6: Update Loki Configuration

The final step is to update the Loki configuration. If you have deployed Loki using this guide, update the helm values file as follows.

loki:
  image:
    registry: docker.io
    repository: grafana/loki
    tag: 3.4.2
  commonConfig:
    replication_factor: 1
  schemaConfig:
    configs:
      - from: "2024-04-01"
        store: tsdb
        object_store: s3
        schema: v13
        index:
          prefix: loki_index_
          period: 24h
  limits_config:
    allow_structured_metadata: true
    retention_period: 168h
    split_queries_by_interval: 15m
    volume_enabled: true

  storage:
    type: s3
    bucketNames:
      chunks: <bucket-name>
      ruler: <bucket-name>
    s3:
      region: <region>
      insecure: false

  auth_enabled: false

storage_config:
  aws:
    region: <region>
    bucketnames: <bucket-name>
    s3forcepathstyle: false
  tsdb_shipper:
    active_index_directory: /data/tsdb-index
    cache_location: /data/tsdb-cache
    cache_ttl: 12h
    flush_interval: 1m
    object_store: s3
    bucket: <bucket-name>

serviceAccount:
  create: true
  annotations:
    "eks.amazonaws.com/role-arn": "<role-arn>"

minio:
  enabled: false
      
deploymentMode: SingleBinary

singleBinary:
  replicas: 1
  persistence:
    storageClass: gp2
    accessModes:
      - ReadWriteOnce
    size: 20Gi

  resources:
    requests:
      cpu: "1"
      memory: "2Gi"
    limits:
      cpu: "2"
      memory: "4Gi"

sidecar:
  image:
    repository: kiwigrid/k8s-sidecar
    tag: 1.30.0
  resources:
    requests:
      cpu: 50m
      memory: 50Mi
    limits:
      cpu: 100m
      memory: 100Mi

backend:
  replicas: 0
read:
  replicas: 0
write:
  replicas: 0

chunksCache:
  allocatedMemory: 500

In the above YAML file, update the region, bucket name, and role arn before using the helm upgrade command.

The additional lines and changes made are given below.

  schemaConfig:
    configs:
      - from: "2024-04-01"
        store: tsdb
        object_store: s3

  limits_config:
    allow_structured_metadata: true
    retention_period: 168h
    split_queries_by_interval: 15m
    volume_enabled: true

  storage:
    type: s3
    bucketNames:
      chunks: <bucket-name>
      ruler: <bucket-name>
    s3:
      region: <region>
      insecure: false
storage_config:
  aws:
    region: <region>
    bucketnames: <bucket-name>
    s3forcepathstyle: false
  tsdb_shipper:
    active_index_directory: /data/tsdb-index
    cache_location: /data/tsdb-cache
    cache_ttl: 12h
    flush_interval: 1m
    object_store: s3
    bucket: <bucket-name>

serviceAccount:
  create: true
  annotations:
    "eks.amazonaws.com/role-arn": "<role-arn>"

Under the schemaConfig the object_store is changed to S3 from the filesystem.
The limits_config section contains the options like retention period, query timeout, etc.
The storage section contains the information about the bucket type, which is S3, and the bucket name.
The storage_config block specifies the S3 bucket and the tsdb_shipper option, like the time interval for pushing chunks to S3.
And, in the serviceAccount block we will specify the ARN of the role we created in the previous step.

Loki may not push the chunks to S3 within the specified time, it may change based on the no of chunks collected, chunk size reached the limit, etc.

Once the values file is updated, run the following command to upgrade the changes.

helm upgrade --install loki grafana/loki -n loki -f loki.yaml

Step 7: Verify if Log Chunks are Pushing to S3

Now, the Loki is configured to push logs to S3, let's verify if the chunks are pushing to S3.

Loki will not push the log chunks immediately once it is configured; it pushes depending on the time interval specified, chunks reaching a certain size, etc.

Loki only pushes the chunks with a time interval; it may take hours for you to see the log chunks on S3.

Once the chunks and index are pushed to S3, you can see the objects in S3.

Common Issues

The common issues you may face during the configuration are given below.

Pod is in pending or crashloopback state, check:
- The cluster has the required CPU and Memory for the pod to deploy.
- If the PV is created and attached to the pod.
Getting S3 permission denied error on Loki pod log, check:
- The policy you created has the required S3 permissions like list, get, put, and delete.
- The policy is added to the role.
- Check OIDC is enabled on the cluster and the correct OIDC ID is specified in the roles trustpolicy.
- Correct role ARN is attached to the service account.
- Loki pod is using the correct service account.
Log Chunks not pushing to S3, check:
- The bucket name you specified is correct.
- The role arn specified on the bucket policy is the same as the role you attached to Loki's service account.