How To Push ec2 logs to Cloudwatch [Logs & Metrics]

application logs to AWS cloudwatch setup

In this blog, I will show you how to push ec2 logs to Cloudwatch using the Cloudwatch agent. You can also send ec2 system metrics to cloudwatch.

This article walks you through the steps involved in configuring the Cloudwatch agent on an ec2 instance and configuring it to push the logs and metrics generated by the applications & system services.

Here is what you are going to learn in this blog.

AWS Cloud Watch Agent Use Cases

AWS Cloudwatch logs service has the capability to store custom logs and process metrics generated from your application instances. Here are some example use cases for custom logs and metrics

  1. Web server (Nginx, Apache, etc ) access or error logs can be pushed to Cloudwatch logs it acts as central log management for your applications running on AWS
  2. Custom application logs (java, python, etc) can be pushed to cloudwatch and you can set up custom dashboards and alerts based on log patterns.
  3. Ec2 instance metrics/custom system metrics/ app metrics can be pushed to cloudwatch.

Application Logs To AWS Cloudwatch Workflow

You can send logs from any number of ec2 sources to Cloudwatch. All you need to have is a Cloudwatch agent running on your instance.

Here is what you have to do

  1. Create a custom ec2 IAM role with Cloudwatch log write access
  2. Install Cloudwatch logs ec2 agent
  3. Configure log sources in the Cloudwatch agent configuration file.
  4. Start the agent with the configuration file.
  5. Validate logs in the Cloudwatch dashboard.

Note: In an actual project implemention the cloudwatch ec2 agent and configuration would be part of the AMI (Golden Image) or AMI packaging tool like packer.

Application Logs To Cloudwatch

Let’s get started with the setup.

Create an IAM role for Cloudwatch Agent

To set up AWS custom logs, first, you need to create and add a custom ec2 IAM role to your instance. This IAM role will have policies with write access to the Cloudwatch service so that all the logs from ec2 instances can be shipped to Cloudwatch.

Before creating a role, you need to create a custom policy.

Step 1: Head over to AWS IAM –> Policies–> Create Policy

Create policy for cloudwatch custom logs

Step 2: Select the JSON option

Step 3: Copy the following content in the policy block. We are allowing the required permissions and the logs arn details. Read AWS arn detailed guide to know more about arn.

  "Version": "2012-10-17",
  "Statement": [
      "Effect": "Allow",
      "Action": [
      "Resource": [

On the next page, add a tag to the policy. give a name, and description for your policy, and click Next

Cloudwatch policy tag

On the next page, add a policy name, and description and click create policy.

create IAM policy for pushing cloudwatch logs

Once you create the policy, you need to create a role with the custom policy you have created.

Step 4: Head over to AWS IAM –> Roles –> Create Role and select options as shown below.

Step 5: From the filter, select “Customer Managed” and select the Policy you created in step 3.

Step 6: Next, enter a role name and create the role.

Add the Cloudwatch Role to the Instance

Follow the steps given below to add the custom IAM role to the ec2 instance where you want to set up the cloud watch agent.

  1. Head over to ec2 and select the instance in which you want to configure the custom logs.
  2. Right-click for options and select Security and then choose Modify IAM Role option.
  3. Select the custom cloud watch IAM role from the dropdown and save it.

Install Cloudwatch Logs Agent

SSH into the ec2 instance and follow the steps given below.

Step 1: Head over to the Cloudwatch agent downloads page. You can select region-wise packages as well.

Step 2: Download the appropriate agent installation file.

In my case it’s ubuntu. I am downloading the latest Ubuntu package and installing it.

sudo dpkg -i amazon-cloudwatch-agent.deb

Redhat users,
rpm -U amazon-cloudwatch-agent.rpm

Configure Cloudwatch Agent

After the installation, you can find all the cloudwatch agent-related config files and executables in the following location.


Here is the tree structure of the files present in the directory.

AWS cloudwatch agent executables and config files

If you are just starting with a cloud watch agent, it is better to run the cloud watch agent wizard that helps you create the log agent configurations.

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard

It prompts you with all the agent-related questions. Execute the following command to start the wizard.

For the question, Do you want to store the config in the SSM parameter store?, select No.

The final config files get stored in the following location


If you want to collect the system metrics, install collected on your server.

sudo apt-get update -y
sudo apt-get install collectd

In my case, I am going to replace the default config.json with a custom config.json that collects the following logs and system metrics using collectd.

  1. Cloudwatch agent logs
  2. System logs from /var/log/messages
  3. Nginx access logs from /var/log/nginx/access.log
  4. Nginx error logs from /var/log/nginx/error.log

Note: Install Nginx, if you want to follow the following configs for testing purposes. Or you can use you can replace the log locations with your applcation log path.

Here is the final cloudwatch agent config.

	"agent": {
	  "metrics_collection_interval": 10,
	  "run_as_user": "root"
	"logs": {
	  "logs_collected": {
		"files": {
		  "collect_list": [
			  "file_path": "/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log",
			  "log_group_name": "/apps/CloudWatchAgentLog/",
			  "log_stream_name": "{ip_address}_{instance_id}",
			  "timezone": "Local"
			  "file_path": "/var/log/messages",
			  "log_group_name": "/apps/system/messages",
			  "log_stream_name": "{ip_address}_{instance_id}",
			  "timestamp_format": "%b %d %H:%M:%S",
			  "timezone": "Local"
			  "file_path": "/var/log/nginx/access.log",
			  "log_group_name": "/apps/webservers/nginx/access",
			  "log_stream_name": "{ip_address}_{instance_id}",
			  "timestamp_format": "%d/%b/%Y:%H:%M:%S %z",
			  "timezone": "Local"
	"metrics": {
	  "aggregation_dimensions": [
	  "append_dimensions": {
		"AutoScalingGroupName": "${aws:AutoScalingGroupName}",
		"ImageId": "${aws:ImageId}",
		"InstanceId": "${aws:InstanceId}",
		"InstanceType": "${aws:InstanceType}"
	  "metrics_collected": {
		"collectd": {
		  "metrics_aggregation_interval": 60
		"disk": {
		  "measurement": [
		  "metrics_collection_interval": 10,
		  "resources": [
		"mem": {
		  "measurement": [
		  "metrics_collection_interval": 10
		"statsd": {
		  "metrics_aggregation_interval": 60,
		  "metrics_collection_interval": 10,
		  "service_address": ":8125"

Now, let’s start the Cloudwatch agent using the following.

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json -s

You can check the agent status using the following command.

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a status

Validating Custom Logs in Cloudwatch Dashboard

Once the setup is done, you can view all the configured logs under the cloudwatch dashboard (under the logs option)

  1. Go to Logs –> Log Groups and you will see the log group you mentioned in the agent configuration.
  2. Select the log group and you should see the instance identified you mentioned in the config.
  3. If you click the instance identifier, it shows all the logs. You can use the cloud watch filter option to filter and query required logs.
cloudwatch ec2 application logs


I have explained the Cloudwatch logs agent setup to push application logs to the Cloudwatch logging service. It is a manual setup.

If you want this to be automated, all the agent configuration has to be baked in the ec2 AMI. Few configurations can be added at the system startup using the user data scripts. Again, it depends on what workflow you are opting for.

Not only just logs, but you can also push custom metrics to cloudwatch for monitoring.

If you face any issues or have any different ideas, do let me know in the comment section.

  1. Hi Bibin Wilson
    I can able to see application logs in cloudwatch but i want to logs directly to cloudwatch and logs should not store in server

  2. Hi
    I want to install cloud watch agent for all my production account instance, without pushing any log to cloud watch when configuring the cloud watch agent.
    Can you help me to do it.

    1. Hi Ngawang,

      Use tools like Anisble to remotely add the cloudwatch agent. Dont start the agent until you finalize the configuration. You can use ansible again to start the agents on all the nodes. You need to test this setup in other environments before you make changes to production.

  3. Hi i have automated the process and install the amazon-cloudwatch-agent in custom ami and created ec2 instance from that custom ami.

    I want to know how we can add the configuration in start up scripts using user data, can you give me idea on this

  4. Hi, i did everything as you describe but i cant see any log group in aws…
    Where can i connect ec2 instance to cloudwatch

    1. Hi Richard,

      It could be an IAM role issue.Check both IAM policy and if the role is attached to the ec2 instance for writing cloudwatch logs.

  5. Hi, I’m trying to ship EMR logs to CloudWatch. Followed your instructions and it worked like a charm. Thank you so much. But when i try to change the log_group_name, its not getting reflected in the cloudWatch. I still see the old name. Any suggestions.

  6. I would like to monitor the application process itself using PID like we view in linux using top cmd On CLOUD WATCH – is that posible?

    1. Hi Ameer,

      You can monitor the application’s processes using the procast plugin. You have to add these configs under the metrics option in the config file. Please refer to this link

  7. sudo python ./ in this command , i would like to give yes for all and no for “More log files to configure?” How can i do that?

    1. Hi Santhosh,

      I have updated the article. You can directly install the agents without the python file. Check the agent downloads page

      For your use case, instead of going through the wizard, you can create the agent config file /opt/aws/amazon-cloudwatch-agent/bin/config.json with required parameters and use it for all the instances.

Leave a Reply

Your email address will not be published. Required fields are marked *

You May Also Like