Amazon CloudWatch is a monitoring and observability service that provides you with data and actionable insights to monitor your applications, respond to system-wide performance changes, optimize resource utilization, and get a unified view of operational health. In this tutorial, we are going to learn how to push EC2 logs to CloudWatch.

Create an IAM role for Cloudwatch

First, we need to create and add a custom EC2 IAM role to your instance. This IAM role will have policies with write access to the Cloudwatch service so that all the logs from ec2 instances can be shipped to CloudWatch. Before creating a role, we need to create a custom policy. To create policy:

Step 1: Go to AWS IAM –> Policies–> Create Policy
Step 2: Select the JSON option
Step 3: Copy and paste the below JSON code

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents",
                "logs:DescribeLogStreams"
            ],
            "Resource": [
                "arn:aws:logs:*:*:*"
            ]
        }
    ]
}

On the next page, add a tag to the policy. Give a name, and description for your policy, and click Next. On the next page, add a policy name, and description and click create policy. Once we create the policy, we need to create a role with the custom policy we have created.

Step 4: Head over to AWS IAM –> Roles –> Create Role and select options as shown below.

alt text

Step 5: From the filter, select “Customer Managed” and select the Policy we created before.

alt text

Step 6: Next, enter a role name and create the role.

Add the CloudWatch Role to the Instance

Now we need add the custom IAM role to the EC22 instance where we want to set up the cloud watch agent.

Step 1: Head over to EC2 and select the instance in which you want to configure the custom logs.
Step 2:Right-click for options and select Security and then choose Modify IAM Role option.
Step 3:Select the custom Cloud wWtch IAM role from the dropdown and save it.

Install CloudWatch Logs Agent

SSH into the EC2 instance and follow the steps given below.

Step 1: Head over to the CloudWatch agent downloads page. You can select region-wise packages as well.
Step 2: Download the appropriate agent installation file.

In here I am using ubuntu. I am downloading the latest Ubuntu package and installing it.

1
2
wget https://s3.amazonaws.com/amazoncloudwatch-agent/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb
sudo dpkg -i -E ./amazon-cloudwatch-agent.deb

Now we have successfully installed the CloudWatch agent on the EC2 instance.

Configure CloudWatch Logs Agent

After the installation, you can find all the CloudWatch agent-related config files and executables in the following location.

/opt/aws/amazon-cloudwatch-agent

If you are just starting with a cloud watch agent, it is better to run the cloud watch agent wizard that helps you create the log agent configurations.

Step 1: Run the below command to start the wizard.

1
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard

It prompts you with all the agent-related questions. Execute the following command to start the wizard. For the question, Do you want to store the config in the SSM parameter store?, select No. The final config files get stored in the following location.

/opt/aws/amazon-cloudwatch-agent/bin/config.json

If you want to collect the system metrics, install collected on your server.

1
2
sudo apt-get update -y
sudo apt-get install collectd

In my case, I am going to use the default config.json. If you need to collect custom logs you can create a custom config.json file and replace the default one.

Step 2: Start the CloudWatch agent.

1
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a status

Verify the Logs

After the agent is started, you can verify the logs in the CloudWatch console. Head over to CloudWatch –> Logs –> Log Groups. You can see the log groups created for the instance.

Conclusion

In this tutorial, we've walked through the process of push EC2 logs to CloudWatch . If you have any issue regarding this tutorial, mention your issue in the comment section or reach me through my E-mail.

Happy Coding