Explore the Power of Account-Level Subscription Filters in Amazon CloudWatch Logs!

CloudWatch logs from all or multiple log groups can now be delivered to the desired destination (Amazon Kinesis Data Stream, Amazon Kinesis Data Firehose, or AWS Lambda) using just one account-level subscription filter.

ℹ️

There is no option in CloudWatch Dashboard UI. We need to use aws logs put-account-policy AWS CLI command or API.

Sample AWS CLI command:

abhijit@AwsJunkie:~$ aws logs put-account-policy --policy-name "Account-sub-test" --policy-type "SUBSCRIPTION_FILTER_POLICY"  --policy-document '{"RoleArn":"arn:aws:iam::141035231386:role/CW2KDFRole","DestinationArn":"arn:aws:firehose:us-west-2:141035231386:deliverystream/MyKDF", "FilterPattern": "", "Distribution": "Random"}' --scope "ALL" --selection-criteria 'LogGroupName NOT IN ["LogGroup1", "LogGroup2"]'
{
    "accountPolicy": {
        "policyName": "Account-sub-test",
        "policyDocument": "{\"RoleArn\":\"arn:aws:iam::141035231386:role/CW2KDFRole\",\"DestinationArn\":\"arn:aws:firehose:us-west-2:141035231386:deliverystream/MyKDF\",\"FilterPattern\":\"\",\"Distribution\":\"Random\"}",
        "lastUpdatedTime": 1707356377773,
        "policyType": "SUBSCRIPTION_FILTER_POLICY",
        "scope": "ALL",
        "selectionCriteria": "LogGroupName NOT IN [\"LogGroup1\", \"LogGroup2\"]"
    }
}

Use --selection-criteria to exclude log groups that you don't want.

You can create only one account-level subscription filter. Let's try to create one with a different --policy-name.

abhijit@AwsJunkie:~$ aws logs put-account-policy --policy-name "Account-sub-test-different" --policy-type "SUBSCRIPTION_FILTER_POLICY"  --policy-document '{"RoleArn":"arn:aws:iam::141035231386:role/CW2KDFRole","DestinationArn":"arn:aws:firehose:us-west-2:141035231386:deliverystream/MyKDF", "FilterPattern": "", "Distribution": "Random"}' --scope "ALL" --scope "ALL" --selection-criteria 'LogGroupName NOT IN ["LogGroup1", "LogGroup2"]'

An error occurred (LimitExceededException) when calling the PutAccountPolicy operation (reached max retries: 2): Resource limit exceeded.

Watch the below video for a demo where I created account-level subscription filters with Amazon Kinesis Data Firehose.

Just in case if you would like to try, I am sharing the IAM Role that I assigned to CloudWatch for permission on Amazon Kinesis Data Firehose.

Trust policy:

{
    "Version": "2008-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "logs.amazonaws.com"
            },
            "Action": "sts:AssumeRole",
            "Condition": {
                "StringLike": {
                    "aws:SourceArn": "arn:aws:logs:us-west-2:141035231386:*"
                }
            }
        }
    ]
}

Permissions:

{
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "firehose:PutRecord"
            ],
            "Resource": [
                "arn:aws:firehose:us-west-2:141035231386:deliverystream/MyKDF"
            ]
        }
    ]
}