Security Monitoring in AWS: CloudTrail, CloudWatch and EventBridge

💡 TLDR

In this post I'll demonstrate how to setup a Security monitoring infrastructure in AWS. The main goal is to leverage AWS Cloudwatch, AWS Lambda and AWS Eventbridge for creating alerts based on specific event types from AWS Cloudtrail. First I'll use Terraform to deploy all required resources and then I'll implement a simple Golang based Lambda function to handle certain events.

Infrastructure

In my Documentation as Code for Cloud series I've described a fictional self-destructing email service which consists of several components, deployed across multiple accounts in AWS (you could do the same in Azure, GCP, etc.).

The big picture:

The PlantUML diagram describes a security monitoring infrastructure comprising three main components: Data Sources, Monitoring & Observability, and Alerting:

Data Sources: This section includes CloudTrail, which is a service that logs AWS API calls made on your account, providing visibility into actions taken within the AWS environment.
Monitoring & Observability: This section contains various tools and services for monitoring and observing the system. It includes CloudWatch Logs for log management and analysis, an S3 Bucket for storing logs and other data, EventBridge for event-driven architecture, EventBridge Rules for defining event routing, and Lambda for serverless computing.
Alerting: This section involves tools for alerting and notification. It includes Slack for real-time communication, SNS (Simple Notification Service) for sending notifications, and SQS (Simple Queue Service) for queuing messages.

The diagram also depicts relationships between these components:

CloudTrail sends logs to S3 for storage and to EventBridge for event processing.
CloudWatch receives logs from CloudTrail for monitoring and analysis.
EventBridge Rules invoke Lambda functions for automated responses and can send messages to SQS and SNS for further processing.
Finally, Lambda functions can directly notify Slack for alerting purposes.

Check out the github repository for the code snippets here

Terraform

As a big fan of "Everything as Code" I'll start by implementing IaC (Infrastructure as Code) in Terraform for activating CloudTrail, setting up the CloudWatch logs and the EventBridge rules.

I'll first define the Terraform version I want to use:

1.7.5

Code Snippet 1: .terraform-version

💡 Use tfenv to manage multiple versions of Terraform.

Bootstrap

Before creating the actual resources, I'll first setup the remote state (using a S3 bucket in AWS) and therefore make sure the Terraform state file is managed safely:

provider "aws" {
  region = "eu-central-1"
}

# Create S3 bucket
resource "aws_s3_bucket" "terraform_state" {
  bucket = "defersec-tfstate"

  lifecycle {
    prevent_destroy = true
  }
}

# Allow versioning
resource "aws_s3_bucket_versioning" "terraform_state" {
    bucket = aws_s3_bucket.terraform_state.id

    versioning_configuration {
      status = "Enabled"
    }
}

Code Snippet 2: remote-state/main.tf

(Optional): Additionally you can add state lock (which is recommended in an enterprise context with large state files) but for now I'll just skip it:

# Use state lock in AWS DynamoDB
resource "aws_dynamodb_table" "terraform_state_lock" {
  name           = "app-state"
  read_capacity  = 20
  write_capacity = 20
  hash_key       = "LockID"

  attribute {
    name = "LockID"
    type = "S"
  }
}

Code Snippet 3: (optional state lock) remote-state/main.tf

I'll use AWS as my main Terraform provider:

provider "aws" {
  region  = "eu-central-1"
  profile = "aws-terraform"

  default_tags {
    tags = {
      app = "cloudtrail-events-using-lambda-and-go"
      managed_by = "terraform"
    }
  }
}

Code Snippet 4: monitoring/provider.tf

Specify a certain version:

terraform {
  required_providers {
    aws = {
      source                = "hashicorp/aws"
      version               = "~> 5.29.0"
    }
  }
}

Code Snippet 5: monitoring/versions.tf

Save the Terraform state file in a S3 bucket:

terraform {
  backend "s3" {
    bucket         = "defersec-tfstate"
    key            = "prod/terraform.tfstate"
    region         = "eu-central-1"
    # You can specify an explicit AWS profile here
    # profile        = "aws-terraform"
  }
}

Code Snippet 6: monitoring/backend.tf

That's it for the bootstrap. Now, let's deep-dive into some Terraform modules.

Modules

Terraform modules are a type of encapsulation for multiple resources that are used together. They server as a way to package and reuse code but also to organize Terraform configurations.

There are different types of Terraform modules:

Root Module: This is the main configuration directory where you run terraform commands. Every Terraform configuration has at least one root module.
Internal Modules: These are modules created for a single purpose, usually to be used from the root module
Child Modules: These are modules created within other modules (e.g. inside your own local modules). You can either create your own child modules or source them from external locations, such as the Terraform Registry or other version control systems.

We'll create some Terraform modules for better reusability.

⮕ Cloudtrail

The infrastructure resources will be created in a single account. In an enterprise (or even for bigger projects) you may want to use a multi-account organization in AWS.

I'll put cloudtrail in it's own module:

resource "aws_cloudtrail" "default" {
  # Put these into variables
  name           = var.trail_name
  s3_bucket_name = var.trail_bucket

  # Create trail in organization master account?
  is_organization_trail = false

  # Use a single S3 bucket for all AWS regions
  is_multi_region_trail = true

  # Add global service events
  include_global_service_events = true

  # Send logs to CloudWatch Logs
  cloud_watch_logs_group_arn = "${aws_cloudwatch_log_group.cloudwatch_log_group.arn}:*"
  cloud_watch_logs_role_arn  = aws_iam_role.cloudtrail_cloudwatch_role.arn

  # We also want Data Events for certain services (such as S3 objects)
  event_selector {
    read_write_type           = "All"
    include_management_events = true

    # TODO: Create S3 bucket GitOps style
    data_resource {
      type   = "AWS::S3::Object"
      values = ["arn:aws:s3:::s3-poc-lambda-golang-eventbridge-cloudtrail/"]
    }

    # You can also exclude certain events
    # exclude_management_event_sources = [
    #   "kms.amazonaws.com",
    #   "rdsdata.amazonaws.com"
    # ]
  }

  depends_on = [
    aws_s3_bucket.cloudtrail_bucket
  ]
}

Code Snippet 7: monitoring/modules/aws-cloudtrail/main.tf

Define the providers (I'll mainly use AWS):

terraform {
  required_providers {
    aws = {
      source                = "hashicorp/aws"
      version               = "~> 5.29.0"
    }
  }
}

Code Snippet 8: monitoring/modules/aws-cloudtrail/providers.tf

Now let's specify some variables:

variable "trail_name" {
  description = "AWS CloudTrail Name"
}

variable "trail_bucket" {
  description = "AWS CloudTrail S3 Bucket to store log data"
}

variable "cloudwatch_log_group_name" {
  description = "CloudWatch Log group name for the Cloudtrail logs"
  type        = string
}

variable "cloudwatch_log_retention_days" {
  description = "The retention length for the Cloudwatch logs"
  default     = 5
  type        = number
}

Code Snippet 9: monitoring/modules/aws-cloudtrail/variables.tf

Create the Cloudwatch log group:

resource "aws_cloudwatch_log_group" "cloudwatch_log_group" {
  name              = var.cloudwatch_log_group_name
  retention_in_days = var.cloudwatch_log_retention_days
}

Code Snippet 10: monitoring/modules/aws-cloudtrail/cloudwatch.tf

Next we continue creating IAM related resources. But first let's fetch some information regarding the current environment/setup:

# The AWS account id
data "aws_caller_identity" "current" {}

# The AWS region currently being used.
data "aws_region" "current" {}

# The AWS partition
data "aws_partition" "current" {}

# The currrent organization
# data "aws_organizations_organization" "current" {}

Code Snippet 11: monitoring/modules/aws-cloudtrail/data.tf

Allow Cloudtrail to assume this IAM role:

# Cloudtrail assume role
data "aws_iam_policy_document" "cloudtrail_assume_role" {
  statement {
    effect  = "Allow"
    actions = ["sts:AssumeRole"]

    principals {
      type        = "Service"
      identifiers = ["cloudtrail.amazonaws.com"]
    }
  }
}

# This role is used by CloudTrail to send logs to CloudWatch.
resource "aws_iam_role" "cloudtrail_cloudwatch_role" {
  name               = "CloudtrailIAMRole"
  assume_role_policy = data.aws_iam_policy_document.cloudtrail_assume_role.json
}

Code Snippet 12: monitoring/modules/aws-cloudtrail/data.tf

Let's add some IAM policies to the role: Allow Cloudtrail to send logs to AWS Cloudwatch

# Allow Cloudtrail to send logs to Cloudwatch
data "aws_iam_policy_document" "cloudtrail_cloudwatch_logs" {
  statement {
    sid    = "WriteCloudWatchLogs"
    effect = "Allow"
    actions = [
      "logs:CreateLogStream",
      "logs:PutLogEvents",
    ]
    resources = ["arn:${data.aws_partition.current.partition}:logs:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:log-group:${var.cloudwatch_log_group_name}:*"]
  }
}

# Policy to be attached to the role
resource "aws_iam_policy" "cloudtrail_cloudwatch_logs" {
  name   = "CloudtrailCloudwatchLogsPolicy"
  policy = data.aws_iam_policy_document.cloudtrail_cloudwatch_logs.json
}

# Attach the policy to a role
resource "aws_iam_policy_attachment" "attach_policy_logs" {
  name       = "CloudtrailCloudwatchLogsPolicy-attachment"
  policy_arn = aws_iam_policy.cloudtrail_cloudwatch_logs.arn
  roles      = [aws_iam_role.cloudtrail_cloudwatch_role.name]
}

Code Snippet 13: monitoring/modules/aws-cloudtrail/data.tf

Allow Cloudtrail to put logs into S3 bucket:

# Bucket policy for the Cloudtrail S3 bucket
data "aws_iam_policy_document" "cloudtrail_bucket_policy" {
  # Allow to fetch bucket ACLs
  statement {
    sid    = "AWSCloudTrailAclCheck"
    effect = "Allow"
    principals {
      type        = "Service"
      identifiers = ["cloudtrail.amazonaws.com"]
    }
    actions = ["s3:GetBucketAcl"]
    resources = [
      "arn:aws:s3:::${var.trail_bucket}",
    ]
    condition {
      test     = "StringEquals"
      variable = "aws:SourceArn"
      values   = ["arn:${data.aws_partition.current.partition}:cloudtrail:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:trail/${var.trail_name}}"]
    }
  }

  # Allow Cloudtrail to put logs into S3 bucket
  statement {
    sid    = "AWSCloudTrailWriteAccount"
    effect = "Allow"
    principals {
      type        = "Service"
      identifiers = ["cloudtrail.amazonaws.com"]
    }
    actions   = ["s3:PutObject"]
    resources = ["arn:aws:s3:::${var.trail_bucket}/AWSLogs/${data.aws_caller_identity.current.account_id}/*"]

    # Conditions
    condition {
      test     = "StringEquals"
      variable = "s3:x-amz-acl"
      values   = ["bucket-owner-full-control"]
    }

    condition {
      test     = "StringEquals"
      variable = "AWS:SourceArn"
      values   = ["arn:aws:cloudtrail:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:trail/${var.trail_name}"]
    }
  }
}

Code Snippet 14: modules/aws-cloudtrail/data.tf

(Optional): If you use AWS organizations you might want Cloudtrail to put logs for the whole organization:

 statement {
   sid    = "AWSCloudTrailWriteOrganization"
   effect = "Allow"
   principals {
     type        = "Service"
     identifiers = ["cloudtrail.amazonaws.com"]
   }
   actions   = ["s3:PutObject"]
   resources = ["arn:aws:s3:::${var.trail_bucket}/AWSLogs/${data.aws_organizations_organization.current.id}/*"]

   # Conditions
   condition {
     test     = "StringEquals"
     variable = "s3:x-amz-acl"
     values   = ["bucket-owner-full-control"]
   }

   condition {
     test     = "StringEquals"
     variable = "AWS:SourceArn"
     values   = ["arn:aws:cloudtrail:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:trail/${var.trail_name}"]
   }
 }

Code Snippet 15: monitoring/modules/aws-cloudtrail/data.tf

Then create the S3 bucket where to put logs to:

# Create S3 bucket
resource "aws_s3_bucket" "cloudtrail_bucket" {
  bucket        = var.trail_bucket
  force_destroy = true
}

# Enable server side encryption (SSE)
resource "aws_s3_bucket_server_side_encryption_configuration" "sse_s3_bucket" {
  bucket = aws_s3_bucket.cloudtrail_bucket.id
  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "aws:kms"
    }
  }
}

resource "aws_s3_bucket_policy" "s3_bucket_policy" {
  bucket = aws_s3_bucket.cloudtrail_bucket.id
  policy = data.aws_iam_policy_document.cloudtrail_bucket_policy.json
}

# Block all public access to the bucket
resource "aws_s3_bucket_public_access_block" "block_public_access" {
  bucket                  = aws_s3_bucket.cloudtrail_bucket.id
  block_public_acls       = true
  block_public_policy     = true
  restrict_public_buckets = true
  ignore_public_acls      = true
}

Code Snippet 16: monitoring/modules/aws-cloudtrail/s3.tf

Finally some outputs:

output "cloudtrail_instance" {
  description = "Cloudtrail instance"
  value       = aws_cloudtrail.default
}
output "cloudtrail_s3_bucket" {
  description = "Cloudtrail S3 bucket"
  value       = aws_s3_bucket.cloudtrail_bucket
}

Code Snippet 17: monitoring/modules/aws-cloudtrail/outputs.tf

⮕ EventBridge

Now we'll setup different modules for EventBride.

⚠ The options for Eventbridge are quite numerous. Check out the Github repo for the whole list.

The block below configures Eventbridge to capture DATA events (related to S3 buckets) and send them to a SNS topic:

# Create Eventbridge rule
module "eventbridge" {
  source = "terraform-aws-modules/eventbridge/aws"
  create_bus = false
  rules = {
    s3_put_object = {
      description = "Triggers when objects are uploaded to certain S3 bucket"
      event_pattern = jsonencode({
        "source": ["aws.s3"],
        "detail-type": ["AWS API Call via CloudTrail"],
        "detail": {
          "eventSource": ["s3.amazonaws.com"],
          "eventName": ["PutObject", "CompleteMultipartUpload"],
            "requestParameters": {
              "bucketName": [var.s3_bucket_name]
          },
        }
      })
    }
  }
  targets = {
    s3_put_object = {
      example_target = {
        arn           = aws_sns_topic.sns_topic.arn
        name          = "send-s3-put-events-to-sns-topic"
      }
    }
  }
}

# Create SNS topic
resource "aws_sns_topic" "sns_topic" {
  name = var.sns_topic
}

Code Snippet 18: monitoring/modules/aws-eventbridge/sns/main.tf

Finally we define the variables for this module:

variable "sns_topic" {
  description = "Name for the SNS topic"
  default     = "poc-lambda-golang-eventbridge-cloudtrail"
}

variable "s3_bucket_name" {
  description = "Name of the S3 bucket"
  default     = "s3-poc-lambda-golang-eventbridge-cloudtrail"
}

Code Snippet 19: monitoring/modules/aws-eventbridge/sns/variables.tf

⮕ Root Module

In the root module (especially main.tf) we glue everything togeter and use our internal modules:

Cloudtrail

# Create Cloudtrail
module "aws-cloudtrail" {
  source = "./modules/aws-cloudtrail"

  # Provide parameters to module
  trail_name             = var.trail_name
  trail_bucket           = var.trail_bucket
  cloudwatch_log_group_name = var.cloudwatch_log_group_name
}

Code Snippet 20: monitoring/main.tf

Let's define the variables:

variable "trail_name" {
  description = "AWS CloudTrail Name"
  default     = "poc-lambda-golang-eventbridge-cloudtrail"
}

variable "trail_bucket" {
  description = "AWS CloudTrail S3 Bucket to store log data"
  default     = "poc-lambda-golang-eventbridge-cloudtrail"
}

variable "cloudwatch_log_group_name" {
  description = "AWS Cloudwatch log group"
  default     = "/security/logs/cloudtrail"
}

Code Snippet 21: monitoring/variables.tf

Eventbridge

# Create Cloudtrail
module "aws-eventbridge-sns" {
  source = "./modules/aws-eventbridge/sns"

  sns_topic             = var.sns_topic
  s3_bucket_name        = var.s3_bucket_name
}

Code Snippet 22: monitoring/main.tf

variable "sns_topic" {
  description = "The SNS topic"
  default     = "poc-lambda-golang-eventbridge-cloudtrail"
}

variable "s3_bucket_name" {
  description = "Name of the S3 bucket"
  default     = "s3-poc-lambda-golang-eventbridge-cloudtrail"
}

Code Snippet 23: monitoring/variables.tf

Deployment

After successful deployment we should have Cloudtrail enabled:

![img](defersec.github.io/public/img/2024/cloudtra.. "Cloudtrail was enabled")

We have a S3 bucket with the log files:

There is a Cloudwatch log group:

The Eventbridge rule has also been created:

The SNS topic has been created and there is an e-mail subscription (which I've created manually):

Testing

Now let's test the Eventbridge rule and upload a file to the S3 bucket s3-poc-lambda-golang-eventbridge-cloudtrail:

$ aws s3 cp file.txt s3://s3-poc-lambda-golang-eventbridge-cloudtrail/
upload: ./file.txt to s3://s3-poc-lambda-golang-eventbridge-cloudtrail/file.txt

Then you should get a notification (via E-Mail) with following content:

{
  "version": "0",
  "id": "02612a4d-bbcb-8050-8bc9-0ce5aa166b5b",
  "detail-type": "AWS API Call via CloudTrail",
  "source": "aws.s3",
  "account": "xxxxxxxxxxxx",
  "time": "2024-04-10T17:32:47Z",
  "region": "eu-central-1",
  "resources": [],
  "detail": {
    "eventVersion": "1.09",
    "userIdentity": {
      "type": "IAMUser",
      "principalId": "xxxxxxxxxxxxxxxxxxxxx",
      "arn": "arn:aws:iam::xxxxxxxxxxxx:user/xxxxxxxx",
      "accountId": "xxxxxxxxxxxx",
      "accessKeyId": "xxxxxxxxxxxxxxxxxxxx",
      "userName": "xxxxxxxx"
    },
    "eventTime": "2024-04-10T17:32:47Z",
    "eventSource": "s3.amazonaws.com",
    "eventName": "PutObject",
    "awsRegion": "eu-central-1",
    "sourceIPAddress": "xxxxxxxxxxxx",
    "userAgent": "[aws-cli/2.15.19 Python/3.11.8]",
    "requestParameters": {
      "bucketName": "s3-poc-lambda-golang-eventbridge-cloudtrail",
      "Host": "s3-poc-lambda-golang-eventbridge-cloudtrail.s3.eu-central-1.amazonaws.com",
      "key": "file.txt"
    },
    "responseElements": {
      "x-amz-server-side-encryption": "AES256"
    },
    "additionalEventData": {
      "SignatureVersion": "SigV4",
      "CipherSuite": "ECDHE-RSA-AES128-GCM-SHA256",
      "bytesTransferredIn": 926,
      "SSEApplied": "Default_SSE_S3",
      "AuthenticationMethod": "AuthHeader",
      "x-amz-id-2": "gYxZ55RgLuT+NtnG0jG9uSm3TBP2ZgzxyXQKkKh/x84hG3KkcCjR41DDY8jNcepL2EoxHM4espCnH89VXaU1iw==",
      "bytesTransferredOut": 0
    },
    "requestID": "H0EC27Z64SE4FCCX",
    "eventID": "91756431-xxxxxxxxxxxxxxxxxxxxxxxxxxx",
    "readOnly": false,
    "resources": [
      {
        "type": "AWS::S3::Object",
        "ARN": "arn:aws:s3:::s3-poc-lambda-golang-eventbridge-cloudtrail/file.txt"
      },
      {
        "accountId": "xxxxxxxxxxxx",
        "type": "AWS::S3::Bucket",
        "ARN": "arn:aws:s3:::s3-poc-lambda-golang-eventbridge-cloudtrail"
      }
    ],
    "eventType": "AwsApiCall",
    "managementEvent": false,
    "recipientAccountId": "xxxxxxxxxxxx",
    "eventCategory": "Data",
    "tlsDetails": {
      "tlsVersion": "TLSv1.2",
      "cipherSuite": "ECDHE-RSA-AES128-GCM-SHA256",
      "clientProvidedHostHeader": "s3-poc-lambda-golang-eventbridge-cloudtrail.s3.eu-central-1.amazonaws.com"
    }
  }
}

Code Snippet 24: CloudTrail event sent via mail

Outlook

In the next post I'll show how to trigger a Lambda function via EventBridge. Instead of sending the whole event to SNS, we might want to implement some custom logic inside the AWS Lambda.

Resources

Here is a list of additional, maybe helpful resources:

DeferSec Engineering