💡 TLDR

In my previous post I've shown how to set up AWS CloudTrail and AWS EventBridge to notify via AWS SNS whenever a certain event in AWS was triggered. With SNS, we cannot process the event before sending it to the recipients. In order to do so, I'll use AWS Lambda and Golang to parse the incoming event and print relevant information.

Motivation

Nowadays for Security monitoring and logging within AWS it's common to rely on modern, robust tools like CloudWatch and Grafana. These solutions have extensive features for data modification, visualization and alerting. For exactly these reasons many organizations choose to implement their monitoring infrastructure. For much simpler use-cases where you want to be alerted only on specific events (inside your infrastructure) there are other approaches.

One approach is to incorporate some business logic into an AWS Lambda function. This way you can create lightweight, highly scalable functions that can handle Security logging tasks without the overhead that comes with more comprehensive monitoring tools. By leveraging the simplicity and performance of Golang, you can setup a system that quickly alerts to potential security issues.

This blog post was a personal challenge and learning opportunity where by leveraging Terraform for infrastructure as code and Golang for the Lambda function, the goal was to demonstrate how one can build an effective, event-driven Security logging solution in AWS.

Benefits

While requirements for each setup may be different, I think this approach has several benefits I'd shortly like to emphasize. By using a Serverless, event-driven approach one can simplify the overall architecture by eliminating the need for more complex tools like Grafana. You also can reduce costs as your Lambda functions only get triggered by specific (CloudTrail) events. In terms of scalability you don't have to worry about bottlenecks as this setup seamlessly adapts to varying loads (volume of incoming events).

Big picture

I like to start with some high-level design before going into details:

Golang

Before we deal with the infrastructure, let's have a look at how one could parse a CloudTrail event in Go. Afterwards, I'll wrap a Lambda function around the Golang application and deploy all necessary resources to AWS.

Makefile

The Makefile controls the build process and builds the binary we need to upload to AWS Lambda. It also creates a Docker image which hosts the Golang application. This way I can choose the environment the application will later run in and test carefully:

build: build-lambda

build-lambda:
    GOOS=linux GOARCH=amd64 go build -ldflags="-s -w" -o ./build/lambda-function.bin lambda/main.go

build-docker:
    docker build --platform linux/amd64 -t security-monitoring-aws:latest .

run-local:
    docker run -d -p 9000:8080 --entrypoint /usr/local/bin/aws-lambda-rie security-monitoring-aws:latest /runtime/lambda-function.bin

Some explanations:

• build-lambda builds the Lambda function (which is a binary)
• build-docker builds the Docker image which should host our Golang application
• run-local runs the Golang application locally

Business logic

First let's initialize the Golang project:

go mod init security-monitoring-aws

This should create a go.mod:

module security-monitoring-aws

go 1.22.0

Now let's download the dependencies:

go mod tidy

Import all required dependencies:

package main

import (
    "context"
    "encoding/json"
    "fmt"
    "os"

    "github.com/sirupsen/logrus"
    "github.com/aws/aws-lambda-go/events"
    "github.com/aws/aws-lambda-go/lambda"
)

In the following I'll define a structure for parsing incoming events. But first let's have a look how we could represent a Cloudtrail event in Go:

// Contains information about an event that was returned by a lookup request. The
// result includes a representation of a CloudTrail event.
type Event struct {
    // The Amazon Web Services access key ID that was used to sign the request. If the
    // request was made with temporary security credentials, this is the access key ID
    // of the temporary credentials.
    AccessKeyId *string

    // A JSON string that contains a representation of the event returned.
    CloudTrailEvent *string

    // The CloudTrail ID of the event returned.
    EventId *string

    // The name of the event returned.
    EventName *string

    // The Amazon Web Services service to which the request was made.
    EventSource *string

    // The date and time of the event returned.
    EventTime *time.Time

    // Information about whether the event is a write event or a read event.
    ReadOnly *string

    // A list of resources referenced by the event returned.
    Resources []Resource

    // A user name or role name of the requester that called the API in the event
    // returned.
    Username *string

    noSmithyDocumentSerde
}

Define the structure meant for parsing the events:

// EventParser has methods to handle different types of AWS EventBridge events.
type EventParser struct {
    log *logrus.Logger
}

// NewEventParser creates a new instance of EventParser.
func NewEventParser(logger *logrus.Logger) *EventParser {
    return &EventParser{
        log: logger,
    }
}

Let's shortly recall how a Cloudtrail event (like the one sent via SNS in my previous post) looks like (I've ommitted some unnecessary details):

{
  "version": "0",
  "id": "02612a4d-bbcb-8050-8bc9-0ce5aa166b5b",
① "detail-type": "AWS API Call via CloudTrail",
② "source": "aws.s3",
  "account": "xxxxxxxxxxxx",
  "time": "2024-04-10T17:32:47Z",
  "region": "eu-central-1",
  "resources": [],
  "detail": {
    "eventVersion": "1.09",
    "userIdentity": {
      "type": "IAMUser",
      "principalId": "xxxxxxxxxxxxxxxxxxxxx",
      "arn": "arn:aws:iam::xxxxxxxxxxxx:user/xxxxxxxx",
      "accountId": "xxxxxxxxxxxx",
      "accessKeyId": "xxxxxxxxxxxxxxxxxxxx",
      "userName": "xxxxxxxx"
    },
    "eventTime": "2024-04-10T17:32:47Z",
    "eventSource": "s3.amazonaws.com",
③   "eventName": "PutObject",
    "awsRegion": "eu-central-1",
    "sourceIPAddress": "xxxxxxxxxxxx",
    "userAgent": "[aws-cli/2.15.19 Python/3.11.8]",
    "requestParameters": {
④     "bucketName": "s3-poc-lambda-golang-eventbridge-cloudtrail",
      "Host": "s3-poc-lambda-golang-eventbridge-cloudtrail.s3.eu-central-1.amazonaws.com",
⑤
     "key": "file.txt"
    },
    [...]
    "resources": [
      {
        "type": "AWS::S3::Object",
        "ARN": "arn:aws:s3:::s3-poc-lambda-golang-eventbridge-cloudtrail/file.txt"
      },
      {
        "accountId": "xxxxxxxxxxxx",
        "type": "AWS::S3::Bucket",
        "ARN": "arn:aws:s3:::s3-poc-lambda-golang-eventbridge-cloudtrail"
      }
    ],
    "eventType": "AwsApiCall",
    "managementEvent": false,
    "recipientAccountId": "xxxxxxxxxxxx",
    "eventCategory": "Data",
    "tlsDetails": {
      "tlsVersion": "TLSv1.2",
      "cipherSuite": "ECDHE-RSA-AES128-GCM-SHA256",
      "clientProvidedHostHeader": "s3-poc-lambda-golang-eventbridge-cloudtrail.s3.eu-central-1.amazonaws.com"
    }
  }
}

Some observations:

• We can see it originates from Cloudtrail ①
• It's an event generated by S3 ②
• The action is PutObject ③
• We can see the
  • bucket name involved in this event ④
  • object name ⑤

The EventParser struct should have methods for parsing CloudTrail events (which are of type events.CloudWatchEvent):

func (e *EventParser) EventHandler(ctx context.Context, event events.CloudWatchEvent) error {
    switch event.DetailType {
    // Make sure it's from Cloudtrail
    case "AWS API Call via CloudTrail":
        e.log.Info("Handling Cloudtrail event")
        return e.HandleEvent(ctx, event)
    default:
        return fmt.Errorf("unhandled detail type: %s", event.DetailType)
    }
}

In EventHandler we make sure the event is coming from Cloudtrail (via EventBridge). Next, in HandleEvent we specify instructions for each type of events:

// HandleEvent deals with different events based on source
func (e *EventParser) HandleEvent(ctx context.Context, event events.CloudWatchEvent) error {
    switch event.Source {

    // Handle S3 events
    case "aws.s3":
        var s3Event events.S3Event
        e.log.Info("Handling S3 event")
        if err := json.Unmarshal(event.Detail, &s3Event); err != nil {
            return fmt.Errorf("could not unmarshal event into S3Event: %w", err)
        }
        return e.handleS3Event(s3Event)

    // Handle EC2 events
    case "aws.ec2":
        e.log.Warn("No event handler defined for EC2")
        return fmt.Errorf("No event handler for EC2")
    default:
        return fmt.Errorf("Unknown event source: %s", event.Source)
    }
}

As an example, let's define what should happen if it's a S3 related event.

// handleS3Event deals with S3 related events (from Cloudtrail)
func (e *EventParser) handleS3Event(event events.S3Event) error {
    e.log.Infof("Event: %#v", event)
    for _, record := range event.Records {
        e.log.Infof("Record: %v", record)
    }
    return nil
}

Finally let's glue everything together in the main function. First, we define a new logger and use it as an argument for the EventParser.

func main() {
    // create a new instance of the logger. You can have any number of instances.
    var log = logrus.New()

    // Log as JSON instead of the default ASCII formatter.
    log.SetFormatter(&logrus.JSONFormatter{})

    // Output to stdout instead of the default stderr
    // Can be any io.Writer, see below for File example
    log.SetOutput(os.Stdout)

    // Only log the warning severity or above.
    log.SetLevel(logrus.InfoLevel)

    // Create new parser
    parser := NewEventParser(log)

Then we check if the binary should run as a CLI or as a Lambda function:

    if len(os.Args) > 1 {
        // Read JSON event from a file specified in the command line arguments
        filePath := os.Args[1]
        fileContent, err := os.ReadFile(filePath)
        if err != nil {
            fmt.Printf("Error reading event file: %v\n", err)
            os.Exit(1)
        }

        var event events.CloudWatchEvent
        if err := json.Unmarshal(fileContent, &event); err != nil {
            fmt.Printf("Error decoding event from file: %v\n", err)
            os.Exit(1)
        }

        // Process the event
        if err := parser.EventHandler(context.Background(), event); err != nil {
            fmt.Printf("Error handling event: %v\n", err)
            os.Exit(1)
        }
    } else {
        // Lambda mode, waiting for CloudWatch events
        log.Info("Starting Lambda function")
        lambda.Start(func(ctx context.Context, event events.CloudWatchEvent) error {
            return parser.EventHandler(ctx, event)
        })
    }
}

If we now run the binary it should run however the Lambda environment is missing:

./build/lambda-function.bin

expected AWS Lambda environment variables [_LAMBDA_SERVER_PORT AWS_LAMBDA_RUNTIME_API] are not defined

Now let's build the environment using Docker.

Testing locally

As described here you'll have to install an emulator in order to test your code locally. I'll be following instructions installing the Go image:

docker run -d -p 9000:8080 --entrypoint /usr/local/bin/aws-lambda-rie security-monitoring-aws:latest /runtime/lambda-function.bin

Invoke the Lambda:

curl "http://localhost:9000/2015-03-31/functions/function/invocations" -d '{"payload":"hello world!"}'

{"errorMessage":"unhandled detail type: ","errorType":"errorString"}

Obvisously, you need to provide the right JSON payload. Next we'll continue deploying additional resources for the Lambda function.

Containerization

Docker image

One reason why I still use Golang is because it compiles natively to a binary. I don't need any dedicated language runtime for shipping my apps. I just compile the binary and copy it to a Docker image. Simple as that!

⚠ Use an AWS OS-only base image:

# Specify the same version you used to build the Golang binary
FROM golang:1.22 as build
WORKDIR /runtime

# Use AWS OS image
FROM public.ecr.aws/lambda/provided:al2023

# Copy artifacts
COPY ./build/lambda-function.bin /runtime/lambda-function.bin

# Specify entrypoint
ENTRYPOINT [ "/runtime/lambda-function.bin" ]

Build a Docker image

Build the Docker image:

make build-docker

You should get something like:

Step 1/5 : FROM golang:1.22 as build
 ---> 0505a58fa464
Step 2/5 : WORKDIR /runtime
 ---> Using cache
 ---> e8ef33706442
Step 3/5 : FROM public.ecr.aws/lambda/provided:al2023
 ---> bcf50e5c3bc4
Step 4/5 : COPY ./build/lambda-function.bin /runtime/lambda-function.bin
 ---> eee7eb07a5dc
Step 5/5 : ENTRYPOINT [ "/runtime/lambda-function.bin" ]
 ---> Running in 2a8d3be4e45f
 ---> Removed intermediate container 2a8d3be4e45f
 ---> a95272eaaaeb
Successfully built a95272eaaaeb
Successfully tagged security-monitoring-aws:latest

Deployment

First create the ECR:

# Create ECR repository
data "aws_caller_identity" "current" {}
locals {
  account_id = data.aws_caller_identity.current.account_id
  region = "eu-central-1"
  ecr_image_tag = "latest"
  docker_image = "security-monitoring-aws"

}

resource "aws_ecr_repository" "cloudtrail-events-repo" {
  name = "cloudtrail-events-repo"
  image_tag_mutability = "MUTABLE"
  image_scanning_configuration {
    scan_on_push = false
  }
}

Now that the Docker repository is created, we also need to upload Docker images to it. Terraform was not meant to actually "build" stuff but rather "define" infrastructure. I'll abuse null_resource to perform actions (without actually provisioning any resources):

# Create ECR repository
resource "null_resource" "ecr_image" {
  # When the Lambda function (a Golang based binary) changes, a new ECR image will be created
  triggers = {
    go_binary = filemd5("../app/build/lambda-function.bin")
  }
  provisioner "local-exec" {
    command = <<EOF
        aws ecr get-login-password  | docker login --username AWS --password-stdin ${local.account_id}.dkr.ecr.${local.region}.amazonaws.com
        docker tag ${local.docker_image}  ${aws_ecr_repository.cloudtrail-events-repo.repository_url}:${local.ecr_image_tag}
        docker push ${aws_ecr_repository.cloudtrail-events-repo.repository_url}:${local.ecr_image_tag}
    EOF
  }
}

AWS Lambda

Function

Create the Lambda function:

# Create Lambda function
resource "aws_lambda_function" "cloudtrail-events-parser" {
  function_name = "cloudtrail-events-parser"
  timeout       = 5
  image_uri     = "${aws_ecr_repository.cloudtrail-events-repo.repository_url}:${local.ecr_image_tag}"
  package_type  = "Image"

  role = aws_iam_role.lambda_role.arn

  # Setup here environment variables
  # environment {
  #   variables = {
  #     ENVIRONMENT = var.env_name
  #   }
  # }
}

You can see the Lambda image gets created from ECR:

IAM

Setup IAM role for Lambda function:

data "aws_iam_policy_document" "assume_lambda_role" {
  statement {
   actions = ["sts:AssumeRole"]

    principals {
      type        = "Service"
      identifiers = ["lambda.amazonaws.com"]
    }
  }

  statement {
    actions = ["sts:AssumeRole"]

    principals {
      type        = "Service"
      identifiers = ["events.amazonaws.com"]
    }
  }
}

// Lambda assume IAM role
resource "aws_iam_role" "lambda_role" {
  name               = "AssumeLambdaRole"
  description        = "Role for lambda to assume lambda"
  assume_role_policy = data.aws_iam_policy_document.assume_lambda_role.json
}

Let's continue with the IAM policies. Keep in mind you'll need a resource-based policy for the lambda function (source).

data "aws_iam_policy_document" "lambda_iam_policy_document" {
  statement {
    effect = "Allow"
    actions = [
      # "logs:CreateLogStream",
      # "logs:CreateLogGroup",
      # "logs:PutLogEvents",
      "logs:*",
    ]
    # TODO: Maybe you want to further restrict the target resources here
    resources = ["arn:aws:logs:*:*:*"]
  }
}

// IAM policy for the AWS Lambda
resource "aws_iam_policy" "lambda_iam_policy" {
  name        = "LambdaIAMPolicy"
  description = "Policy for the AWS Lambda"
  policy      = data.aws_iam_policy_document.lambda_iam_policy_document.json
}

# Attach lambda_iam_policy to Lambda's IAM role
resource "aws_iam_role_policy_attachment" "lambda_policy" {
  role       = aws_iam_role.lambda_role.name
  policy_arn = aws_iam_policy.lambda_iam_policy.arn
}

This should create a resource based policy for EventBridge:

Now allow EventBridge to invoke our Lambda function:

resource "aws_lambda_permission" "allow_eventbridge" {
  statement_id  = "AllowExecutionFromEventBridge"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.cloudtrail-events-parser.arn
  principal     = "events.amazonaws.com"
  source_arn    = module.aws-eventbridge-Lambda.rule_arn
}

At the end you should have a Lambda function should be triggered by EventBridge:

Logging

Create a CloudWatch log group for the Lambda function:

// Create log group in Cloudwatch
resource "aws_cloudwatch_log_group" "cloudwatch_log_group" {
  name              = "/aws/lambda/${aws_lambda_function.cloudtrail-events-parser.function_name}"
  retention_in_days = 7
  lifecycle {
    prevent_destroy = false
  }
}

AWS EventBridge

Here a EventBridge module is setup which will help us with the creation of specific EventBridge rules:

# Create Eventbridge rule
module "eventbridge" {
  source = "terraform-aws-modules/eventbridge/aws"

  # Don't create a new bus
  create_bus = false

  ### IAM role settings
  # Don't create a new IAM role
  # We have already setup a resource-based polcy
  create_role = false
  attach_lambda_policy = false

  # role_name = "EventBridgeRuleForLambda"
  # role_description = "IAM Role to be attached to the EventBridge rule"

  lambda_target_arns = [var.lambda_function_arn]

  # Create a new rule
  rules = {
    s3_put_object_lambda = {
      description = "Triggers a Lambda function when objects are uploaded to certain S3 bucket"
      event_pattern = jsonencode({
        "source": ["aws.s3"],
        "detail-type": ["AWS API Call via CloudTrail"],
        "detail": {
          "eventSource": ["s3.amazonaws.com"],
          # Specify which actions to look for
          "eventName": ["PutObject", "CompleteMultipartUpload"],
          # Also limit to events to our previously created S3 bucket
          "requestParameters": {
            "bucketName": [var.s3_bucket_name]
          },
        }
      })
    }
  }

  # Create a new Lambda based EventBridge target
  targets = {
    s3_put_object_lambda = {
      lambda_target = {
        arn           = var.lambda_function_arn
        name          = "send-s3-put-events-to-lambda"
      }
    }
  }
}

And the variables:

variable "s3_bucket_name" {
  description = "Name of the S3 bucket"
  default     = "s3-poc-lambda-golang-eventbridge-cloudtrail"
}

variable "lambda_function_arn" {
  description = "ARN of the Lambda function"
  type = string
}

This will create a new EventBridge rule:

Finally some outputs:

output "rule_arn" {
  value = module.eventbridge.eventbridge_rule_arns.s3_put_object_lambda
}

main

Glue everything together:

# Create Cloudtrail
module "aws-cloudtrail" {
  source = "./modules/aws-cloudtrail"

  # Provide parameters to module
  trail_name             = var.trail_name
  trail_bucket           = var.trail_bucket
  cloudwatch_log_group_name = var.cloudwatch_log_group_name
}

# Create EventBridge for SNS
module "aws-eventbridge-sns" {
  source = "./modules/aws-eventbridge/sns"

  sns_topic             = var.sns_topic
  s3_bucket_name        = var.s3_bucket_name
}

# Create EventBridge for Lambda
module "aws-eventbridge-Lambda" {
  source = "./modules/aws-eventbridge/lambda"

  s3_bucket_name        = var.s3_bucket_name
  lambda_function_arn   = aws_lambda_function.cloudtrail-events-parser.arn
}

Proof of Concept

Now, that we have setup everything we need for this to work let's run a small test:

aws s3 cp --recursive . s3://s3-poc-lambda-golang-eventbridge-cloudtrail/

This command is triggering "PUT" events against the AWS API to upload new objects to the S3 bucket. Let's check if EventBridge and see if our rule was triggered. It may take a while to see the events so be patient:

Indeed, we have 70 matched events! 🎉 Let's check for the Lambda function invocations:

The EventBridge rule also triggered our Lambda function. Let's check the CloudWatch logs:

Using CloudWatch we can also see the debug messages sent by our Lambda function.

Resources

• https://dev.to/feregri_no/infrastructure-with-terraform-tweeting-from-a-lambda-1ael
  • Found out about the null_resource resource
  • Used the same hack to upload Docker images to AWS ECR
• AWS Lambda and Golang
  • Community | How do I run existing Go applications on AWS Lambda - Part 1
  • Community | How do I run existing Go applications on AWS Lambda - Part 2