Build Cicd Pipeline For Aws Lambda With Terraform, Github & Circleci - Part 2, Terraform

0

In the previous article, we created a lambda function using nodejs and typescript. Now we would configure cloud resources to expose our lambda to the internet using Terraform.

Terraform is an open-source infrastructure as a code tool for managing resources in the cloud. It is easy to describe resources using the Hashicorp Configuration Language(HCL) in concise reproducible blocks of resource descriptions.

We will use Terraform to create and manage our cloud resources. These resources include:

1. Lambda functions
2. API Gateway (For exposing our functions to the internet)
3. AWS S3 (Storage for our functions)
4. AWS Cloudwatch (Monitoring and logging)

First, you need a working installation of Terraform CLI. Follow the instructions here to install the Terraform CLI specific to your OS on your PC.

To be able to create and manage resources on Amazon Web Services, You need an account. You can create a free account here and set up your access key id and secret access key here.

We can go ahead to create our terraform configurations.

Modules

Terraform modules are buckets of modularized code that help us concisely manage our resources to not repeat ourselves over and over again. They help us create lightweight abstractions so we can treat infrastructure in terms of our architecture.

In our project, we need three modules. We would create them in the following format:

The first of them would represent the minimal resources required to manage our lambda function in AWS. Each module may ship with output and variable files depending on our requirements.

# terraform/modules/lambda-function/main.tf

# -----------------------------------------------------------------------------
# Resource: Lambda function
# -----------------------------------------------------------------------------

resource "aws_lambda_function" "lambda_fn" {
  function_name = var.function_name

  s3_bucket = var.s3_bucket_id
  s3_key    = var.s3_key

  runtime = "nodejs12.x"
  handler = var.handler

  role = var.iam_role

  depends_on = [
    var.dependency,
  ]
}

# -----------------------------------------------------------------------------
# Resource: Lambda function logs
# -----------------------------------------------------------------------------

resource "aws_cloudwatch_log_group" "cwatch_logs" {
  name = "/aws/lambda/${aws_lambda_function.lambda_fn.function_name}"

  retention_in_days = 30
}

# -----------------------------------------------------------------------------
# Resource: Api Exec Permissions
# -----------------------------------------------------------------------------

module "api_gw_perm" {
# This third party module helps us define minimal permissions for our
# API gateway
  source  = "Cloud-42/lambda-permission/aws"
  version = "2.0.0"

  function_name = aws_lambda_function.lambda_fn.function_name
  statement_id  = var.apigw_statement_id
  source_arn    = var.source_arn
}

Variables are how we define resources we intend to use in multiple locations in our terraform infrastructure. The variables are usually clear and concise. Variables can be defined in this format:

variable "name_of_var" {
  # optional `type` can be a string, number, bool, list, map and null 
  type = string 
  # optional `default` value, can be reassigned 
  default = "greatness"
  # brief optional `description` for documentation purposes
  description = "A simple demo variable"
}

# This variable can be accessed in HCL using the `var.name_of_var` format

Here we define variables for our lambda-function module:

# terraform/modules/lambda-function/variables.tf

# -----------------------------------------------------------------------------
# Variables: Main
# -----------------------------------------------------------------------------

variable "function_name" {
  description = "AWS lambda resource name"
}

variable "s3_bucket_id" {
  description = "AWS S3 resource id"
}

variable "s3_key" {
  description = "AWS S3 resource object key"
}

variable "handler" {
  description = "AWS lambda resource handler"
}

variable "iam_role" {
  description = "AWS lambda resource iam role"
}

variable "dependency" {
  description = "AWS lambda depends on for creation"
}

variable "apigw_statement_id" {
  type = string
  default = "AllowExecutionFromAPIGateway"
  description = "Api gateway statement id"
}

variable "source_arn" {
  description = "Source arn for the apigatewayv2 api gateway"
}

Next up we specify outputs for our function. Outputs help us relay information in the CLI or expose information about our infrastructure to other parts of our terraform configuration file. They can be defined in the following format:

output "instance_ip_addr" {
  # required `value`
  value       = aws_instance.server.private_ip
  # optional `description`
  description = "The private IP address of the main server instance."
  # sensitive is optional, suppresses sensitive values e.g api keys in the cli
  sensitive   = true   
 # optional `depends_on` for dependencies
  depends_on = [
    # Security group rule must be created before this IP address could
    # actually be used, otherwise the services will be unreachable.
    aws_security_group_rule.local_access,
  ]
}

Here is what ours would look like:

# terraform/modules/lambda-function/outputs.tf

# Output variable definitions 

output "function_name" {
  description = "AWS lambda fn name"
  value       = aws_lambda_function.lambda_fn.function_name
}

output "function_handler" {
  description = "AWS lambda fn handler"
  value       = aws_lambda_function.lambda_fn.handler
}

output "invoke_arn" {
  description = "AWS lambda fn invoke arn"
  value       = aws_lambda_function.lambda_fn.arn
}

Because we need more than one route to access our lambda, we also need a module for our API gateway routes that we would create:

# terraform/modules/apigatewayv2-route/main.tf

# -----------------------------------------------------------------------------
# Resource: Api Gateway Integration Routes
# -----------------------------------------------------------------------------

resource "aws_apigatewayv2_route" "fn_rte" {
  api_id = var.api_id

  route_key = var.route_key
  target    = var.lambda_id
}

And the corresponding variables required to use the module:

# terraform/modules/apigatewayv2-route/variables.tf

variable "api_id" {
  description = "API Gateway V2 api id"
}

variable "route_key" {
  description = "route key for the route being created"
}

variable "lambda_id" {
  description = "AWS lambda resource id"
}

Another module required in our setup is one for our Cloudwatch logs. With this we log requests to our resource:

# terraform/modules/cloudwatch-log-group/main.tf

resource "aws_cloudwatch_log_group" "cwatch_logs" {
  name = var.log_name
 
  retention_in_days = 30
}

And the corresponding variable required by the module:

# terraform/modules/cloudwatch-log-group/variables.tf

# -----------------------------------------------------------------------------
# Variables: Main
# -----------------------------------------------------------------------------

variable "log_name" {
  description = "AWS cloudwatch resource name"
}

Output variable:

# terraform/modules/cloudwatch-log-group/outputs.tf

output "arn" {
  description = "Cloudwatch arn"
  value       = aws_cloudwatch_log_group.cwatch_logs.arn
}

Main

Now it’s time to create our main file:

# terraform/main.tf

terraform {
# providers are responsible for understanding API interactions and exposing 
# resources 
  required_providers { 
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3.48.0"
    }
  }

  required_version = "~> 1.0"
}

locals {
# global name of the resource we are managing, useful for tracking resources
  resource_name = "${var.namespace}-${var.resource_tag_name}"
}

# Here, we define aws credentials for accessing their APIs 
provider "aws" {
  region     = var.provider_region
  access_key = var.provider_access_key
  secret_key = var.provider_secret_key
}

# Because some resource names are globally unique in our AWS account, we 
# need a utility resource for generating random strings to attach to our resource 
# names:
# -----------------------------------------------------------------------------
# Resources: Random string generator
# -----------------------------------------------------------------------------
resource "random_string" "postfix" {
  length  = 6
  number  = false
  upper   = false
  special = false
  lower   = true
}

# -----------------------------------------------------------------------------
# Resources: S3, where our functions would be stored
# -----------------------------------------------------------------------------
resource "aws_s3_bucket" "artifact_store" {
  bucket        = "${local.resource_name}-artifact-store-${random_string.postfix.result}"
  acl           = "private"
  force_destroy = true
# tags for cost tracking
  tags = {
    Environment = var.namespace
    Name        = var.resource_tag_name
  }
}

# Our first artifact. This ensures our functions are created and deployed 
# the first time terraform is applied. This artifact is just 
# a zipped folder containing our lambda function build
resource "aws_s3_bucket_object" "artifact" {
  bucket = aws_s3_bucket.artifact_store.id

  key    = "${var.resource_tag_name}.zip"
  source = "../build/terraform-lambda.zip"

  depends_on = [
    aws_s3_bucket.artifact_store,
  ]
}

Main Variables

# terraform/variables.tf

# -----------------------------------------------------------------------------
# Variables: Cloud Provider
# -----------------------------------------------------------------------------

variable "provider_region" {
  description = "AWS region"
  default = "us-east-1"
}

variable "provider_access_key" {
  description = "AWS access key"
}

variable "provider_secret_key" {
  description = "AWS secret key"
}

# -----------------------------------------------------------------------------
# Variables: Main
# -----------------------------------------------------------------------------

variable "namespace" {
  description = "AWS resource namespace/prefix"
  default = "prod"
}

variable "resource_tag_name" {
  description = "Resource tag name for cost tracking"
  default = "terraform-lambda"
}


# -----------------------------------------------------------------------------
# Variables: API gateway
# -----------------------------------------------------------------------------

variable "apigw_statement_id" {
  type = string
  default = "AllowExecutionFromAPIGateway"
  description = "Api gateway statement id"
}


# -----------------------------------------------------------------------------
# Variables: Cloudwatch
# -----------------------------------------------------------------------------

variable "retention_in_days" {
  description = "AWS cloudwatch resource log retention"
  default = 30
}

Main Outputs

# terraform/outputs.tf

output "function_name" {
  description = "Name of the Lambda function."

  value = module.fn.function_name
}

output "function_url" {
  description = "Invoke url of the Lambda function."

  value = "${aws_apigatewayv2_stage.lambda_gw_stg.invoke_url}"
}

output "artifact_store" {
  description = "S3 bucket ID"

  value = aws_s3_bucket.artifact_store.id
}

output "artifact" {
  description = "Key of the artifact"

  value = aws_s3_bucket_object.artifact.key
}

Policy Template

This template would allow us to create policies for roles that need to be assumed

# terraform/policies/assume-role.tpl

	{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "${service}"
      },
      "Action": "${action}"
    }
  ]
}

Functions

In this file, we create our function and all necessary resources to expose it to the internet. We make use of the reusable modules created in our previous steps:

# terraform/functions.tf

# -----------------------------------------------------------------------------
# Resource: Lambda Role and Policy Document
# -----------------------------------------------------------------------------
data "template_file" "assume-role-template" {
  template = file("${path.module}/policies/assume-role.tpl")

  vars = {
    service = "lambda.amazonaws.com"
    action  = "sts:AssumeRole"
  }
}

resource "aws_iam_role" "lambda_exec" {
  name = "${local.resource_name}-role"

  assume_role_policy = data.template_file.assume-role-template.rendered
}

resource "aws_iam_role_policy_attachment" "lambda_policy" {
  role       = aws_iam_role.lambda_exec.name
policy_arn="arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
}

# -----------------------------------------------------------------------------
# Resource: Lambda Functions
# -----------------------------------------------------------------------------

module "fn" {
  # `source` references the lambda function module we created earlier
  source        = "./modules/lambda-function"
  function_name = "event"

  s3_bucket_id = aws_s3_bucket.artifact_store.id
  s3_key       = aws_s3_bucket_object.artifact.key

  handler = "index.createEvent"

  iam_role = aws_iam_role.lambda_exec.arn

  dependency = aws_s3_bucket_object.artifact

  source_arn = "${aws_apigatewayv2_api.lambda_api.execution_arn}/*/*"
}

# -----------------------------------------------------------------------------
# Resource: Api Gateway Routes
# -----------------------------------------------------------------------------
module "createEvent" {
 # `source` references the module we created earlier
  source = "./modules/apigatewayv2-route"
  api_id = aws_apigatewayv2_api.lambda_api.id

  route_key = "POST /createEvent"
  lambda_id = "integrations/${aws_apigatewayv2_integration.fn_integration.id}"
}

module "getEvent" {
  source = "./modules/apigatewayv2-route"
  api_id = aws_apigatewayv2_api.lambda_api.id

  route_key = "GET /getEvent"
  lambda_id = "integrations/${aws_apigatewayv2_integration.fn_integration.id}"
}

# -----------------------------------------------------------------------------
# Resource: API Gateway Integration
# -----------------------------------------------------------------------------
resource "aws_apigatewayv2_integration" "fn_integration" {
  api_id = aws_apigatewayv2_api.lambda_api.id


  integration_uri    = module.fn.invoke_arn
  integration_type   = "AWS_PROXY"
  integration_method = "POST"
}

# -----------------------------------------------------------------------------
# Resource: API Gateway API
# -----------------------------------------------------------------------------
resource "aws_apigatewayv2_api" "lambda_api" {
  name          = "${local.resource_name}-api"
  protocol_type = "HTTP"
}

# -----------------------------------------------------------------------------
# Resource: Api Gateway Logs
# -----------------------------------------------------------------------------
module "api_gw_lg" {
  source = "./modules/cloudwatch-log-group"

  log_name = "/aws/api_gw/${aws_apigatewayv2_api.lambda_api.name}"
}


# -----------------------------------------------------------------------------
# Resource: API Gateway Stage
# -----------------------------------------------------------------------------
resource "aws_apigatewayv2_stage" "lambda_gw_stg" {
  api_id = aws_apigatewayv2_api.lambda_api.id

  name        = local.resource_name
  auto_deploy = true

  access_log_settings {
    destination_arn = module.api_gw_lg.arn

    format = jsonencode({
      requestId               = "$context.requestId"
      sourceIp                = "$context.identity.sourceIp"
      requestTime             = "$context.requestTime"
      protocol                = "$context.protocol"
      httpMethod              = "$context.httpMethod"
      resourcePath            = "$context.resourcePath"
      routeKey                = "$context.routeKey"
      status                  = "$context.status"
      responseLength          = "$context.responseLength"
      integrationErrorMessage = "$context.integrationErrorMessage"
      }
    )
  }
}

Now that we have all our resources defined, It’s time to create them in the cloud. Let’s give it a shot.

Taking a shot, image credit - unsplash

From the root of your project, run the following commands:

1. cd terraform to checkout to the terraform directory
2. terraform init to initialize terraform locally and install dependencies
3. terraform plan -var "provider_access_key=<awsaccesskeyidhere>" -var "provider_secret_key=<awssecretaccesskeyhere>" to create a terraform plan, ie dry run.
4. terraform apply -var "provider_access_key=<awsaccesskeyidhere>" -var "provider_secret_key=<awssecretaccesskeyhere>" and then when prompted to enter a value, type in yes to create the necessary resources on aws.

Terraform would go ahead and create all the resources on AWS. The result of the last step is the following output:

artifact = <artifact object key>
artifact_store = <artifact store id>
function_name = <name of the function>
function_url = <The endpoint i.e function invocation url>

To test your new function, run the command below:

curl --header "Content-Type: application/json" --data '{"username":"xyz","password":"xyz"}' <function_url>

Congratulations! Your first lambda function is live. You can log into the AWS Management console to review all the resources created e.g Lambda, S3 and API Gateway.

Now that you have tested your lambda functions, you need to clean the stack i.e delete all the resources created. Let's do so:

1. terraform plan -destroy would let you know resources about to be cleaned and if you are okay, proceed to clean up with the following command
2. terraform destroy -var "provider_access_key=<awsaccesskeyidhere>" -var "provider_secret_key=<awssecretaccesskeyhere>" and then when prompted to enter a value, type in yes to clean up the resources on aws.

Now that you know how to manage AWS services using Terraform IAC, it is time to bring everything together in a cicd pipeline. In the next article, we would configure a pipeline for our project using CircleCI and Github, set up and automate our Terraform configurations using Terraform Cloud.

devops awssite reliability engineering 0 0