Deploying a python package to AWS Lambda via Cloudformation, CodeBuild and CodePipeline

Milan Cermak Source

I want to setup a CI/CD pipeline for my AWS infrastructure and a AWS Lambda function. The idea is to have everything in code, version-controlled and automated. I just want to git push to a repository and have CodePipeline take over from there, updating my infrastructure, running tests and, if successful, updating my Lambda function with the latest code.

I'm basing my CloudFormation template on this excellent example. It looks like this:

AWSTemplateFormatVersion: 2010-09-09
Description: playground pipeline 1
Parameters:
  SourceRepositoryName:
    Type: String
    Default: lambda-playground
  SourceBranchName:
    Type: String
    Default: master

Resources:
  ArtifactsBucket:
    Type: AWS::S3::Bucket
    DependsOn: CloudFormationRole
    DeletionPolicy: Delete
    Properties:
      BucketName: lambda-playground-artifacts

  CodeBuildRole:
    Type: AWS::IAM::Role
    DependsOn: CloudFormationRole
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action:
              - sts:AssumeRole
            Principal:
              Service:
                - codebuild.amazonaws.com
      Policies:
        - PolicyName: ServiceRole
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Sid: CloudWatchWriteLogsPolicy
                Effect: Allow
                Action:
                  - logs:CreateLogGroup
                  - logs:CreateLogStream
                  - logs:PutLogEvents
                Resource: '*'
              - Sid: CodeCommitPullPolicy
                Effect: Allow
                Action:
                  - codecommit:GitPull
                Resource: '*'
              - Sid: S3GetObjectPolicy
                Effect: Allow
                Action:
                  - s3:GetObject
                  - s3:GetObjectVersion
                Resource: '*'
              - Sid: S3PutObjectPolicy
                Effect: Allow
                Action:
                  - s3:PutObject
                Resource: '*'

  CodePipelineRole:
    Type: AWS::IAM::Role
    DependsOn: CloudFormationRole
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action:
              - sts:AssumeRole
            Principal:
              Service:
                - codepipeline.amazonaws.com
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/AdministratorAccess

  CloudFormationRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action:
              - sts:AssumeRole
            Principal:
              Service:
                - cloudformation.amazonaws.com
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/AdministratorAccess

  CodeCommitRepository:
    Type: AWS::CodeCommit::Repository
    Properties:
      RepositoryName: !Ref SourceRepositoryName

  CodeBuildProject:
    Type: AWS::CodeBuild::Project
    DependsOn: CloudFormationRole
    Properties:
      Description: A playground of Lambda
      Artifacts:
        Type: CODEPIPELINE
      Environment:
        ComputeType: BUILD_GENERAL1_SMALL
        Image: aws/codebuild/python:2.7.12
        Type: LINUX_CONTAINER
      Name: lambda-playground
      ServiceRole: !GetAtt CodeBuildRole.Arn
      Source:
        Type: CODEPIPELINE
      TimeoutInMinutes: 5

  CodePipeline:
    Type: AWS::CodePipeline::Pipeline
    Properties:
      ArtifactStore:
        Type: S3
        Location: !Ref ArtifactsBucket
      Name: !Ref AWS::StackName
      RestartExecutionOnUpdate: true
      RoleArn: !GetAtt CodePipelineRole.Arn
      Stages:
        - Name: Source
          Actions:
            - Name: Source
              ActionTypeId:
                Category: Source
                Owner: AWS
                Provider: CodeCommit
                Version: 1
              Configuration:
                RepositoryName: !Ref SourceRepositoryName
                BranchName: !Ref SourceBranchName
              OutputArtifacts:
                - Name: SourceOutput
        - Name: PipelineDeploy
          Actions:
            - Name: UpdatePipeline
              ActionTypeId:
                Category: Deploy
                Owner: AWS
                Provider: CloudFormation
                Version: 1
              Configuration:
                ActionMode: CREATE_UPDATE
                Capabilities: CAPABILITY_IAM
                RoleArn: !GetAtt CloudFormationRole.Arn
                StackName: !Ref AWS::StackName
                TemplatePath: SourceOutput::infra.yml
              InputArtifacts:
                - Name: SourceOutput
        - Name: Build
          Actions:
            - Name: BuildAndTest
              ActionTypeId:
                Category: Build
                Owner: AWS
                Provider: CodeBuild
                Version: 1
              Configuration:
                ProjectName: !Ref CodeBuildProject
              InputArtifacts:
                - Name: SourceOutput
              OutputArtifacts:
                - Name: BuildOutput

  LambdaFunction:
    Type: AWS::Lambda::Function
    Properties:
      Code:
        S3Bucket: !Ref ArtifactsBucket
        S3Key: !Ref BuildOutput # DOES NOT WORK
      FunctionName: playground-fc
      Handler: src.main.handler
      # TODO: Role: foo
      Runtime: python2.7

Outputs:
  ArtifactsBucketURL:
    Description: Artifacts bucket URL
    Value: !GetAtt ArtifactsBucket.WebsiteURL
  RepositoryURL:
    Description: SSH URL of the repository
    Value: !GetAtt CodeCommitRepository.CloneUrlSsh

So I have a CodePipeline with 3 stages - Source, which grabs code from a CodeCommit repo, PipelineDeploy, which updates my CloudFormation stack if necessary and Build, which runs the configured CodeBuild project.

My buildspec.yml is here:

version: 0.1

phases:
  install:
    commands:
      - pip install -r requirements.txt -t lib
  pre_build:
    commands:
      - python lib/pytest.py src
artifacts:
  type: zip
  files:
    - src/**/*
    - lib/**/*

It just installs the necessary libraries, runs the tests via pytest and creates a deployment zip. This zip file is then the OutputArtifact of the Build stage and gets stored in the ArtifactsBucket. However, each time, it gets a unique name (e.g. dfVV6Uh), which makes sense, but I don't know how to reference it in the LambdaFunction -> Properties -> Code -> S3Key field.

So my question is, how can I create a stack/pipeline, that after doing all the steps, will deploy the latest version to my AWS Lambda function? Is there a way to maybe use CodeDeploy to do this? What is the best practice here?

amazon-web-servicesaws-lambdaamazon-cloudformationaws-codepipelineaws-codebuild

Answers

answered 2 years ago awsnitin #1

There is an example of how to do achieve something similar (deploying lambda functions via CodePipeline/CodeBuild). http://docs.aws.amazon.com/lambda/latest/dg/automating-deployment.html

This example is for lambda functions written in NodeJS, but the basic idea is the same. You use CloudFormation to deploy/update your lambda functions after you've build your artifact via CodeBuild and let CodePipeline manage artifact propagation within stages.

Let me know if this helps.

answered 2 years ago wjordan #2

You can use a Parameter Override with Fn::GetArtifactAtt and the ObjectKey attribute to dynamically provide the the name of the artifact .zip generated by AWS CodePipeline to your CloudFormation deploy action.

Using your example, the configuration for your UpdatePipeline CloudFormation deploy action would look something like this:

Configuration:
  ActionMode: CREATE_UPDATE
  Capabilities: CAPABILITY_IAM
  RoleArn: !GetAtt CloudFormationRole.Arn
  StackName: !Ref AWS::StackName
  TemplatePath: SourceOutput::infra.yml
  ParameterOverrides:
    {
      "LambdaKey" : { "Fn::GetArtifactAtt" : ["LambdaFunctionSource", "ObjectKey"]}
    }
InputArtifacts:
- Name: SourceOutput
- Name: BuildOutput

Then, declare and then reference the LambdaKey Parameter within your CloudFormation stack template:

Parameters:
  LambdaKey:
    Type: String
  # ...
Resources:
  LambdaFunction:
    Type: AWS::Lambda::Function
    Properties:
      Code:
        S3Bucket: !Ref ArtifactsBucket
        S3Key: !Ref LambdaKey
      # ...

comments powered by Disqus