GitOps implementation using GitLabCI and AWS

GitOps is nice and all, but how to implement GitOps may not be so easy. In this article we'll see an example implementation using GitLabCI and AWS.

---

We'll talk a lot about GitOps, GitLabCI and AWS. If you haven't already heard about GitOps, here are some nice articles from Weaveworks and GitLab.

---

Minimal example

Let's consider the following context:

• Our application is a small static website on S3 deployed with CloudFormation.
• A GitLab project is hosting our Infra as Code (CloudFormation template and website static content)
  • See CloudFormation template source code here
• Deployments are done manually by an operator via a command like aws cloudformation deploy

We want to implement GitOps such as:

• Any change on master must trigger a deployment to production
• Changes on master must be done via Merge Request in order to be reviewed and validated

Let's add .gitlab-ci.yml such as:

# Run our Infra as Code and deploy our website
# This is a Gitlab hidden job we can exend in concrete job
deploy:
  stage: deploy
  # Use AWS CLI image to deploy with CloudFormation
  image: 
    name: amazon/aws-cli:2.2.13
    entrypoint: [""]
  script:
  # Deploy CloudFormation stack
  # HostedZoneName is the name of the AWS Hosted Zone to use, such as devops.crafteo.io
  # WebsiteFQDN is our website domain name, such as gitlabci-gitops-s3-website.devops.crafteo.io
  - >
    aws cloudformation deploy 
    --template-file cloudformation.yml 
    --stack-name gitops-s3-static-website-$CI_COMMIT_REF_SLUG
    --parameter-overrides 
        HostedZoneName="devops.crafteo.io." 
        WebsiteFQDN="gitlabci-gitops-s3-website.devops.crafteo.io"

  # Copy our website static content to S3 bucket
  # S3 bucket is named after website FQDN
  - aws s3 sync --delete website/ s3://gitlabci-gitops-s3-website.devops.crafteo.io
  # We only want to deploy on master branch
  rules:
   - if: $CI_COMMIT_REF_NAME == "master"

We now need AWS credentials on CI for our job to run. Let's create an IAM user, affect it proper policy for deployment (see example policy here) and generate AWS credentials.

We can now configure them as CI variables. For additional security, protect the master branch and set variables as masked and protected to get something like:

Type	Key	Value	Protected	Masked	Environment
Variable	AWS_ACCESS_KEY_ID	***	✓	✓	production
Variable	AWS_SECRET_ACCESS_KEY	***	✓	✓	production

See Run AWS commands from GitLab CI/CD for details.

That's it! Now, any change on master (typically via Merge Request) will trigger deployment.

Multi-environment GitOps implementation

Let's consider a multi-environment and multi-AWS account deployment using a GitFlow-like pattern:

• Production environment is based on master branch and deployed on AWS Production account
  • Only dev or hotfixes can be merged in master
• Dev environment is based on dev branch and deployed on AWS Development account
  • Dev is used to test our application before deployment to production
  • Features branches are created from and merged to dev
• Ephemeral review environments can be deployed from any other branches on AWS Development account
  • New features are implemented on these branches
  • Ephemeral environments can be used to test the feature implementation before being merged in dev
  • They must be cleaned-up automatically or on-demand

Let's update our .gitlab-ci.yml with jobs:

• .deploy: a generic hidden job with we can extend for each environment
• deploy-dev extends .deploy to manage Dev and ephemeral environment deployment from dev
• deploy-prod extends .deploy to manage Production environment deployment from master
• undeploy-dev extends .deploy to manage dev / ephemeral environments cleanup after some time

# Generic job for deployment
# This is a Gitlab hidden job we can extend in concrete job 
# for each environment
.deploy:
  stage: deploy
  image: 
    name: amazon/aws-cli:2.2.13
    entrypoint: [""]
  variables:
    # Name of CloudFormation stack
    # Use the ref slug (i.e. sluggified branch or tag name)
    # to ensure our stack name is unique across branches and environments
    CLOUDFORMATION_STACK_NAME: gitops-s3-static-website-$CI_COMMIT_REF_SLUG
    # Route53 Hosted zone and Fully Qualified Domain Name (FQDN)
    # used to manage DNS records
    HOSTED_ZONE_NAME: devops.crafteo.io.
    # This variable is not set yet but must be set on extending jobs
    # to define our website FQDN
    # WEBSITE_FQDN: <REQUIRED ON EXTENDING JOBS>
  script:
  # Deploy CloudFormation stack
  # HostedZoneName is the name of the AWS Hosted Zone such as devops.crafteo.io
  # WebsiteFQDN is our website domain name such as gitlabci-gitops-s3-website.devops.crafteo.io
  - >
    aws cloudformation deploy 
    --template-file cloudformation.yml 
    --stack-name $CLOUDFORMATION_STACK_NAME
    --parameter-overrides 
      HostedZoneName="$HOSTED_ZONE_NAME" 
      WebsiteFQDN="$WEBSITE_FQDN"

  # Copy our website static content to S3 bucket
  # S3 bucket is named after website FQDN
  - aws s3 sync --delete website/ s3://$WEBSITE_FQDN

# Dev job to deploy both dev environment and ephemeral environments
# Dev environment will be deployed from dev branch using FQDN
#   gitlabci-gitops-s3-website.dev.devops.crafteo.io
# Ephemeral environment will be deployed from other branches using FQDN
#   gitlabci-gitops-s3-website.<branch name>.devops.crafteo.io
deploy-dev:
  extends: .deploy
  variables:
    WEBSITE_FQDN: gitlabci-gitops-s3-website.$CI_COMMIT_REF_SLUG.devops.crafteo.io
  # Define a GitLab environment
  # Will be automatically destroyed after 1 day using undeploy-dev job
  environment:
    name: review/$CI_COMMIT_REF_NAME
    url: http://gitlabci-gitops-s3-website.$CI_COMMIT_REF_SLUG.devops.crafteo.io
    on_stop: undeploy-dev
    auto_stop_in: 1 day
  # Auto deploy on dev (dev environment)
  # Manual deployment on non-master and non-dev branches (ephemeral environments)
  rules:
  - if: $CI_COMMIT_REF_NAME == "dev"
  - if: $CI_COMMIT_REF_NAME != "dev" && $CI_COMMIT_REF_NAME != "master"
    when: manual
    allow_failure: true

# Stop job for dev / ephemeral deployments
# Extends deployment job to inherit variables, CI environment and other configs
undeploy-dev:
  extends: deploy-dev
  # Tell GitLabCI this job is for stopping our environment
  # As we extend deploy-dev, we don't need to specify environment name again
  environment:
    action: stop
  script:
  # Delete CloudFormation stack, but bucket will be kept as we use Retained deletion policy
  - aws cloudformation delete-stack --stack-name $CLOUDFORMATION_STACK_NAME
  # Cleanup remaining bucket (won't be deleted by CF)
  - aws s3 rb s3://$WEBSITE_FQDN --force
  # Only run manually on non-master branch (dev and any other branches)
  rules:
  - if: $CI_COMMIT_REF_NAME != "master"
    when: manual
    allow_failure: true

# Production job to deploy Production to 
# gitlabci-gitops-s3-website.devops.crafteo.io
# Only run on master
deploy-prod:
  extends: .deploy
  variables:
    WEBSITE_FQDN: gitlabci-gitops-s3-website.devops.crafteo.io
  environment:
    name: production
    url: http://gitlabci-gitops-s3-website.devops.crafteo.io
  rules:
  - if: $CI_COMMIT_REF_NAME == "master"

We now need to configure AWS credentials for each account (Dev and Production). Create an AWS IAM user per account (with related IAM policy, see an example here) and generate credentials for each.

For Dev AWS account, configure CI variables and set Environments as review/* to match the environment: keyword from .gitlab-ci.yml. Do the same for production environment using Prod credentials, and set them as protected. You'll get something like:

Type	Key	Value	Protected	Masked	Environment
Variable	AWS_ACCESS_KEY_ID	***	X	✓	review/*
Variable	AWS_SECRET_ACCESS_KEY	***	X	✓	review/*
Variable	AWS_ACCESS_KEY_ID	***	✓	✓	production
Variable	AWS_SECRET_ACCESS_KEY	***	✓	✓	production

See Run AWS commands from GitLab CI/CD for details.

We now manage a multi-environment application the GitOps way using GitLabCI & AWS!

Full example on Github

This GitOps implementation example is based on GitLabCI and AWS along with Infra as Code, Merge Request and CI, but lots of different ways and tools can be used to implement GitOps. We'll see some in upcoming articles.

You can checkout the full example on GitLab. For more example implementation of DevOps tools and practices, see DevOps examples on my GitHub.