GitOps implementation using GitLabCI and AWS
GitOps is nice and all, but how to implement GitOps may not be so easy. In this article we'll see an example implementation using GitLabCI and AWS.
We'll talk a lot about GitOps, GitLabCI and AWS. If you haven't already heard about GitOps, here are some nice articles from Weaveworks and GitLab.
Let's consider the following context:
- Our application is a small static website on S3 deployed with CloudFormation.
- A GitLab project is hosting our Infra as Code (CloudFormation template and website static content)
- Deployments are done manually by an operator via a command like
aws cloudformation deploy
We want to implement GitOps such as:
- Any change on
mastermust trigger a deployment to production
- Changes on
mastermust be done via Merge Request in order to be reviewed and validated
.gitlab-ci.yml such as:
# Run our Infra as Code and deploy our website # This is a Gitlab hidden job we can exend in concrete job deploy: stage: deploy # Use AWS CLI image to deploy with CloudFormation image: name: amazon/aws-cli:2.2.13 entrypoint: [""] script: # Deploy CloudFormation stack # HostedZoneName is the name of the AWS Hosted Zone to use, such as devops.crafteo.io # WebsiteFQDN is our website domain name, such as gitlabci-gitops-s3-website.devops.crafteo.io - > aws cloudformation deploy --template-file cloudformation.yml --stack-name gitops-s3-static-website-$CI_COMMIT_REF_SLUG --parameter-overrides HostedZoneName="devops.crafteo.io." WebsiteFQDN="gitlabci-gitops-s3-website.devops.crafteo.io" # Copy our website static content to S3 bucket # S3 bucket is named after website FQDN - aws s3 sync --delete website/ s3://gitlabci-gitops-s3-website.devops.crafteo.io # We only want to deploy on master branch rules: - if: $CI_COMMIT_REF_NAME == "master"
We now need AWS credentials on CI for our job to run. Let's create an IAM user, affect it proper policy for deployment (see example policy here) and generate AWS credentials.
We can now configure them as CI variables. For additional security, protect the master branch and set variables as masked and protected to get something like:
See Run AWS commands from GitLab CI/CD for details.
That's it! Now, any change on
master (typically via Merge Request) will trigger deployment.
Multi-environment GitOps implementation
Let's consider a multi-environment and multi-AWS account deployment using a GitFlow-like pattern:
- Production environment is based on
masterbranch and deployed on AWS Production account
devor hotfixes can be merged in
- Dev environment is based on
devbranch and deployed on AWS Development account
- Dev is used to test our application before deployment to production
- Features branches are created from and merged to
- Ephemeral review environments can be deployed from any other branches on AWS Development account
- New features are implemented on these branches
- Ephemeral environments can be used to test the feature implementation before being merged in
- They must be cleaned-up automatically or on-demand
Let's update our
.gitlab-ci.yml with jobs:
.deploy: a generic hidden job with we can extend for each environment
.deployto manage Dev and ephemeral environment deployment from
.deployto manage Production environment deployment from
.deployto manage dev / ephemeral environments cleanup after some time
# Generic job for deployment # This is a Gitlab hidden job we can extend in concrete job # for each environment .deploy: stage: deploy image: name: amazon/aws-cli:2.2.13 entrypoint: [""] variables: # Name of CloudFormation stack # Use the ref slug (i.e. sluggified branch or tag name) # to ensure our stack name is unique across branches and environments CLOUDFORMATION_STACK_NAME: gitops-s3-static-website-$CI_COMMIT_REF_SLUG # Route53 Hosted zone and Fully Qualified Domain Name (FQDN) # used to manage DNS records HOSTED_ZONE_NAME: devops.crafteo.io. # This variable is not set yet but must be set on extending jobs # to define our website FQDN # WEBSITE_FQDN: <REQUIRED ON EXTENDING JOBS> script: # Deploy CloudFormation stack # HostedZoneName is the name of the AWS Hosted Zone such as devops.crafteo.io # WebsiteFQDN is our website domain name such as gitlabci-gitops-s3-website.devops.crafteo.io - > aws cloudformation deploy --template-file cloudformation.yml --stack-name $CLOUDFORMATION_STACK_NAME --parameter-overrides HostedZoneName="$HOSTED_ZONE_NAME" WebsiteFQDN="$WEBSITE_FQDN" # Copy our website static content to S3 bucket # S3 bucket is named after website FQDN - aws s3 sync --delete website/ s3://$WEBSITE_FQDN # Dev job to deploy both dev environment and ephemeral environments # Dev environment will be deployed from dev branch using FQDN # gitlabci-gitops-s3-website.dev.devops.crafteo.io # Ephemeral environment will be deployed from other branches using FQDN # gitlabci-gitops-s3-website.<branch name>.devops.crafteo.io deploy-dev: extends: .deploy variables: WEBSITE_FQDN: gitlabci-gitops-s3-website.$CI_COMMIT_REF_SLUG.devops.crafteo.io # Define a GitLab environment # Will be automatically destroyed after 1 day using undeploy-dev job environment: name: review/$CI_COMMIT_REF_NAME url: http://gitlabci-gitops-s3-website.$CI_COMMIT_REF_SLUG.devops.crafteo.io on_stop: undeploy-dev auto_stop_in: 1 day # Auto deploy on dev (dev environment) # Manual deployment on non-master and non-dev branches (ephemeral environments) rules: - if: $CI_COMMIT_REF_NAME == "dev" - if: $CI_COMMIT_REF_NAME != "dev" && $CI_COMMIT_REF_NAME != "master" when: manual allow_failure: true # Stop job for dev / ephemeral deployments # Extends deployment job to inherit variables, CI environment and other configs undeploy-dev: extends: deploy-dev # Tell GitLabCI this job is for stopping our environment # As we extend deploy-dev, we don't need to specify environment name again environment: action: stop script: # Delete CloudFormation stack, but bucket will be kept as we use Retained deletion policy - aws cloudformation delete-stack --stack-name $CLOUDFORMATION_STACK_NAME # Cleanup remaining bucket (won't be deleted by CF) - aws s3 rb s3://$WEBSITE_FQDN --force # Only run manually on non-master branch (dev and any other branches) rules: - if: $CI_COMMIT_REF_NAME != "master" when: manual allow_failure: true # Production job to deploy Production to # gitlabci-gitops-s3-website.devops.crafteo.io # Only run on master deploy-prod: extends: .deploy variables: WEBSITE_FQDN: gitlabci-gitops-s3-website.devops.crafteo.io environment: name: production url: http://gitlabci-gitops-s3-website.devops.crafteo.io rules: - if: $CI_COMMIT_REF_NAME == "master"
We now need to configure AWS credentials for each account (Dev and Production). Create an AWS IAM user per account (with related IAM policy, see an example here) and generate credentials for each.
For Dev AWS account, configure CI variables and set
review/* to match the
environment: keyword from
.gitlab-ci.yml. Do the same for
production environment using Prod credentials, and set them as protected. You'll get something like:
See Run AWS commands from GitLab CI/CD for details.
We now manage a multi-environment application the GitOps way using GitLabCI & AWS!
Full example on Github
This GitOps implementation example is based on GitLabCI and AWS along with Infra as Code, Merge Request and CI, but lots of different ways and tools can be used to implement GitOps. We'll see some in upcoming articles.
You can checkout the full example on GitLab. For more example implementation of DevOps tools and practices, see DevOps examples on my GitHub.