Flox + Novops: development packages, secrets and multi-environment management all-in-one !
Setting up development environment often turns into a developer horror story. Bring secrets, environment variables, authentication and consistency between each developer's machines and CI pipelines and it becomes a nightmare. Thankfully, modern tools like Novops and Flox can help streamline most of this boilerplate.
Flox: reproducible and shareable development environment
Flox automates creation of development environment with all required dependencies. Developers can quickly get started on a project without having to install a bunch of tools - and it ensures everyone's on the same page !
Suppose you're working on this NodeJS project, with a bit of Python on the side and a few other packages. Sure you have npm to install Node dependencies, but what about Node itself and other non-Node packages ?
Using Flox you can define all of your dependencies via a manifest.toml such as:
[install]
# Core project dependencies
# Node and Python deps can be installed using their respective package managers
nodejs.pkg-path = "nodejs"
python3.pkg-path = "python3"
# A few more tools useful for development
just.pkg-path = "just" # Task runner
jq.pkg-path = "jq" # JSON processor
podman.pkg-path = "podman" # Container engine for local development with containers
podman-compose.pkg-path = "podman-compose" # Running our development environment is a breeze:
$ flox activate
✅ You are now using the environment 'super-app'.
To stop using this environment, type 'exit'
flox [super-app] $ node --version
22.10.0
flox [super-app] $ python --version
Python 3.10.9
flox [super-app] $ just --version
just 1.9.0
# ...In short, Flox is a universal virtual environment manager working with any technology out of the box. You can combine it with language-specific package manager like Node's npm and package.json or Python pip.
We could go further and automatically run npm install on startup by adding a hook in our manifest.toml:
[hook]
on-activate = """
npm install
"""But what about things like secrets, environment variables and managing multiple environments (Staging, Preprod, Prod...) ?
Novops: universal secret and configuration manager
Novops is a universal secret and configuration manager for development, applications and CI. It can load secrets from various sources (Hashicorp Vault, AWS / Azure / GCP Secret Manager, password managers like BitWarden...) - and also manage multi-environments contexts.
Considering our NodeJS application, let's say we need to provide easy access to our developer to a development and staging environment deployed on AWS with secrets stored in Hashicorp Vault. Setting up required config is generally cumbersome: installing required tools (AWS, Kubernetes and Vault CLI), logging-in, escalating to related IAM Roles, etc. - all of that by switching between environments. A Developer Hell 😨
With Novops, all of this becomes easy. We can create a .novops.yml such as:
#
# .novops.yml content
#
environments:
# Dev environment configs
# Environment variables and secrets
dev:
variables:
#
# Plain variables specific to our environment
#
# Name of current environment
- name: SUPER_APP_ENVIRONMENT
value: dev
# AWS region to use
- name: AWS_DEFAULT_REGION
value: eu-central-1
#
# Fetch and generate secrets from Hashicorp Vault
#
# Database password in Hashicorp Vault
# Fetch value "password" from secret "superapp/dev/database"
# And save it as environment variable SUPER_APP_DB_PASSWORD
- name: SUPER_APP_DB_PASSWORD
value:
hvault_kv2:
path: superapp/dev/database
key: password
- name: SUPER_APP_DB_USER
value:
hvault_kv2:
path: superapp/dev/database
key: user
# Load secret files directly in memory
files:
- variable: SUPER_APP_TOKEN
content:
hvault_kv2:
path: superapp/dev/api
key: token
# Generate AWS credentials (impersonate an IAM role "dev_role")
# Will set 3 AWS built-in environment variables:
# AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN
hashivault:
aws:
name: dev_role
role_arn: arn:aws:iam::111122223333:role/dev_role
# Staging environment config
staging:
variables:
# [...]
# Setup Vault address to use globally
config:
hashivault:
address: https://vault.supercompany.org:8200On load Novops will:
- Fetch database user/password from Vault and set
SUPER_APP_DB_PASSWORDandSUPER_APP_DB_USERenvironment variables - Generate temporary AWS credentials using Vault AWS Secret Engine
- Setup a few environment variables specific to our environment
Run source <(novops load) to load environment. All secrets are stored securely in memory and discarded when the session ends:
$ source <(novops load)
? Select environment
❯ dev
staging
$ echo $SUPER_APP_DB_PASSWORD
Super!p3sSw0rd
$ aws sts get-caller-identity
{
"UserId": "AROAEXAMPLEID123:xxx",
"Account": "111122223333",
"Arn": "arn:aws:sts::111122223333:assumed-role/dev_role/xxx"
}
$ exit # Secret environment variables are discarded as they were stored in shell's session
$ echo SUPER_APP_DB_PASSWORD
# <nothing>
$ aws sts get-caller-identity
# <not logged in>Novops and Flox: greater together !
Putting both Flox and Novops together, we can use the above .novops.yml with a Flox hook to automatically setup our environment:
#
# manifest.toml content
#
[install]
# [...] Other app packages
# Add our packages
awscli2.pkg-path = "awscli2" # AWS CLI
kubectl.pkg-path = "kubectl" # Kubernetes CLI
vault.pkg-path = "vault" # Hashicorp Vault CLI
# Alternatively, run:
#
# flox install awscli2 kubectl vault
#
# Setup hook to login to Vault and
# load Novops automatically on startup
[hook]
on-activate = """
# [ ... ]
# Let user login to vault via web UI.
# Will set environment variable VAULT_TOKEN
# used by Novops to authenticate with Vault.
#
# Novops can also authenticate with Vault directly
# See https://novops.dev/config/hashicorp-vault.html
#
export VAULT_TOKEN=$(vault login -method=oidc -path=oidc -token-only role="default" 2> /dev/null)
# Load secrets and variables with Novops
source <(novops load)
echo "Environment $SUPER_APP_ENVIRONMENT ready!"
"""Let's run it!
$ flox activate
# ✅ You are now using the environment 'flox-novops-demo'.
# To stop using this environment, type 'exit'
# ? Select environment
# ❯ dev
# staging
# Environment dev ready!
flox [super-app] $ echo $SUPER_APP_DB_PASSWORD
# Super!p3sSw0rd
flox [super-app] $ aws sts get-caller-identity
# {
# "UserId": "AROAEXAMPLEID123:xxx",
# "Account": "111122223333",
# "Arn": "arn:aws:sts::111122223333:assumed-role/dev_role/xxx"
# }
flox [super-app] $ exit
$ echo $SUPER_APP_DB_PASSWORD
# <nothing>
$ aws sts get-caller-identity
# <not logged in>Consistency between developers' environments, CI pipelines and automated scripts
Using Flox and Novops, we can now ensure consistency between developer's environments: Flox uses a .lock file and is backed by Nix, a powerful package manager, ensuring that all sessions use the exact same versions of our packages.
Even better: we can integrate all of this into our CI pipelines and automated scripts! Simply configure your CI pipeline to run commands such as:
flox activate -- echo 'Running my command with $SUPER_APP_ENVIRONMENT'Passing -- [COMMAND] to flox activate will start the environment and run a one-off command, allowing usage in scripts and CI pipelines.
Transform development environment setup into a painless experience
Using both Novops and Flox allow for seamless development, putting all together packages, environment variables, secrets and multi-environment management. To get started install Flox and Novops:
Enjoyed this article? Feel free to give Novops and Flox a ⭐ on GitHub!