Working with Docker Compose? Use environmental variables

Intro to Docker Compose

Much like the Docker runtime, Compose is not limited to the Docker Enterprise platform. Compose also works with Amazon Elastic Container Service (ECS), Azure Container Instances (ACI) and Kubernetes.

Additionally, Docker made the Compose specification, a cloud-agnostic community project that works with other packaging formats such as Helm charts and Kubernetes YAML manifests. Furthermore, Compose can deploy applications on ACI, Amazon ECS and AWS Fargate, and can configure Kubernetes resources via the Kompose conversion tool. For example, the Azure integration lets developers use native Docker commands to run applications in ACI via the Docker CLI or Virtual Studio Code extension.

Compose is included in Docker Desktop for Windows and MacOS and can be installed manually on Linux via the Docker website.

The easiest and recommended way to get started using Docker Compose is by installing Docker Desktop. The prerequisites for running Compose are Docker Engine, which is an open-source container runtime, and Docker CLI, which is the command line application used to interact with Docker Engine.

The Compose specification defines the following as the essential elements of a multi-container application Compose file:

• Services. Services are based on one or more container images and run within a Docker runtime to provide application functionality.
• Networks. Networks provide IP connectivity and routing between containers.
• Volumes. Volumes store persistent data that can be shared among containers via a filesystem mount point.
• Configs. Configs enable services to change behavior without an image file rebuild via options stored in a file mounted to the container's file system.
• Secrets. Secrets containing sensitive configuration information such as server certificates.

Together, these components form an application project.

Compose file and syntax

The following three steps are required to deploy applications with Compose:

1. Define the application images and environment using a Dockerfile.
2. Define and configure the components of a multi-container application using a Docker Compose file.
3. Run docker-compose up to deploy and run the application.

A typical, albeit simple, example of a composite application is a two-tier website consisting of a front-end web server, a back-end database server and an associated disk volume. When implemented with containers, this consists of the following:

• Two services and related image files (web and database).
• One secret (HTTPS certificate for the web front end).
• One configuration (HTTP, also for the front end).
• One persistent volume mounted by the database.
• Two networks (front end to the internet; back end between the web and database servers).

The YAML Compose file for this project looks like the following code.

services:
  frontend:
    image: awesome/webapp
    ports:
      - "443:8043"
    networks:
      - front-tier
      - back-tier
    configs:
      - httpd-config
    secrets:
      - server-certificate

  backend:
    image: awesome/database
    volumes:
      - db-data:/etc/data
    networks:
      - back-tier

volumes:
  db-data:
    driver: flocker
    driver_opts:
      size: "10GiB"

configs:
  httpd-config:
    external: true

secrets:
  server-certificate:
    external: true

networks:
  # The presence of these objects is sufficient to define them
  front-tier: {}
  back-tier: {}

Compose files are typically named docker-compose.yml and use keywords defined in the specification. Runtime options can be set by command-line flags or environment variables in an env_file, typically carrying a .env suffix.

Compose environment variables

Environment variables are used in every programming language as an abstraction layer to let IT admins change parameters in a piece of software or configuration without changing the underlying code itself.

In Docker Compose, IT admins can use environment variables to generalize configurations for different situations, deployment environments and security contexts without editing the main project file(s) manually. Variables can either be passed as command-line arguments -- suitable for only a few parameters -- or via a .env file, preferred for more complicated configurations.

Using a .env file makes it easier to store and provide environment variables for multiple environments. Instead of providing the environment variables as command line arguments, a simple file is used where the environment variables are provided line by line. See the example below for the format.

SERVICE_PORT=8080
IMAGE_VERSION=1.23.4

Maintaining multiple .env files -- perhaps named .env.dev and .env.prod -- makes it quick to ensure the proper environment variables are passed to the docker compose file when running your application in the respective environment. The command for starting docker compose with a .env file is docker compose --env-file .env.dev up.

Docker Compose YAML files are of key-value pairs. Compose environment files support both map and array syntaxes, as illustrated in Figure 1.

Because variables can be defined in several places, developers must be careful not to override items defined in an environment file. The order of evaluation precedence for variables is compose file, shell environment variables, environment file and Dockerfile.

As the syntax example in Figure 1 indicates, environment variables are best used to specify deployment-specific parameters:

• An application name.
• Container image.
• Volume names and mount points.
• IP addresses and ports.
• Time intervals.
• Boolean values and runtime variables -- for example, DB_USERNAME=webapp.

Do not store security information, such as passwords, SSH private keys, SSL certificates or any other data that should not be stored as clear text, as environment files and variables. Instead, use Swarm secrets, Kubernetes Secrets or a cloud service like AWS Secrets Manager or Google Cloud Secret Manager, which then integrates with the other secret providers above.

The main Compose file uses variable substitution to refer to parameters defined in an environment file. This file supports both $VARIABLE and ${VARIABLE} syntax.

# database server
db-container:
IMAGE:"mysql:${mysql version}"
PORT="{db_port}"

Environment variables are a critical element of Docker Compose, as the prime advantage is reusable multi-container application configurations. Variables enable developers to modify parameters quickly for different deployment and resource needs without adjusting the underlying code and, thus, should be part of every production Docker Compose environment.

Editor's Note: This article was originally written by Kurt Marko in 2021. Matthew Grasberger revised it in 2024 to improve the reader experience.

Matthew Grasberger is a DevOps engineer at Imperfect Foods. He has experience in test automation, DevOps engineering, security automation and open-source mobile testing frameworks.