Docker

Setting Up a Python Development Environment with Docker Compose

Zachary Carciu • • 6 min read
Table of Contents

Contents

Setting Up a Python Development Environment with Docker Compose

This guide walks you through setting up a robust Python development environment using Docker Compose. The setup includes services for PostgreSQL, Jupyter Notebook, Streamlit, and pytest, ensuring a seamless workflow for development, testing, and data management.

Project Structure

The directory structure for the project is organized as follows:

.
├── docker-compose.yml
├── dockerfiles/
│   ├── Dockerfile_postgres
│   ├── Dockerfile_notebook
│   ├── Dockerfile_streamlit
│   └── Dockerfile_pytest
├── src/
│   ├── notebooks/
│   └── streamlit_dashboard/
│       └── streamlit_app.py
├── data/
│   ├── db/
│   └── sql/
├── requirements.txt
└── .env

Service Components

1. PostgreSQL Database

The PostgreSQL service provides a persistent database using the official PostgreSQL Alpine image. Below is the Dockerfile:

FROM postgres:15.7-alpine3.19

RUN apk update && apk add vim less

ENV PAGER less
ENV EDITOR vim

Key configuration in docker-compose.yml:

Note the volumes here! /var/lib/postgresql/data is where your database is written on disk, and we create a directory called ./data/db under the context dir. Our new ./data/db directory on the host machine is mapped to the /var/lib/postgresql/data in the container. This allows your data to persist between restarts and shutdowns of docker compose.

database:
  build:
    dockerfile: dockerfiles/Dockerfile_postgres
    context: .
  volumes:
    - ./data/db:/var/lib/postgresql/data
  environment:
    - POSTGRES_USER=postgres
    - POSTGRES_PASSWORD=postgres
    - POSTGRES_DB=testdb
  ports:
    - 5432:5432

Now, that our database is defined, let’s move on to setting up the notebook service!

2. Jupyter Notebook

The notebook service uses the Jupyter scipy-notebook base image, with additional Python packages installed for database interaction and project dependencies:

FROM jupyter/scipy-notebook

USER root
RUN apt update -y && apt install -y libpq-dev python3-dev gcc g++
USER jovyan

COPY ./requirements.txt .
RUN pip install psycopg2-binary
RUN pip install -r requirements.txt

Configuration in docker-compose.yml:

notebook:
  build:
    dockerfile: dockerfiles/Dockerfile_notebook
    context: .
  volumes:
    - ./src:/src
    - ./src/notebooks:/home/jovyan/work
  ports:
    - 8888:8888
  command: jupyter notebook

Note: The volume mapped with ./src/notebooks:/home/jovyan/work allows us to persist the notebooks we create in jupyer. The ./src:/src mapping allows us to access our code.

3. Streamlit Dashboard

The Streamlit service uses Python 3.11 as the base image and installs necessary packages for web application hosting. Below we will install our requirements along with a few other libraries, and we will set the PYTHONPATH environment variable. This needs to align with where we mapped our volume for us to be able to import our code into the notebooks!

FROM python:3.11

WORKDIR /app

COPY requirements.txt .
RUN pip install -r requirements.txt
RUN pip install streamlit streamlit_modal rich

ENV PYTHONPATH=/app/src

Configuration in docker-compose.yml:

streamlit:
  build:
    dockerfile: dockerfiles/Dockerfile_streamlit
    context: .
  volumes:
    - ./:/src
    - ./requirements.txt:/src/src/requirements.txt
  ports:
    - 8501:8501
  command: streamlit run /src/src/streamlit_dashboard/streamlit_app.py

4. Testing Environment

The pytest service is configured to run automated tests within the project:

FROM python:3.11

WORKDIR /src

COPY requirements.txt .
RUN pip install -r requirements.txt
RUN pip install pytest

Configuration in docker-compose.yml:

pytest:
  profiles:
    - test
  build:
    context: .
    dockerfile: dockerfiles/Dockerfile_pytest
  volumes:
    - .:/src
  working_dir: /src/test
  entrypoint: ["pytest", "-s", "-o", "log_cli=true"]

Network Configuration

All services are connected via an internal bridge network for seamless communication:

networks:
  internal-network:
    driver: bridge

Getting Started

Step 1: Create Necessary Directories

Run the following commands to set up the required directories:

mkdir -p src/notebooks src/streamlit_dashboard data/{db,sql}

Step 2: Create an .env File

Define environment variables in the .env file:

touch .env

Step 3: Start the Services

Use Docker Compose to start all services in detached mode:

docker compose up -d

Access Points:

Step 4: Run Tests

To execute tests with pytest:

docker compose --profile test run pytest

Best Practices

  1. Secure Sensitive Information: Store credentials and sensitive data in the .env file and avoid hardcoding.
  2. Use Volume Mounts: Leverage volume mounts for persistent data and easy access during development.
  3. Isolate Build Contexts: Keep separate build contexts for each service to optimize Docker performance.
  4. Pin Image Versions: Use specific version tags for base images to ensure reproducibility.
  5. Configure PYTHONPATH: Set PYTHONPATH for clean imports and better module organization.

This setup offers a complete development environment with:

  • Persistent database management
  • Interactive notebook development
  • Streamlit-based web application hosting
  • Automated testing with pytest

By following this guide, you’ll have a scalable and maintainable Python development environment powered by Docker Compose.