Skip to content

Building a Universal Container System (So I Never Have to Write Another Dockerfile)

  • Docker
  • DevOps
  • Infrastructure
  • Automation

TL;DR: Built a modular Dockerfile system that lets you compose dev/prod containers using build arguments instead of writing custom Dockerfiles. Includes 28 feature modules with 100+ tools, weekly automated version updates with testing, and support for Python/Node/Rust/Go/Kubernetes and more. Saves me between a few hours and a day or two per project setup in addition to a lot of downstream effort with updates and coordinating environments.

The Problem That Wouldn’t Go Away

You know that feeling when you’re starting a new project and you think “oh great, I get to set up devcontainer again”? That was me, every single time. Another day lost to writing yet another Dockerfile, configuring Python, Node, databases, cloud tools, dev tools… rinse and repeat.

After several projects, I realized something depressing: I was literally copying and pasting the same 300 lines of Dockerfile with minor tweaks. The only thing changing was whether I needed Python 3.12 or 3.13, or if this project used Postgres or MySQL.

But the real pain hit when maintenance season arrived. Python releases a security patch? Great, now I get to update six different repos. New team security practice? Update six Dockerfiles. New teammate starts a project? They copy the Dockerfile from the last project (which was already out of date), and now we have seven slightly different configurations.

This is ridiculous, I thought. I’m a designer and understand software engineering best practices and patterns. I should be able to solve this.

The core problem wasn’t just the repetition. It was the maintenance burden. I don’t have time to manually track when Python 3.13.7 comes out, or when kubectl updates, or when some npm package has a critical vulnerability. I needed automation that would handle the routine stuff and only bother me when something actually broke.

So I built it.

The Core Idea

What if, instead of writing custom Dockerfiles, you could just declare what you want?

docker build -t myproject:dev \
  -f containers/Dockerfile \
  --build-arg INCLUDE_PYTHON_DEV=true \
  --build-arg INCLUDE_NODE_DEV=true \
  --build-arg INCLUDE_POSTGRES_CLIENT=true \
  --build-arg NODE_VERSION=20 \
  --build-arg PYTHON_VERSION=3.13 \
  .

That’s it. No custom Dockerfile to write or maintain. Just build arguments that compose pre-tested features into exactly what you need.

Feature composition diagram

  • Base layer: “Debian Slim” (gray box)
  • Arrow down with “+INCLUDE_PYTHON_DEV=true” label
  • Second layer: “Base + Python 3.13 + Poetry + Pytest + Black + Mypy” (blue boxes stacking)
  • Arrow down with “+INCLUDE_NODE_DEV=true” label
  • Third layer: Previous + “Node 20 + TypeScript + ESLint + Jest” (green boxes adding)
  • Arrow down with “+INCLUDE_KUBERNETES=true” label
  • Final layer: Previous + “kubectl + helm + k9s” (purple boxes adding)

The solution ended up being a single, modular Dockerfile that can create any development or production container through build-time configuration. I designed it as a git submodule so I could add it to any project and immediately have access to dozens of pre-built features. This means you get the benefits of a centralized, maintained Dockerfile without losing the ability to customize per-project.

Want Python with all the dev tools? INCLUDE_PYTHON_DEV=true. Need to add Kubernetes tools later? INCLUDE_KUBERNETES=true. The Dockerfile doesn’t change. The project doesn’t change. You just flip a switch.

Repository: https://github.com/joshjhall/containers

Let me show you what this actually looks like in practice.

What This Actually Solves

Let me be concrete about what changes when you use this.

Setup Time: Days to Minutes

Before: Starting a new Python API project meant 1-2 days of Dockerfile work. Copy an old one, update Python version, fix broken apt packages, research how to install AWS CLI, configure poetry, set up pytest, add black and mypy, configure non-root user, set up entrypoint scripts… you know the drill.

Now: Add the git submodule, set INCLUDE_PYTHON_DEV=true, and you’re done in 10 minutes. Everything is already there and tested.

  • Left side (Before): Timeline showing “Day 1: Research Docker setup, write Dockerfile” → “Day 2: Debug apt packages, fix Python install” → “Day 3: Add dev tools, configure entrypoint” → “Done: 2-3 days”
  • Right side (After): Single timeline showing “Minute 1: Add git submodule” → “Minute 5: Set build args” → “Minute 10: Done ✓”
  • Large “2-3 days → 10 minutes” callout

The time savings alone justify this. But honestly, the bigger win is the mental overhead. I no longer dread starting new projects because of Docker setup.

Consistency: Stop the Drift

Before: Project A has Python 3.11 with poetry 1.4. Project B has Python 3.12 with poetry 1.5. Project C has Python 3.13 with poetry 2.0. They all work… differently. New developer joins? Good luck figuring out which version of which tool you need for which project.

Now: All projects use the same foundation. Update the submodule, and all projects move forward together. Same Python version. Same tool versions. Same configurations. It’s so much saner.

Dev/Prod Parity: One Dockerfile, Two Environments

Before: Separate Dockerfiles for dev and prod. Try to keep them in sync. Fail. Ship to production. Discover your dev environment had a dependency that prod doesn’t. Debug in production. Not fun.

Now: Same Dockerfile for both. Just different build arguments:

# Development: Full tooling
docker build --build-arg INCLUDE_PYTHON_DEV=true ...

# Production: Minimal runtime
docker build --build-arg INCLUDE_PYTHON=true ...

If it works in dev, it works in prod. Same base, same versions, same everything.

Adding Features: From Hours to Seconds

Before: Need to add Redis to your project? Time to research the correct apt package name, figure out how to configure the client, update environment variables, test it… 2 hours later you have Redis support.

Now: --build-arg INCLUDE_REDIS_CLIENT=true. Done. Tested. Works.

This is the kind of thing that makes development feel fast again.

Security & Updates: Set It and Forget It

Before: Python security patch comes out. Now you get to manually update six different Dockerfiles in six different repos. Miss one? Hope your security team doesn’t notice.

Now: The automation handles it. Update happens automatically, gets tested, merges if everything passes. You wake up and it’s done. Or you get notified if something broke and you need to intervene.

Security best practices are built into the system. Non-root users. Minimal base images. Vulnerability scanning. It’s all there by default.

The Automation Philosophy (Or: How I Stopped Worrying and Learned to Trust CI)

Here’s the thing that really makes this system practical: I genuinely don’t have time to track version updates manually. Python 3.13.7 comes out? I’ll find out… eventually. kubectl 1.34 releases? I’m probably three versions behind already. Security patch for some npm package? I might hear about it on Hacker News if it’s bad enough.

This is a terrible way to manage infrastructure. So I automated it completely.

The Problem with Manual Updates

Every tool in your container has a version. Python, Node, kubectl, Terraform, AWS CLI, poetry, npm… the list goes on. Each one gets updated regularly. Some weekly, some monthly. Tracking all of them manually? That’s not a job, that’s a punishment.

And it’s not just tracking. You need to test each update. Does the new Python version break your linter? Does the new kubectl version have API changes? Does the new Node.js version introduce a breaking change in how it handles modules?

I needed a system that would handle the boring parts and only bug me when something actually needed my attention.

The Solution: Sunday Morning Automation

Every Sunday at 2am UTC, the system wakes up and checks every pinned tool version against the latest releases:

  • Python, Node, Rust, Go, Ruby, Java
  • kubectl, helm, Terraform, AWS CLI, Google Cloud SDK
  • Poetry, npm, cargo, and all the other package managers
  • Development tools, database clients, everything

If it finds updates, it creates a new branch with all the version bumps, updates the Dockerfile and CHANGELOG, commits everything, and pushes to GitHub.

No human intervention. No ticket in my inbox. Just automatic detection and branch creation.

The Test Suite That Makes It Possible

This is where it gets good. That new branch triggers the full CI gauntlet—and I mean full:

  • 535+ unit tests on every bash script (one for each feature installation, version check, cache configuration, error handling path, and Debian compatibility scenario)
  • Shellcheck for code quality and common bash pitfalls
  • Gitleaks scanning for accidentally committed secrets
  • Full Docker builds for all six variants (minimal, python-dev, node-dev, cloud-ops, polyglot, rust-golang)
  • Integration tests that actually use the tools—compile code, run tests, execute commands
  • Debian compatibility checks spot-testing across versions 11, 12, and 13
  • Security scanning with Trivy for known vulnerabilities

If the update breaks something, the tests catch it. If it introduces a security vulnerability, Trivy catches it. If it has Debian compatibility issues, the matrix testing catches it.

Automation flow diagram

  • Start: “Sunday 2am UTC” (clock icon)
  • Step 1: “Check for Updates” → “Found: Python 3.13.7, kubectl 1.31.2” (magnifying glass icon)
  • Step 2: “Create Branch ‘auto-update-2025-10-27’” (git branch icon)
  • Step 3: “Run Full CI Pipeline” (gear icon) with sub-bullets: “535+ unit tests”, “Build 6 variants”, “Security scan”
  • Decision Diamond: “All Tests Pass?”
    • YES path (green): “Auto-merge to main” → “Create tag v1.2.3” → “Notify: ✅ Patch Release v1.2.3 Deployed”
    • NO path (red): “Preserve branch” → “Notify: ❌ CI Failed - Python 3.13.7 breaks black formatter - Manual review required”

What Actually Happens Next

Here’s the magic part:

If everything passes: The system auto-merges to main, creates a version tag, and sends me a Pushover notification on my phone: ”✅ Patch Release v1.2.3 Deployed”. I wake up to updated containers. I did nothing. It’s beautiful.

  • Pushover notification at top of phone screen
  • App icon, title “Container System”
  • Message: ”✅ Patch Release v1.2.3 Deployed”
  • Subtext: “Updated: Python 3.13.6→3.13.7, kubectl 1.31.1→1.31.2”
  • Time: “6:47 AM”

If anything fails: I get a high-priority Pushover notification (the kind that makes noise even in Do Not Disturb mode): ”❌ CI Failed - Python 3.13.7 breaks black formatter”. The branch is preserved for manual review. Now I actually need to get involved.

This means 95% of version updates happen automatically. I only get involved when something actually breaks. That’s the level of automation I was looking for.

Important note on update control: This automation updates the containers repository itself. Projects that include containers as a git submodule maintain full control over when they adopt new versions. You can pin to a specific version for stability and test updates on your schedule (treating it like any other dependency managed via npm, cargo, etc.). Or you can automate pulling updates if you want to stay current automatically. The choice is yours—you get the benefits of automated testing and version tracking without being forced to adopt updates before you’re ready.

How It Works Under the Hood

I designed this as a git submodule that you add to any project:

git submodule add https://github.com/joshjhall/containers.git containers

One Dockerfile, Many Configurations

A single Dockerfile accepts build arguments to enable features:

# Dockerfile (simplified)
ARG INCLUDE_PYTHON_DEV=false
ARG INCLUDE_NODE_DEV=false
ARG INCLUDE_RUST_DEV=false
# ... dozens more features

RUN if [ "$INCLUDE_PYTHON_DEV" = "true" ]; then \
      /tmp/build-scripts/features/python-dev.sh; \
    fi

Modular Features

I broke everything into self-contained installation scripts:

lib/
  features/
    python.sh          # Python runtime
    python-dev.sh      # + poetry, pytest, black, mypy
    node.sh            # Node.js runtime
    node-dev.sh        # + TypeScript, ESLint, Jest
    rust.sh            # Rust toolchain
    docker.sh          # Docker CLI (for Docker-in-Docker)
    kubernetes.sh      # kubectl, helm, k9s
    aws.sh             # AWS CLI
    postgres-client.sh # psql
    # ... 28 feature modules total

Each script validates its installation, configures caching, handles Debian version differences automatically, and follows security best practices. Critically, each is independently testable.

The 28 feature modules install 100+ individual tools. For example, golang-dev.sh alone installs 34 Go development tools (gopls, dlv, golangci-lint, staticcheck, etc.), while rust-dev.sh installs 11 Rust tools, and dev-tools.sh adds 10+ productivity utilities.

Smart Caching Strategy

BuildKit cache mounts are configured for every package manager:

RUN --mount=type=cache,target=/cache/pip \
    --mount=type=cache,target=/cache/npm \
    --mount=type=cache,target=/cache/cargo \
    pip install poetry && \
    npm install -g typescript
  • X-axis: “First Build” and “Rebuild with Cache”
  • Y-axis: Time in minutes (0-10)
  • First Build: Bar reaching 8.5 minutes (red)
  • Rebuild with Cache: Bar reaching 1.2 minutes (green)
  • Large callout: “7x faster rebuilds”
  • Below: Icons showing cached items: pip, npm, cargo, go modules

Rebuilds are fast even when switching features. I’ve had builds that took 8 minutes the first time complete in under 90 seconds on subsequent runs.

Runtime Initialization

First-time setup scripts run on container start:

lib/runtime/
  first-time-setup.d/   # Run once per container
    20-aws-setup.sh     # Check AWS credentials
    20-kubernetes-setup.sh
  startup.d/            # Run every time
    10-docker-socket-fix.sh

This means users get helpful setup messages instead of cryptic errors when something’s misconfigured.

Testing: Confidence Instead of Crossing Fingers

Here’s something that still surprises people: I have 535+ unit tests for bash scripts. Yeah, really.

Most Dockerfile projects have zero tests. You write it, build it, hope it works, and find out it doesn’t work when someone else tries to use it three months later. That’s not acceptable for production infrastructure.

So I built a complete testing framework specifically for bash. It has assertions, mocking, container testing utilities, and detailed reporting:

#!/usr/bin/env bash
source "../../framework.sh"
init_test_framework

test_python_installs_correctly() {
    # Mock external commands
    mock_function "curl" "echo 'mocked download'"

    # Run the installation
    source lib/features/python.sh

    # Assert expected behavior
    assert_success "Installation should succeed"
    assert_file_exists "/usr/local/bin/python"
}

test_python_version() {
    local image="test-python:latest"
    assert_command_in_container "$image" "python --version" "Python 3."
    assert_executable_in_path "$image" "poetry"
}

run_test test_python_installs_correctly "Python installs correctly"
generate_report

Testing framework output

Running tests for features/python.sh...
✓ test_python_installs_correctly (0.3s)
✓ test_python_version_check (0.2s)
✓ test_poetry_available (0.4s)
✓ test_cache_configuration (0.2s)
✗ test_rust_compiles (1.2s)
  Expected: rustc command available
  Got: command not found

Tests: 447 passed, 3 failed, 0 skipped
Time: 2m 14s
Coverage: 94.2%

The framework provides assertion functions (assert_success, assert_equals, assert_file_exists), container testing (assert_command_in_container), a mocking system, and detailed reporting. Each test runs in isolation.

What Gets Tested

Unit Tests (535+): Every bash script tested in isolation—base system setup, all 28 feature modules, runtime scripts, and user-facing commands. Each script has tests for successful installation, version verification, error handling, cache configuration, and Debian compatibility.

Integration Tests: Full container builds for six real-world scenarios:

  • minimal: Base system only, for when you want to start from scratch
  • python-dev: Python stack with databases, for API development
  • node-dev: Node.js stack with test frameworks, for web development
  • cloud-ops: Kubernetes + Terraform + AWS + GCloud, for infrastructure work
  • polyglot: Python + Node.js together, for full-stack projects
  • rust-golang: Rust + Go, for systems programming

Each integration test builds the container, verifies tools are installed correctly, runs version checks, tests actual functionality (compile code, run tests), and verifies cache configuration.

Debian Matrix Testing: The CI pipeline spot-checks compatibility across Debian 11 (Bullseye), 12 (Bookworm), and 13 (Trixie) to catch compatibility issues before they ship. The system can be configured for more thorough cross-version testing when needed.

Security Testing: Shellcheck for static analysis, Gitleaks for secret scanning, Trivy for vulnerability scanning of the final images.

When the CI pipeline runs, if something fails, I know exactly which feature broke, what assertion failed, which Debian version it affects, and whether there are security implications. No more “it works on my machine” mysteries.

By the Numbers: What 42,000 Lines of Code Looks Like

Let me show you what’s actually in this repository, because the scope surprised even me when I looked at it recently:

Repository stats:

  • Total size: 4.8 MB
  • Total files: 178
  • Lines of code: ~42,000 (including documentation)

Code breakdown:

  • Shell scripts: 117 files, 35,776 lines total
    • 28 feature installation modules: 13,013 lines
    • Test framework + unit tests: 15,263 lines
    • Runtime/startup scripts: 1,966 lines
    • Core utilities: 1,565 lines
    • User-facing scripts: 1,337 lines
  • Documentation: 17 markdown files, 4,434 lines
  • CI/CD workflows: 1,369 lines
  • Docker configs: 412 lines

What this actually means:

The 1:1 ratio of feature code to test code (13K lines each) isn’t an accident. When I said I have comprehensive testing, I meant it. For every line of feature installation code, there’s roughly a line of test code validating it.

The feature modules are 36% of the codebase. The tests are another 36%. The remaining 28% is split between documentation, core utilities, runtime scripts, and CI/CD. This is what production-ready infrastructure looks like.

The efficiency angle:

Here’s what makes this interesting: 42,000 lines supporting 28 feature modules covering 100+ tools, with full CI/CD, comprehensive testing, and extensive documentation, all in under 5MB.

Most enterprise Dockerfile collections I’ve seen would be 5-10x this size for similar functionality. They’d have separate Dockerfiles for every combination, duplicated setup code across files, and minimal or no testing. This modular approach is genuinely more maintainable.

For context, a typical enterprise setup might have:

  • 30+ separate Dockerfiles (one per project/team)
  • Each 200-500 lines
  • 6,000-15,000 lines of duplicated Docker code
  • Maybe 500 lines of tests if you’re lucky
  • Inconsistent versions across all of them

This system replaces all of that with one Dockerfile, modular features, and more tests than most teams have for their entire Docker infrastructure.

What’s Actually Included

Over time, I’ve built 28 feature modules that install 100+ tools to cover basically everything I need across different projects. Here’s how they group by use case:

If you’re building APIs or web services:

  • Python with FastAPI/Flask (runtime + poetry, pytest, black, mypy, ruff)
  • Node.js with Express/Next.js (runtime + TypeScript, ESLint, Prettier, Jest)
  • Database clients for PostgreSQL, Redis, and SQLite
  • All the testing frameworks and linters you actually use

If you’re doing cloud operations or infrastructure:

  • Kubernetes tools: kubectl, helm, and k9s for cluster management
  • Terraform + Terragrunt for infrastructure as code
  • AWS CLI, Google Cloud SDK, and Cloudflare Workers tooling
  • Docker CLI for Docker-in-Docker workflows

If you’re doing ML or data science work:

  • Python data science stack with all the usual suspects
  • Ollama for running local LLMs (because apparently that’s a thing we do now)
  • R for statistical computing
  • Java for Spark/Hadoop work

If you’re doing systems programming:

  • Rust toolchain with cargo, clippy, and rustfmt
  • Go with all the build tools
  • C/C++ compilers and build systems

For everyone:

  • Git with GitHub CLI for version control
  • 1Password CLI for secrets management (so you stop committing API keys)
  • All the basic dev utilities you forget you need until you don’t have them

Every feature is independently toggleable through build arguments. All versions are pinned and automatically tracked for updates by the weekly automation I described earlier.

Architecture support: Works on ARM64 (Apple Silicon M1/M2/M3, AWS Graviton) and AMD64 (traditional x86_64). The same Dockerfile builds correctly on both.

Architecture: How I Organized This Thing So I Could Actually Find Stuff Later

Here’s the folder structure that evolved as this project grew:

containers/
├── Dockerfile              # Universal, feature-based
├── lib/
│   ├── base/              # Core system setup
│   │   ├── setup.sh       # Base system config
│   │   ├── user.sh        # Non-root user creation
│   │   └── apt-utils.sh   # Debian version detection
│   ├── features/          # Individual feature modules (28)
│   │   ├── python.sh, python-dev.sh
│   │   ├── node.sh, node-dev.sh
│   │   └── ... all other features
│   └── runtime/           # Container initialization
│       ├── first-time-setup.d/
│       └── startup.d/
├── tests/
│   ├── unit/              # Feature-level tests
│   └── integration/       # Full build scenarios
└── examples/              # Docker Compose templates
  • Root: “containers/” folder icon
  • Green section: “lib/base/” with shield icon - “Core system, security, user setup”
  • Blue section: “lib/features/” with puzzle piece icons - “28 modular features, 100+ tools” with mini-icons for Python, Node, Kubernetes, etc.
  • Purple section: “lib/runtime/” with play button icon - “Initialization scripts”
  • Orange section: “tests/” with checkmark icon - “535+ tests”
  • Arrows showing: “Build time uses lib/base + lib/features” and “Runtime uses lib/runtime”

Handling Debian Version Compatibility (Or: That One Time Debian Broke Everything)

Remember when Debian 13 (Trixie) removed the apt-key command in 2024? If you maintain Docker images, you probably remember. Container builds across the entire ecosystem broke overnight. HashiCorp tools? Broken. Kubernetes? Broken. Terraform? Broken. Every single image that added third-party repositories the “old way” just… stopped working.

The error looked like this:

bash: line 1: apt-key: command not found
✗ Adding HashiCorp GPG key failed with exit code 127

I saw this coming (the deprecation warnings had been around for a while), so I built automatic Debian version detection into the system. The scripts detect which Debian version they’re running on and use the appropriate method:

# lib/features/terraform.sh (simplified)
source /tmp/build-scripts/base/apt-utils.sh

if command -v apt-key >/dev/null 2>&1; then
    # Debian 11/12: Legacy method
    curl -fsSL https://apt.releases.hashicorp.com/gpg | apt-key add -
else
    # Debian 13+: Modern signed-by method
    curl -fsSL https://apt.releases.hashicorp.com/gpg | \
        gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
    echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] ..."
fi
Package/CommandDebian 11Debian 12Debian 13
apt-key✓ Available✓ Available✗ Removed
lzma-dev✓ Available✓ Available→ liblzma-dev
GPG key methodapt-key addapt-key addsigned-by=
Highlight cells that changed in red, add checkmarks in green

I also built utility functions that feature authors can use:

if is_debian_version 13; then
    # Trixie-specific logic
fi

apt_install_conditional 11 12 lzma-dev  # Only Debian 11/12
apt_install liblzma-dev                  # Works on all versions

The system handles package migrations (like lzma-dev to liblzma-dev in Debian 13) automatically. The CI pipeline spot-checks compatibility across Debian 11, 12, and 13, catching major issues before they ship—without the overhead of testing every possible combination on every run.

This saved me when Debian 13 released. While everyone else was scrambling to fix broken builds, mine just… worked. That felt good.

Cache Strategy Deep Dive

I configured persistent cache volumes for every package manager that matters:

/cache/
  ├── pip/       # Python packages
  ├── npm/       # Node packages
  ├── cargo/     # Rust crates
  ├── go/        # Go modules
  └── bundle/    # Ruby gems

Mount these as Docker volumes for fast rebuilds:

docker run -v project-cache:/cache myproject:dev

The first build downloads everything. Subsequent builds reuse the cache, even if you change which features are enabled. I’ve seen build times drop from 8+ minutes to under 2 minutes just from proper cache configuration.

Common Pitfalls I Learned the Hard Way

Let me save you some pain by sharing mistakes I made while building this:

Why I Chose Debian Over Alpine

Alpine seems attractive at first—tiny base image, minimal attack surface. And for many use cases, Alpine is excellent (I use it for database containers and simple services all the time).

But for development containers with lots of tools, I ran into friction:

  • Different package manager (apk vs apt) meant learning a new ecosystem and maintaining two versions of scripts
  • Musl libc instead of glibc caused occasional compatibility issues with pre-compiled binaries
  • Many Python packages need compilation from source on Alpine (no pre-built wheels)
  • Some tools have better support and documentation for Debian/Ubuntu

After spending time debugging Alpine-specific issues, I switched to Debian slim for development containers. The image size difference was about 50MB, but the development experience improved significantly. Your mileage may vary—Alpine is great for production workloads and simpler images, but Debian slim gave me fewer surprises when installing development tools.

The Day Debian 13 Broke Everything

I already mentioned this, but it’s worth emphasizing: always plan for breaking changes in base images. I learned to:

  • Pin Debian versions in CI testing (debian:11-slim, debian:12-slim, debian:13-slim)
  • Test new Debian versions before they become stable
  • Build version detection into installation scripts
  • Never assume commands will exist forever

The apt-key deprecation taught me this lesson hard.

Why I Don’t Use ARG for Secrets

Early on, I tried using build arguments for API keys and credentials. Don’t do this:

# DON'T DO THIS
ARG AWS_ACCESS_KEY_ID
ARG AWS_SECRET_ACCESS_KEY

Build arguments end up in the image history. Anyone with access to the image can read them with docker history. Use secrets management (1Password CLI, AWS Secrets Manager) or mount them at runtime instead.

The Submodule Update Trap

Git submodules are great but have one big gotcha: they don’t auto-update. When you git pull your main project, the submodule stays at its old commit unless you explicitly update it.

I now include a note in every project’s README:

# Update the container system
git submodule update --remote containers

Better yet, I’m working on a pre-commit hook that warns when the submodule is more than a week behind main.

Keeping Your Projects Updated

Here’s the workflow for keeping your containers current:

Updating to Latest Versions

# In your project directory
cd containers
git checkout main
git pull origin main
cd ..
git add containers
git commit -m "Update container system to v1.2.3"

Your existing containers keep running. Next time you rebuild, they’ll use the new versions. If you want to force an update immediately:

docker compose down
docker compose build --no-cache
docker compose up

What Happens During Updates

When you update the submodule:

  1. No immediate effect - Running containers keep running
  2. Next build uses new versions - Python 3.13.6 → 3.13.7, etc.
  3. Tests run during build - If something breaks, the build fails (not your running container)
  4. Cache mostly survives - Only changed features need re-downloading

Rolling Back if Needed

Git submodules make rollbacks trivial:

cd containers
git checkout v1.2.2  # Previous version
cd ..
git add containers
git commit -m "Rollback container system to v1.2.2"

Then rebuild. This is one of the big advantages of the submodule approach—every version is one git checkout away.

What’s Actually Built

Here’s what the system includes right now:

Feature Coverage:

  • 28 feature modules that install 100+ individual tools
  • 8 programming languages (Python, Node.js, Rust, Go, Ruby, R, Java, Mojo)
  • 5 cloud platform CLIs (AWS, GCloud, Cloudflare, Kubernetes, Terraform)
  • 3 database clients (PostgreSQL, Redis, SQLite)
  • Comprehensive dev tools (Git, GitHub CLI, Docker CLI, 1Password CLI, and more)

Test Coverage:

  • 535+ unit tests covering every feature installation
  • 6 integration test scenarios (minimal, python-dev, node-dev, cloud-ops, polyglot, rust-golang)
  • Debian compatibility spot-checks across versions 11, 12, and 13
  • Security scanning with Trivy for all built images
  • Shellcheck validation on all bash scripts
  • Secret scanning with Gitleaks

Automation:

  • Weekly automated version checks (Sunday 2am UTC)
  • Full CI pipeline on every update (build 6 variants, run all tests, scan for vulnerabilities)
  • Auto-merge on success, high-priority notification on failure
  • Pushover notifications keep you informed without requiring constant monitoring

Architecture Support:

  • ARM64 (Apple Silicon M1/M2/M3, AWS Graviton, Raspberry Pi)
  • AMD64 (traditional x86_64, Intel/AMD processors)
  • Same Dockerfile builds correctly on both architectures

The system is production-ready with comprehensive testing, documentation, and automation. It’s being used across multiple projects, but the numbers that matter are the ones above—those demonstrate the engineering rigor built into the foundation.

Why Not Just Use…?

You might be wondering why not just use existing solutions. Fair question.

Custom Dockerfiles: This is what I was doing. Full control is great until you have ten projects and need to update something in all of them. The maintenance burden became untenable. Every project drifts slightly differently, and keeping them in sync is a losing battle.

Dev Container Features: These are actually pretty good! Microsoft’s dev container features are well-designed and solve similar problems. But they’re VS Code specific. I wanted something that works with VS Code, Docker Compose, plain Docker, CI/CD pipelines, and production environments. Also, dev container features don’t solve the version tracking and automation problem—you still need to manually update your feature versions.

Pre-built Images (python:3.13, node:20, etc.): Fast to pull from Docker Hub, but you get what you get. Need Python + Node.js? That’s not a standard combination. Need Python + Kubernetes tools + PostgreSQL client? Good luck finding that exact image. And you definitely don’t get automatic version updates with comprehensive testing. You’re trusting someone else’s build process and update cadence.

Configuration Management (Ansible, Chef, Puppet): Different problem space. Those are for runtime configuration of running systems (mutable infrastructure). This is build-time configuration for immutable containers. Both have their place, but they’re solving different problems.

Docker Official Images + Multistage Builds: This gets closer, but you still end up maintaining multistage Dockerfiles for every project. The complexity moves but doesn’t disappear. And you still need to manually track version updates.

The unique value here is the combination: modular features + automated updates + comprehensive testing + production-ready defaults. I haven’t found another solution that does all of this together.

Documentation

I wrote comprehensive documentation because I got tired of answering the same questions (and because I’d forget details myself):

Core docs:

  • README.md - Quick start and common use cases
  • CLAUDE.md - Architecture guidance and design decisions (yes, I wrote docs specifically for Claude to understand the codebase)
  • CONTRIBUTING.md - How to add new features
  • CHANGELOG.md - Version history with breaking changes highlighted

Detailed guides in docs/:

  • Troubleshooting common issues (the “it doesn’t work” guide)
  • Writing tests for new features
  • Security best practices
  • Architecture decisions and rationale
  • Version tracking and automated releases
  • Security scanning with Trivy

Examples in examples/:

  • Docker Compose templates for common scenarios
  • Build context patterns
  • Environment configurations
  • Multi-service setups

This means new developers can onboard without waiting for me to explain things, and troubleshooting is self-service. The CLAUDE.md file has been particularly useful—it means debugging can be quickly handed off to an AI agent to track down issues. Given the rigor of the testing system, this has been quite effective. I still review the changes, of course, but an agent running something like Claude Sonnet can usually identify and fix problems while I’m mostly focused on something else.

Future Plans

I’m actively working on several improvements:

Performance optimizations:

  • Parallel feature installation (run independent installs concurrently)
  • More aggressive layer caching
  • Build time metrics tracking

Plugin system:

  • Allow custom features without modifying core
  • Company-specific tools (internal VPNs, proprietary CLIs)
  • Private registry support
  • Local feature overrides

Configuration templates:

  • Pre-built combinations for common stacks (Python FastAPI + PostgreSQL, Next.js + Redis, etc.)
  • Quick-start templates
  • Best practices baked in

Observability:

  • Build-time metrics (what takes the most time?)
  • Image size tracking over time
  • Security vulnerability trends
  • Feature usage analytics

Advanced runtime features:

  • Environment templating (generate configs from 1Password/AWS Secrets)
  • Health checks
  • Auto-update mechanisms for running containers
  • Graceful feature degradation

Enterprise features:

  • SBOM (Software Bill of Materials) generation for compliance
  • License scanning
  • Air-gapped environment support
  • Custom registry integration

The roadmap is driven by real usage. If you have feature requests, open an issue on GitHub.

Getting Started

Want to try it? The setup is straightforward.

Step 1: Add as Submodule

cd your-project
git submodule add https://github.com/joshjhall/containers.git containers

Step 2: Create Docker Compose Configuration

Create .devcontainer/docker-compose.yml (or use it anywhere you’d normally use a Dockerfile):

services:
  devcontainer:
    build:
      context: ../containers
      dockerfile: Dockerfile
      args:
        PROJECT_NAME: myproject
        INCLUDE_PYTHON_DEV: "true"
        INCLUDE_POSTGRES_CLIENT: "true"
        INCLUDE_DOCKER: "true"
    volumes:
      - ..:/workspace/myproject
      - myproject-cache:/cache

volumes:
  myproject-cache:
  • Arrow pointing to INCLUDE_PYTHON_DEV: “This enables Python + poetry + pytest + black + mypy”
  • Arrow pointing to INCLUDE_POSTGRES_CLIENT: “Adds psql command”
  • Arrow pointing to INCLUDE_DOCKER: “Docker-in-Docker support”
  • Arrow pointing to myproject-cache:/cache: “Persistent cache for fast rebuilds”
  • Green checkmark icon: “That’s it - no custom Dockerfile needed!”

Step 3: Build and Run

docker compose -f .devcontainer/docker-compose.yml up

That’s it. You now have a fully configured Python development environment with PostgreSQL client and Docker-in-Docker support.

VS Code Dev Container Integration

If you’re using VS Code, you can use Microsoft’s devcontainer base images for a cleaner integration. This avoids the Docker-in-Docker plugin complications and their questionable security implications. Here’s what that looks like:

.devcontainer/docker-compose.yml:

services:
  devcontainer:
    build:
      context: ../containers
      dockerfile: Dockerfile
      args:
        BASE_IMAGE: mcr.microsoft.com/devcontainers/base:trixie
        PROJECT_NAME: myproject
        USERNAME: vscode
        WORKING_DIR: /workspace/myproject
        INCLUDE_PYTHON_DEV: 'true'
        INCLUDE_POSTGRES_CLIENT: 'true'
        INCLUDE_DEV_TOOLS: 'true'
      cache_from:
        - type=local,src=/tmp/.buildx-cache
      cache_to:
        - type=local,dest=/tmp/.buildx-cache,mode=max
    volumes:
      - ..:/workspace/myproject
    environment:
      - TZ=${TZ:-America/Chicago}
      - ENVIRONMENT=${ENVIRONMENT:-development}
    command: sleep infinity
    networks:
      - containers-network

networks:
  containers-network:
    driver: bridge

.devcontainer/devcontainer.json:

{
  "name": "My Project",
  "dockerComposeFile": "docker-compose.yml",
  "service": "devcontainer",
  "workspaceFolder": "/workspace/${localWorkspaceFolderBasename}",

  "postCreateCommand": "poetry install",

  "customizations": {
    "vscode": {
      "extensions": [
        "ms-python.python",
        "ms-python.vscode-pylance",
        "charliermarsh.ruff"
      ],
      "settings": {
        "terminal.integrated.defaultProfile.linux": "zsh",
        "python.defaultInterpreterPath": "/usr/local/bin/python"
      }
    }
  },

  "remoteUser": "vscode"
}

Notice the key differences:

  • BASE_IMAGE arg lets you use Microsoft’s devcontainer base images
  • USERNAME: vscode integrates with VS Code’s expectations
  • Simple devcontainer.json without Docker-specific plugin configurations
  • The same features work whether you use debian:13-slim for production or mcr.microsoft.com/devcontainers/base:trixie for development

This flexibility means you can optimize for each environment: Microsoft’s devcontainer images for local VS Code development, standard Debian LTS images (11, 12, or 13) for production and QA. Same features, same tools, just different base images.

Adding More Features

Want to add Node.js? Just update the build args:

args:
  INCLUDE_PYTHON_DEV: "true"
  INCLUDE_NODE_DEV: "true"      # Add this line
  INCLUDE_POSTGRES_CLIENT: "true"
  INCLUDE_DOCKER: "true"

Rebuild:

docker compose down
docker compose build
docker compose up

Common Use Cases

Python API development:

args:
  INCLUDE_PYTHON_DEV: "true"
  INCLUDE_POSTGRES_CLIENT: "true"
  INCLUDE_REDIS_CLIENT: "true"

Node.js web development:

args:
  INCLUDE_NODE_DEV: "true"
  INCLUDE_POSTGRES_CLIENT: "true"

Cloud operations:

args:
  INCLUDE_KUBERNETES: "true"
  INCLUDE_TERRAFORM: "true"
  INCLUDE_AWS: "true"

Full-stack development:

args:
  INCLUDE_PYTHON_DEV: "true"
  INCLUDE_NODE_DEV: "true"
  INCLUDE_POSTGRES_CLIENT: "true"
  INCLUDE_REDIS_CLIENT: "true"
  INCLUDE_DOCKER: "true"

All available build arguments are documented in the repository README.

Security

Security best practices are built into the foundation:

  • Non-root user by default - All processes run as a non-root user unless explicitly needed
  • Minimal Debian slim base images - Smallest viable attack surface
  • Automated security updates - Weekly automation checks for and applies security patches
  • Secret scanning with Gitleaks - Prevents accidentally committed credentials
  • Vulnerability scanning with Trivy - Catches known CVEs in dependencies
  • Validated installations - Each feature verifies correct installation
  • Proper file permissions - No world-writable files or directories

The automated update system is designed with security in mind. Security patches are prioritized and tested immediately. If a critical vulnerability is detected, the system can create emergency updates outside the normal Sunday schedule.

Contributing

Want to add a feature? The process is straightforward:

1. Create the Feature Script

Create lib/features/your-feature.sh following this template:

#!/usr/bin/env bash
set -euo pipefail

# Detect latest version
TOOL_VERSION="1.2.3"

# Install
apt-get update
apt-get install -y your-tool

# Validate
if ! command -v your-tool >/dev/null 2>&1; then
    echo "Installation failed"
    exit 1
fi

echo "Successfully installed your-tool $TOOL_VERSION"

2. Add Unit Tests

Create tests/unit/features/your-feature.sh:

#!/usr/bin/env bash
source "../../framework.sh"
init_test_framework

test_your_feature_installs() {
    source lib/features/your-feature.sh
    assert_success "Installation should succeed"
}

test_your_feature_version() {
    local image="test-image:latest"
    assert_command_in_container "$image" "your-tool --version"
}

run_tests
generate_report

3. Update Documentation

Add your feature to:

  • README.md - Available features list
  • CHANGELOG.md - Under “Unreleased”
  • docs/FEATURES.md - Detailed feature documentation

4. Submit Pull Request

The CI pipeline will automatically test your changes on all Debian versions and run the full test suite.

See CONTRIBUTING.md for detailed guidelines, coding standards, and best practices.

Wrapping Up

I built this out of frustration. I was tired of writing the same Dockerfile repeatedly. Tired of tracking version updates manually. Tired of fixing the same Docker issues in six different repos. Tired of spending days on infrastructure instead of building actual products.

This system solves all of those problems for me:

  • New projects set up in 10 minutes instead of 2 days
  • All my projects stay in sync automatically
  • Version updates happen while I sleep (and only wake me if something breaks)
  • Dev and production environments use identical foundations
  • I have actual confidence that builds work because of comprehensive testing

It started as a side project to scratch my own itch. But it’s become the foundation for everything I build with containers now. I haven’t written a custom Dockerfile in over a year. I don’t miss it.

The best part? The automation. That weekly CI run that updates everything, tests it, and either deploys or notifies me? That’s the difference between managing infrastructure and just using it. I wake up to patched containers. I spend time building products instead of maintaining Docker configs.

If you’re dealing with the same pain points—multiple projects, manual version tracking, inconsistent environments, time-consuming setup—this might help you too.

Try It On Your Next Project

Don’t overcommit. Just try it on one project:

  1. Add the git submodule
  2. Enable the features you need
  3. Build the container
  4. Spend your time building instead of configuring Docker

If you run into issues or have questions, open an issue on GitHub. I’m actively maintaining this and usually respond within a day.

If it saves you even half the time it’s saved me, that’s hours back in your week. Hours you can spend on things that actually matter.

Repository: https://github.com/joshjhall/containers

Documentation: See the docs/ directory for detailed guides

Examples: See the examples/ directory for Docker Compose templates

Issues/Questions: Open an issue on GitHub - I’m responsive and want this to work well for others too