Devstack

Originally Published:

Tags: [ #DX, #DevOps, #Kubernetes, #Helm ]

The Problem

In the move to microservices and Kubernetes, we strive to create small, well-defined services with minimal coupling with other services. But there will always be a need to test in an integrated system. You need to prove to yourself that you did in fact build the right system that integrates with other services. At the same time, you cannot build a rigorous system when you only work in a fully integrated way.

A traditional software delivery cycle might be:

  1. Write code on your local (LOCAL) system.
  2. Push the code to a remote server to build.
  3. Publish the changes to a development integrated testing (DIT) environment.
  4. Verify the changes in DIT.
  5. Publish the changes to a systems integrated testing (SIT) environment.
  6. Verify the changes in SIT.
  7. Publish the changes to production.
  8. Create new requirements for development.
  9. Repeat.

This cycle could be presented as the following pipeline of environments:

LOCAL > DIT > SIT > PROD

This traditional approach can make the DIT environment a bit contentious among developers in a team. Developers can inadvertently make changes in DIT that unintentionally impact other developers. On the other hand, if developers do not regularly verify their services integrate with other services, then they're more likely to build the wrong solution. Also, by the nature of some services such as a cache (Redis), a database (PostgreSQL), and a queue (RabbitMQ), a developer may need to spin up a local network of services anyways.

LOCAL as DIT

One solution I have experienced to this problem is to mirror the entire DIT environment on a developer's LOCAL system. The developer is responsible to:

  1. keep their source repositories up to date
  2. install all of the toolchains necessary to build and run all of the services
  3. mount the code through to their cluster of services.

This works fine for a small cluster of services, but this does not scale well. A local developer's system likely cannot handle 100s of services. The compute requirements in memory, storage, and processing will be easily overburdened. This solution also discourages experimenting with new tech stacks and tool chains, as all developers must be aligned at all times.

Enter Devstack

The devstack is a cluster of highly related services running in Kubernetes on a developer's system. Typically, the services are placed in the same namespace and share a single helm chart as the unit of deployment. The most important piece to defining a devstack is custom routing that can be configured differently for deployment downstream (DIT and beyond). Once you have built custom routing, you can add in other capabilities to improve the development experience.

A simple devstack could include a dev-only Kubernetes gateway that configures routing to your single namespace of services.

A more advanced devstack could be a Helm chart with the following requirements:

  1. Add a Kubernetes gateway with TLS certs for a adhoc domain (e.g. https://local.mysupercoolnewapp.com/) and a route to your development web server.
  2. Update your hosts file to map that domain to your local system.
  3. Add features and capabilities that are configurable with Helm values in a Helm chart. Some options may include:
    1. Provide a fake auth service that exposes a profile cookie in the http messages in the cluster.
    2. Provide a dev home page to document the local devstack.
    3. Provide a local CDN with static assets available to your cluster.
    4. Provide cross-cutting services, such as logging (via Grafana & Loki).
    5. Provide breakglass services such as PGAdmin4 to manage the PostgreSQL database.

The Helm chart could then be used as a subchart to your chart under development.

With the devstack, we can now setup the following pipeline:

LOCAL > DEVSTACK > DIT > SIT > PROD

Awesome! Now, we have a way to develop a small stack of services on our local systems. Obviously, there is still a need to test the interfaces between namespaces, but at least the boundary of more highly coupled services can be tested in isolation.

Don't Forget: LOCAL

The importance of working locally with a single service cannot be overstated. The different levels of integration downstream are great for prototyping and verification, but a lot can go wrong when you have many services in the mix. At the most basic level, developers should have the ability to work on a service in complete isolation from all other services. This isolation is crucial for rapid prototyping and iteration, allowing developers to focus solely on the internal logic and functionality of their service without external dependencies.

The rule I try to follow is:

90 > 9 > 1

Where:

  1. 90% of time is spent on LOCAL development,
  2. 9% in devstack development,
  3. and 1% downstream.

Update Log

Update 2026-02-13

My original post about the devstack left off the importance of working locally with a single service. In my drive for rigor over the last several years, I have since placed greater emphasis on local development. The devstack is still critical to my overall flow, but it's importance has been lowered.