Dockerizing Our Heroku Apps

What we’re built on

The engineering team behind Ambassify isn’t a large one, so using Heroku was an obvious choice. Heroku allowed us to focus our energy on building the applications rather than building the infrastructure that runs the applications. Today, Ambassify is made out of nearly 25 microservices, each created using javascript and node.js to run the code. 

Heroku provides a consistent environment where we can run our applications, so we’ve automated getting our code to run on it. Each and every one of our microservices is exposed to a large set of tests and scans. If these succeed, Heroku picks up on this, builds it into an image, and deploys it into our staging environment.

Keeping our options open

At the end of 2020, we started exploring what options we would have if we decided to move away from the Heroku platform. The reason is that we were having some issues implementing some of our more complex features.

Most of the options we encountered on our journey involved using docker containers. For us to test our applications on some of the other platforms, we started looking for a way to package our services into docker images without having to modify them.

This led us to a solution using herokuish and two scripts that could be used across all of our apps to get our code packaged up into a docker image.

Building the image

Some of us on the team had experience with dokku, the self-hosted and open-source PaaS solution; it uses the gliderlabs/herokuish docker image to build and run your applications as if they were running on Heroku.

Gliderlabs/herokuish

The herokuish docker image provides an environment as close as possible to the server environments on Heroku. This is how the tool is described on GitHub:

The goal is to be the definitive, well maintained and heavily tested Heroku emulation utility shared by all. It is based on the Heroku:18, Heroku:20, and Heroku:22 system images. Together they form a toolkit for achieving Heroku compatibility.

Building and running your application within the herokuish image is really simple but requires some massaging to get working. Most importantly, you’ll need to know that your app’s source code must be available during the build at /tmp/app, but may not be there when calling the /start command.

docker run -ti -v /path/to/app:/tmp/source:ro gliderlabs/herokuish /bin/bash
cp -a /tmp/source /tmp/app
/build
rm -Rf /tmp/app
/start web

Using these steps you could bundle your app into a docker image using this simple Dockerfile and docker build.

FROM gliderlabs/herokuish
COPY . /tmp/app
RUN /build && rm -Rf /tmp/app
ENTRYPOINT [ "/start" ]
CMD [ "web" ]

Optimizing image

The above Dockerfile will however result in an image that is larger than required. Below are two changes we can make to our Dockerfile that will reduce the size to essentials.

Multi-stage build

Docker has support for multi-stage builds, this allows us to build all our assets without them being included in the second stage (our final image) of the build. A new stage is denoted by a new FROM command in the Dockerfile.

FROM gliderlabs/herokuish AS build
COPY . /tmp/app
RUN /build && rm -Rf /tmp/*
RUN herokuish slug generate && \
    herokuish slug export > /app.tar.gz

FROM gliderlabs/herokuish
COPY --from=build /app.tar.gz
RUN herokuish slug import < /app.tar.gz
ENTRYPOINT [ "start" ]
CMD [ "web" ]

You might notice that the first section of this file looks familiar, except for the fact that we use the herokuish command to generate and export a slug containing only the built version of our app. This excludes the buildpacks that were downloaded or temporary resources that were used during the build.

We copy this slug over to our second stage using the COPY command. Finally, we import the slug back into our final image to generate an image that can run our app.

Unpack slug

In our previous Dockerfile, we still copied over our app slug while we never use it again after the import. Would there be a way to remove this slug from our final result? This slug can quickly become a few hundred megabytes, after all.

As it turns out, not only is it possible to copy over artifacts from a previous stage. When using docker with BuildKit you can actually mount resources from a previous stage while building your next stage.

FROM gliderlabs/herokuish AS build
COPY . /tmp/app
RUN /build && rm -Rf /tmp/*
RUN herokuish slug generate && \
    herokuish slug export > /app.tar.gz


FROM gliderlabs/herokuish
RUN --mount=type=bind,from=build,source=/app.tar.gz,target=/app.tar.gz \
    herokuish slug import < /app.tar.gz
ENTRYPOINT [ "start" ]
CMD [ "web" ]

Here you can see that we supply the mount argument to the RUN command that will be extracting our app into the final image. We need to supply 4 parameters to the mount argument. type=bind specifies we want to reference a directory from a previous stage. The ‘from’ parameter specifies the stage from which we want to mount images; in our case that is ‘build.’ Finally, ‘source’ and ‘target’ specify the source file or directory and ‘target’ where to mount the file in the new stage.

Compatibility

To improve compatibility, we’ve included a line in our initial build stage to load custom buildpacks from ‘app.json’. Without this line, only the automatically detected buildpacks will work.

RUN (cat /tmp/app/app.json || echo '{}') | jq -r '.buildpacks[].url' > /tmp/app/.buildpacks && \
    (test -s /tmp/app/.buildpacks || rm /tmp/app/.buildpacks)

This will read the buildpacks from your ‘app.json’ file using the jq utility and write them into .buildpacks, which is the file that herokuish will read to determine which buildpacks to use.

Improving developer experience

Now we just need a way to name our images consistently, test them and eventually also deploy them. For simplicity, we’ve chosen to use a Makefile in the project’s root directory. make is easily accessible on most platforms and is easy to understand. Our Makefile has a few parameters that can be changed during setup. After that, building, testing, and deploying can be controlled with a few make commands.

APP_NAME?=some-app
APP_NAMESPACE?=apps
IMAGE_NAME?="ambassify/$(APP_NAME)"

export DOCKER_BUILDKIT = 1

up: build
    docker run -t -i \
        -e PORT=3000 \
        -p 3000:3000/tcp \
        $(IMAGE_NAME)

pull:
    docker pull gliderlabs/herokuish

build: pull
    docker build --build-arg NPM_TOKEN -t $(IMAGE_NAME) .

debug: build
    docker run -t -i --entrypoint /bin/bash $(IMAGE_NAME)

publish: build
    docker tag $(IMAGE_NAME) ghcr.io/$(IMAGE_NAME)
    docker push ghcr.io/$(IMAGE_NAME)

deploy: publish
    kubectl rollout restart deployment $(APP_NAME) -n $(APP_NAMESPACE)

This is a generic Makefile that only requires us to change the three parameters at the top of the file. APP_NAME controls the name of the service we are building. APP_NAMESPACE is only required in combination with Kubernetes and references the namespace into which the app is deployed. IMAGE_NAME is the name used to publish the image.

The export DOCKER_BUILDKIT=1 command ensures that any docker commands are run using BuildKit, which is required to use the mount argument to RUN discussed earlier.

Future improvements

Herokuish is a great tool to help us test and transition without having to sacrifice our ongoing development. It allows us to continue using heroku and use a familiar software stack until we are ready to make the switch. 

Herokuish vs. specialized image

Using herokuish has one disadvantage, the images produced are quite large in comparison to some of the regular node images or even larger than the node-alpine docker base images. Even the smallest applications generate an image with a total size of over 1GB, causing slow startup or restarts on occasion.

Once we are ready to move away from Heroku we will most likely modify our workflow to use the regular node images for specific programming languages, such as the node image.

Conclusion

Herokuish is a great tool to deploy a Heroku code base onto the many different container platforms that use Docker, such as Kubernetes. It, however, comes with the downside of producing images that are larger than they need to be to run your applications.

If you are not intending to use Heroku for your code base, you are most likely better off going with a specialized image for your programming language. The full scripts used in this blog post can be found on our GitHub gist.