Contributing

As explained in Tools and images and in the User Guide, multiple collections of images are provided. For each collection, a set of base images is provided, which are to be used for building and for runtime. These are defined in base.dockerfile. See, for instance, debian-bookworm/base.dockerfile. All the images in the ecosystem are based on these:

  • 'build/base container image size' Debian Bookworm, Debian Bullseye or Rocky Linux 8, with updated ca-certificates, curl and Python 3.

  • 'build/build container image size' based on base, includes clang and make.

  • 'build/dev container image size' based on build, includes cmake, libboost-all-dev and python3-dev.

Then, for each project/tool there are dockerfiles (one for each collection), a GitHub Actions workflow, and one or more test scripts. Those are used for:

  • Tools are built using REGISTRY/[ARCHITECTURE/][COLLECTION/]build images.

  • Package images based on scratch (and/or other reusable packages) are produced.

  • Ready-to-use images based on the runtime base image (REGISTRY/[ARCHITECTURE/][COLLECTION/]build/base) are produced.

  • Ready-to-use images are tested before uploading.

The Package images created in some dockerfiles/workflows are based on scratch and contain pre-built assets. Therefore, they are not really useful per se, but meant to be used for building other. In fact, multiple tools are merged into ready-to-use images for common use cases (such as impl, formal or prog).

Important

Before working on adding or extending the support for a tool, please check the issues and pull requests; open an issue or let us know through the chat. Someone might be working on that already!

Note

Currently, many projects don’t use containers at all, hence, all images are generated in this repository. However, the workload is expected to be distributed between multiple projects in the ecosystem.

Dockerfiles

Two kinds of Dockerfile definitions are supported by the utils used in this repository: single-file solutions, or aided by additional assets. Nonetheless, all dockerfiles use, at least, two or three stages:

  • A global argument named REGISTRY defines the default registry path and collection to be used.

  • One stage, named build, based on $REGISTRY/build/base or $REGISTRY/build/build or $REGISTRY/build/dev, is used to (optionally) install build dependencies, and to actually build the tool.

    • The tool/project is built using the standard PREFIX, but installed to a custom location using DESTDIR. See Package images.

  • If the tool/project is to be packaged, one stage based on image scratch is named pkg and it contains the artifacts of the build stage only.

    • Tool artifacts are copied from the build stage using COPY --from=STAGE_NAME.

  • If the tool/project is to be used standalone, a final stage based on image $REGISTRY/build/base is used to (optionally) install runtime dependencies, to copy the build artifacts from the build stage, and to (optionally) set the default command for the image.

Note

In some dockerfiles, global argument IMAGE is used to compose stages on top of multiple intermediate images.

Single-file example

Some tools are defined in a dockerfile only, without any additional assets. In those cases, the dockerfile is named after the name of the main tool/image built there.

Listing 1 Reference Dockerfile to build a TOOL and generate a regular image and a package image.
ARG REGISTRY='gcr.io/hdl-containers/debian/bookworm'

#--

FROM $REGISTRY/build/build AS build

RUN apt-get update -qq \
 && DEBIAN_FRONTEND=noninteractive apt-get -y install --no-install-recommends \
    ... \
 && apt-get autoclean && apt-get clean && apt-get -y autoremove \
 && rm -rf /var/lib/apt/lists/*

RUN git clone REPOSITORY_URL /tmp/TOOL \
 && cd /tmp/TOOL \
 && ./configure \
 && make -j$(nproc) \
 && make DESTDIR=/opt/TOOL install

#---

FROM scratch AS pkg
COPY --from=build /opt/TOOL /TOOL

#---

FROM $REGISTRY/build/base

RUN apt-get update -qq \
 && DEBIAN_FRONTEND=noninteractive apt-get -y install --no-install-recommends \
    ... \
 && apt-get autoclean && apt-get clean && apt-get -y autoremove \
 && rm -rf /var/lib/apt/lists/*

COPY --from=build /opt/TOOL /
CMD ["TOOL"]

Note

Typically, stage pkg is used as a target to build a package image, and the dockerfile is used without a target in order to build the regular image for the tool. However, some tools/groups require additional stages, and some other don’t have the package or the regular stage.

Example with HDLC script

On the other hand, the dockerfiles to build tools with additional assets are named Dockerfile and located in a subdir under the collection directory, named after the name of the main tool/image built there. Those are typically used along with a shell script named HDLC. The structure of these dockerfiles is similar to the Single-file example, however, BuilKit’s --mount feature is used to source a helper script (HDLC) without creating additional stages/steps (see BuildKit). A similar strategy can be used to run or copy additional assets into the images.

Listing 2 Reference HDLC script to decouple dependency lists and build steps from the Dockerfile.
makedepends=(
  ...
)

build() {
  mkdir /tmp/TOOL
  cd /tmp/TOOL
  git clone REPOSITORY_URL ./
  ./configure
  make -j$(nproc)
  make DESTDIR=/opt/TOOL install
}

depends=(
  ...
)

Note

The default shell in the collections used in this repository is set to bash. Therefore, arrays and other bash extensions are supported in these scripts.

Listing 3 Reference Dockerfile to build a TOOL and generate a regular image and a package image, using an HDLC script and BuildKit mount features.
ARG REGISTRY='gcr.io/hdl-containers/debian/bookworm'

#---

FROM $REGISTRY/build/build AS build

RUN --mount=type=bind,target=/tmp/ctx . /tmp/ctx/HDLC \
 && mkdir -p /usr/share/man/man1/ \
 && apt-get update -qq \
 && DEBIAN_FRONTEND=noninteractive apt-get -y install --no-install-recommends ${makedepends[@]} \
 && apt-get autoclean && apt-get clean && apt-get -y autoremove \
 && update-ca-certificates \
 && rm -rf /var/lib/apt/lists/*

RUN --mount=type=bind,target=/tmp/ctx . /tmp/ctx/HDLC && build

#---

FROM scratch AS pkg
COPY --from=build /opt/TOOL /TOOL

#---

FROM $REGISTRY/build/base

RUN --mount=type=bind,target=/tmp/ctx . /tmp/ctx/HDLC \
 && apt-get update -qq \
 && DEBIAN_FRONTEND=noninteractive apt-get -y install --no-install-recommends ${depends[@]} \
 && apt-get autoclean && apt-get clean && apt-get -y autoremove \
 && rm -rf /var/lib/apt/lists/*

COPY --from=build /opt/TOOL /
CMD ["TOOL"]

BuildKit

The usage of dockerfiles in this repository relies on the image build engine making an analysis and pruning of the stages. Furthermore, option --mount used in some of the dockerfiles requires docs.docker.com: BuildKit. Therefore, enabling BuildKit is required in order to build the images.

Depending on the version of Docker on your host or CI service, BuildKit features might need to be enabled explicitly. As explained in docs.docker.com: To enable BuildKit builds, either set the DOCKER_BUILDKIT=1 environment variable, or set the daemon feature to true in the JSON configuration file: ({ "features": { "buildkit": true } }).

Find further details about BuildKit’s mount syntax in moby/buildkit: frontend/dockerfile/docs/syntax.md.

Note

In order to use those features, apart from using BuildKit # syntax=docker/dockerfile:1.3 might need to be added as the first line of the Dockerfile. That depends on the version of Docker. Recent versions should not require it.

Step by step checklist

  1. Create or update dockerfile(s).

  • For each tool and collection, a Dockerfile recipe exists. Find details about the requirements when writing dockerfiles in Dockerfiles.

    • Optionally, add tools to multiple collections at the same time. That is, create one dockerfile for each collection.

  • Some tools are to be added to existing images which include several tools (coloured BROWN in the Graphs). After creating the dockerfile where the corresponding package image is defined, add COPY --from=$REGISTRY/pkg/TOOL_NAME statements to the dockerfiles of multi-tool images.

  1. Build and test the dockerfile(s) locally. Use helper scripts from subdir utils/, as explained in Utils.

  • If a new tool was added, or a new image is to be generated, a test script needs to be added to test/. See Smoke-tests for naming guidelines.

  • Be careful with the order. If you add a new tool and include it in one of the multi-tool images, the package image needs to be built first.

  1. Create or update workflow(s).

  • For each tool or multi-tool image, a GitHub Actions workflow is added to .github/workflows. In each workflow, multiple images produced from stages of the corresponding dockerfile are built, tested and pushed. Scripts from utils/ are used. Find details at Structure.

    • Copying some of the existing workflows in this repo and adapting it is suggested.

    • If necessary, update the config.yml to override the defaults or to define new job/task lists.

  1. Update the documentation.