Skip to content

Building and Testing Containers

Choose a container base from DockerHub

Projects should build containers by starting with a well-known application base container on a public registry. Projects should add a minimum of additional project software and packages so that the container is clearly built for a specific purpose. Containers built for one specific batch job, either a data transformation or analysis, are examples of this approach. Container builds that assemble groups of tools and then used to run a variety of tasks should be avoided. Additionally, container builds that start from generic distributions such as Debian or Ubuntu should also be avoided as leaner and more focussed application and language containers are already available.

Examples of batch job container bases are Python and PyTorch, and other language specific and ML software stacks.

Add SHS file system directories

Container images built to run in the SHS GPU Cluster must implement the following line in the Dockerfile to interface with the project data and the SHS file system:

RUN mkdir /safe_data /safe_outputs /scratch

The project’s private /safe_data/<project id> directory is mapped to the container’s /safe_data directory using BeeGFS. A unique container job output directory is created for the user and mapped to /safe_outputs in the container. And /scratch is a temporary workspace that is deleted when the container exits. If the container processing uses the TMP environment variable, it should be set to the /scratch volume mount. Hence, containers have access to three directories located on the host system as detailed in the following table:

Directory on BeeGFS Directory in container Intended use
/mnt/beegfs/<safe_heaven>/<project_id>/shared /safe_data Read-only access if required by IG, or read-write access, to data and other project files.
/mnt/beegfs/<safe_heaven>/<project_id>/users/<username> /safe_outputs Will be created as an empty directory. Intended for any outputs: logs, data, models.
~/scratch /scratch Temporary directory that is removed after container termination on the host system. Any temporary files should be placed here.

Currently, temporary files can also be written into any directory in the container’s internal file system. This is allowed to prevent container software failure when it is dependent on the container file system being writable. However, the use of /scratch is encouraged for two reasons:

  1. In the future, write access to the container file system might be prevented for security reasons. Further, the space available on the container’s internal file system is limited compared to the space available on /scratch. Writing only to /scratch at runtime is therefore future-proof.
  2. Use of /scratch can be more efficient if the service is able to mount it on high-performing storage devices.

Install and copy project code into container

Research software stacks are complex and dependent on many package sets and libraries, and sometimes on specific-version combinations of these. The container build process presents the project team with the opportunity to manage and resolve these dependencies before contact with the project data in the restricted SHS setting.

Unlike the SHS desktop servers, containers do not have access to external network repositories, and do not have access to external licence servers. Any required licence must be copied into the container at build-time and imported with the container. EPCC are not responsible for verifying that the appropriate licences are installed or that licence terms are being met.

Projects using configuration files for multiple containers, for example ML models, can also import these to the SHS directly by copying them to the project /safe_data file system.

Batch jobs built to run as containers should start directly from the ENTRYPOINT or CMD section of the Dockerfile. Batch jobs should run without any user interaction after start, should read input from the /safe_data directory and write outputs to the /scratch and /safe_outputs directories.

Test the container for SHS use

When the container is running in the SHS GPU Cluster it will not have any external network or internet access. If the code, or any of its dependencies, rely on data files downloaded at runtime (for example machine learning models) this will fail in the SHS. Code must be modified to load these files explicitly from a file path which is accessible from inside the container.

An example of SHS network isolation significance and the considerations arising from this is provided by Hugging Face, where models are cached in the user local ~/.cache/huggingface/hub/ directory in the container. The environment variable HF_HOME must be set to a directory in a /safe_data project folder and the cache_dir option of the from_pretrained() call used at runtime.

If a model is downloaded from Hugging Face, the advice is to set the environment variable HF_HUB_OFFLINE=1 to prevent attempts to contact the Hugging Face Hub. Any connection attempts in the SHS will take a significant time to elapse and then fail.

It is recommended that the checklist for Dockerfile composition be followed: Container Build Guide

Before using a specific container in the TRE, it may be necessary to test the security risk and gain Information Governance approval. Information Governance may require a security scan of both following artefacts:

  1. The Dockerfile used to build the container image

  2. The container image itself, once it is built.

Trivy is a tool that can help with these tasks. Trivy inspects container images to find items which have known vulnerabilities and produces a report that may be used to help assess the risk.

1. Scanning the container Dockerfile

The Trivy misconfiguration tool on Dockerfiles can scan the container Dockerfile. This tool option will highlight many common security issues in the Dockerfile. Example execution is as follows:

docker run --rm -v $(pwd):/repo ghcr.io/aquasecurity/trivy:latest config "/repo/Dockerfile"

The security posture of containers and the build process may be of interest to Information Governance teams, however it is not expected that security issues indicated by the tool need to be addressed before the container is run in the SHS unless the Information Governance team issues specific guidance on vulnerability and configuration remediation and mitigation.

2. Scanning the container image using Trivy CI

Trivy can be run manually on the built image but it is easier to have it run automatically whenever you update your container image. An example GitHub Actions workflow to run Trivy and publish the outputs can be found in our documentation.

The Trivy report can be downloaded as an artefact from the job summary page.