Article: Preview the future of the ue4-docker project

Tags: Docker, Unreal Engine

Posted: 12 March 2021

The ue4-docker project is the pioneering source of Dockerfiles for both Windows and Linux container images that encapsulate the Unreal Engine. Since its creation in 2018, the project has grown to encompass a wide variety of patches and tweaks to improve the Unreal Engine build experience inside containers, in addition to a suite of testing and diagnostic tools to identify issues in host system build environments. As a result of this organic growth, the codebase has begun pushing up against the limitations of its fundamental design, and a complete redesign is planned for the future.

Although a lack of feature parity between the available tooling for Windows containers and their more mature Linux counterparts is currently blocking implementation of the new design, one of the the most important features of the planned redesign has recently been added to the current ue4-docker codebase in a preliminary form. Although the new functionality is rather limited when used with Windows containers, it provides several extremely powerful capabilities for Linux containers that developers can benefit from today.

Contents

• Limitations of the current design
• Benefits of the planned redesign
• Using the new features today
  • Example: building a Linux container image with BuildKit

Limitations of the current design

In its current form, ue4-docker is a monolithic tool that stores Dockerfiles inside its internal codebase and is responsible for all aspects of building container images from those Dockerfiles. This design was chosen for security purposes, since the manner in which Docker builds container images makes it inherently difficult to leverage sensitive data (such as authentication credentials) without risking its exposure. As discussed in the article Build secrets and SSH forwarding in Docker 18.09 by Tõnis Tiigi, sensitive data stored in files or environment variables during the build process of a container image will become embedded in the metadata of that image, and even if a multi-stage build is used to isolate the relevant build steps the sensitive data will still remain in the local build cache of the Docker daemon on the host filesystem.

The official solution to this build-time security dilemma is to use the special BuildKit features discussed in the article linked above, which allow secrets to be securely mounted during the build process without leaking into the resulting container image. However, the BuildKit build backend was designed for Linux containers and does not currently support Windows containers, which precludes the use of its features by ue4-docker when protecting user-supplied credentials during the build process. In order to keep credentials secure when building both Windows and Linux containers, ue4-docker implements a HTTP endpoint (called the “credential endpoint”) that is queried for credentials on-demand during the build process, as illustrated in the diagram below:

The current architecture used by ue4-docker to prevent user credentials from leaking into built container images.

To prevent other local processes from accessing the credential endpoint and stealing credentials, a security token is randomly generated and injected into the container. This token is inevitably leaked in the metadata of the built images, but the one-time nature of its use renders this merely an annoyance rather than a security vulnerability. This convoluted process is currently the simplest mechanism that I am aware of which facilitates the secure use of sensitive data across both Windows and Linux containers. It is the single most brittle component of the build process and a major source of issues for developers working with ue4-docker (second only to the myriad problems that have plagued Windows containers over the past few years, particularly those that prevent the creation of large filesystem layers.)

The inclusion of the credential endpoint tightly couples the build process to ue4-docker itself, preventing the use of its Dockerfiles in environments where ue4-docker is not present or cannot run. This monolithic nature imposes a number of frustrating limitations:

• Container images which encapsulate the Unreal Engine cannot be built in cloud CI/CD environments that lack the ability to run ue4-docker (e.g. due to an inability to access a Docker daemon or to configure firewall rules that permit access to the credential endpoint) or inside Kubernetes clusters that are running containerised build systems such as BuildKit or kaniko.

• The Dockerfiles within the ue4-docker source tree contain a large number of patches that only apply to specific use cases or versions of the Unreal Engine, and developers cannot easily determine which subset of any given Dockerfile is relevant to them without also reading the corresponding ue4-docker source code and/or documentation. Extraneous patches can be disabled at build-time but they cannot be removed to improve readability without impacting other users, and this problem continues to be compounded as ue4-docker grows in scope.

• Developers cannot modify the Dockerfiles without modifying the source code of ue4-docker itself, and any custom changes will be overwritten when a newer version of the ue4-docker Python package is installed.

• Dockerfiles cannot be stored in a version control system to act as a source of truth for the container images that they produce without also storing the source code of ue4-docker itself.

• The random security token changes each time a build is performed, invalidating the build cache for filesystem layers that include it and thus forcing a re-build of those layers in circumstances where their build cache could otherwise be reused to speed up builds of other images (e.g. when building separate image variants for different use cases that use the same version of the Unreal Engine.) This is particularly problematic given that the filesystem layers in question are those which contain the Unreal Engine source code and its bundled third-party dependencies, and rebuilding them involves downloading many gigabytes of data from remote servers.

These limitations have motivated the planned redesign of ue4-docker, which is discussed in the section below.

Benefits of the planned redesign

The planned redesign reframes ue4-docker as primarily taking the role of a plugin-based Dockerfile generation tool rather than a build tool. In this new design, ue4-docker provides developers with functionality to generate and customise Dockerfiles, which are then exported to the filesystem so that users can perform any desired modifications and commit the results to a version control system. These Dockerfiles can then be built independently of ue4-docker itself, in any desired build environment. This provides a wide array of benefits:

• Container images which encapsulate the Unreal Engine can be built in any build environment, including cloud CI/CD environments and inside Kubernetes clusters that are running containerised build systems.

• Generated Dockerfiles can be stripped of extraneous code or patches that are not required for their intended use case, significantly improving the readability of the Dockerfiles that end up getting built.

• Separate sets of Dockerfiles can be generated for different use cases, then stored and used independently of one another.

• Custom modifications can be made easily, either through manual editing or automatically through the use of ue4-docker plugins or scripted post-processing steps.

• Dockerfiles can be stored in a version control system to act as a source of truth for the container images that they produce, without any imposed directory structure or extraneous files. This allows developers to make use of the version control practices established by popular open source base images from Docker Hub such as Microsoft .NET and Node.js.

• Build-time secrets do not affect the calculation of filesystem layer hashes for caching purposes, and so the build cache generated when building one image variant can be reused to speed up the build process for other image variants that use the same version of the Unreal Engine.

Unfortunately, it will only be feasible to implement the new design once container build tools support secure build-time secret injection for Windows containers as well as Linux containers. In particular, support from BuildKit is critical for maintaining compatibility with Docker. As a result, the redesign is currently blocked until Windows container support is added to BuildKit. However, a subset of its functionality is available today, as discussed in the section below.

Using the new features today

At the time of writing, a large amount of development work is still required to add Windows container support to BuildKit and achieve feature parity with Linux containers. As such, implementation of the ue4-docker redesign is likely to be postponed for quite some time. To help bridge the gap in the interim, I added basic support for template-driven Dockerfile generation to ue4-docker version 0.0.79 and subsequently added support for combining generated Dockerfiles into a single output file in ue4-docker version 0.0.80. These new features do not address all of the limitations of the current codebase, but serve as a starting point to begin an incremental transition towards the new design.

The simplest change is the introduction of a flag called -layout, which allows the user to specify a directory to which the Dockerfiles from the ue4-docker codebase should be copied without being built. (The name of this flag is borrowed from the Microsoft Visual Studio installer, where it provides roughly analogous functionality, and was chosen to avoid confusion with the ue4-docker export command. The name generate was also considered, but it lacked the satisfying irony of borrowing terminology from a component used exclusively in Windows container images to denote a feature that is primarily of benefit to Linux containers.) In isolation, this flag would be of little to no use, since the Dockerfiles are designed to be built by ue4-docker and by default will not function correctly in an external build environment. The key to making this export functionality useful is the introduction of a templating system that permits the introduction of alternate behaviour for use outside of ue4-docker itself.

Dockerfiles in the ue4-docker codebase are now pre-processed using the Jinja templating system prior to being exported or built, which facilitates the introduction of conditional blocks which are not possible when using the regular Dockerfile syntax. These conditional blocks are now used to gate patches and tweaks behind options that the user can specify with a flag called --opt, allowing developers to strip out code which is not necessary for newer versions of the Unreal Engine without impacting the compatibility of ue4-docker with older Engine versions. More importantly, conditional blocks are now used to provide alternative implementations of the logic for retrieving the Unreal Engine source code, and users have the ability to select which implementation will be used. The default implementation still relies on the ue4-docker credential endpoint, but the following alternative implementations can be used in external build environments:

• Copy from the host filesystem: this approach requires that the user has already obtained the Unreal Engine source code and placed it inside the build context on the host filesystem. This is far from ideal with respect to convenience or performance, but it can be used to build both Windows and Linux container images without requiring ue4-docker itself and is the only approach which can be used in an environment which does not have access to GitHub due to security or network limitations. This approach is enabled by specifying the flags --opt source-mode=copy.

• Clone from GitHub with credentials provided by BuildKit build secrets: this approach only works for building Linux container images, but it allows developers to start enjoying many of the benefits of the future ue4-docker design immediately. This approach is enabled by specifying the flags --opt source-mode=git --opt credential-mode=secrets.

In addition to the ability to generate customised Dockerfiles that can be built in environments where ue4-docker itself is not present, ue4-docker version 0.0.80 introduced a flag called --combine which causes the generated Dockerfiles to be combined into a single output Dockerfile. In this mode, the contents of each original Dockerfile are transformed into stages within a large multi-stage build. This is useful for scenarios where developers are only interested in the final image in a sequence (typically either ue4-minimal or ue4-full) and are not interested in retaining the intermediate ue4-build-prerequisites and ue4-source images. The generated Dockerfile will be prefixed with a comment indicating which images were included and which Jinja template options were used to generate them, which is otherwise stored in a separate JSON file when not combining generated Dockerfiles.

To demonstrate these new features, the section below walks through an example of generating a combined Dockerfile for Linux container images and building it with BuildKit. You can also find detailed usage information in the relevant section of the documentation.

Example: building a Linux container image with BuildKit

First, use the ue4-docker build command to generate a combined Dockerfile for the ue4-build-prerequisites, ue4-source and ue4-minimal images:

# Generate a combined Dockerfile for ue4-build-prerequisites, ue4-source and ue4-minimal with BuildKit build secrets support
# (Note that the Unreal Engine version specified here is actually unused, since the version will be specified at build time)
ue4-docker build 4.26.1 -layout "/path/to/outdir" --combine --no-engine --no-full --opt credential-mode=secrets

This will create a Dockerfile in the directory /path/to/outdir/combined, accompanied by the shell scripts and Python scripts used by the various patches and tweaks implemented by ue4-docker. The BuildKit build secret implementation expects to find the user’s git username and password in text files named username.txt and password.txt, which will be mounted as secrets during the build process. Note that if you are cloning the Unreal Engine source code from GitHub then you will need to use a personal access token rather than a password, but this is not the case if you are cloning from a self-hosted git repository (e.g. an internal mirror operated by your organisation.)

Once the git credentials have been placed in text files alongside the Dockerfile, you will need to build the container image using BuildKit. If you have the latest version of Docker installed then the simplest way to do this is to use the docker buildx build command:

docker buildx build \
	-t "adamrehn/ue4-minimal:4.26.1-opengl" \
	--build-arg "BASEIMAGE=nvidia/opengl:1.0-glvnd-devel-ubuntu18.04"
	--build-arg "GIT_REPO=https://github.com/EpicGames/UnrealEngine.git"
	--build-arg "GIT_BRANCH=4.26.1-release" \
	--build-arg "BUILD_DDC=true" \
	--build-arg "EXCLUDE_DEBUG=0" \
	--build-arg "EXCLUDE_TEMPLATES=0" \
	--secret id=username,src=username.txt \
	--secret id=password,src=password.txt \
	--progress=plain \
	.

You need to specify values for all of the Docker build arguments that ue4-docker would ordinarily provide, including:

• Which base image to use (BASEIMAGE)
• The git repository URL and branch name (GIT_REPO and GIT_BRANCH)
• Whether to build the DDC for the Installed Build of the Engine (BUILD_DDC=true or BUILD_DDC=false)
• Whether to exclude debug symbols from the final image (EXCLUDE_DEBUG=0 or EXCLUDE_DEBUG=1)
• Whether to exclude template projects from the final image (EXCLUDE_TEMPLATES=0 or EXCLUDE_TEMPLATES=1)

BuildKit will then build the image as one large multi-stage build, mounting the credential text files as secrets during the step that clones the Unreal Engine source code. Once the build is complete, the finished image will be tagged as adamrehn/ue4-minimal:4.26.1-opengl (or whatever tag value you specify when running the command) and the local build cache will contain the filesystem layers of each intermediate step for reuse by subsequent builds. Neither the finished image or the build cache will contain any trace of the sensitive git credentials.

It is worth noting that the default Docker configuration for BuildKit at the time of writing truncates log output which exceeds 1MiB in size, so it may be necessary to modify the configuration files for the Docker daemon to increase this limit in order to view the full log output during the build process. The exact steps required to perform this configuration will vary across different Linux distributions.

When you’re ready to further explore the new ue4-docker features, check out the relevant section of the documentation for full usage details.