What Are Immutable Tags And Can They Protect You From Supply Chain Attacks?

Artifacts, such as container images, are referenced during the development lifecycle using tags – a readable short name (usually a version like v1.0). In this blog we demonstrate that some tags cannot be trusted to reference the same object all the time, and can be changed without the users’ knowledge, opening the door to a supply chain attack. While some controls exist – such as making a tag immutable – they are often still flawed because of the way they are implemented in artifact registries

We will end this blog describing the safest approach to reference images reliably.

What are Immutable Tags?

An Immutable object is an object whose state cannot be modified after it is created.

Tags are pointers, simply put. In the container images world, for example, a tag is a pointer to a specific container image (More specifically, to a manifest file, which is a JSON document describing the contents of the image).

Put together, an immutable tag is a tag that, once created, cannot be changed. Thus, guarantees that the original image, which was pointed by the tag, will be pointed by it forever. Unfortunately, it is not that simple, and most artifact tags today are in fact mutable.

Mutable Tags: A Software Supply Chain Attack Surface

A mutable tag is the opposite of an immutable tag, meaning the pointed object of it can be changed at any given time. Consider the following scenario (as shown in diagram below):

1. Developer creates a pull request
   1. Goes through a standard code review and gets an approval to merge
2. Pull request gets merged, and triggers a basic CI/CD flow
3. The build process starts
   1. Container image is built
   2. Container image is tagged with the 1.0 tag
   3. Container image is then pushed to the registry with the 1.0 tag, pointing to the container image that we just built
4.  An attacker gets push access to your container registry
   1. Builds a malicious container image
   2. Tags the malicious image with 1.0 (It is mutable, after all)
   3. Pushes malicious image & tag
   4. Tag 1.0 now points to the malicious image
5. The deploy phase, which runs the container in the runtime production environment, executes and pulls whatever the 1.0 tag points to, in this example, a malicious image

As you can see, mutable tags are pointers that can be manipulated. while a regular image hash can uniquely point to a container (it is a hash – so it can always be verified against the image blob), a mutable tag can change under your feet. There is no guarantee that a version tag ‘v3.6’ will always point to the same object.

How Can I Make Tags Immutable?

We have established so far that it is safer to create tags immutable so that once they are created they will always point to the same image.

The availability of this option depends on the type of artifact, where and how it is being used, but it is primarily up to the registry that hosts the artifact, to decide that behavior, so make sure this configuration exists in the specific registry that you use. Some registries do not have this option which means tags will always be mutable.

For example, the AWS ECR (Elastic Container Registry) has the option to make tags immutable. If an attacker tries to override a tag in this particular registry, an "ImageTagAlreadyExistsException" will be raised.

Which leads us to the next question… 

Can Immutable Tags Be Trusted?

In theory, tagged artifacts guarantee that they cannot be altered in ways such as re-pushing or re-tagging. Unfortunately, the answer to this question is ambiguous. If an attacker gets access to change the registry configuration, he can then turn the tag immutability setting off, which immediately removes the above-mentioned immutable tags benefits. When it comes to supply chain security, it is better to keep ambiguity out of the equation whenever possible, so we will have to go with a "no".

To further explain, and while staying on the same “1.0” example that accompanies us throughout this post, imagine the registry had tag immutability enabled. We said the attacker with push access would get the following error while trying to overwrite the 1.0 tag:

“Tag invalid: The image tag '1.0' already exists in the repository and cannot be overwritten because the repository is immutable.”

Sounds perfect, right? Exactly what we expected… but not really. If the attacker obtained push access to the repository, he may as well have enough access to disable the tag immutability option:

aws ecr put-image-tag-mutability --repository-name my-repository --image-tag-mutability MUTABLE

Which would then allow him to freely push a different image digest using the very same 1.0 tag, leading to the same result. The attacker can later restore the immutability configuration as if nothing happened. What it means is that with an access key and little more effort than before, an attacker can maliciously replace a tag even though the registry may seem immutable.

Pros and Cons of Tags in your Software Supply Chain

As you noticed by now, using tags in general while pulling artifacts can pose a risk. We are no longer referring to immutable tags only, but tags in general.

The main reason for using a mutable tag is to allow developers to push important security and non-breaking critical bug fixes onto the same tag without requiring any changes from the consumer side. However, the cons could be worse - relying on a tag while pulling artifacts exposes you to a risk that is out of your control - if any 3rd party image that you pull via a tag gets compromised, so will you.

The trade-off is between getting security updates (and other important fixes) without having to do anything, to potentially pulling malicious code (also without having to do anything).

Given the rise of software supply chain attacks and the growing dependence on open-source software, linking to a version tag should, in our opinion, be immutable by default. It is an important way to strengthen trust in the software supply chain, and it should be enforced by all popular registries out there.

A More Secure Approach for your Software Supply Chain

Currently, the safest practice would be using container image digest (or the alternative, in other artifact types) instead of using a tag, which effectively pins an image to a specific version in time and stops pulling any updated version of an image without your explicit action. If you want to pull an updated image, you will need to change the digest accordingly.

So, in our example, we will change the original pull command from:

pull my-registry/my-image:1.0

to:

pull my-registry/my-image@sha256:e62e73dd7f24578c82f40f15b3e6c40b49e33ccb86

This obviously adds some complexity and operational overhead. For example, pinning to a container image digest will bring additional overhead of maintaining updated versions that may also include important security updates, so it’s important to be aware of that and stay on top of any new releases to your dependencies.

How Legit Can Help

Our platform can help you detect any potential misconfigurations in your registry, notify you of any artifacts that may be exposed to a risk such as discussed above, create relevant issues, provide you with easy-to-follow remediation steps, and much more.

To learn more best practices in software supply chain security, check out our guide: Best Practices to Secure Your Software Supply Chains.