Your Best Versioning Strategy for Secure Deployments
Containers have simplified how we develop and deploy software. The way in which container images are managed is a critical component of a smooth CI/CD process. Container Registries act as trusted intermediaries between development and runtime environments for containerized applications.
Organizations that build and deploy container-native applications need to choose between various tools to build, run and manage containers in production and image versioning is a key decisions that engineers need to master in order to support that process. The central purpose of a container registry is to serves as the central repository for deploying, versioning, and managing container-native artifacts. By utilizing the built-in functions of the container registry, release engineers can easily manage deployments to multiple environments.
As in most development processes, versioning container images in a repository is necessary to denote changes made over a product’s lifecycle. Using a combination of timestamps, commit IDs, and pre-defined strings, a clear picture of what is deployed in what environment can be seen. Versioning allows engineers to trace changes made in that particular software or image back to changes in source code.
The concept of versioning or tagging as it is named in the container space, differs from the concepts we usually understand when referring to versioning in traditional software.
The main difference is that tags are not stable and can point to some other image layers over the course of time. Also, multiple tags can all point to the same set of layers.
We can observe this concept applied on almost all popular open source container images on Docker Hub. Below is the screenshot of the Nginx tag descriptions on Docker Hub.
Each row in the list refers to a distinct image of Nginx probably containing a different extension or application version. The first row for example has the tags 1.19.6, mainline, 1, 1.19 and :latest. All the tags point to the same image layers for now
Applying multiple tags on the same image is an established convention for image consumers to provide context regarding the image tag stability. Some tags will change frequently some seldom and some never changed.
A common practice is to tag every release with :latest independent if it is a development snapshot or the final release. Every successful run of the CI/CD pipeline will be tagged with :latest and consequently point to different image layers. Although very handy for development this is not desired for the productive rollout. To provide image consumers with more control over the image change frequency and the impact of a change, the tag name carries a context. For example the meaning of 1.19 indicated that this tag will always point to the latest patch release 1.19.6,1.19.7,1.19.8 and so on.
As touched on previously, including the proper version for a release is critical to ensure DevOps and other members of the team have a method to determine what changed from the prior release. Because of this fact, it is recommended to be explicit when using images based off publicly available containers. The same is true for images that may reference other containers in order to run properly.
If an engineer depends on an image with the :latest tag for their upstream image, they run the risk of introducing an unwanted side effect in their container because the :latest tag will point to different image layers somewhere in the future unnoticed by our engineer as his declaration in Dockerfile will still stay the same. By explicitly stating a version, an engineer can make sure that he always receive the desired layers.
These explicit versions should also be formatted and integrated in a way that points back to changes in the release. If a feature released to production started as a project management task, this version should line up with the intended release cycle that was part of that project. By doing so, an engineer can easily see what could be considered “breaking changes.”
Another aspect of proper versioning and usage of container images is the ability to roll back to a previous state in a very short amount of time. In some cases, multiple versions of the same product may be required to be active. For example, a company may need to allow backwards compatibility for web services they offer to clients. Orchestrating multiple versions of the image is what makes this type of scenario possible.
Part of an engineering team’s adoption of container technology is to determine what type of strategy they will use for image tagging. By determining this strategy up-front, integration with CI/CD pipelines can be made easier thanks to the ability to follow a standard. Using this strategy, developers and the operations team have a usable method for identifying artifacts.
Every image has a unique ID, also known as a digest. It is a hash of its manifest that is generated using the SHA265 algorithm.
Remember that images and layers are not synonyms. One image may have multiple layers. Each of them would have its own digest, and the parent image – to be precise, its manifest list – will have its own digest, too.
Any modifications of the image will result in a new digest.
In addition to the image identification, the digest also serves the purpose of image verification. You can recalculate the digest using image data and verify that the contents have not been modified.
Digests are the safest way to identify images. Using digests, you can be sure what you are deploying. They are immutable.
Indeed, the only problem with them is that they are barely readable. You cannot expect someone to memorize an image ID.
Image tags are named references to an image version at a specific point in time. Tags allow an easy lookup of image revisions by using human-readable strings.
Additionally, tags can be used to store helpful information about a specific image version. Most often, the image tag consists of an indication of the underlying service artifacts, supported architectures, and applicable environments.
Unfortunately, any tags are mutable, so that the image represented by the tag might change over time. Tags can not get you back to a build, and the contents of the build. So, it’s hard to tell exactly what you’ll get when you pull an image by its tag.
Consequently, using tags as your versioning strategy alone won’t allow you to roll back to a previous version should the current one not be working.
We must mention two widely used universal Docker image tags. They are assigned to images on a rolling basis: with every new release, the tag rolls from the previous version to the newest.
These tags comes in handy if you need to grab the most relevant image but do not know which one you need. You can also use rolling tags to setup your orchestration tool to pull images with these tags to ensure that you get all updates in time.
The drawback of this approach is that you may get an incompatible image one day. Although this is fine for the testing environment, it is not acceptable for production. Here you’d better rely on unique tags that will be described later.
One of the rolling tags is :latest.
Never deploy any images with the :latest tag. Apart from the fact that tags are not mutable, this one is generated when you do not specify any other tag.
Other popular canonical name that developers use as a tag is :stable. The tag was supposed to demonstrate that the contents of an image are stable and safe to be deployed to production environments and be consumed by downstream containers in orchestrated environments.
As in the case of :latest, the pitfall here is the impossibility to roll back to a prior version. Since all releases are marked as “stable,” those that are replaced with the newest version are in a semi-archival state. While still usable, the non-uniqueness of that tag can spoil image orchestration.
We have identified two difficulties associated with using “native” Docker image attributes as your image versioning strategy:
Image tags are easy to reference but are challenging to operate. The mutability of tags creates the risk of deploying wrong containers into production which can cause a myriad of issues, such as releasing untested features, broken APIs, bypassing security checks, production failures, etc. Thus, developer teams need an effective strategy that can work for any specific use cases.
Instead of tagging your images chaotically with some catchy names, you can stick to the old good system called Semantic Versioning, also known as SemVer. In fact, this can be treated as a special case of the stable tag strategy mentioned before.
According to this, your tag would be the version number of your software that consists of three parts:
For instance, 2.3.4.. In this manner, you overcome the tag mutability since you are not going to tag images of another patch with the number from a different patch.
Nonetheless, it does not make the tags immutable. Someone pushes a different image with the same tag, and – voilà! – the same confusion is there again.
Every time you do a commit, a new Docker image is generated. Therefore, teams can tag images using the short Git hash to trace back changes in a reliable manner.
Although Git hashes are way shorter than the image digests, they are still difficult to memorize and are not self-explanatory.
Unique identifiers allow users to tag each image with semi-immutable references.
Identifiers like this are generated automatically and, if you use the same system to generate them, are produced only once.
This method still does not exclude the possibility of someone tempting with your images. Although the identifiers are unique and immutable within their native systems, the image tags are not.
There are several ways to generate unique tags for an image. We give you the two most popular methods.
A timestamp is, indeed, an easy solution. But it has more drawbacks than advantages. It lacks a correlation to the included changeset(s) for the container image release since you cannot match it with the respective build.
Do not forget about the evil timezone: which time was it actually in yours? Moreover, what if you created more than one image at exactly the same time? Last but not least, someone can push an image with the same tag just by adding it manually.
A build ID is a unique identification of binaries. It is not only truly authentic but directly refers to a certain build. It also cannot be faked. But, similarly to the image digest, it is long and does not give any hint about the changesets it includes. Neither it is helpful if you want to search for specific image.
Use rolling tags for picking base images and pulling updates constantly.
Use unique tags when you deploy containers in production.
A predictable SemVer style is always a good idea. The approach tightly couples an image to the underlying changesets and can be managed using automation. Moreover, any downstream users are also assured of receiving an image that is compatible with their existing systems.
Rolling and SemVer tags, as in the example of NGINX above, when used in combination provide a solid tagging strategy for every use case.
In small teams but sensitive deployments and for workflows of a manually manageable size, feel free to use digests, GitCommits, timestamps, and build IDs.
Additionally, you can enable tag immutability in your container image registry, such as Harbor. This will ensure that only unique tags are pushed to the repository and that the images cannot be deleted or altered in any way, such as by re-pushing, re-tagging, or replicating them from a different registry.
You can even use regular expressions to create tag immutability rules as described here. Regex allows you to catch patterns of tags created following the semantic versioning rules: three parts separated by two commas. As a result, you make your SemVer tags immutable.
For particularly busy container enthusiasts, we offer a managed container registry service, which we call simply Container Registry.
It is hosted in a cloud, offers all must-have features plus an SSO, RBAC, vulnerability scanners, etc., and has a storage-based pricing model.
Find out how our Harbor-inspired Container Registry can streamline container releases by trying it for free.
Published — September 9, 2021
