Google & Apache Found Vulnerable to GitHub Environment Injection

In this blog post, we'll discuss a new type of GitHub Actions workflow vulnerability we called "GitHub Environment Injection". We've found a couple of top-tier open-source organizations vulnerable to this attack: Google Firebase and Apache. Both Google and Apache acknowledged the issues we reported and fixed their workflows accordingly. Those organizations contained repositories that had workflows that download an untrusted artifact and dumped its content into a GitHub environment file. As we’ve seen previously, using untrusted data in a privileged context can open the door to CI/CD pipeline takeover risk and allow a malicious actor to take control of the pipeline.

 In this write-up, we’ll cover the vulnerable Google Firebase workflows and how they can lead to a software supply chain attack. To keep this blog within a reasonable length and due to similarities between the cases, we won’t cover the Apache case, but you can find their fix here.

This post is the 3rd in the series of Vulnerable GitHub Actions Workflows, following:

• Part 1: Privilege Escalation Inside Your CI/CD Pipeline

• Part 2: Actions That Open the Door to CI/CD Pipeline Attacks

Google Firebase - The Vulnerable Workflows Found

The following firebase repositories: https://github.com/firebase/friendlyeats-web & https://github.com/firebase/codelab-friendlychat-android/ contain a workflow named “preview_deploy.yml” that deploys a preview version of the application to Firebase in order to test it before merging:

The workflow works as follows:

• It is triggered whenever a workflow named “Generate Preview” finishes

• It downloads two artifacts (zip files) that were uploaded by the “Generate Preview” workflow: pr.zip and firebase-android.zip

• It then reads the pull-request number from ‘pr.zip' and writes it into the ‘pr_number' environment variable

• lastly, it unzips the firebase app and deploys it

What is $GITHUB_ENV?

From GitHub documentation:

“You can make an environment variable available to any subsequent steps in a workflow job by defining or updating the environment variable and writing this to the GITHUB_ENV environment file.”

This is a special file that whenever input is redirected to it, the runner engine takes it and sets environment variables accordingly.

For example, in the above workflow, line 47 sets an environment variable named “pr_number” to the “NR” file content.

Yet another case of a vulnerable GitHub-Actions workflow

The workflow downloads artifacts that were uploaded by a pull request triggered workflow and writes its content into $GITHUB_ENV. Since anyone can create a pull request and upload artifacts for public repositories, an attacker can craft an artifact payload that would set arbitrary environment variables in the privileged context (for more information about workflow_run and how privilege escalation attacks can take place when it is used insecurely, take a look at Part 1).

For example, if the file in line 47 (“NR”) had the following content: “123\nTEST=BLA”, the command would translate to 

resulting in two environment variables being set instead of only “pr_number” as the workflow author intended.

OK, so we can set environment variables as we wish. Now let’s leverage this to execute code in the privileged workflow.

Linux has many special environment variables that control how programs behave which we can modify to execute code. For example, we can use “LD_PRELOAD” to load some malicious binary that we add in the pull request, or alternatively, use the more convenient “NODE_OPTIONS”. The “NODE_OPTIONS” env is used to provide additional command line parameters to the node engine. For example, the below value will tell the node engine to print “123” at the beginning of every node program:

With this information in our hands, let’s build the final exploitation workflow:

1. Fork one of the vulnerable Firebase repositories

2. Create the below workflow:

3. 

4. Create a pull request to the original repository

5. The “NODE_OPTION” payload would dump the process environment variables to the console with all the sensitive secrets and keys

An adversary that wishes to initiate a supply chain attack could create a more sophisticated payload that would modify the repository build artifacts, releases, and tags, and in some cases, the main branch source code to deliver a poisoned version of this code.

Surprisingly, a variation of this vulnerability was previously discovered by Google

Felix Wilhelm from Project Zero found a similar issue a couple of years ago. He discovered that GitHub Actions set-env mechanism is broken and makes thousands of workflows vulnerable. Back then, to set an environment variable while the workflow was executing, one had to log a specific character sequence to STDOUT which the Action runner would pick up and change the container environment accordingly

This implementation of workflow commands was fundamentally insecure since logging to STDOUT is a widespread practice, and attackers could inject a malicious payload that would trigger the set-env command quite easily. The ability to modify environment variables creates multiple paths to RCE with the most obvious payload is the one shown before:

To mitigate this risk, GitHub took the following actions:

1. Deprecate the set-env command in favor of the file command: https://github.blog/changelog/2020-10-01-github-actions-deprecating-set-env-and-add-path-commands/, which makes it harder for attackers to modify environment variables, like in the line below: 

   1. 
2. Create a list of blocked environment variables that cannot be changed: 
   1. 

Different symptoms but same disease

The problem with the new implementation is:

1. Although the migration from STDOUT commands to file commands helped mitigate this risk, we did find quite a few repositories that write unsafe data to the environment file, such as the ones from Firebase described above. With control on the payload written to the env-file attackers can modify environment variables as they wish.

2. GitHub didn’t migrate the blocklist code from the STDOUT commands to file commands https://github.com/actions/runner/blob/main/src/Runner.Worker/FileCommandManager.cs doesn’t contain the code listed above

So, once again, if an attacker manages to modify environment variables, the path to attacking the organization’s software supply chain is very short.

We reported these findings to GitHub, but unfortunately, they decided not to fix it, claiming they “consider users to be responsible for any vulnerabilities arising from these insecure Actions workflows. So it’s up to the repositories maintainers to make sure they are safe. Google Firebase and Apache aren’t the only cases we found. Since using the GITHUB_ENV file is currently considered the safe way to change environment variables in GitHub Actions, many repositories are using workflows that write untrusted data into the environment file, leaving them exposed to pipeline takeover risk.

What Can You Do to Protect Yourselves

• Never write untrusted input data to the environment file

• Restrict the GitHub token permissions only to the required ones, this way, even if an attacker will succeed in compromising your workflow, they won’t be able to do much

• Prefer using Actions output parameters instead of environment variables

• If possible, check that the triggering workflow doesn’t belong to a forked repository, and if it does require human approval as explained in this blog post: Using Environment Protection Rules to Secure Secrets When Building External Forks with pull_request_target

Timeline

• April 5th: report the issue to Apache through ASF Security Team

• April 5th (a few hours later): Apache fixed the issue

• April 30th: reported issue to Google through their bug bounty program

• May 2nd: received first response from Google

• June 29th: Google fixed the issue and provided approval for the write-up (b2f24955a5d6909706f7528bc5a6d7250f9373a5 & df65aefd24cf6f092a27a5576067ff9f29aa2ef1)

Legit Security Can Help You Prevent Software Supply Chain Attacks

The Legit Security platform connects to your GitHub organization and detects vulnerable workflows such as this one in real-time and much more. If you are concerned about these vulnerabilities and others across your software supply chain, please contact us or request a demo on our website.