HOWTO: VMware workaround steps to address CVE-2021-44228 and CVE-2021-45046 in Tanzu Application Service v2.7 - v2.12

CVE-2021-44228 and CVE-2021-45046 has been determined to impact Tanzu Application Service via the Apache Log4j open source component it ships.  This KB describes how to mitigate your TAS environment for CVE-2021-44228 and CVE-2021-45046.

This vulnerability and its impact on VMware products are documented in the following VMware Security Advisory (VMSA). Review VMware Response to CVE-2021-44228: Apache Log4j Remote Code Execution (87068) and CVE-2021-44228 – VMSA-2021-0028 before continuing.

Impact / Risks

After this User Account and Authentication (UAA) and CredHub mitigation is applied to the Tanzu Application Service (TAS) Tile Metadata on Operations Manager, upgrading the TAS Tile to a version that does not include a fix for CVE-2021-44228 and CVE-2021-45046 will revert the mitigation.

If for some reason you do need to upgrade TAS to a version that does not include the CVE fix, the mitigation procedure can be re-applied to the vulnerable TAS tile before applying the updated version.

Note: While the CredHub and UAA release substitution changes are applied to the TAS Tile Metadata, any Apply Changes may take additional compilation time, as the replacement releases are not pre-compiled.

Resolution 

A fix has been released and is included in the following Tanzu Application Service releases:

• 2.12.5 (https://docs.pivotal.io/application-service/2-12/release-notes/runtime-rn.html#2.12.5 )
• 2.11.12 (https://docs.pivotal.io/application-service/2-11/release-notes/runtime-rn.html#2.11.12 )
• 2.10.24 (https://docs.pivotal.io/application-service/2-10/release-notes/runtime-rn.html#2.10.24)
• 2.7.44 (https://docs.pivotal.io/application-service/2-7/release-notes/runtime-rn.html#2.7.44 )

These patch versions include the upgraded log4j 2.16 with fix for CVE-2021-44228 and CVE-2021-45046.

Workaround (if upgrade of TAS tile cannot be immediately performed)

The workaround consists of understanding and mitigating the 4 components of TAS, which all potentially have vulnerable versions of log4j. Additionally, this document describes how to use platform features to attempt to mitigate apps on the platform, which may use/specify vulnerable libraries.

In summary, to work around this issue, we recommend:

1. Swapping the UAA and Credhub releases in the TAS Tile to newer versions which have updated log4j libraries.

2. Identifying whether you could be affected by vulnerable 3rd party agent jars, which ship in the java buildpack and are triggered by service bindings and consider action.

3. Identifying whether you could be affected by the one vulnerable 3rd party agent jars in the PHP buildpack, which is unlikely, but could also call for action.

4. Rollout remediation strategies to possibly vulnerable applications using log4j directly on the platform

Note: We recommend you apply all these steps, but they can be applied in any order.

To apply the workaround for CVE-2021-44228 and CVE-2021-45046 to Tanzu Application Service, perform the following detailed steps.

Step 1: Patching TAS UAA & Credhub 

1. SSH into the Ops Manager VM. For more information, refer to Logging Into Ops Manager VMs with SSH.

2. Download the patched CredHub and UAA BOSH releases appropriate to your TAS version to the Ops Manager VM:

Note:  The current mitigation release of credhub and uaa include log4j 2.16 exclusively

• For TAS 2.7, the CredHub release is at https://storage.googleapis.com/credhub-final-release-tarballs/credhub-2.5.15.tgz  and the UAA release is at https://uaa-release-tarballs.s3.us-west-1.amazonaws.com/releases/uaa-release-73.4.36-rc.2.tgz
• For TAS 2.8-2.12, the CredHub release is at https://storage.googleapis.com/credhub-final-release-tarballs/credhub-2.9.7.tgz  and the UAA release is at https://uaa-release-tarballs.s3.us-west-1.amazonaws.com/releases/uaa-release-74.5.29-rc.5.tgz

sudo -u tempest-web wget -P /var/tempest/releases/ https://THE-URL-OF-THE-RELEASE
3. Find the file paths of the YAML files that define all the versions of the TAS tile in your library; you want the .yml file from the following command It should look something like:

• “/var/tempest/workspaces/default/metadata/c5c28c298f9f.yml“
• “/var/tempest/workspaces/default/metadata/product-template-c5c28c298f9f.yml“

Run this command:

sudo grep -l "^name: cf" /var/tempest/workspaces/default/metadata/* 

4. Confirm the version of TAS you’re using with the following command on each full file path; if there’s more than one file returned by the above, run it on each to identify the version that you have currently deployed, which you’ll need to edit in next steps.

sudo head FULL-FILE-PATH

5. Make a backup of this YAML file, into your home directory. You can restore this backup over the file you’re about to edit in order to revert the workaround if needed later.

sudo cp FULL-FILE-PATH ~ubuntu/

6. Edit the YAML file (using “sudo editor-of-choice”, such as “emacs”, “vi”, or “nano”) to replace the 2 relevant release sections for credhub and UAA release. You’ll find the 2 sections you need to edit in the “releases:” part of this very large YAML file.

Make the following changes for both Credhub and UAA releases:

• Update the “version” and “file” keys with new validate indicated below.
• Remove the entire “exported_from” block.

EXAMPLE OLD CREDHUB SECTION THAT YOU’RE REPLACING:

- name: credhub 
  version: 2.9.4 
  file: credhub-2.9.4-ubuntu-xenial-621.160.tgz 
  exported_from: 
  -  os: ubuntu-xenial 
     version: ‘621.160’

NEW SECTIONS FOR TAS 2.7:

- name: credhub 
  version: 2.5.15 
  file: credhub-2.5.15.tgz 
... 
- name: uaa 
  version: 73.4.36-rc.2 
  file: uaa-release-73.4.36-rc.2.tgz

NEW SECTIONS FOR TAS 2.8 AND ABOVE:

- name: credhub 
  version: 2.9.7 
  file: credhub-2.9.7.tgz
... 
- name: uaa 
  version: 74.5.29-rc.5 
  file: uaa-release-74.5.29-rc.5.tgz

7. Apply Changes at least to the TAS Tile! When you review pending changes for the TAS tile, it should look something like this:

Versions in may not match exactly but the overall changes should be similar to the following:

Note: If you see an error like “Packages must be exported from stemcell ‘ubuntu-xenial/621.160’, but some packages are not compiled for this stemcell”, then you likely forgot to remove the “exported_from” block from a previous step. You can fix that and try again.

Your UAA and Credhub are now patched. They will remain patched, even if VM resurrection takes place, you upgrade the stemcell, or you reconfigure the tile. If you upgrade the TAS tile to a new version this mitigation will be lost and may need to be reapplied.

For reference, here is an example of what the Operations Manager change logs will look like when applying this patch:

Task 175 done
 releases:
 - name: credhub
-   exported_from:
-   - os: ubuntu-xenial
-     version: '621.176'
-   url: file:///var/tempest/releases/credhub-2.9.4-ubuntu-xenial-621.176.tgz
+   url: file:///var/tempest/releases/credhub-2.9.7.tgz
-   version: 2.9.4
+   version: 2.9.7
 - name: uaa
-   exported_from:
-   - os: ubuntu-xenial
-     version: '621.176'
-   url: file:///var/tempest/releases/uaa-74.5.27-ubuntu-xenial-621.176.tgz
+   url: file:///var/tempest/releases/uaa-release-74.5.29-rc.5.tgz
-   version: 74.5.27
+   version: 74.5.29-rc.5

Task 176

Task 176 | 23:21:29 | Preparing deployment: Preparing deployment (00:00:15)
Task 176 | 23:21:44 | Preparing deployment: Rendering templates (00:01:05)
Task 176 | 23:22:52 | Preparing package compilation: Finding packages to compile (00:00:01)
Task 176 | 23:22:53 | Compiling packages: luna-hsm-client-7.4/746f3c30aadc0af7afc2d5cddcc16d8836a8f845
Task 176 | 23:22:53 | Compiling packages: openjdk_1.8.0/225f67373c9ad0a1da464aeb92f06207bd3e8da1
Task 176 | 23:22:53 | Compiling packages: uaa/30afa631c973566fadf4bd2da82180dcef9077b7ce390ae66166da6a074b8ecf
Task 176 | 23:24:12 | Compiling packages: openjdk_1.8.0/225f67373c9ad0a1da464aeb92f06207bd3e8da1 (00:01:19)
Task 176 | 23:24:12 | Compiling packages: credhub/f1dc27702ec869336ae30ab5e489f09852b3f6ee
Task 176 | 23:24:12 | Compiling packages: luna-hsm-client-7.4/746f3c30aadc0af7afc2d5cddcc16d8836a8f845 (00:01:19)
Task 176 | 23:24:27 | Compiling packages: credhub/f1dc27702ec869336ae30ab5e489f09852b3f6ee (00:00:15)
Task 176 | 23:24:36 | Compiling packages: uaa/30afa631c973566fadf4bd2da82180dcef9077b7ce390ae66166da6a074b8ecf (00:01:43)
Task 176 | 23:25:11 | Updating instance uaa: uaa/de19e667-0d52-4810-8423-182057981ccb (0) (canary)
Task 176 | 23:25:19 | L executing pre-stop: uaa/de19e667-0d52-4810-8423-182057981ccb (0) (canary)
Task 176 | 23:25:19 | L executing drain: uaa/de19e667-0d52-4810-8423-182057981ccb (0) (canary)
Task 176 | 23:25:20 | L stopping jobs: uaa/de19e667-0d52-4810-8423-182057981ccb (0) (canary)
Task 176 | 23:25:34 | L executing post-stop: uaa/de19e667-0d52-4810-8423-182057981ccb (0) (canary)
Task 176 | 23:25:44 | L installing packages: uaa/de19e667-0d52-4810-8423-182057981ccb (0) (canary)
Task 176 | 23:25:47 | L configuring jobs: uaa/de19e667-0d52-4810-8423-182057981ccb (0) (canary)
Task 176 | 23:25:47 | L executing pre-start: uaa/de19e667-0d52-4810-8423-182057981ccb (0) (canary)
Task 176 | 23:25:50 | L starting jobs: uaa/de19e667-0d52-4810-8423-182057981ccb (0) (canary)
Task 176 | 23:26:21 | L executing post-start: uaa/de19e667-0d52-4810-8423-182057981ccb (0) (canary) (00:01:20)
Task 176 | 23:26:39 | Updating instance credhub: credhub/91593213-3100-4182-a826-9036932e126c (0) (canary)
Task 176 | 23:26:43 | L executing pre-stop: credhub/91593213-3100-4182-a826-9036932e126c (0) (canary)
Task 176 | 23:26:43 | L executing drain: credhub/91593213-3100-4182-a826-9036932e126c (0) (canary)
Task 176 | 23:26:45 | L stopping jobs: credhub/91593213-3100-4182-a826-9036932e126c (0) (canary)
Task 176 | 23:26:58 | L executing post-stop: credhub/91593213-3100-4182-a826-9036932e126c (0) (canary)
Task 176 | 23:27:08 | L installing packages: credhub/91593213-3100-4182-a826-9036932e126c (0) (canary)
Task 176 | 23:27:11 | L configuring jobs: credhub/91593213-3100-4182-a826-9036932e126c (0) (canary)
Task 176 | 23:27:11 | L executing pre-start: credhub/91593213-3100-4182-a826-9036932e126c (0) (canary)
Task 176 | 23:27:13 | L starting jobs: credhub/91593213-3100-4182-a826-9036932e126c (0) (canary)
Task 176 | 23:27:43 | L executing post-start: credhub/91593213-3100-4182-a826-9036932e126c (0) (canary) (00:01:05)

Step 2: Identify java applications using agents which may be vulnerable 

The java Offline buildpack itself is written in ruby and not vulnerable, but in some circumstances it will inject libraries into built applications in order to help them utilize services such as APMs.

The following dependencies that can be installed by the Java Offline Buildpack contain Log4J libraries:

Dependency	Vendor	Vulnerable	Security Bulletin	Fixed Version
AppDynamics	Cisco	Yes	Link	21.11.2
Apache Skywalking	Apache	No
Elastic APM	Elastic	Yes	Link	1.28.1
NewRelic	New Relic	Yes	Link	7.4.2
Introscope / Wily	SAP	No	Link
Contrast Security	Contrast	No	Link
ProtectApp Security	Thales	Yes
Tanzu GemFire	VMware	Yes	Link

The Java Offline buildpack will install a vulnerable dependency if there is a bound service with a service name, label or tag that contains one of the following strings:

• protectapp
• introscope
• app-dynamics
• appdynamics
• elastic-apm
• newrelic
• session-replication

These services would usually be provided by a broker, and you can check if you have any of these brokered services even available to your developers at all. Unfortunately, devs could use a user-provided service with tags to also trigger the behavior.

How to mitigate the Java Offline Buildpack vulnerable dependencies

Option A

• Go to Tanzu Network and download Java Buildpack offline version 4.45  or later
• Login to cf cli using admin credentials
• Update the Java Offline Buildpack using cf cli and specify “–lock” to ensure this buildpack will not get overwritten when upgrading TAS.  You can unlock this buildpack when you upgrade to a TAS version that includes JAVA Offline Buildpack 4.45 or later.

  • cf update-buildpack java_buildpack_offline -p java-buildpack-offline-v4.45.zip

  • cf update-buildpack java_buildpack_offline --lock

• Restage all applications that use the Java Offline Buildpack to generate new droplets

Option B

If you are not able to upgrade the Java Offline BuildPack then we suggest you unbind these vulnerable services from any applications exposed by them.  Once these dependencies are unbound, restage all applications that were modified to generate new droplets.

Step 3: Identify php applications using agents which may be vulnerable 

Like the Java Offline buildpack, the PHP buildpack contains an AppDynamics agent which includes a JRE and a potentially vulnerable version of log4j. The log4j vulnerability would only affect apps that are bound to an AppDynamics user-provided service (which would have initially been created by the user via the `cf create-user-provided-service`/`cf cups` command and bound using the `cf bind-service`/`cf bs` command). The name of the user-provided service would have been given by the user at the time of creation.

To see a list of created services, run cf services.

How to mitigate the PHP Buildpack vulnerable dependencies

Option A

To fix this vulnerability upgrade the PHP buildpack to version 4.4.53 which include log4j 2.16

1. Go to Tanzu Network and download PHP buildpack offline version 4.4.53  or later
2. Login to cf cli using admin credentials
3. Update the PHP Buildpack using cf cli and specify “–lock” to ensure this buildpack will not get overwritten when upgrading TAS.  You can unlock this buildpack when you upgrade to a TAS version that includes PHP Buildpack 4.4.53 or later.

   1. cf update-buildpack php_buildpack -p php_buildpack-cached-cflinuxfs3-v4.4.53+1639605932.zip​​​

   2. cf update-buildpack php_buildpack --lock

4. Restage all applications that use the PHP Buildpack and have the user-provided AppDynamics service bound to generate new droplets.

Option B

To mitigate the effects of the vulnerability without upgrading the PHP buildpack for apps using AppDynamics, disable AppDynamics for that app. To do this:

1. unbind any user-provided AppDynamics services using cf unbind-service <app name> <service name>.
2. Then, restage the app using cf restage <app name>.

Step 4: Rollout broad remediation for potentially vulnerable java applications 

Each JAVA application in the Tanzu platform will need to be analyzed individually for this vulnerability.  Identified applications need to apply Apache recommended remediation steps as soon as possible.

Please refer to the Apache log4j security vulnerability disclosure for the most up to date information on how to remediate vulnerable java applications that are running in your Tanzu Application Service platform.

If you have already applied the partial mitigation by setting environment variable “LOG4J_FORMAT_MSG_NO_LOOKUPS” or JVM property “log4j2.noFormatMsgLookup”, then we do not recommend removing the change or rolling it back.  Your application is likely less vulnerable with this partial mitigation in place, however we do recommend applying one of the Apache approved mitigations as soon as possible.

If you can not yet upgrade all of your applications and still wish to use the partial mitigation by setting the “LOG4J_FORMAT_MSG_NO_LOOKUPS” environment variable on your app then we recommend using environment variable groups feature available through cf cli and then restarting affected applications.

To verify the workaround for CVE-2021-44228 and CVE-2021-45046 has been correctly applied to TAS, perform the following steps: 

1. Open the “/debug/files” URL in the OpsManager UI, such as https://pcf.mycorp/debug/files

2. Click the link called “Deployed BOSH Manifest for Product: cf”
3. Find the “releases:” section and verify that the UAA and credhub releases include patch versions. Patched versions, depending on TAS version, include:
   * TAS 2.7: Credhub 2.5.15
   * TAS 2.7: UAA 73.4.36-rc.2
   * TAS 2.8+: Credhub 2.9.7
   * TAS 2.8+: UAA 74.5.29-rc.5

To revert the workaround for CVE-2021-44228 and CVE-2021-45046 in TAS, perform the following steps:

• Any normal TAS tile upgrade will revert the Step 1 workaround. Alternatively, the definition  file copy created in part 5 of step 1 can be restored to its original path and then an Apply Changes will finish the revert.
• To remove the running environment variable group run cf set-running-environment-variable-group, removing the value of LOG4J_FORMAT_MSG_NO_LOOKUPS