HOWTO: Remediation Steps for CVE-2021-44228 and CVE-2021-45046 in vRealize Operations 8.x (87076)

CloudNerve.com -

SYMPTOMS:

Notice: On December 14, 2021 the Apache Software Foundation notified the community that their initial guidance for CVE-2021-44228 workarounds was not sufficient. We believe the instructions in this article to be an effective mitigation for CVE-2021-44228, but in the best interest of our customers we must assume this workaround may not adequately address all attack vectors.

We expect to fully address both CVE-2021-44228 and CVE-2021-45046 by updating Log4j to version 2.16 in forthcoming releases of vRealize Operations, as outlined by our software support policies. VMSA-2021-0028 will be updated when these releases are available. In the interim, we have updated this Knowledge Base article with revised guidance to remove all JndiLookup classes per Apache Software Foundation guidance.

Please subscribe to this article to be informed when updates are published.

PURPOSE

CVE-2021-44228 has been determined to impact vRealize Operations 8.0.x – 8.6 via the Apache Log4j open source component it ships.  This vulnerability and its impact on VMware products are documented in the following VMware Security Advisory (VMSA), please review this document before continuing:

Notes:

  • For Application Remote Collector (ARC) versions 8.1.x – 8.3.x, where the standalone Application Remote Collector appliance is available, use the Analytic (Primary, Replica, Data), Remote Collector and Witness node workaround instructions in this article.  There is no need to run the scripts against the Application Remote Collector appliance, as it does not have any Log4j dependencies or jars.
  • For Application Remote Collector (ARC) versions 8.4 and above, where a Cloud proxy is in use, follow the Cloud Proxy workaround instructions in this article.
  • For those still using End Point Operations agents, End Point Operations is based on Log4j 1.2.x which may be impacted by CVE-2021-45046, however we have no components using the JMSAppender configuration.  End Point Operations agents are not impacted.

Impact / Risks

It is highly recommended to take snapshots of the vRealize Operations nodes following How to take a Snapshot of vRealize Operations.

Note: These snapshots are required if you should have to revert the workaround for any reason.

The mitigation will be undone if Management PAK’s are installed or Activated after applying the workaround, which bring Log4j-core components below version 2.16.  The below workaround steps must be reapplied after installing any Management PAKs.

The mitigation will be undone if the any vRealize Operations nodes, or Cloud Proxies are freshly deployed or updated to a version which doesn’t contain the fix.  The below workaround steps must be reapplied after deploying new vRealize Operations nodes or Cloud Proxies, or updating to a release that does not contain the fix.

RESOLUTION:

The workarounds described in this document are meant to be a temporary solution only.
Upgrades documented in the aforementioned advisory should be applied to remediate CVE-2021-44228 and CVE-2021-45046 when available.

 

WORKAROUND:

Notice: The below content has been updated as of 12/15/2021 to add workaround steps for the related CVE-2021-45046 as noted above.  Please re-run all of the below steps even if you have already implemented the original CVE-2021-44228 workaround steps by running the data-rc-witness-log4j-fix.sh and cp-log4j-fix.sh scripts.

Quick Links

To apply the workaround for CVE-2021-44228 and CVE-2021-45046 to vRealize Operations, perform the following steps:

For Analytic (Primary, Replica, Data), Remote Collector and Witness nodes:

  1. Log into the vRealize Operations Manager Admin UI as the local admin user.
  2. Click Take Offline under Cluster Status.

Note: Wait for Cluster Status to show as Offline.

  1. Copy the attached data-rc-witness-log4j-fix.sh and vrops-log4j-fix.sh files to the /tmp directory on all Analytic, Remote Collector and Witness nodes in the cluster using an SCP utility.
  2. Log into each Analytic, Remote Collector and Witness node as root via SSH or Console, pressing ALT+F1 in a Console to log in.
  3. Change to the /tmp directory on all nodes:

cd /tmp

  1. Run the following command on all nodes to make the data-rc-witness-log4j-fix.sh script executable:

chmod +x data-rc-witness-log4j-fix.sh

  1. Run the following command on all nodes to make the vrops-log4j-fix.sh script executable:

chmod +x vrops-log4j-fix.sh

  1. Run te following command on all nodes to execute the data-rc-witness-log4j-fix.sh script:

./data-rc-witness-log4j-fix.sh

Note: Ensure there are no ERROR messages in the script output.

  1. Run the following command on all nodes to execute the vrops-log4j-fix.sh script:

./vrops-log4j-fix.sh

Note: Ensure there are no ERROR messages in the script output.

  1. Run the following command on all nodes to restart the CaSA service:

service vmware-casa restart

  1. Log into the vRealize Operations Manager Admin UI as the local admin user.
  2. Click Bring Online under Cluster Status.

Note: Wait for Cluster Status to show as Online.

For Cloud Proxies

  1. Copy the attached cp-log4j-fix.sh and vrops-log4j-fix.sh files to the /tmp directory on all Cloud Proxies using an SCP utility.
  2. Log into each Cloud Proxy as root via SSH or Console, pressing ALT+F1 in a Console to log in.
  3. Change to the /tmp directory on all Cloud Proxies:

cd /tmp

  1. Run the following command on all Cloud Proxies to make the cp-log4j-fix.sh script executable:

chmod +x cp-log4j-fix.sh

  1. Run the following command on all Cloud Proxies to make the vrops-log4j-fix.sh script executable:

chmod +x vrops-log4j-fix.sh

  1. Run the following command on all Cloud Proxies to execute the cp-log4j-fix.sh script:

./cp-log4j-fix.sh

Note: Ensure there are no ERROR messages in the script output.

  1. Run the following command on all Cloud Proxies to execute the vrops-log4j-fix.sh script:

./vrops-log4j-fix.sh

Note: Ensure there are no ERROR messages in the script output.

  1. Run the following command on all Cloud Proxy nodes to restart the CaSA and Collector services:

service vmware-casa restart; service collector restart
To verify the workaround for CVE-2021-44228 has been correctly applied to vRealize Operations, perform the following steps:

  1. Log into each node as root via SSH or Console, pressing ALT+F1 in a Console to log in.
  2. Run the following command to verify if the data-rc-witness-log4j-fix.sh script was successful:

ps axf | grep –color log4j2.formatMsgNoLookups | grep -v grep

Note: There should be output from the above command.  If there was no output on any particular node(s), that node(s) was not successfully modified.  Re-run the script on that node(s) following the instructions above.

  1. Run the following command to verify if the vrops-log4j-fix.sh script was successful:

./tmp/vrops-log4j-fix.sh

Note: You should receive output reading:
Searching for impacted .jar files. Please wait…
No impacted .jar files found

Related Information
To revert the workaround for CVE-2021-44228 and CVE-2021-45046 to vRealize Operations perform the following steps:

  1. Revert to the snapshot taken of each node prior to implementing the workaround.

Change Log:

  • December 10th 2021 – 14:22 MST:    Drafted initial document with initial workaround.
  • December 15th 2021 – 13:30 MST:    Added the vrops-log4j-fix.sh script and instructions to remove vulnerable JndiLookup classes from all .jar files.
  • December 16th 2021 – 10:56 MST:    Added notes pertaining to Application Remote Collector (ARC) in the Purpose section.
  • December 16th 2021 – 11:30 MST:    Added “CVE-2021-45046” to the title.  Added note about End Point Operations in the Purpose section.

For workaround steps for vRealize Operations Cloud see Workaround instructions to address CVE-2021-44228 in vRealize Operations Cloud.