Cybersecurity

HOWTO: VMware NSX-T Data Center log4j Remediation (2.5.0-3.1.3) (87086) CVE-2021-44228

3 years ago   •   3 min read

By CloudNerve.com

Notice:   On December 14, 2021 the Apache Software Foundation notified the community that their initial guidance for CVE-2021-44228 workarounds was not sufficient. We believe the instructions in this article to be an effective mitigation for CVE-2021-44228, but in the best interest of our customers we must assume this workaround may not adequately address all attack vectors.
We expect to fully address both CVE-2021-44228 and CVE-2021-45046 by updating log4j to version 2.16 in forthcoming releases of NSX-T Data Center, as outlined by our software support policies.

This Knowledge Base article and VMSA-2021-0028 will be updated when these releases are available.

Please subscribe to this article to be informed when updates are published.

CVE-2021-44228 has been determined to potentially impact VMWare NSX-T Data Center via the Apache Log4js open-source component it ships.  This vulnerability and its impact on VMware products are documented in the following VMware Security Advisory (VMSA), please review this document before continuing:

 Impact / Risks

A malicious actor with network access to an impacted VMware product may exploit this issue to invoke remote code execution.  However, there are multiple layers of protection in NSX-T that will make exploiting CVE-2021-44228 difficult:

  • Internal NSX-T processes/JVM’s are prevented from making external connections via the iptables configuration in the NSX-T Unified Appliance.
  • The version of Java JDK in NSX-T Datacenter version 3.0.2 and above prevents the most common remote code execution publicized (ie. LDAP attack vector).

All versions of NSX-T Data Center contain the log4js.  Further exploit of the log message lookup feature may be possible.

Note: NSX-T Edge VM’s and Bare Metal Edge Nodes are not affected by this issue. 

Note: If the below workaround is applied to an NSX-T Manager, and that NSX-T Manager is subsequently upgraded to a newer, vulnerable version of NSX-T, the workaround must be re-applied post upgrade. 

HOWTO: NSX-T Data Center log4j Remediation (2.5.0-3.1.3) (87086) CVE-2021-44228

 Resolution

The workarounds described in this document are meant to be a temporary solution only.
Upgrades documented in the aforementioned advisory should be applied to remediate CVE-2021-44228 when available. Supported versions of NSX-T Data Center will be updated in the near future.

Workaround

To apply the workaround for CVE-2021-44228, connect to the VMWare NSX-T Data Center Manager or NSX-T Cloud Service Manager and perform the following steps.

Important: Apply this steps to one manager at a time and allow time for your management cluster to stabilize before moving onto the next manager.

1. Login to the NSX-T Manager or NSX-T Cloud Service Manager via SSH as admin and check cluster status:

get cluster status
Note: Above command is only pertinent to NSX-T Manager and NOT NSX-T Cloud Service Manager

2. Switch to root user:

st en
Enter the root password when prompted.

3. First make a copy of the tanuki conf files in case we need to revert the patch:

mkdir /root/tanuki-confcp -p /usr/tanuki/conf/*-wrapper.conf   /root/tanuki-conf/

4. Then apply the workaround with:

find /usr/tanuki/conf/ -name ‘*-wrapper.conf’ | xargs -n 1 -I {} sh -c ‘echo “wrapper.java.additional.100=-Dlog4j2.formatMsgNoLookups=true” >> {}’

5. Finally reboot the system:

/sbin/reboot

6. Login as admin and check cluster status:

get cluster status
Note: Above command is only pertinent to NSX-T Manager and NOT NSX-T Cloud Service Manager

7. When cluster is stable again, proceed to the next NSX-T Manager and start at step 1 again.

Note:  Step 7 is not applicable for NSX-T Cloud Service Manager.

Note:  This workaround will have to be re-applied on the post-restore NSX-T Managers or NSX-T Cloud Service Managers if an environment is restored to backup.

To revert the workaround for CVE-2021-44228 to VMware NSX-T Data Center perform the following steps on each NSX-T Manager or NSX-T Cloud Service Manager.

cp -p /root/tanuki-conf/* /usr/tanuki/conf/
/sbin/reboot
 Related Information

Change Log:

  • December 13th 2021 – 11:00 PST: Removed validation test steps as it was only applicable to one attack vector, specifically for NSX-T 3.0.1 and earlier.
  • December 13th 2021 – 11:30 PST: Clarified that NSX-T Edge Nodes, VM or Bare Metal, are not affected by this issue.
  • December 13th 2021 – 12:30 PST: Removed erroneous word ‘exit’ from workaround.
  • December 13th 2021 – 13:00 PST:  Added note regarding re-applying the workaround if an environment is restored to backup.
  • December 13th 2021 – 16:00 PST: Minor edits to include NSX-T Cloud Service Manager to this KB as well.
  • December 14th 2021 – 05:00 PST: Minor edit to add -p on cp commands.
  • December 14th 2021 – 12:30 PST: Added clarity regarding upgrading to a newer, vulnerable version of NSX-T after applying the workaround.
  • December 15th 2021 – 11:30 PST: Added notice acknowledging CVE-2021-45046 and an impending release containing log4j version 2.16.

Spread the word

Keep reading