Customers of Ivanti Connect Secure VPN had rough start of the year due to recently discovered high severity vulnerabilities that were already being exploited in the wild prior to being discovered by Security researchers. The combination of two vulnerabilities made overall discovery lethal for the users of Ivanti VPN servers. First one was an authentication bypass vulnerability whereas second allowed remote users to execute arbitrary code on VPN server. This post gives more information about the vulnerabilities, impact and mitigation techniques.

On 10th of January Volexity broke news with a report and follow up tweet about discovery of two critical vulnerabilities that were already being exploited in the wild as zero-day for Ivanti Connect Secure VPN (f.k.a Pulse Secure VPN from Juniper). With the combination of these two vulnerabilities a threat actor can complete control of Connect Secure VPN servers, even if they were fully patched. However, as per the report, they had seen exploitation of these vulnerabilities at least since 3rd of December 2023. When the news broke, CVEs were already published for both the vulnerabilities. First vulnerability CVE-2023-46805 (CVSS v3: 8.2, High) which is used bypass authentication and second vulnerability CVE-2024-21887 (CVSS v3: 9.1, Critical) was used to perform remote code execution on fully patched Connect Secure VPN Servers.

Threat Actor Attribution
Due the way these vulnerabilities were identified and exploited by threat actor, Mandiant suspects the threat actor to be espionage motivated, and is tracking the threat actor with code name of UNC5221, however on the other hand Volexity has named this threat actor as UTA0178 and has claimed to have reasons to believe that the threat actor is China based & state sponsored. 

Authentication Bypass Vulnerability (CVE-2023-46805)
Rapid7, in their analysis of these vulnerabilities, stated that the API endpoint "/api/v1/totp/user-backup-code" was cause of authentication bypass vulnerability which was caused due to improper usage of "strncmp" function in "doAuthCheck" section of the code. The vulnerability can be exploited using classical "Path Traversal" or "../.." type of attack to invoke other regions of the code using this unauthenticated API.

Command Injection Vulnerability (CVE-2024-21887)
Rapid7 discovered two command injection vulnerabilities one each in files "restservice/api/resources/license.py" and "restservice\api\resources\awsazuretestconnection.py”. Both the vulnerabilities had were result of improper usage of “Popen” function, which led to command injection in multiple endpoints of their API eco-system. This was again was the result of lack of input sanitization at API level.

These two command injection vulnerabilities when combined with earlier mentioned authentication bypass vulnerability, become a strong weapon to infiltrate an enterprise using Ivanti Connect Secure VPN solution and was enough to grant threat actors access to a large part of an enterprise network.
The table given below summarizes what and where the vulnerabilities were discovered.
 

Ivanti Connect Secure VPN Vulnerability and Exploitation
The investigation done by Volexity of the initial breach  revealed that once the threat actor (UTA0178) got complete access to Connect Secure VPN appliances, they were known to largely use living off the land binaries (or native OS executable files) to further expand their control and laterally move within the network. The threat actor used more than one way to gain access to valid user credentials once they got access to the VPN appliance.

1. The threat actor had compromised JavaScript that was used by the SSL VPN login page of the VPN appliance to capture any user credentials that were entered in it.
2. The threat actor was also seen logging into various workstations and servers, dump the memory of LSASS process to disk using task manager, and later exfiltrated the dump for further credential harvesting using offline methods.
3. The threat actor also extracted credentials from active directory backup files, which they had exfiltrated after compressing it using 7-zip utility
4. Lastly, they had also used an open source script from Github to dump credentials from target’s backup software.

Having harvested credentials in multiple ways the threat actor also deployed three different versions of webshells in the target network to make sure that they are able to retain a strong foothold on to the target’s network.

1. First webshell was the result of modified existing web components of VPN appliance with name compcheckresult.cgi which would help threat actors execute commands from remote.
2. Other two webshells were modified versions of the single basic webshell code, where the first one was used on systems that were internet facing, and the second one was configured in a way to provide threat actor access to non-internet facing systems.

Detection, Mitigations and Patches
While there are several indicators of compromise released so far for performing detection of infected VPN applications, we will focus only on vendor recommended approaches for detecting, mitigating and patching a breached VPN appliance.

Mitigation XML
It was noticed that, Ivanti had released Mitigation XML immediately after vulnerabilities were disclosed, these XML based mitigation technique, prevented threat actors from abusing known vulnerabilities. However, as published in the blog post from Veloxity, if old backups were imported post application of this XML based mitigation, the devices again become vulnerable, and there have been instances where devices again got compromised due to such implementation of XML based mitigation.

Integrity Checker
Unlike the XML based mitigation method mentioned above, integrity checker is a detection mechanism which helps enterprises check if their VPN appliance has been compromised. While Ivanti appliances do provide internal integrity checker tool that are built into appliance as part of security tools, Ivanti has recommended against relying on output of internal integrity checker as they had noticed threat actors also trying to compromise logic of the tool. Hence they had suggested enterprises to use external integrity checker tool which they had published on 20th Jan 2024. However, in one of their KB articles they had also warned customers that if the threat actor had returned the appliance in its original state after compromising it, the external integrity checker also fails to detect the compromise. Ivanti had also cautioned their customers that were planning to use this integrity checker, that they must take complete backup of appliance, including memory. As running integrity checker tool forces the appliance to reboot which results in loss of valuable forensic evidences that may be needed later, should the integrity checker shows appliance as compromised.

Hardening appliance
Other than XML Mitigation tool based hardening, Ivanti recommended a few more hardening configurations to their customers which enables them to either protect the appliance or detect the compromise faster than they otherwise could have detected.

1. Enabling unauthenticated requests logging which is disabled by default, this provides more telemetry to your SIEM solution which can be further leveraged to write better detection rules and generate alerts in case of suspected compromise.
2. Enable logging of various appliances activities to an external logging server rather than maintain logs within the appliance itself.
3. Lockdown administrative access to internal or management interfaces only, and make sure that admin access on external port is disabled which is the default setting anyway.

Patching
Ivanti had faced significant challenges in providing timely and quality patches that could fix these vulnerabilities. This can be easily noticed by following the sequence of events that had occurred while publishing  patches to their customers.

1. Along with their vulnerability disclosure back in 10th Jan, Ivanti had promised to release patches on 22nd Jan, while releasing patches after 12 days of known exploitation itself brings down their customer’s confidence on their abilities, but they failed to meet even that stretched deadline.
2. Later Ivanti released first patch on 31st January after long wait, while many KB articles of Ivanti refer to patches released on “31st January and 1st February” as of writing this we didn’t find any evidence of any actual patch being released on 1st Feb 2024, but as per KB article from Ivanti, they had just updated “Updated patch release information” on 1st Feb and had recommended their customers to perform factory reset of VPN appliances.
3. However, as if this was not enough, Ivanti had disclosed 2 more vulnerabilities on 31st January. We will be covering details about other vulnerabilities in upcoming blog posts in this topic.
4. Later again on 8th February, Ivanti re-released patches to fix same two vulnerabilities, which they had claimed as patched on 31st Jan itself, forcing their customers to re-patch their appliances with newer fixes. 

The Saga Continues..
Despite critical patches were released on 8th Feb, the saga of Ivanti vulnerabilities, their exploitation and fixes still continues, we will cover this and more in upcoming part of this blog post.