X-Force uncovers global NetScaler Gateway credential harvesting campaign



X-Force uncovers global NetScaler Gateway credential harvesting campaign






























advisory document containing guidance on detection, incident response, mitigations and validating security controls. However, through multiple incident response investigations, X-Force discovered a new exploitation artifact related to CVE2-2023-3519 and developed additional guidance to be used in conjunction with CISA’s detection and response recommendations.

This post will cover the initial incident that led to uncovering the larger campaign, the credential harvesting campaign, as well as the new artifact, considerations and recommendations for responding to remediating an incident involving CVE-2023-3519.

Incident overview

X-Force identified the campaign through an incident response engagement where a client had discovered the script after investigating reports of slow authentications on the NetScaler device. The script which is appended to the legitimate “index.html” file loads an additional remote JavaScript file that attaches a function to the “Log On” element in the VPN authentication page that collects the username and password information and sends it to a remote server during authentication.

As part of the initial exploit chain, the attackers sent a web request to “/gwtest/formssso? event=start&target=” triggering the memory corruption documented in CVE-2023-3519 to write a simple PHP web shell to /netscaler/ns_gui/vpn. With interactive access established through the PHP web shell, the attacker retrieved the contents of the “ns.conf” file on the device and appended custom HTML code to “index.html” which references a remote JavaScript file hosted on attacker-controlled infrastructure.

Figure 1_exploit_adc.png

To facilitate the credential harvesting, the JavaScript code appended to “index.html” retrieves and executes additional JavaScript code that attaches a custom function to the “Log_On” element which collects the form data containing the username and password information and sends it to a remote host through a HTTP POST method upon authentication.

Figure 2_cred_harvest.png

Larger credential harvesting campaign

From that initial engagement, X-Force identified multiple domains created by the threat actor – jscloud[.]ink, jscloud[.]live, jscloud[.]biz, jscdn[.]biz, and cloudjs[.]live – registered on August 5th, 6th and 14th, and leveraging Cloudflare to mask where the domains were hosted. After identifying the threat actor’s C2, X-Force was able to identify almost 600 unique victim IP addresses hosting modified NetScaler Gateway login pages, with concentrations in the United States and Europe. The earliest modification time stamp X-Force has identified for NetScaler Gateway login pages is on August 11th, 2023, although the campaign could have begun closer to when the domains were registered.

Figure 3_optic-geospatial-2023-09.png

Despite variations in the URL, all domains hosted an almost identical JavaScript file, with the only difference being the C2 domain listed in the file, and all captured credentials were sent to the same URI/items/accounts/“. X-Force has also observed in some instances the threat actor appending the same URL, or a URL using one of the other domains, to the same victim login page, indicating this threat actor is likely opportunistically compromising vulnerable NetScaler Gateways.

Figure 4_JavaScript file.png

While public reporting has highlighted how various threat actors, including suspected Chinese threat actors and FIN8, have exploited these vulnerabilities, X-Force has not observed follow-on activity and is unable to attribute this campaign at this time.

CVE-2023-3519 detection and investigation guidance

Considerations for evidence collection from NetScaler device:

In the default configuration, the NetScaler device will rewrite the “ns_gui” folder upon boot resulting in the directory being destroyed during shutdown. Throughout various investigations, X-Force has recovered valuable evidence from the “ns_gui” directory including samples of web shells and modified versions of “index.html”. Organizations should be careful not to shut down the device prior to collecting an image or other evidence.

New artifact for CVE-2023-3519 detection: NetScaler application crash logs

Through X-Force incident response engagements involving CVE-2023-3519, X-Force analysts identified that the NetScaler Packet Processing Engine (NSPPE) crash files can contain evidence of the exploitation of the vulnerability. The crash files are located with “/var/core/<number>/NSPPE*”. Similar to the default log files on the NetScaler device, the crash files are stored in “.gz” archives so they will need to be extracted prior to analysis.

Crash file path example: /var/core/6/NSPPE-01-9502.gz

The crash files are not human readable by default however X-Force discovered that the crash files do contain string data that can be extracted using strings, PowerShell or any other tool that can print the strings of printable characters in files.

X-Forced observed that the NSPPE crash file timestamps aligned with the filesystem timestamps of the PHP web shells created through exploitation. In other instances, X-Force was able to recover commands being passed to the web shells as part of post-exploitation activities.

Figure_5_Crash file timestamps.png

Figure 6_Commands to web shells.png

Note on NetScaler log backups:

X-Force has observed that the default NetScaler audit configuration is to leverage circular logging and retain the last 25 log files with a maximum size of 100 Kb. When logs are rolled, NetScaler will retain older log files in “.gz” archives. X-Force has observed that some of the available CVE-2023-3519 detection tools available on the Internet, do not consider the log data within the “.gz” archives. Organizations should ensure to extract the log files from the archives or leverage a tool such as “zgrep” which can search within compressed files.

Figure 7_Compressed files.png

Considerations for detection strategies within NetScaler access logs:

X-Force recreated the exploit for CVE-2023-3519 by sending a GET request to “https://<VulnerableGateway>/gwtest/formssso?event=start&target=” however X-Force was not able to recover a record of any of the web requests associated with exploitation attempts in the access logs. It is not clear whether the lack of a log entry for the connection to the “formssso” endpoint is due to a configuration issue on the X-Force test instance or if the “formssso” does not log connections by design.

X-Force recommends that clients analyze the following log sources for evidence of post-exploitation activity in the following files with a particular focus on identifying entries indicative of interacting with a web shell:

  • /var/log/httpaccess.log
  • /var/log/httperror.log
  • /var/log/httpaccess-vpn.log

X-Force recommends that organizations assess their access logs for POST/GET requests and anomalous PHP files. Examples of post-exploitation interactions with a PHP web shell observed by X-Force:

Figure 8_Post-exploit interactions.png

While during exploitation tests, X-Force was not able to recover the details of the commands executed via the PHP web shells recovered from incident response engagements with the access logs, X-Force still recommends organizations assess their NetScaler access logs for evidence of command execution in the event different web shells were used.

Considerations for detection strategies within NetScaler command history logs:

The CISA advisory recommends organizations assess bash.log and sh.log for evidence of malicious activity leveraging the following keywords:

  • database.php
  • ns_gui/vpn
  • /flash/nsconfig/keys/updated
  • LDAPTLS_REQCERT
  • ldapsearch
  • openssl + salt

In addition to the CISA advisory, X-Force recommends organizations also consider assessing “/var/log/notice.log”, “/var/log/bash.log” and” /var/log/sh.log” (including the associated rollover “.gz” archives) for signs of post-exploitation activity using the following additional keywords:

  • Whoami
  • base64 –decode
  • /flash/Nsconfig/keys
  • &>> index.html
  • echo <?php
  • echo <script
  • /nsconfig/ns.conf

It is important for organizations to analyze command history logs in the correct context of the attacker’s operations. Evidence gathered from command history within the context of an attack involving CVE-2023-3519 will be focused on post-exploitation activity. Organizations should analyze process execution data sources (including command history logs on the device) for commands associated with reconnaissance, credential harvesting, lateral movement and downloading/uploading of data and not restrict their assessment to just what is provided within the keywords.

Considerations for remediation:

As noted in the CISA advisory, attackers were observed viewing NetScaler configuration files /flash/nsconfig/keys/updated/* and /nsconfig/ns.conf which “contain an encrypted password that can be decrypted by the key stored on the ADC appliance”.

X-Force also noted that there were multiple credentials and certificates stored in the NetScaler configuration files so organizations should consider changing certificates as well as all passwords as part of incident remediation.

Indicators

Indicator Indicator Type Context
jscloud[.]ink Domain C2
jscloud[.]live Domain C2
jscloud[.]biz Domain C2
jscdn[.]biz Domain C2
cloudjs[.]live Domain C2
cloud-js[.]cloud Domain C2

Scroll to view full table

References

To learn how IBM Security X-Force can help with anything regarding cybersecurity including incident response, threat intelligence or offensive security services, schedule a meeting here: IBM Security X-Force Scheduler.

If you are experiencing cybersecurity issues or an incident, contact IBM Security X-Force for help: US hotline 1-888-241-9812 | Global hotline (+001) 312-212-8034.

More from Defensive Security

A diverse group of professionals having a discussion in a modern cyber defense office with large digital screen wall
A diverse group of professionals having a discussion in a modern cyber defense office with large digital screen wall

Why federal agencies need a mission-centered cyber response

4 min read – Cybersecurity continues to be a top focus for government agencies with new cybersecurity requirements. Threats in recent years have crossed from the digital world to the physical and even involved critical infrastructure, such as the cyberattack on SolarWinds and the Colonial Pipeline ransomware attack. According to the IBM Cost of a Data Breach 2023 Report, a breach in the public sector, which includes government agencies, is up to $2.6 million from $2.07 million in 2022. Government agencies need to move…

data transfer concept, internet server connection icon
data transfer concept, internet server connection icon

X-Force releases detection & response framework for managed file transfer software

5 min read – How AI can help defenders scale detection guidance for enterprise software tools If we look back at mass exploitation events that shook the security industry like Log4j, Atlassian, and Microsoft Exchange when these solutions were actively being exploited by attackers, the exploits may have been associated with a different CVE, but the detection and response guidance being released by the various security vendors had many similarities (e.g., Log4shell vs. Log4j2 vs. MOVEit vs. Spring4Shell vs. Microsoft Exchange vs. ProxyShell vs.…

An anonymous man working at a desk on a computer with a man and two coffee cups next to him
An anonymous man working at a desk on a computer with a man and two coffee cups next to him

X-Force certified containment: Responding to AD CS attacks

6 min read – This post was made possible through the contributions of Joseph Spero and Thanassis Diogos. In June 2023, IBM Security X-Force responded to an incident where a client had received alerts from their security tooling regarding potential malicious activity originating from a system within their network targeting a domain controller. X-Force analysis revealed that an attacker gained access to the client network through a VPN connection using a third-party IT management account. The IT management account had multi-factor authentication (MFA) disabled…

Topic updates

Get email updates and stay ahead of the latest threats to the security landscape, thought leadership and research.

Subscribe today

Source: https://securityintelligence.com/posts/x-force-uncovers-global-netscaler-gateway-credential-harvesting-campaign/

Tags: LEARN, VPN, CLOUD, BREACH, TOOL, GOVERNMENT, VULNERABILITY, PASSWORD