CheckMesh: Hidden Threats in Your FW

Contents: 1

 

CheckMesh: Unveiling the Hidden Threats in Your Firewall

At HackersEye, tackling the most complex cybersecurity challenges is a daily routine and cornerstone of our operations. As Israel faces ongoing geopolitical tensions and an escalation in cyber incidents due to the current state of conflict, our team stands at the forefront, defending critical infrastructure against sophisticated and persistent threats.

 

"Our highly skilled cybersecurity experts specialize in identifying and mitigating advanced persistent threats (APTs), utilizing state-of-the-art tools and methodologies to stay ahead of malicious actors," says Tal Raveh, CEO of HackersEye.

 

Recently, we encountered an exceptionally advanced cyber-attack targeting an Israeli enterprise utilizing a Check Point firewall. Dubbed as CheckMesh, for obvious reason the attack successfully deploys & executes a Mesh agent on the CheckPoint firewall Linux system. The Attacker transformed the CheckPoint firewall into a stealthy Command and Control (C2) node using a malicious MeshAgent-based ELF implant. 

 

"The sophistication and persistence of this threat actor, likely an APT group rather than mere hacktivists, demonstrated their advanced capabilities in maintaining covert control over compromised systems, explains Yaron Adika, HackersEye CTO. "This case study not only underscores the advanced techniques employed by the threat actors but also highlights our meticulous approach to incident response, crisis management, and remediation."

 

Join us as we delve into the intricacies of this incident, uncovering the strategies employed by the attackers and the robust measures we implemented to safeguard our clients against such persistent threats.

 

"Our team of cybersecurity experts, equipped with extensive technical knowledge and a relentless dedication to excellence, is prepared to defend against the continually evolving landscape of cyber threats," Raveh adds. "Through detailed forensic analysis, real-time detection, and comprehensive incident response, HackersEye continues to lead the charge in protecting organizations from the increasing cyber warfare in today’s digital battlefield."

 

Explaining the breach - The story in a nutshell

The attack on this organization unfolded in a series of calculated steps, setting the stage for a significant breach:

• Late June: Potential compromise of Admin access.
  The attacker may have used one of several methods to gain admin access:
      1. Brute forcing.
      2. Password spraying.
      3. Leaked credentials.
      4. Exploiting a known CheckPoint vulnerability.
  There is no conclusive evidence indicating a specific method used by the attacker in order to gain access.
• Initial Connection via GAIA HTTP: In late June, the attacker began their operation by connecting to the GAIA web interface using the stolen credentials. This initial connection allowed them to gather more information and plan their next move.
• Upgrading to SSH CLI: With enough reconnaissance done, the attacker upgraded their access by initiating an SSH CLI session on the firewall. This move provided them with a more powerful and flexible control interface.
• Installing the C2 Agent: Once they had SSH access, the attacker deployed a malicious ELF implant. This C2 agent was installed and disguised as a legitimate process, enabling encrypted communication with their C2 server and ensuring persistent access.
• Operating as Root on the Linux Box: With the C2 agent in place, the attacker operated as root on the firewall's Linux system. This high-level access allowed them to manipulate the firewall's functions, maintain their foothold, and further their objectives within the compromised network.
• Domain users and lateral movement: with port forwarding abilities on the organizational FW, the attack obtained domain credentials via password spraying, Brute-Force, and other methods, and created a RDP tunnel to internal network servers and endpoints.

Threat Actor Attribution, TTPs, and Motivation

Threat Actor Attribution

The attack on this organization shows significant similarities to the activities attributed to LilacSquid APT group, Observed by CISCO Talos.

Below is a detailed comparison of the tactics, techniques, and procedures (TTPs) used in both cases, supporting the attribution of this sophisticated attack to LilacSquid.

 

Modus Operandi

The modus operandi of the LilacSquid APT group involves a multi-stage approach to compromise and persist within targeted networks:

1. Initial Exploitation: Exploiting vulnerabilities in public-facing applications or devices to gain initial access.
2. Persistence: Deploying persistent implants, such as MeshAgent, to maintain long-term access.
3. Credential Theft and Lateral Movement: Stealing credentials to move laterally within the network and escalate privileges.
4. C2 Communication: Establishing secure C2 channels using encrypted protocols like WebSocket over TLS (wss) to communicate with compromised devices.
5. Service Control: Using Windows service control commands (sc create, sc start) to manage malicious services and ensure persistence.

 

Tactics, Techniques, and Procedures (TTPs)

 

Example Comparison: SC Command

“SC” Execution on the internal network of the attacked organization:

Notice the service name is “IPSHelper”, that’s crucial for later on the investigation.

“SC” Execution performed by LilacSquid observed by CISCO Talos:

Motivation

The LilacSquid APT group is likely motivated by geopolitical and economic factors, aiming to gather intelligence and disrupt critical infrastructure. Their targeting of high-profile entities and use of sophisticated tools and techniques indicate a well-resourced and strategically driven operation, typical of state-sponsored actors. The similarity in TTPs between the recent Check Point firewall attack and known activities of LilacSquid suggests that the same motivations underpin both sets of activities, focusing on long-term access, data exfiltration, and potential sabotage of critical systems.

 

Understanding MeshAgent:

What is MeshAgent

MeshAgent is an open-source project available on GitHub, designed for legitimate remote management purposes. Its key functionalities include:

• Remote Shell Access: Allows administrators to execute shell commands remotely.
• File Transfer: Facilitates the transfer of files between the server and the managed device.
• Process Management: Enables remote monitoring and control of system processes.
• System Information: Collects and reports system details back to the management server.

These features, while useful for legitimate administration, also provide a powerful toolkit for an attacker once they gain control of the agent.

Technical goodies: Dissecting the Attack

The Initial Clues

Our investigation began with an unusual observation: an RDP connection to the domain controller originating from a NAT IP address associated with the GAIA firewall.

 

After a successful login, the user tried to install a service with a version of mesh-agent for windows, which was blocked by the crowdstrike edr.

Diving a bit more, we find that there is an external machine, not yet to be known, that is attacking our internal assets with credentials attacks such as password spraying, and brute force:

A Host named “ADMIN4025” that is not known and is not a part of the client’s internal network, raising suspicious on our side, taking us to the only place that can explain this kind of behavior, the organizational FW. (checkpoint)

 

These actions raised critical questions. Deciding to SSH into the firewall, we prepared to dissect it completely.

 

Following the breadcrumbs.

Upon gaining access to the FW system, we conducted a series of standard diagnostic commands, including ls, ps -auxf, and netstat -antop.

These initial actions revealed the presence of a particularly noteworthy directory named /private located in the root (/) path of the firewall.

Within this directory, we identified an executable file named IPHelper, which warranted further investigation.

The IPHelper folder, contained a bash script named meshinstall.sh

We start by examining the bash script to understand its purpose and actions.

The script was designed to automate the installation and ensure the persistence of the malicious ELF implant (MeshAgent).

Here’s a detailed breakdown of the script and its functionality:

• Initialization Information: The script includes init information that describes the service, including dependencies and run levels.
• Variables: Key variables like SCRIPT, RUNAS, PIDFILE, and LOGFILE are defined for later use.
• Start Function: This function checks if the service is already running by verifying the presence of a PID file and then starts the service if it's not running. The CMD variable runs the meshagent in a persistent loop to ensure it automatically restarts if it stops unexpectedly.
• Stop Function: This function stops the service by killing the process identified by the PID file.
• Restart Function: This simply calls the stop and start functions sequentially.
• Status Function: This checks if the service is running by verifying the process status through the PID file.
• Case Statement: This handles the command-line arguments to start, stop, restart, or check the status of the service.

Meshinstall.sh - b1b15e09ea98228203e110456d514327ce6b7438
“Usage: service meshagent {start|stop|restart|status}"

 

Finding the first execution:

Executing the bash install script with the C2 address, and a token, for the installation of the agent itself.

Deployment and Execution

Upon discovering the IPSHelper executable on the Check Point firewall, our investigation quickly pivoted to focus on this binary. Notably, the same process name had been earlier detected on a Windows machine, where it was blocked by CrowdStrike. Recognizing that IPSHelper is typically associated with a different firewall platform (Fortigate), we prioritized the examination of this ELF binary to understand its role in the current incident and to determine the extent of its involvement. This examination led to the discovery of the ELF implant on the Check Point firewall, revealing a sophisticated and persistent threat.

 

Technical Analysis: examining the ELF

During our investigation, we used several diagnostic commands to understand the state of the compromised firewall. Here’s a step-by-step breakdown of how we identified and analyzed the malicious ELF implant.

 

Initial Discovery

1. Process Analysis with ps auxf: We began by listing all running processes to identify any anomalies. The command revealed one suspicious processes: IPSHelper with the “start” argument. – That defently took our attention, as we remember this process name from fortigates, not checkpoint.

$ ps auxf

Network Connections with netstat

Next, we examined the network connections to understand if these processes were communicating externally.

We execute “netstat -antop”:

The IPSHelper process was listening on port 58885, which raised further suspicion, and had a live connection to the attacker-controlled server C2

Delving Deeper: MeshAgent

Within this directory, we found the IPSHelper ELF binary to confirm its legitimacy, we compared its hash with the official MeshAgent release from GitHub.

The hash matched the official release, indicating that the attacker utilized an unmodified version of MeshAgent.

Configuration File Inspection

The IPSHelper.msh configuration file provided crucial details about the implant’s operation.

Detailed Breakdown

1. MeshName=Remote

● Description: This parameter defines the name of the mesh network to which the agent belongs. In this instance, it is named "Remote".

● Significance: The name indicates the agent’s group or network, potentially used for organizational or categorization purposes within the attacker's infrastructure.

2. MeshType=2

● Description: This parameter specifies the type of mesh network. The value 2 typically indicates a specific configuration or operational mode defined by the MeshAgent framework.

● Significance: Understanding the type helps in assessing the operational characteristics and capabilities of the deployed agent.

3. MeshID=0xAFEEB2DF3591A4320E1860B*****

      ● Description: This is a unique identifier for the mesh network. It is represented as a hexadecimal string.

● Significance: The MeshID uniquely identifies the network, allowing the attacker to manage and communicate with specific groups of compromised devices.

4. ServerID=310FB318A0A0FF51CF91DA3F5D6******

● Description: This parameter provides a unique identifier for the C2 server managing the agent.

● Significance: The ServerID helps in distinguishing between different C2 servers, which could be crucial for load balancing or redundancy in the attacker’s infrastructure.

5. MeshServer=wss://api.gupdate.net:443/agent.ashx

● Description: This parameter specifies the WebSocket Secure (wss) URL for the Command and Control (C2) server. The agent communicates with the C2 server at api.gupdate.net on port 443, using the path /agent.ashx.

● Significance: The use of a secure WebSocket (wss) connection ensures encrypted communication between the agent and the C2 server, making it more challenging to intercept and analyze the traffic. The specific URL indicates the endpoint used for managing the compromised device.

 

Reviewing the C2, redirected us to a login page, with mesh-central, confirming our findings, and the C2:

Detections in VirusTotal, for the domain and IP of the C2:

 

Recommendations

Incident Response Recommendations

Tal Raveh, CEO of HackersEye, on the Importance of Specialized Incident Response Teams

"Dealing with advanced persistent threats (APTs) like those from the LilacSquid APT group requires a specialized Incident Response (IR) team," states Tal Raveh, CEO of HackersEye. "A dedicated IR team proficient in investigating Linux-based firewall systems is crucial for identifying, containing, and eradicating hidden threats. Their ability to perform comprehensive forensic analysis ensures a thorough understanding of an attack's scope, which is essential for effective remediation."

"An effective IR team must have advanced tools and methodologies for detection, containment, and eradication of threats," Raveh explains. "Their capability to quickly neutralize sophisticated threats is vital for protecting critical assets. This specialized expertise is indispensable in managing and mitigating complex cyber incidents."

"Engaging a specialized IR team significantly enhances an organization's security posture," Raveh concludes. "It ensures timely and effective response to cyber threats, reinforcing defenses against future attacks. Having a skilled IR team is fundamental for safeguarding against persistent and sophisticated cyber threats."

 

Detection, IOCs, and YARA rules:

YARA Rules for Detection

To assist in the initial stages of this investigation, we developed YARA rules aimed at detecting the MeshAgent ELF binary and its configuration file out of the box. These rules are specifically tailored for the Check Point firewall's Linux environment. It is important to note that these YARA rules are rudimentary and can be easily evaded by sophisticated attackers. Their primary purpose was to quickly scan and identify potential indicators of compromise at the onset of the incident. As the investigation progressed, these initial detections guided our deeper and more detailed forensic analysis.

rule MeshAgent_ELF
{
meta:
description = "Detects the CheckMesh attack"
author = "HackersEye"
date = "2024-07-04"
strings:
$elf_magic = { 7f 45 4c 46 02 01 01 00 }
$mesh_string1 = "meshcore/KVM/Linux/linux_kvm.c" ascii
$mesh_string2 = "meshcore: %s" ascii
$mesh_string3 = "meshcore/agentcore.c" ascii
$mesh_string4 = "meshagent" ascii
$mesh_string5 = "--meshServiceName=" ascii
$mesh_string6 = "/var/run/meshagent.pid" ascii
condition:
uint32(0) == 0x464c457f and
filesize < 10MB and
all of ($elf_magic, $mesh_string1, $mesh_string2, $mesh_string3, $mesh_string4,
$mesh_string5, $mesh_string6)
}

 

Msh configuration file yara:

rule MeshAgent_Config
{
meta:
description = "Detects the CheckMesh configuration file"
author = "HackersEye"
date = "2024-07-04"
strings:
$config_string1 = "MeshName=Remote" ascii
$config_string2 = "MeshType=2" ascii
$config_string3 = "MeshID=0x" ascii
$config_string4 = "ServerID=" ascii
$config_string5 = "MeshServer=wss://" ascii
$config_string6 = "\"agent\":\"Agent\"" ascii
$config_string7 = "\"install\":\"Install\"" ascii
$config_string8 = "\"setup\":\"Setup\"" ascii
condition:
filesize < 10KB and
all of ($config_string1, $config_string2, $config_string3, $config_string4, $config_string5) and
any of ($config_string6, $config_string7, $config_string8)
}

IOCs:

 

References

• Cisco Talos Blog on LilacSquid: LilacSquid: The stealthy trilogy of PurpleInk, InkBox and InkLoader (Cisco Talos Blog)​
• Analysis of MeshAgent Usage: GitHub - MeshCentral
• Detailed TTPs and Exploit Techniques: MITRE ATT&CK Framework