Ghosts in /proc: Manipulation and Timeline Corruption

hendryadrian.com

hendryadrian.com

Introduction

In our previous blog, “Hiding in plain sight: Techniques and defenses against `/proc` filesystem manipulation in Linux” we explored techniques for concealing malicious processes from forensics triage tools. Forensic analysts often rely on the Linux virtual filesystem /proc to enumerate processes, reconstruct timelines, and attribute activity to specific executables. Utilities such as ps, top, and various triage scripts extract process metadata from files located under /proc/<pid>/, including cmdline and stat. The integrity of these files is therefore critical to many incident response workflows.

Our research demonstrates how the /proc view can be manipulated by threat actors with privileged access, or certain types of malicious code, to create misleading forensic artifacts. By altering this in-kernel interface, a threat actor can obfuscate process identities by replacing displayed command-line arguments, and corrupt temporal data such as the recorded process start time, which undermines timeline analysis.

Key Discoveries

• /proc/<pid>/cmdline can be substituted or edited to present arbitrary command strings, complicating process identification.
• The starttime field in /proc/<pid>/stat can be altered to produce timeline discrepancies, including future timestamps that corrupt event correlation.
• Because manipulation requires elevated privileges or kernel modification, successful tampering is a strong indicator of serious host compromise.
• Reliance on /proc in isolation increases risk of triage false negatives and attribution errors; cross-validation with kernel and remote telemetry is necessary.

Who may find this blog interesting:

• Cybersecurity analysts
• Malware and Threat intelligence professionals
• Incident Responders
• Security Operation teams

Group-IB Threat Intelligence Portal:

Group-IB customers can access our Threat Intelligence portal for more information how attackers could manipulate the Linux /proc filesystem to hide malicious processes and distort forensic timelines.

Technical Walkthrough

Environment and Objective

The experiment was performed on a Linux host using a minimal test script (the “evasion” script) ran from an interactive shell. The objective was twofold:

• Demonstrate how substituting /proc/<pid>/cmdline can change the command string shown by ps/top.
• Highlight how editing the starttime field in /proc/<pid>/stat can produce temporally inconsistent (including “future”) process start times when converted to human-readable form.

Test Process: Small Interactive Script

The target process is a minimal script named evasion:

After launching the script, the process appears in process listings (example excerpt from ps aux):

Clone /proc Data for the Target process

We created a workspace and copy the live /proc/<pid> fields to the workspace:

After copying, the content of the workspace is empty until the artifacts are placed there. Our intention is to use this folder as the bind mount target later.

Bind-mount the Cloned Data Over /proc/<pid>

By bind-mounting /tmp/evil onto /proc/18316, the kernel-exported view of the process for userland utilities can be redirected to the substituted files:

After the bind, ps aux no longer reports the original ./evasion string because the process view has been replaced by the mounted artifacts. For example, the initial ps aux showed the process, but after mounting the workspace the ps output no longer shows evasion, only the grep process appears.

Place Edited Artifacts into the Mounted Folder

Next, the previously copied files are written into the mounted folder and edited. The following is an example of commands performed during our investigation.

At this point, userland tools reading /proc/18316/cmdline and /proc/18316/stat will read the versions stored in /tmp/evil. The evasion process is now backed by those substituted artifacts.

Demonstration: Replacing the Command String

The cmdline file in /tmp/evil was edited to contain a single token safeprocess:

After this substitution, ps aux shows the new command string for PID 18316:

And so, the live display of the process name was altered from /usr/bin/bash ./evasion to safeprocess purely by replacing the cmdline content exposed at /proc/18316/cmdline. This demonstrates how /proc-based identification can be trivially spoofed when adversaries control the exported files.

The Stat Artifact: Raw and Parsed

The stat file contains many numeric fields. For this experiment, the captured stat line for the process was:

Thanks to proc_pid_stat(5), we can translate those numeric fields into a more understandable order:

Interpreting these fields requires referring to proc(5); in particulat, field (22) is starttime, the time the process started after system boot, expressed in clock ticks.

To simplify interpretation, the experiment used a small helper program procstatto translate the numeric fields into human-readable form. The following is an example output:

The procstat tool converts the starttime clock-tick value into a readable timestamp and a relative age (for example, 698.43s), which is useful for timeline correlation.

Converting starttime into Absolute Times

The starttime value must be interpreted relative to system boot. In this experiment, the system’s uptime was captured as follows:

Using the numeric starttime value present in the original stat file (48156368), the report shows that the process started approximately 85 seconds before the observation point (calculation performed in the lab converts clock ticks to seconds via _SC_CLK_TCK and compares against uptime). The procstat tool then reported start_time: 10.12 22:06 (698.43s), indicating how the raw stat data is converted for human interpretation.

Tampering with starttime to Create Temporal Disinformation

To illustrate how the timeline is manipulated, the starttime field in /tmp/evil/stat was edited from 48156368 to 98156368:

After modifying the starttime and leaving the cmdline substitution in place, ps aux now reports the PID with a calendar date that lies in the future relative to the actual system time. During our tests, the system date/time read: Sun Oct 12 10:24:07 PM +07 2025. After changing starttime, the ps output displayed the process start date as Oct18 (i.e., a future date) for PID 18316.

This demonstrates how editing the starttime numeric field in the stat record can lead utilities to display misleading calendar timestamps — effectively “moving” a process into the future and corrupting timeline correlation.

Conclusion

The /proc pseudo-filesystem is a convenient conduit for process introspection but not an immutable evidence store in adversarial contexts. Privileged actors can manipulate /proc entries to hide identities and corrupt temporal data relied upon for timeline reconstruction. Effective detection requires treating /proc as one evidence source among many and validating process events against kernel-rooted telemetry and remote immutable logs.

Detection & Defense Recommendations

• Monitor mount and umount syscalls: Use tools such as auditd or eBPF to detect suspicious mounting activity.
• Alert on abnormal /proc mounts: Generate alerts if /proc/[pid] is mounted to tmpfs or any external filesystems.
• Cross-validate /proc data: Compare /proc entries against kernel and remote telemetry sources such as auditd, EDR kernel event streams, systemd journal timestamps, and remote logs to maintain a trusted timeline.
• Restrict privileged access: Apply the principle of least privilege, enforce RBAC, and allow only signed or authorized kernel modules to load.
• Implement immutable audit forwarding: Forward critical audit logs to remote, write-once storage to preserve evidence in case of hosts tampering.
• Enable kernel integrity controls:Enable secure boot, kernel module signing, and host attestation to ensure integrity where possible.
• Enhance triage tooling: Update triage scripts to flag inconsistencies between /proc and independent telemetry sources, and to treat /proc entries as “untrusted by default.”

Source: https://www.group-ib.com/blog/ghosts-in-proc/