OS-Level Forensics
Incident Response & Digital Investigation
Master the art of investigating compromised Linux systems. Learn to analyze kernel-level indicators, preserve evidence, conduct root cause analysis, and respond to security incidents. Discover advanced forensic techniques for detecting rootkits, analyzing malicious activity, and reconstructing attack timelines. Complete the trilogy with enterprise incident handling strategies.
Linux Forensic Fundamentals
Building Blocks of Digital Investigation
Incident Response at OS Level
From Detection to Containment
🚨 Phase 1: Detection & Verification
Incident begins with alert or observation. Alert from monitoring system: unusual process, failed authentication spike, filesystem changes, network connection to known bad IP. Your job: verify it's real, not false positive. Check the source: is monitoring correct? Could it be misconfiguration? Verify by examining underlying data directly. If alert says "process using 100% CPU," check with ps and top. If alert says "unauthorized file change," check actual file and timestamp.
⛔ Phase 2: Containment (Conceptual)
Once incident confirmed, contain damage immediately. Isolation strategy depends on situation. Suspect user account compromised? Change password, revoke SSH keys, kill user's active sessions. Suspect service compromised? Stop service, don't delete anything yet (evidence preservation). Suspect entire system compromised? Isolate from network (disconnect network cable or firewall block), keep power on (memory contains evidence). Containment's goal: prevent attacker from spreading, accessing other systems, or deleting evidence. Balance speed (stop attack immediately) with evidence preservation (don't destroy forensic data).
🔒 Phase 3: Evidence Preservation
Before analysis, secure evidence chain. Live evidence (memory, process state, active connections) vanishes when system powers down. Capture: running processes and open files (lsof), network connections (netstat/ss), memory dump (if suspected malware in memory), system state snapshots. Hash critical files before touching them: sha256sum file > file.sha256. Document everything: when evidence collected, who collected it, how collected, tool versions used. Create forensic copy of filesystem: dd if=/dev/sda of=disk-image.img. Never work on original, only copy.
🔎 Phase 4: Analysis & Investigation
With evidence preserved, analyze offline. Examine logs for attack timeline. Identify what happened, when, and by whom. Find malicious files and decode them. Trace process execution from initial compromise through attacker actions. Document findings: what was compromised, how attack occurred, when, evidence trail supporting conclusions. Analysis might reveal: attacker accessed system via known CVE, installed backdoor, exfiltrated data, covered tracks by deleting logs.
🧹 Phase 5: Remediation & Recovery
After analysis complete, remove attacker access and recover system. Update kernel with patches that closed vulnerability attacker exploited. Remove malicious accounts, files, processes. Review and update access controls. If system severely compromised, rebuild from clean image. Before bringing system back online, verify no malware remnants exist. Run filesystem integrity checks. Retest security controls. Monitor closely post-recovery for re-compromise indicators.
Kernel-Level Indicators
Detecting Rootkits and Malicious Behavior
🔧 Suspicious Kernel Module Behavior
Kernel modules (drivers) run in privileged mode. Malicious modules (rootkits) hide attacker presence:
- Hidden Processes: Rootkit intercepts process listing syscalls. 'ps' shows processes attacker allows you to see. Hidden processes not shown. Forensic indicator: PIDs in /proc/stat don't match ps output. Use /proc directly rather than tools: cat /proc/*/status | grep Name. Hidden processes won't be there
- Hidden Files: Rootkit hides malware files. 'ls' doesn't show them but raw filesystem does. Indicator: filesystem size (du) doesn't match visible files (ls -la). Difference = hidden files
- Hidden Network Connections: Rootkit hides backdoor connections. 'netstat' shows attacker-allowed connections. Hidden connections don't appear. Indicator: /proc/net/tcp raw format shows connections netstat hides. Compare outputs
- File Hooks: Rootkit intercepts file access. Reading file through system calls (cat) shows attacker-hidden version. Reading raw disk directly shows truth. Indicator: file content differs depending on how you read it
- System Call Interception: Rootkit replaces syscall handlers. Legitimate syscalls redirected to rootkit code. Forensic indicator: syscall timing anomalies, unexpected system behavior, errors that shouldn't occur
🎯 System Integrity Verification Concepts
Detect system compromise through integrity checking:
- File Hashing Baseline: Before system goes into production, compute SHA256 hash of all system files. Store hashes securely (readonly filesystem, separate server). Periodically re-hash all files, compare to baseline. Any hash mismatch = file changed. Could be legitimate update or compromise
- Package Manager Verification: System packages have embedded hashes. 'apt verify-all-packages' checks if installed files match package manifest. Missing tool = likely deleted by attacker. Changed files = modification
- Immutable Bit: Mark critical files immutable: chattr +i /etc/passwd. Even root can't modify. Forensic value: if immutable file changed, indicates kernel-level tampering or filesystem bypass
- SELinux/AppArmor Denied Operations: MAC systems log denied syscalls. Spike in denials indicates attack. Attacker trying to access files policy forbids. Analysis: what's attacker trying to access? Why forbidden? Gives insight into attack chain
- System Call Auditing: auditd logs detailed syscalls. Configure to audit sensitive operations: file modifications in /etc, process creation, network connections. Audit log shows exactly what occurred when
| Indicator | Forensic Value | Detection Method | Rootkit Bypass Difficulty |
|---|---|---|---|
| Process List Mismatch | Hidden processes indicate rootkit | Compare ps vs /proc vs lsof | Medium (raw /proc bypass difficult) |
| File Hash Change | Modified files = compromise | Compare SHA256 to baseline | High (requires disk-level tampering) |
| Suspicious Network Conn | Outbound backdoor channel | Raw /proc/net/tcp analysis | Medium (network layer detection) |
| SELinux Denials | Policy violations = attacks | ausearch -m AVC audit logs | High (policy prevents attack) |
| Audit Log Gaps | Intentional log deletion/tampering | Check auditd sequence numbers | High (requires kernel compromise) |
🛡️ Investigation Workflow Example
Scenario: Suspected rootkit installed via kernel exploit. How to investigate?
- Step 1: Collect Live Data - Don't reboot (rootkit might not persist across reboot). Capture: ps aux, lsof, netstat, dmesg, kernel log. Dump memory if possible. This is snapshot of current state
- Step 2: Offline Analysis - Boot from forensic USB with clean OS. Mount compromised disk read-only. Compare rootkit-collected data to raw filesystem reads. Discrepancies indicate rootkit presence
- Step 3: Module Detection - Run 'lsmod' to list loaded kernel modules. Known rootkits have signatures—check against threat intelligence. Unknown modules suspicious. Check loaded module source: lsmod shows file path. Does it exist? Is it signed?
- Step 4: Syscall Analysis - auditd logs show syscalls rootkit intercepted. Pattern analysis: what syscalls modified? What operations bypassed? Identifies rootkit hooks
- Step 5: Root Cause - Determine how rootkit was installed. Likely: kernel CVE exploit. Check kernel version against CVE database. Find corresponding audit entries showing exploitation attempt
Enterprise Incident Handling
Professional Investigation & Documentation
🔍 Root Cause Analysis Framework
Professional incident investigation requires systematic root cause analysis:
- 5 Whys Technique: Start with symptom. Ask "why did it happen?" Each answer generates new "why" question. Continue until reaching true root cause. Example: "Attacker accessed system" → Why? "Used unpatched CVE" → Why? "Patch wasn't applied" → Why? "Patching process broken" → Why? "No patch management policy" → Root cause: organizational gap
- Timeline Reconstruction: Create detailed timeline of attack. Extract timestamps from logs, filesystem, audit data, memory dump. Chronological sequence: when did attacker first access? What was first command? When did persistence established? When discovered? Timeline might show vulnerability present for months before exploitation
- Scope Assessment: Determine full breach scope. Which systems affected? What data accessed? How many users compromised? If attacker on one system, likely on others. Scan similar systems for indicators. Log analysis shows if attacker pivoted to other systems
- Attribution Attempt: Try to identify attacker. APT groups have signatures: toolsets, TTPs (tactics/techniques/procedures), targets. Compare indicators to known groups. Attribution difficult—attackers use stolen tools and cover tracks. Avoid over-confident attribution
- Remediation Effectiveness: After fixes applied, verify they worked. Did patch installation actually happen? Are systems rebooted? Is monitoring enabled to detect re-exploitation? Verify security gaps causing breach actually closed
📊 Documentation and Reporting
Professional incident investigation produces thorough documentation:
- Executive Summary: One-page high-level summary. What happened? When? Impact? What was done? Designed for non-technical leadership who need facts without technical details
- Incident Details: Comprehensive technical report. Timeline of events. Evidence collected. Findings and conclusions. Rootkits detected? Accounts compromised? Data exfiltration? Evidence cited for each conclusion
- Evidence Chain of Custody: Track evidence from collection to analysis. Who collected? When? Where stored? Any handling that might compromise evidence? Required for legal proceedings. Digital signature/hash verification ensures evidence integrity
- Forensic Methodology: Describe investigation process. Tools used and versions. Commands executed. Results obtained. Reproducibility matters—other investigators should be able to repeat analysis and reach same conclusions
- Recommendations: Based on root cause, propose fixes. Patch policies? Vulnerability management? Monitoring gaps? User training? Recommendations address prevention of similar breaches. Prioritize by impact and feasibility
- Lessons Learned Session: Team meeting after incident resolved. What worked well? What could improve? How to detect earlier? Incidents are learning opportunities. Document improvements for future readiness
🎯 Professional IR Best Practices
Enterprise incident response teams follow established standards:
- NIST IR Framework: National Institute of Standards and Technology incident response standard. Four phases: Preparation (readiness), Detection & Analysis, Containment/Eradication/Recovery, Post-Incident Activity. Used by US government and many enterprises
- Incident Classification: Categorize by severity. Severity 1 (critical): data breach, ransomware, system down. Severity 2: unauthorized access contained, some data risk. Severity 3: minor vulnerability, quickly patched. Classification determines response urgency
- Escalation Procedures: Clear escalation chain. Team lead → Security Manager → CISO → Executive Leadership. Each level knows when to escalate. Ensures appropriate resources and visibility at right level
- Communication Plan: Internal stakeholders (IT, security, management) need updates on incident status. External stakeholders (customers, regulators, press) might need notification. Communication plan specifies who says what and when
- Legal & Compliance: Incident investigation might trigger legal obligations. GDPR requires breach notification. Litigation hold preserves evidence for lawsuits. Legal team involved early to ensure investigation preserves admissibility
- Incident Metrics: Track: Time to detect incident, Time to contain, Time to remediate, Data affected count, Systems affected count. Metrics show program effectiveness. Trend over time indicates improvement
Forensics & IR Resources
Industry Standards & Professional Tools
📚 Incident Response Frameworks & Standards
- NIST SP 800-61: Computer Security Incident Handling Guide - Federal standard for incident response procedures
- SANS Incident Handler's Handbook - Practical IR procedures and best practices
- FBI Cyber Division Incident Response - Federal law enforcement IR guidance
- CISA Incident Response Assistance - US cybersecurity agency IR support and resources
🔬 Forensic Tools & Analysis
- Volatility 3 - Advanced memory forensics and malware analysis
- The Sleuth Kit - Filesystem forensics and disk analysis
- Kali Linux - Penetration testing and forensics platform
- RegRipper - Windows registry forensics (for cross-platform investigations)
- Foremost - Data recovery and carving tool
🌐 Linux-Specific Forensics
- Linux Man Pages Online - Complete Linux command reference
- Linux Kernel Administration Guide - Kernel-level investigation techniques
- Linux Audit Framework (auditd) - System call auditing and logging
- AIDE (Advanced Intrusion Detection Environment) - File integrity monitoring tool
- osquery - Operating system instrumentation and forensics
⚠️ Threat Intelligence & CVE Databases
- NVD (National Vulnerability Database) - Official CVE database and vulnerability analysis
- CVE MITRE Database - Common Vulnerabilities and Exposures list
- Exploit Database - Public exploits and proof-of-concepts
- MITRE ATT&CK Framework - Tactics, techniques, and procedures database for threat analysis
Claim Your Certificate
Complete Your Journey as a Linux Security Expert
Course Curriculum Mastered:
✅ Module 1: Kernel Security Foundations & Attack Vectors
✅ Module 2: Privilege Escalation Defense & System Hardening
✅ Module 3: OS-Level Forensics & Incident Response
Your certificate verifies expertise in Linux kernel security, defensive hardening strategies, and forensic investigation. Share with your professional network and employers to demonstrate your advanced cybersecurity knowledge.
Program Status: ✅ All Modules Complete
Certificate Ready for Download & Verification