Logs & System Investigation

/var/log, journalctl, who/last/history - covering both the attacker's and defender's view.

Medium 16 minlogsjournalctlforensics

Logs are a system's memory. They record who connected, what commands ran, which files changed, and when authentication failed. As a defender, logs are your intrusion detection system. As an attacker on an authorized engagement, knowing what logs record - and what they miss - is essential for understanding your own footprint.

The /var/log directory

Most system logs live under /var/log. Start there whenever you're investigating a system.

ls -lt /var/log/        # List by modification time - newest first
ls -lh /var/log/*.log   # Log files with human-readable sizes

kali@vr4cs: ~

Logs rotate automatically - .1 is yesterday, .2.gz is the week before, etc. If you're investigating an incident from last week, check the compressed archives too.

Key log files

Log file	What it records
`/var/log/auth.log`	All authentication events: SSH logins, sudo usage, PAM
`/var/log/syslog`	General system messages
`/var/log/kern.log`	Kernel messages (driver errors, hardware events)
`/var/log/apache2/access.log`	HTTP requests to Apache
`/var/log/apache2/error.log`	Apache errors
`/var/log/nginx/access.log`	HTTP requests to Nginx
`/var/log/mysql/error.log`	MySQL errors and queries (if query logging is on)
`/var/log/dpkg.log`	Package installs and removals
`/var/log/faillog`	Failed login attempts (read with `faillog` command)
`/var/log/wtmp`	Login history (read with `last`)
`/var/log/btmp`	Failed logins (read with `lastb`)

auth.log - the security heartbeat

/var/log/auth.log is usually the first place you look after a suspected intrusion.

kali@vr4cs: ~

In a single glance: alice logged in with a key, then ran sudo bash (suspicious - who runs /bin/bash with sudo?), and an external IP tried root three times and then succeeded. That's a brute-force followed by a successful login - an active incident.

Parsing auth.log with text tools

# All successful SSH logins
grep "Accepted" /var/log/auth.log
 
# All failed SSH logins (brute force detection)
grep "Failed password" /var/log/auth.log | awk '{print $11}' | sort | uniq -c | sort -rn
 
# All sudo commands run on the system
grep "sudo:" /var/log/auth.log | grep "COMMAND"
 
# Find who ran bash/sh as root via sudo
grep "COMMAND=.*\/bin\/\(bash\|sh\|zsh\)" /var/log/auth.log

kali@vr4cs: ~

IP 203.0.113.7 made 245 failed attempts - textbook SSH brute force.

journalctl - systemd's log aggregator

Modern Linux systems (running systemd) store logs in a binary journal, queried with journalctl.

journalctl                           # All journal entries (very verbose)
journalctl -n 50                     # Last 50 lines
journalctl -f                        # Follow (like tail -f)
journalctl -u ssh                    # Logs for the SSH service
journalctl -u apache2                # Logs for Apache
journalctl --since "2024-05-12 10:00" --until "2024-05-12 11:00"
journalctl -p err                    # Only error-level and above
journalctl -p warn -n 100            # Last 100 warnings and errors
journalctl _UID=1000                 # Logs from a specific user ID
journalctl -b                        # Logs since last boot
journalctl -b -1                     # Logs from previous boot

kali@vr4cs: ~

Who is logged in right now?

who          # Users currently logged in (tty/pts, time, IP)
w            # Like who, but shows what they're doing
last         # Login history (reads /var/log/wtmp)
last -n 20   # Last 20 login events
lastb        # Failed login history (reads /var/log/btmp, requires root)
last reboot  # System reboot history

kali@vr4cs: ~

Root is logged in from an external IP - after a brute-force - and is actively running bash. That's an active compromise.

Command history

Bash saves commands to ~/.bash_history. By default it stores the last 1000-2000 commands.

history                      # Print current session history
cat ~/.bash_history          # Read the saved file directly
cat /home/*/.bash_history    # Root can read all users' histories

kali@vr4cs: ~

This history tells a story: alice read the database config, connected to MySQL, read /etc/shadow, and then downloaded and ran a privilege escalation script. The attacker used Alice's account and ran standard post-exploitation steps.

History limitations (the attacker's view)

export HISTSIZE=0          # Set this before a session to disable history
export HISTFILE=/dev/null  # Redirect history to the void
unset HISTFILE             # Unset the history file variable
history -c                 # Clear in-memory history
cat /dev/null > ~/.bash_history  # Wipe the history file

History evasion is a forensic red flag

Clearing or disabling history is itself suspicious. Forensic investigators and SIEM tools look for history file truncation events, and many systems ship with PROMPT_COMMAND hooks that write to immutable audit logs regardless of HISTFILE. Assume your commands are being recorded somewhere.

The defender's playbook

Detecting SSH brute force

# Count failed logins per IP, sorted
grep "Failed password" /var/log/auth.log \
  | grep -oP 'from \K[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+' \
  | sort | uniq -c | sort -rn
 
# Find IPs that failed then succeeded (compromise indicator)
failed_ips=$(grep "Failed password" /var/log/auth.log | grep -oP 'from \K\S+' | sort -u)
for ip in $failed_ips; do
  if grep -q "Accepted.*$ip" /var/log/auth.log; then
    echo "[!] $ip failed then succeeded - possible brute force compromise"
  fi
done

Detecting suspicious sudo usage

# Who ran a shell as root?
grep "COMMAND" /var/log/auth.log | grep -E "COMMAND=.*(bash|sh|python|perl|ruby|nc|ncat)"

Detecting new users created

grep "useradd\|adduser" /var/log/auth.log
grep "new user" /var/log/auth.log

Checking for crontab modifications

grep "crontab" /var/log/syslog
grep "CRON" /var/log/syslog | grep -v "CMD"   # non-job cron events

Auditing recently modified files

# Files modified in the last 24 hours outside home directories
find / -mtime -1 -type f 2>/dev/null \
  | grep -v "^/proc\|^/sys\|^/run\|^/dev"
 
# Recently installed packages
grep " install " /var/log/dpkg.log | tail -20

Setting up persistent logging - auditd

For serious monitoring, install auditd - the Linux audit daemon. It can log every file access, exec, network connection, and user action at the kernel level.

sudo apt install auditd
 
# Watch all writes to /etc/passwd
sudo auditctl -w /etc/passwd -p w -k passwd_watch
 
# Watch all executions of suspicious tools
sudo auditctl -a always,exit -F arch=b64 -S execve -k exec_audit
 
# Read audit logs
sudo ausearch -k passwd_watch
sudo aureport --auth

Logs are only useful if you're watching them

Configure log forwarding to a central SIEM (Splunk, Elastic, Graylog). An attacker who has root can delete local logs - but they can't delete logs that were already shipped off the machine.

Key takeaways

/var/log/auth.log records all authentication events - SSH logins, sudo commands, PAM activity.
journalctl -u servicename queries the systemd journal for a specific service.
w and last show who is logged in now and historically - cross-reference with auth.log for anomalies.
~/.bash_history is the attacker's accidental diary - read it for every user on a compromised system.
The defender pattern: find IPs that failed then succeeded (grep "Failed"→grep "Accepted"), and find unexpected sudo shell invocations.
Forward logs to an external SIEM before an attacker can clear them. Local logs are evidence, not guarantees.

Linux Privilege Escalation