linux
Teaches Linux filesystem, processes, systemd, permissions, SSH, networking, performance (CPU/memory/disk), SELinux/AppArmor, and production ops. Use when administering Linux hosts, debugging system is
SKILL.mdskills/extras/linux/SKILL.md
GitHubLinux
1. Overview
What it is: The OS kernel + userland that runs almost all cloud VMs, containers' host, and CI runners.
Why it matters for DevOps: Containers and Kubernetes sit on Linux primitives (cgroups, namespaces, netfilter). Debugging production almost always ends at processes, disks, memory, and sysctl.
When to use: Always — host hardening, capacity, incident response.
When NOT to treat as app platform: Prefer containers/PaaS for apps; bare metal Linux for data planes, agents, and specialized workloads.
2. Concepts
Filesystem
Everything is a file. Key paths: /etc (config), /var/log (logs), /proc (kernel/process API), /sys (devices/cgroups), /tmp.
Processes & signals
PID, PPID, threads. Signals: SIGTERM (graceful), SIGKILL (forced), SIGHUP (reload often).
systemd
Unit files manage services: start/stop/restart/status, journals via journalctl.
Permissions
Owner/group/other + rwx; umask; ACL (setfacl) for finer grants; capabilities for privileged ops without full root.
SSH
Key-based auth, sshd_config, bastions/SSM Session Manager preferred over open SSH to every node.
Cron / timers
cron and systemd.timer for scheduled jobs — prefer idempotent jobs + locking.
Security modules
SELinux / AppArmor — mandatory access control beyond DAC permissions.
3. Architecture
graph TB
Userspace[User Processes / Containers] --> Syscalls
Syscalls --> Kernel[Linux Kernel]
Kernel --> CPU
Kernel --> Mem[Memory]
Kernel --> Disk
Kernel --> Net[Network Stack]
systemd --> Userspace
4. Production Best Practices
- Immutable / cattle hosts (golden images) over pet servers
- Centralize auth (SSO/IAM); disable password SSH
- Unattended security updates or bake patches into images
- chrony/NTP time sync (TLS and logs depend on it)
- Log to journal + ship off-box; never only local disk
- Monitor disk inodes and space; alert before 100%
- Separate OS disk vs data disks
5. Common Mistakes
| Mistake | Symptoms | Fix |
|---|---|---|
| Full disk | Services fail writes | Clean logs; expand volume |
| Out of inodes | "No space" with free bytes | Find tiny-file dirs |
| Ignoring OOM killer | Random process death | dmesg; raise RAM/limits |
| World-writable scripts | Privilege abuse | Fix modes 750/640 |
| Disabling SELinux permanently | Audit gaps | Fix policies properly |
6. Debugging Guide
# CPU / load
uptime; top; pidstat 1
# Memory
free -h; vmstat 1; cat /proc/meminfo
# Disk
df -h; df -i; iostat -xz 1
# Process
ps auxf; systemctl status <svc>; journalctl -u <svc> -e
# Network
ss -lntp; ip route; ping; traceroute
# Kernel
dmesg -T | tail
7. Security
- Minimal packages; no compilers on prod if policy requires
- Firewall (nftables/UFW); only needed ports
- Fail2ban or cloud SG + WAF for SSH/HTTP
- Auditd for privileged actions
- Kernel hardening sysctls (rp_filter, kptr_restrict, etc.) carefully tested
8. Performance
- Understand load average vs CPU count
- Distinguish iowait vs CPU bound
- Tune
nofile/nprocfor high-connection services - Use XFS/ext4 appropriately; mount options for DBs per vendor guidance
9. Real Production Example
Bastion-less: SSM Session Manager to EC2; CloudWatch agent; CIS-hardened AMI rebuilt weekly; systemd units for sidecars; journald → CloudWatch Logs; nftables default deny.
10. Interview Questions
Beginner: What does chmod 640 mean? SIGTERM vs SIGKILL?
Intermediate: How does the OOM killer choose victims? systemd dependency types?
Senior: Debug intermittent latency (steal time, noisy neighbor, softirq). Design patch cadence for 10k nodes.
11. Checklist
- Time sync, disk alerts, SSH keys only
- Automatic patching strategy
- journald retention + shipping
- Firewall + minimal listeners (
ss -lntp) - SELinux/AppArmor enforcing where required
12. Cheat Sheet
| Command | Purpose |
|---|---|
systemctl restart x |
Restart service |
journalctl -fu x |
Follow unit logs |
ss -lntp |
Listening ports |
lsof -i :443 |
Who uses port |
chmod / chown / getfacl |
Permissions |
See also: hardening-checklist.md
Related Skills
Author & maintainer: Shanmukha Kumar Karra
Created and maintained as part of AgentOps Kit.