Skip Navigation
ShareGPT @reddthat.com
PumpkinDrama @reddthat.com

How do you debug system issues on Linux?

cross-posted from: https://reddthat.com/post/52207927

I use Manjaro Linux with the Cinnamon desktop and sometimes run into system-level issues, but I have no idea how to properly debug them. It doesn’t feel as straightforward as debugging a normal program. What’s the best way or resource to learn system debugging on Linux?

1 comments

  • Here’s a structured map + curated resources to help you learn how to debug system-level issues on Manjaro (Cinnamon) or any Linux. You won’t master it overnight, but these will give you direction and tools.

    1. Concepts, approaches, mindset

    You need to treat the system like layers (hardware, kernel, init, services, user space). When something fails, you trace upward or downward from symptom to root cause.

    Key strategies:

    • Reproduce reliably, with minimal state.
    • Check logs early (kernel, systemd, service logs).
    • Isolate subsystems (network, disk, memory, UI).
    • Use interactive and passive tracing tools.
    • When reporting bugs, collect as much context as possible (versions, logs, stack traces).
    • Understand that for some deep bugs you’ll need debug symbols (which Manjaro doesn’t provide by default). (ArchWiki)

    2. Foundational references / web resources

    These are guide‐level or canonical references you should read and keep bookmarked:

    ResourceWhat it gives you
    ArchWiki — DebuggingHow to gather traces, core dumps, diagnosing faults in programs & kernel. (ArchWiki)
    “Logging and Diagnostics in Linux” (Cycle.io)Basics of Linux logging, log files, how to interpret them. (cycle.io)
    “The Ultimate Guide to Troubleshooting in Linux”A general walkthrough of common classes of errors and methods. (Steven P. Sanderson II, MPH)
    Manjaro forum threads (e.g. “Debug strategies on system failures”)Real user cases, patterns, community knowledge. (Manjaro Linux Forum)

    3. Tools & commands you must know

    You’ll use these regularly. Practice them so they’re second nature.

    Logging / logs

    • journalctl — read systemd journal. Use --list-boots, -b, -k (kernel) etc. (Manjaro Linux Forum)
    • Log files in /var/log/ (syslog, kernel logs, app logs) (cycle.io)
    • dmesg — kernel ring buffer messages
    • systemctl status <service> / systemctl logs

    System inspection

    • top, htop, free, vmstat — memory/CPU monitoring
    • iostat, iotop — disk I/O
    • lspci, lsusb, lsmod, lshw — hardware & modules info (CBT Nuggets)
    • smartctl — check hard drive health
    • memtester — test RAM (in user space) (YouTube)
    • strace, ltrace — track syscalls / library calls
    • gdb + core dumps — backtraces, debugging crashes (ArchWiki)
    • systemtap (for dynamic tracing) (Wikipedia)

    Benchmark / stress / diagnostics suites

    • Phoronix Test Suite — benchmarking various subsystems. (Wikipedia)
    • Various stress / diagnostic tools (CPU, memory, I/O) — e.g. stress, stress-ng, sysbench (Baeldung on Kotlin)

    4. Manjaro / Arch‐specific challenges & tips

    • Because Manjaro is based on Arch but with its own packaging, debug symbols are not generally provided for stable packages. You’ll often need to compile yourself or switch branches to testing/unstable to get matching symbols. (Manjaro Linux Forum)
    • For serious debugging (crashes, backtraces) users often switch the mirror branch to “unstable” and fetch debug symbols from Arch repos. (Manjaro Linux Forum)
    • When opening bug reports, include: package version, logs, backtrace, steps to reproduce.

    5. Learning path & practice

    1. Pick a subsystem you care about (e.g. UI, GPU, networking).
    2. Introduce a minor fault (e.g. misconfigure something) and try to trace it using logs + tools.
    3. Read ArchWiki’s debugging articles.
    4. Explore journalctl and logs.
    5. Try capturing a crash and using gdb or coredumpctl to get backtrace.
    6. Practice using strace, ltrace on small programs.
    7. When you do hit a real bug, do a write‐up: collect all context, logs, trace, then try submit upstream.

    If you like, I can send you a custom curated “Debugging Linux / Manjaro” cheat sheet (PDF or text) you can use offline. Do you want me to prepare that for you?