Core Dump Complete Guide | Generation, Analysis & Debugging

Core Dump Complete Guide | Generation, Analysis & Debugging

· 50분 읽기 · 수정 Advanced Guide
Core Dump Complete Guide | Generation, Analysis & Debugging

이 글의 핵심

Complete guide to Core Dump generation, GDB analysis, Segmentation Fault debugging, and production cases. Solve production crash issues with ulimit configuration, symbol files, and backtrace analysis.

Introduction: Why Core Dump is Needed

How do you debug when a program suddenly crashes in production? Core Dump is essential for solving hard-to-reproduce bugs, intermittent Segmentation Faults, and memory corruption issues. Value of Core Dump:

  • Complete memory snapshot at crash time
  • All variable values and stack traces
  • Post-mortem analysis without reproduction
  • Real data from production environment What This Guide Covers:
  • Core Dump generation configuration
  • Core Dump analysis with GDB
  • Practical debugging scenarios
  • Automation and monitoring

Reality in Production

When learning development, everything seems clean and theoretical. But production is different. Wrestling with legacy code, chasing tight deadlines, facing unexpected bugs. The content covered here was initially learned as theory, but through applying it to real projects, I realized “ah, that’s why it’s designed this way.” Particularly memorable are the trials and errors from my first project. I followed what I learned from books but couldn’t figure out why it didn’t work, spending days stuck. Eventually discovered the problem through senior developer code review, learning a lot in the process. This guide covers not just theory but also pitfalls and solutions you might encounter in practice.

Table of Contents

  1. Core Dump Basics
  2. Core Dump Generation Setup
  3. GDB Core Dump Analysis
  4. Practical Debugging Scenarios
  5. Automation and Collection
  6. Production Environment Setup
  7. Advanced Analysis Techniques
  8. Troubleshooting

1. Core Dump Basics

What is Core Dump?

Core Dump is saving memory contents to a file when a program terminates abnormally (crashes).

flowchart TB
    Program[Program Running] --> Crash["Crash Occurs\nSIGSEGV, SIGABRT etc"]
    
    Crash --> Kernel[Kernel Receives Signal]
    
    Kernel --> Check{Core Dump\nEnabled?}
    
    Check -->|Yes| Dump["Generate Memory Dump\ncore.12345"]
    Check -->|No| Exit[Program Terminates]
    
    Dump --> File["Core Dump File\n- Stack\n- Heap\n- Registers\n- Variable Values"]
    
    File --> Debug[Analyze with GDB]

Information in Core Dump

1. Stack Memory
   - Function call stack
   - Local variables
   - Function arguments
2. Heap Memory
   - Dynamically allocated memory
   - Objects created with malloc/new
3. Register State
   - PC (Program Counter)
   - SP (Stack Pointer)
   - General purpose registers
4. Global Variables
   - Static variables
   - Global objects
5. Shared Library Mappings
   - Loaded .so file list
   - Memory address mappings

Signals That Cause Crashes

// Signals that generate Core Dump
SIGQUIT    // Quit (Ctrl+\)
SIGILL     // Illegal Instruction
SIGABRT    // Abort (assert failure)
SIGFPE     // Floating Point Exception (divide by zero)
SIGSEGV    // Segmentation Fault (invalid memory access)
SIGBUS     // Bus Error (alignment error)
SIGTRAP    // Trace/Breakpoint Trap
SIGSYS     // Bad System Call

2. Core Dump Generation Setup

ulimit Configuration

# Check current Core Dump size limit
ulimit -c
# 0 → Core Dump disabled
# unlimited → No limit
# Enable Core Dump (current session)
ulimit -c unlimited
# Permanent setting (all users)
echo "* soft core unlimited" | sudo tee -a /etc/security/limits.conf
echo "* hard core unlimited" | sudo tee -a /etc/security/limits.conf
# Specific user only
echo "myuser soft core unlimited" | sudo tee -a /etc/security/limits.conf
# For systemd services
# /etc/systemd/system/myapp.service
[Service]
LimitCORE=infinity

Core Dump Save Location

# Check current setting
cat /proc/sys/kernel/core_pattern
# Default: core (creates 'core' file in current directory)
# Set custom pattern
sudo sysctl -w kernel.core_pattern=/var/crash/core.%e.%p.%t
# Permanent setting
echo "kernel.core_pattern=/var/crash/core.%e.%p.%t" | sudo tee -a /etc/sysctl.conf
# Pattern variables:
# %e - Executable name
# %p - PID
# %t - Timestamp (Unix time)
# %s - Signal number
# %u - UID
# %g - GID
# %h - Hostname
# Example: core.myapp.12345.1234567890
# Install systemd-coredump (Ubuntu)
sudo apt install systemd-coredump
# Configuration
sudo nano /etc/systemd/coredump.conf
[Coredump]
Storage=external
Compress=yes
ProcessSizeMax=2G
ExternalSizeMax=2G
MaxUse=10G
# Core Dump save location
# /var/lib/systemd/coredump/
# List Core Dumps
coredumpctl list
# Specific Core Dump info
coredumpctl info <PID>
# Extract Core Dump
coredumpctl dump <PID> -o core.dump
# Analyze directly with GDB
coredumpctl debug <PID>

3. GDB Core Dump Analysis

Basic Analysis Flow

# 1. Compile with debug symbols
g++ -g -O0 myapp.cpp -o myapp
# 2. Configure to generate Core Dump
ulimit -c unlimited
# 3. Run program (crash occurs)
./myapp
# Segmentation fault (core dumped)
# 4. Open Core Dump with GDB
gdb ./myapp core.12345
# Or use systemd-coredump
coredumpctl debug 12345

GDB Basic Commands

# After loading Core Dump
# 1. Backtrace (stack trace)
(gdb) bt
#0  0x00005555555551a9 in crash_function () at myapp.cpp:10
#1  0x00005555555551c8 in main () at myapp.cpp:15
# Detailed backtrace
(gdb) bt full
#0  0x00005555555551a9 in crash_function () at myapp.cpp:10
        ptr = 0x0
        value = 42
#1  0x00005555555551c8 in main () at myapp.cpp:15
        argc = 1
        argv = 0x7fffffffe0a8
# 2. Move to specific frame
(gdb) frame 0
(gdb) frame 1
# 3. Check source code
(gdb) list
5       void crash_function() {
6           int* ptr = nullptr;
7           int value = 42;
8           
9           // Segmentation Fault!
10          *ptr = value;
11      }
# 4. Check variable values
(gdb) print ptr
$1 = (int *) 0x0
(gdb) print value
$2 = 42
# 5. All local variables
(gdb) info locals
ptr = 0x0
value = 42
# 6. Register state
(gdb) info registers
rax            0x0                 0
rbx            0x555555557d60      93824992247136
rsp            0x7fffffffddd0      0x7fffffffddd0
rip            0x5555555551a9      0x5555555551a9 <crash_function()+15>
# 7. Check memory contents
(gdb) x/10x $rsp
0x7fffffffddd0: 0xffffddf0      0x00007fff      0x555551c8      0x00005555
# 8. Thread information
(gdb) info threads
  Id   Target Id         Frame
* 1    Thread 0x7ffff7fc1740 (LWP 12345) crash_function () at myapp.cpp:10
  2    Thread 0x7ffff77c0700 (LWP 12346) 0x00007ffff7bc8e2d in poll ()
# 9. Switch to specific thread
(gdb) thread 2
(gdb) bt

Practical Example 1: Null Pointer Dereference

// crash_null.cpp
#include <iostream>
void process_data(int* data) {
    std::cout << "Processing: " << *data << std::endl;  // Crash!
}
int main() {
    int* ptr = nullptr;
    process_data(ptr);
    return 0;
}
# Compile (with debug symbols)
g++ -g -O0 crash_null.cpp -o crash_null
# Run
./crash_null
# Segmentation fault (core dumped)
# GDB analysis
gdb ./crash_null core
(gdb) bt
#0  0x00005555555551b2 in process_data (data=0x0) at crash_null.cpp:4
#1  0x00005555555551e1 in main () at crash_null.cpp:9
(gdb) frame 0
(gdb) print data
$1 = (int *) 0x0
(gdb) list
1       #include <iostream>
2
3       void process_data(int* data) {
4           std::cout << "Processing: " << *data << std::endl;  // Crash here!
5       }
# Cause: data is nullptr
# Solution: Add nullptr check

4. Practical Debugging Scenarios

Scenario 1: Double Free

// crash_double_free.cpp
#include <cstdlib>
#include <iostream>
int main() {
    int* ptr = new int(42);
    
    std::cout << "Value: " << *ptr << std::endl;
    
    delete ptr;
    
    // Double free!
    delete ptr;
    
    return 0;
}
# Compile
g++ -g -O0 crash_double_free.cpp -o crash_double_free
# Run
./crash_double_free
# Value: 42
# free(): double free detected in tcache 2
# Aborted (core dumped)
# GDB analysis
gdb ./crash_double_free core
(gdb) bt
#0  0x00007ffff7a42387 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007ffff7a43a78 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#5  0x0000555555555234 in main () at crash_double_free.cpp:12
(gdb) frame 5
(gdb) print ptr
$1 = (int *) 0x555555559eb0  # Already freed address
# Solution: Set ptr = nullptr after delete

5. Automation and Collection

Automatic Core Dump Collection Script

Manually hunting for core files after every crash doesn’t scale — a collection script keeps every crash, its binary metadata, and a first-pass backtrace together for later triage.

#!/bin/bash
# collect_core.sh — run from a crash hook or cron watching /var/crash
CORE_DIR="/var/crash"
ARCHIVE_DIR="/var/crash/archive"
mkdir -p "$ARCHIVE_DIR"
for core in "$CORE_DIR"/core.*; do
    [ -f "$core" ] || continue
    ts=$(date +%Y%m%d_%H%M%S)
    binary=$(echo "$core" | cut -d. -f2)
    dest="$ARCHIVE_DIR/${binary}_${ts}"
    mkdir -p "$dest"
    mv "$core" "$dest/core"
    cp "/usr/local/bin/$binary" "$dest/binary" 2>/dev/null
    gdb -batch -ex "bt full" "$dest/binary" "$dest/core" > "$dest/backtrace.txt" 2>&1
    echo "Collected crash: $dest"
done

Using systemd-coredump Hooks

systemd-coredump can run a script automatically on every crash via Storage=external plus a custom handler, avoiding a separate polling cron job.

# /etc/systemd/coredump.conf.d/hook.conf
[Coredump]
Storage=external
Compress=yes
# Trigger on every coredumpctl-registered crash
coredumpctl list --json=short | jq -r '.[].pid' | while read pid; do
    coredumpctl info "$pid" >> /var/log/crash-summary.log
done

Grouping Crashes by Signature

Deduplicating crash reports by their top-frame backtrace signature turns thousands of raw core files into a small number of distinct bugs to fix.

#!/bin/bash
# group_crashes.sh — group by top 3 stack frames
for dir in /var/crash/archive/*/; do
    sig=$(grep -A3 "^#0" "$dir/backtrace.txt" | md5sum | cut -d' ' -f1)
    echo "$sig $dir" >> /tmp/crash_signatures.txt
done
sort /tmp/crash_signatures.txt | uniq -c -w32 | sort -rn

6. Production Environment Setup

Disk Space Management

A crashing service under heavy load can generate core dumps fast enough to fill a disk — cap total usage and compress aggressively.

# /etc/systemd/coredump.conf
[Coredump]
Storage=external
Compress=yes
ProcessSizeMax=2G      # skip dumps larger than this
ExternalSizeMax=2G     # per-dump cap after compression
MaxUse=10G              # total space systemd-coredump may use
KeepFree=5G             # always leave this much free

Alerting on New Crashes

Wiring coredumpctl into a monitoring pipeline turns “a crash happened three days ago and nobody noticed” into a page within minutes.

#!/bin/bash
# crash_alert.sh — run periodically via cron/systemd timer
LAST_CHECK_FILE="/var/lib/crash_alert/last_check"
LAST_CHECK=$(cat "$LAST_CHECK_FILE" 2>/dev/null || echo 0)
NOW=$(date +%s)
NEW_CRASHES=$(coredumpctl list --since="@$LAST_CHECK" --no-legend | wc -l)
if [ "$NEW_CRASHES" -gt 0 ]; then
    curl -X POST "$SLACK_WEBHOOK_URL" \
         -d "{\"text\": \"⚠️ $NEW_CRASHES new crash(es) detected on $(hostname)\"}"
fi
echo "$NOW" > "$LAST_CHECK_FILE"

Symbol Management for Stripped Binaries

Production binaries are usually built without -g for size/security reasons — keep a separate debug-symbol package installable alongside the stripped binary so gdb can still resolve a useful backtrace.

# Build: keep symbols in a separate file, strip the shipped binary
g++ -g -O2 myapp.cpp -o myapp
objcopy --only-keep-debug myapp myapp.debug
strip --strip-debug --strip-unneeded myapp
objcopy --add-gnu-debuglink=myapp.debug myapp
# GDB automatically finds myapp.debug next to myapp, or via debug-file-directory
gdb ./myapp core.12345

7. Advanced Analysis Techniques

Analyzing Multi-Threaded Crashes

When the crashing thread isn’t obvious, check every thread’s backtrace before assuming the signal-receiving thread caused the actual bug — a race condition often corrupts state from one thread and crashes a different one.

(gdb) thread apply all bt
# Compare stacks across threads; look for the thread holding
# a lock or touching the same memory as the crashing frame
(gdb) info threads
(gdb) thread 3
(gdb) bt full

Inspecting Memory Corruption with x

When a crash is caused by heap corruption rather than a clean null dereference, examining raw memory around the faulting address helps distinguish “use-after-free” (freed-memory poison patterns) from “buffer overflow” (adjacent allocation overwritten).

(gdb) print $rip
(gdb) x/20xg $rsp                 # 20 giant words from the stack pointer
(gdb) x/4xg (void*)ptr - 32       # inspect memory just before a suspect pointer
(gdb) x/s (char*)ptr              # interpret as a string, useful for corrupted buffers

Reverse Debugging with rr

For crashes that are hard to reproduce, Mozilla’s rr records a full execution trace and lets you step backwards through it — instead of guessing where a corruption started, you can run to the crash, then reverse-step to the exact instruction that first wrote bad data.

rr record ./myapp
rr replay
(rr) continue      # run forward to the recorded crash
(rr) reverse-step   # step backwards from the crash point
(rr) watch *ptr     # then reverse-continue to find where *ptr was last written
(rr) reverse-continue

Scripting GDB for Batch Analysis

gdb -batch combined with a command file turns repetitive manual analysis into a one-line report generator, useful when triaging a large archive of collected core dumps.

# analyze.gdb
set pagination off
bt full
info registers
info threads
thread apply all bt
quit
gdb -batch -x analyze.gdb ./myapp core.12345 > report.txt

8. Troubleshooting

Core Dump Not Generated at All

# Check each layer that can suppress a core dump:
ulimit -c                           # must not be 0
cat /proc/sys/kernel/core_pattern   # must point somewhere writable
df -h /var/crash                    # must have free space
# suid/sgid binaries don't dump by default unless explicitly allowed:
cat /proc/sys/fs/suid_dumpable      # set to 2 ("suidsafe") if needed

”No symbol table is loaded” in GDB

# Confirm the binary was compiled with -g and matches the core exactly
file myapp                    # should mention "with debug_info"
gdb ./myapp core.12345
(gdb) info sharedlibrary      # check if libraries also need debug symbols

Rebuild with -g if the binary is stripped, or install matching -dbg/-debuginfo packages for system libraries (apt install libc6-dbg on Debian/Ubuntu, debuginfo-install glibc on RHEL-family).

Backtrace Shows ?? for Every Frame

Usually means the binary and the core don’t match, or the process was dynamically linked against libraries GDB can’t locate.

(gdb) info sharedlibrary
# Look for "No" in the "Syms Read" column and install the matching debug package

Core File Truncated or Incomplete

Large processes can exceed ProcessSizeMax/ulimit -c, silently producing a partial (unusable) core file.

ulimit -c unlimited
# Also raise the systemd-coredump caps:
sudo sed -i 's/^ProcessSizeMax=.*/ProcessSizeMax=8G/' /etc/systemd/coredump.conf
sudo sed -i 's/^ExternalSizeMax=.*/ExternalSizeMax=8G/' /etc/systemd/coredump.conf

GDB Can’t Attach: “ptrace: Operation not permitted”

Containerized environments (Docker, Kubernetes) often restrict ptrace by default; grant it explicitly for debugging containers.

docker run --cap-add=SYS_PTRACE --security-opt seccomp=unconfined ...

Summary

Core Dump Checklist

Development Environment

  • Set ulimit -c unlimited
  • Compile with -g flag
  • Disable optimization (-O0)
  • Use AddressSanitizer (-fsanitize=address)
  • Memory check with Valgrind

Production Environment

  • Set core limit in /etc/security/limits.conf
  • Configure kernel.core_pattern
  • Install and configure systemd-coredump
  • Monitor disk space
  • Set up automatic collection script
  • Integrate notification system

Analysis

  • Install GDB and learn usage
  • Keep debug symbols
  • Auto-extract backtrace
  • Analyze crash patterns
  • Deduplicate and group

GDB Command Cheatsheet

# Basic
gdb <executable> <core>     # Load Core Dump
bt                          # Backtrace
bt full                     # Detailed backtrace
frame <N>                   # Move to frame N
list                        # Source code
# Variables
print <var>                 # Variable value
print *<ptr>                # Value pointed to by pointer
print <array>[0]@10         # 10 array elements
info locals                 # All local variables
info args                   # Function arguments
# Memory
x/10x <addr>                # Memory (hexadecimal)
x/10s <addr>                # Strings
x/10i <addr>                # Instructions (assembly)
# Threads
info threads                # Thread list
thread <N>                  # Switch to thread N
thread apply all bt         # Backtrace all threads
# Registers
info registers              # All registers
print $rip                  # Specific register
# Other
info sharedlibrary          # Loaded libraries
info proc mappings          # Memory map

Common Crash Causes

flowchart TB
    Crash[Program Crash]
    
    Null["Null Pointer\nDereference"]
    Overflow[Buffer Overflow]
    UseAfter[Use After Free]
    DoubleFree[Double Free]
    StackOver[Stack Overflow]
    Race[Race Condition]
    Assert[Assert Failure]
    
    Crash --> Null
    Crash --> Overflow
    Crash --> UseAfter
    Crash --> DoubleFree
    Crash --> StackOver
    Crash --> Race
    Crash --> Assert
    
    Null --> Fix1[nullptr check]
    Overflow --> Fix2[Bounds checking]
    UseAfter --> Fix3[Smart pointers]
    DoubleFree --> Fix4[nullptr after delete]
    StackOver --> Fix5[Recursion limit]
    Race --> Fix6[Mutex]
    Assert --> Fix7[Condition validation]

References

Frequently Asked Questions (FAQ)

Q. When would I use this in practice?

A. Complete guide to Core Dump generation, GDB analysis, Segmentation Fault debugging, and production cases.

Q. What should I read before this?

A. Follow the previous article or related articles links at the bottom of each post to learn in sequence.

Q. Where can I study this more deeply?

A. Check cppreference and the relevant library’s official documentation. The reference links at the end of the article are also worth using.

Other articles related to this topic.

Keywords Covered in This Article (Related Search Terms)

This article covers CoreDump, GDB, debugging, SegmentationFault, C++, Linux, SystemProgramming.