SlideShare a Scribd company logo
2
Most read
3
Most read
8
Most read
Advanced Debugging with gdbDavid KhosidSept 21, 2009david.kh@gmail.com
AgendaTechniques for debugging big, modern software:STL containers and algorithms, Boost Ex: how to see containersSignalsMulti-threaded (ex.: how to follow a thread?)Repetitive tasks on the almost unchanging code baseRemote debuggingExamples2
GDB was first written by Richard Stallman in 1986 as part of his GNU systemRichard Stallman, “Debugging with gdb”www.gnu.org/software/gdb/documentationHelp: 	$gdb –h 		(gdb) h 		(gdb) aproposCommand names may be truncated if the abbreviation is unambiguous. TAB completion. Command Cheat Sheetwww.yolinux.com/TUTORIALS/GDB-Commands.htmlLast GDB version is 6.8, new 7.0 soon: 2009-09-233Sources of information
Item #1: C++ and STL - ContainersHow to see container’s content?Commands file, ex. .gdbinithttp://www.yolinux.com/TUTORIALS/src/dbinit_stl_views-1.03.txtLimitations: a littlelibstdc++ compiled in debug modeLimitations: - different product , not for QA, not for client, not in performance tuning stage- performance 4
Item #1: C++ and STL - ContainersHow to see container’s content?Auxiliary functionstypedef map<string, float> MapStringFloat;void mapPrint(const MapStringFloat& m){       for(MapStringFloat::const_iterator pos = m.begin(); pos != m.end(); ++pos){                     cout << pos->first << " : " << pos->second << "\n"; }Limitations: - you can’t do that without a process to debug (investigating core files)- optimization of unused functions. Solution: ‘volatile’Pretty-printing of STL containers in future versions of GDB5
Item #2: Extending GDB – User-defined commands(gdb) show user commandnameExample: (gdb)define adder	print $arg0 + $arg1 + $arg2 end(gdb) adder 1 2 3 6
Item #3: Automating repetitive tasks What GDB Does During Startup1. Executes all commands from system init file 2. Executes all the commands from ~/.gdbinit3. Process command line options and operands 4. Executes all the commands from ./.gdbinit5. reads command files specified by the `-x' option 6. …7
Automating tasks - history, recordingcontinueWhat GDB Does During Startup… 6. Reads the command history recorded in the history file. (gdb) set history filename fname(gdb) set history save on/off(gdb) show history(gdb) show commands8
Item #4: Signals‘i handle’ or ‘i signals’Print a table of all the signals and how gdb has been told to handle each one.handle signal [keywords...]keywords: nostop|stop,  print|noprint and pass|nopassEx: handle SIG35 nostop print pass 	handle SIG36 stop (implies the ‘print’ as well)	handle SIG37 nostop print nopass	handle SIG38 nostopnoprintnopass9
Item #5: Multi-threadsUse case: debugging specific thread, while controlling behavior of others.facilities for debugging multi-thread programs:• automatic notification of new threads• ‘thread threadno’, to switch among threads• ‘info threads’, to inquire about existing threads• thread-specific breakpoints• set mode for locking scheduler during execution 	(gdb) set scheduler-locking step/on/offothers: Interrupted System CallsExample:(gdb) i threads(gbd) b foo.cpp:13 thread 28 if x > lim10
Item #5: Remote debuggingUse case: - GDB runs on one machine (host) and the program  being debugged (exe.verXYZ.stripped ) runs on another (target). - GDB communicates via Serial or TCP/IP.- Host and target: exactly match between the executables and libraries, with one exception: stripped on the target.- Complication: compiling on one machine (CC view), keeping code in different place (ex. /your/path/verXYZ)Solution: - Connect gdb to source in the given place:(gdb) set substitute-path /usr/src /mnt/cross (gdb) dir /your/path/verXYZ11
Remote debugging - exampleUsing gdbserver through TCP connection: remote (10.10.0.225)>  gdbserver :9999 program_strippedor remote> ./gdbserver :9999 –attach <pid>host> gdb programhost>(gdb) handle SIGTRAP nostopnoprint pass			 to avoid pausing when launching the threadshost> (gdb) target remote 10.10.0.225:9999TARGET (Android Dev phone)	 HOST (Fedora Linux)12
Item #6: Back to the past Convenience variables are used to store values that you may want to refer later. Any string preceded by $ is regarded as a convenience variable.Ex.:set $table = *table_ptr	(gdb) show convCheckpoint - a snapshot of a program’s state(gdb) checkpoint(gdb) i checkpoint(gdb) restartcheckpoint-idValue history- values printed by the print command.13
Small Items: #7, #8 #7. 	How to see macros?	$ g++ -gdwarf-2 -g3 a.cpp -o prog#8. 	64 bit .vs. 32bit -m32 flagOn 64-bit machine, install another 32-bit version of GDB	$ ls -l `which gdb32`	/usr/bin/gdb32 ->  ‘/your/install/path’14
Lightweight how-to'sHow to remove a symbol table from a file?A: 	stripHow to supply arguments to your program in GDB?A1: With --args option 	#sudo gdb -silent --args /bin/ping google.comA2: As arguments to run: (gdb) runarg1 arg2run without arguments uses the same arguments used by the previous run. A3: With set args  command: 		(gdb) set args arg1 arg2(gdb) show args set args without arguments – removes all arguments. How to know where you are (file, next execution line)?A:  	(gdb) f15
Lightweight how-to's - continueHow to find out the crash file executable? A1: #file core.1234A2: #gdb core.1234A3: use /proc/sys/kernel/core_pattern#echo "core_%e.%p" > /proc/sys/kernel/core_pattern 	if the program foo dumps its core, 	the core_foo.1234 will be created.How to find out why your program stopped?A: (gdb) iprogWhich command(s) can be used to exit from loops?A:(gdb)until lineNo‘print’, ‘info’, ‘show’- what is a difference?‘print’ – print value of expression‘info’ – showing things about the program being debugged‘show’ – showing things about the debugger16

More Related Content

PPT
Debugging Applications with GNU Debugger
PDF
Q2.12: Debugging with GDB
PPTX
Gnu debugger
PDF
UM2019 Extended BPF: A New Type of Software
PPTX
PDF
GDB Rocks!
PDF
PDF
BPF / XDP 8월 세미나 KossLab
Debugging Applications with GNU Debugger
Q2.12: Debugging with GDB
Gnu debugger
UM2019 Extended BPF: A New Type of Software
GDB Rocks!
BPF / XDP 8월 세미나 KossLab

What's hot (20)

PPTX
Linux Kernel Module - For NLKB
PDF
Embedded Linux from Scratch to Yocto
PDF
eBPF Perf Tools 2019
PDF
Intel dpdk Tutorial
PPTX
Understanding eBPF in a Hurry!
PDF
Module 4 Embedded Linux
PDF
Open vSwitch - Stateful Connection Tracking & Stateful NAT
PDF
PPT
GUI design using JAVAFX.ppt
PDF
introduction to linux kernel tcp/ip ptocotol stack
PDF
containerd summit - Deep Dive into containerd
PDF
Part 02 Linux Kernel Module Programming
PDF
Yocto - Embedded Linux Distribution Maker
PPT
101 2.5 use rpm and yum package management
PDF
qemu + gdb: The efficient way to understand/debug Linux kernel code/data stru...
PDF
Embedded Recipes 2019 - LLVM / Clang integration
PDF
How it's made: C++ compilers (GCC)
PPTX
Linux Network Stack
PDF
Performance Analysis Tools for Linux Kernel
PDF
Learning Git and GitHub - BIT GDSC.pdf
Linux Kernel Module - For NLKB
Embedded Linux from Scratch to Yocto
eBPF Perf Tools 2019
Intel dpdk Tutorial
Understanding eBPF in a Hurry!
Module 4 Embedded Linux
Open vSwitch - Stateful Connection Tracking & Stateful NAT
GUI design using JAVAFX.ppt
introduction to linux kernel tcp/ip ptocotol stack
containerd summit - Deep Dive into containerd
Part 02 Linux Kernel Module Programming
Yocto - Embedded Linux Distribution Maker
101 2.5 use rpm and yum package management
qemu + gdb: The efficient way to understand/debug Linux kernel code/data stru...
Embedded Recipes 2019 - LLVM / Clang integration
How it's made: C++ compilers (GCC)
Linux Network Stack
Performance Analysis Tools for Linux Kernel
Learning Git and GitHub - BIT GDSC.pdf
Ad

Viewers also liked (20)

PDF
Crash Dump Analysis 101
PDF
Gdb remote debugger
PDF
From printk to QEMU: Xen/Linux Kernel debugging
PPT
Introduction to gdb
PDF
Advanced Tracing features using GDB and LTTng
PDF
Debugger Principle Overview & GDB Tricks
PDF
Aag c45 697761
PDF
Reverse, Multi-Process and Non-Stop Debugging come to the CDT
PPT
Real time debugging: using non-intrusive tracepoints to debug live systems
PDF
Working Remotely (via SSH) Rocks!
PDF
LAS16-403: GDB Linux Kernel Awareness
PPTX
Linux booting process!!
PDF
Vim Rocks!
PPTX
Linux booting Process
PDF
DPDK Summit - 08 Sept 2014 - Futurewei - Jun Xu - Revisit the IP Stack in Lin...
PPT
Linux installation and booting process
PDF
kubernetes, pourquoi et comment
PDF
Simple Belief - Mosky @ TEDxNTUST 2015
PDF
Virtualization with KVM (Kernel-based Virtual Machine)
Crash Dump Analysis 101
Gdb remote debugger
From printk to QEMU: Xen/Linux Kernel debugging
Introduction to gdb
Advanced Tracing features using GDB and LTTng
Debugger Principle Overview & GDB Tricks
Aag c45 697761
Reverse, Multi-Process and Non-Stop Debugging come to the CDT
Real time debugging: using non-intrusive tracepoints to debug live systems
Working Remotely (via SSH) Rocks!
LAS16-403: GDB Linux Kernel Awareness
Linux booting process!!
Vim Rocks!
Linux booting Process
DPDK Summit - 08 Sept 2014 - Futurewei - Jun Xu - Revisit the IP Stack in Lin...
Linux installation and booting process
kubernetes, pourquoi et comment
Simple Belief - Mosky @ TEDxNTUST 2015
Virtualization with KVM (Kernel-based Virtual Machine)
Ad

Similar to Advanced Debugging with GDB (20)

PDF
ELC-E Linux Awareness
PDF
gdb-tutorial.pdf
PDF
lab1-ppt.pdf
PPTX
Debugging Modern C++ Application with Gdb
PPTX
GNU Debugger
PDF
LAS16-403 - GDB Linux Kernel Awareness
PPTX
Debuging like a pro
PDF
Usage of GDB
PDF
Gdb tutorial-handout
PDF
Anatomy of ROCgdb presentation at gcc cauldron 2022
PDF
Debugging of (C)Python applications
PDF
Debugging Python with gdb
PDF
Analyze corefile and backtraces with GDB for Mysql/MariaDB on Linux - Nilanda...
PDF
Debugging Hung Python Processes With GDB
PDF
Linux kernel debugging
PPTX
Wavedigitech gdb
PPTX
Roll your own toy unix clone os
PPT
Linux Kernel Debugging
PPTX
Debug generic process
PDF
Apache Airflow® Best Practices: DAG Writing
ELC-E Linux Awareness
gdb-tutorial.pdf
lab1-ppt.pdf
Debugging Modern C++ Application with Gdb
GNU Debugger
LAS16-403 - GDB Linux Kernel Awareness
Debuging like a pro
Usage of GDB
Gdb tutorial-handout
Anatomy of ROCgdb presentation at gcc cauldron 2022
Debugging of (C)Python applications
Debugging Python with gdb
Analyze corefile and backtraces with GDB for Mysql/MariaDB on Linux - Nilanda...
Debugging Hung Python Processes With GDB
Linux kernel debugging
Wavedigitech gdb
Roll your own toy unix clone os
Linux Kernel Debugging
Debug generic process
Apache Airflow® Best Practices: DAG Writing

Recently uploaded (20)

PDF
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
PDF
NewMind AI Weekly Chronicles - August'25 Week I
PDF
Optimiser vos workloads AI/ML sur Amazon EC2 et AWS Graviton
PDF
Bridging biosciences and deep learning for revolutionary discoveries: a compr...
PDF
Advanced IT Governance
PDF
Advanced Soft Computing BINUS July 2025.pdf
PDF
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
PDF
GDG Cloud Iasi [PUBLIC] Florian Blaga - Unveiling the Evolution of Cybersecur...
PPT
Teaching material agriculture food technology
PDF
Mobile App Security Testing_ A Comprehensive Guide.pdf
PDF
Review of recent advances in non-invasive hemoglobin estimation
PPTX
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
PDF
[발표본] 너의 과제는 클라우드에 있어_KTDS_김동현_20250524.pdf
PDF
Diabetes mellitus diagnosis method based random forest with bat algorithm
PPTX
PA Analog/Digital System: The Backbone of Modern Surveillance and Communication
PDF
Peak of Data & AI Encore- AI for Metadata and Smarter Workflows
PDF
Machine learning based COVID-19 study performance prediction
PDF
CIFDAQ's Market Insight: SEC Turns Pro Crypto
PDF
Unlocking AI with Model Context Protocol (MCP)
PPTX
20250228 LYD VKU AI Blended-Learning.pptx
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
NewMind AI Weekly Chronicles - August'25 Week I
Optimiser vos workloads AI/ML sur Amazon EC2 et AWS Graviton
Bridging biosciences and deep learning for revolutionary discoveries: a compr...
Advanced IT Governance
Advanced Soft Computing BINUS July 2025.pdf
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
GDG Cloud Iasi [PUBLIC] Florian Blaga - Unveiling the Evolution of Cybersecur...
Teaching material agriculture food technology
Mobile App Security Testing_ A Comprehensive Guide.pdf
Review of recent advances in non-invasive hemoglobin estimation
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
[발표본] 너의 과제는 클라우드에 있어_KTDS_김동현_20250524.pdf
Diabetes mellitus diagnosis method based random forest with bat algorithm
PA Analog/Digital System: The Backbone of Modern Surveillance and Communication
Peak of Data & AI Encore- AI for Metadata and Smarter Workflows
Machine learning based COVID-19 study performance prediction
CIFDAQ's Market Insight: SEC Turns Pro Crypto
Unlocking AI with Model Context Protocol (MCP)
20250228 LYD VKU AI Blended-Learning.pptx

Advanced Debugging with GDB

  • 2. AgendaTechniques for debugging big, modern software:STL containers and algorithms, Boost Ex: how to see containersSignalsMulti-threaded (ex.: how to follow a thread?)Repetitive tasks on the almost unchanging code baseRemote debuggingExamples2
  • 3. GDB was first written by Richard Stallman in 1986 as part of his GNU systemRichard Stallman, “Debugging with gdb”www.gnu.org/software/gdb/documentationHelp: $gdb –h (gdb) h (gdb) aproposCommand names may be truncated if the abbreviation is unambiguous. TAB completion. Command Cheat Sheetwww.yolinux.com/TUTORIALS/GDB-Commands.htmlLast GDB version is 6.8, new 7.0 soon: 2009-09-233Sources of information
  • 4. Item #1: C++ and STL - ContainersHow to see container’s content?Commands file, ex. .gdbinithttp://www.yolinux.com/TUTORIALS/src/dbinit_stl_views-1.03.txtLimitations: a littlelibstdc++ compiled in debug modeLimitations: - different product , not for QA, not for client, not in performance tuning stage- performance 4
  • 5. Item #1: C++ and STL - ContainersHow to see container’s content?Auxiliary functionstypedef map<string, float> MapStringFloat;void mapPrint(const MapStringFloat& m){ for(MapStringFloat::const_iterator pos = m.begin(); pos != m.end(); ++pos){ cout << pos->first << " : " << pos->second << "\n"; }Limitations: - you can’t do that without a process to debug (investigating core files)- optimization of unused functions. Solution: ‘volatile’Pretty-printing of STL containers in future versions of GDB5
  • 6. Item #2: Extending GDB – User-defined commands(gdb) show user commandnameExample: (gdb)define adder print $arg0 + $arg1 + $arg2 end(gdb) adder 1 2 3 6
  • 7. Item #3: Automating repetitive tasks What GDB Does During Startup1. Executes all commands from system init file 2. Executes all the commands from ~/.gdbinit3. Process command line options and operands 4. Executes all the commands from ./.gdbinit5. reads command files specified by the `-x' option 6. …7
  • 8. Automating tasks - history, recordingcontinueWhat GDB Does During Startup… 6. Reads the command history recorded in the history file. (gdb) set history filename fname(gdb) set history save on/off(gdb) show history(gdb) show commands8
  • 9. Item #4: Signals‘i handle’ or ‘i signals’Print a table of all the signals and how gdb has been told to handle each one.handle signal [keywords...]keywords: nostop|stop, print|noprint and pass|nopassEx: handle SIG35 nostop print pass handle SIG36 stop (implies the ‘print’ as well) handle SIG37 nostop print nopass handle SIG38 nostopnoprintnopass9
  • 10. Item #5: Multi-threadsUse case: debugging specific thread, while controlling behavior of others.facilities for debugging multi-thread programs:• automatic notification of new threads• ‘thread threadno’, to switch among threads• ‘info threads’, to inquire about existing threads• thread-specific breakpoints• set mode for locking scheduler during execution (gdb) set scheduler-locking step/on/offothers: Interrupted System CallsExample:(gdb) i threads(gbd) b foo.cpp:13 thread 28 if x > lim10
  • 11. Item #5: Remote debuggingUse case: - GDB runs on one machine (host) and the program being debugged (exe.verXYZ.stripped ) runs on another (target). - GDB communicates via Serial or TCP/IP.- Host and target: exactly match between the executables and libraries, with one exception: stripped on the target.- Complication: compiling on one machine (CC view), keeping code in different place (ex. /your/path/verXYZ)Solution: - Connect gdb to source in the given place:(gdb) set substitute-path /usr/src /mnt/cross (gdb) dir /your/path/verXYZ11
  • 12. Remote debugging - exampleUsing gdbserver through TCP connection: remote (10.10.0.225)> gdbserver :9999 program_strippedor remote> ./gdbserver :9999 –attach <pid>host> gdb programhost>(gdb) handle SIGTRAP nostopnoprint pass to avoid pausing when launching the threadshost> (gdb) target remote 10.10.0.225:9999TARGET (Android Dev phone) HOST (Fedora Linux)12
  • 13. Item #6: Back to the past Convenience variables are used to store values that you may want to refer later. Any string preceded by $ is regarded as a convenience variable.Ex.:set $table = *table_ptr (gdb) show convCheckpoint - a snapshot of a program’s state(gdb) checkpoint(gdb) i checkpoint(gdb) restartcheckpoint-idValue history- values printed by the print command.13
  • 14. Small Items: #7, #8 #7. How to see macros? $ g++ -gdwarf-2 -g3 a.cpp -o prog#8. 64 bit .vs. 32bit -m32 flagOn 64-bit machine, install another 32-bit version of GDB $ ls -l `which gdb32` /usr/bin/gdb32 -> ‘/your/install/path’14
  • 15. Lightweight how-to'sHow to remove a symbol table from a file?A: stripHow to supply arguments to your program in GDB?A1: With --args option #sudo gdb -silent --args /bin/ping google.comA2: As arguments to run: (gdb) runarg1 arg2run without arguments uses the same arguments used by the previous run. A3: With set args command: (gdb) set args arg1 arg2(gdb) show args set args without arguments – removes all arguments. How to know where you are (file, next execution line)?A: (gdb) f15
  • 16. Lightweight how-to's - continueHow to find out the crash file executable? A1: #file core.1234A2: #gdb core.1234A3: use /proc/sys/kernel/core_pattern#echo "core_%e.%p" > /proc/sys/kernel/core_pattern if the program foo dumps its core, the core_foo.1234 will be created.How to find out why your program stopped?A: (gdb) iprogWhich command(s) can be used to exit from loops?A:(gdb)until lineNo‘print’, ‘info’, ‘show’- what is a difference?‘print’ – print value of expression‘info’ – showing things about the program being debugged‘show’ – showing things about the debugger16
  • 17. Problem Determination Tools for Linux-Wall Code reviewProgram’s traces, syslog, profilersStatic Source Code Analysis:scan.coverity.com – free for FOSSFlexelintDynamic analysis: Valgrind, strace, /proc filesystem, lsof, ldd, nm, objdump, wireshark17
  • 18. SummaryStart from thinking of Use Case, then look in the manual, use ‘apropos’ and ‘help’Productivity:Stepping through a program is less productive than thinking harder and adding output statements and self-checking code at critical places.When to use GDB? - core file, - when a problem can be reproduced, repeating errors - self-educatingWhen not?Other tools, tracesQuestions?18