| Main topics | Introduction to software debugging, debugging tools and techniques on Linux-based systems, introduction to debugging with GDB, kernel debugging by analyzing logs and crash messages (oops and panic), Linux kernel tracing tools and techniques, kernel debugging with GDB, Linux kernel memory leak detection, kernel lockup detection, crash and core dump analysis in user space applications, user space tracing tools, remote debugging user space applications with GDB, debugging multithreaded applications, debugging memory leaks in user space applications, performance analysis in Linux systems |
| Duration | 3 days - 24 hours - 40% theory and 60% practice |
| Trainer | Sergio Prado |
| Audience | Students, engineers, developers and team leaders that work with software development for embedded systems |
| Prerequisites | Basic knowledge of GNU/Linux command line tools (ls, cp, mv, cat, grep, find, vi, etc), intermediate knowledge in C and C++ languages, build tools such as make and gcc, embedded Linux system architecture (toolchain, bootloader, kernel, rootfs), Linux kernel configuration and compilation |
| Training material | The training material contains the slides and exercises documents, some reference guides and support files. All material is provided in electronic form at the beginning of the training |
| Development Kit | Development kits are loaned to students during training to carry out practical activities. By default, a System-on-Module from Toradex is used (Colibri i.MX6). If required by the contracting party, the training can be ported to other hardware platforms |
Presentation: Introduction to software debugging, the software debugging process, main types of problems and debugging mechanisms, tools and techniques to debug an embedded Linux system.
Exercise: Preparing the lab environment - In this exercise, we will prepare the training environment and boot a Linux system on the target platform.
Presentation: Introduction to GDB, debugging native applications with GDB, basic GDB commands, controlling the execution with GDB, breakpoints, watchpoints, catchpoints, printing expressions, displaying memory, stack backtrace, TUI mode, integration with front-end tools like DDD and Eclipse.
Exercise: Debugging with GDB - In this exercise, we will study the GDB command line interface and debug some native C applications.
Presentation: Introduction to Linux kernel debugging, collecting information from virtual filesystems (procfs, sysfs, debugfs), analyzing kernel logs, understanding and analyzing kernel oops and panic messages, debugging kernel modules, pstore, kdump.
Exercise: Analysing kernel logs and crashes - In this exercise, we will learn some of the main tools and techniques to analyze error messages and crashes in kernel space.
Presentation: Linux kernel tracing mechanisms, static and dynamic tracing, tracing tools and techniques (ftrace, trace-cmd, kernelshark, systemtap), introduction to ftrace, function tracing, measuring latency and stack usage, introduction to trace-cmd, trace data visualization with kernelshark.
Exercise: Tracing the execution of the Linux kernel - In this exercise, we will learn how to use ftrace to trace the execution of the Linux kernel.
Presentation: Linux kernel debugging with GDB (kgdb, gdbclient and agent-proxy), GDB and code optimization in kernel space, KDB frontend, hardware level debugging with JTAG.
Exercise: Debugging the Linux kernel with GDB - In this exercise, we will learn how to debug the Linux kernel with GDB.
Presentation: Linux kernel debugging frameworks, the “Kernel Hacking” configuration menu, kernel memory leak detection (kmemleak), Magic Sysrq Key, kernel lockup detection.
Exercise: Memory and lockup issues in the kernel - In this exercise, we will learn how to identify and analyze memory leaks and lockups in the Linux kernel.
Presentation: Debugging Linux applications, user space logging tools and techniques, crash and core dump analysis in user space applications and libraries.
Exercise: Crash analysis in user space applications - In this exercise, we will learn how to analyze crashes in user space applications.
Presentation: Tracing tools for user space applications, strace, ltrace, etrace, uprobe, perf, systemtap, lttng-ust.
Exercise: Tracing user space applications - In this exercise, we will study how to trace the execution of user space applications.
Presentation: Linux applications remote debugging with gdbserver, GDB and dynamic libraries, GDB and code optimization, debugging multithreaded applications with GDB.
Exercise: Remote debugging user space applications - In this exercise, we will study the process of remote debugging user space applications with GDB.
Presentation: Debugging memory usage in Linux applications, memory usage analysis tools (free, procrank, pmap), debugging and identifying memory leaks with valgrind, mtrace, memwatch and dmalloc.
Exercise: Memory leak analysis in user space - In this exercise, we will analyze memory leak problems in user space applications.
Presentation: Linux performance analysis, CPU consumption analysis (uptime, cpuload, top, schedule, time), memory usage analysis (free, procrank, pmap), performance analysis with perf, oprofile, strace, ltrace and gprof.
Exercise: Analysing performance issues - In this exercise, we will study some techniques and tools for performance analysis on Linux systems.
Presentation: Final considerations, study links and references, books, conclusion.