White Paper: Red Hat Crash Utility


< Prev Contents Next >

The Command Set

Each crash command generally falls into one of the following categories:

The remainder of this section breaks the command set into categories, and gives a short description of each command in that category. However, for complete details and examples, recall that the crash utility has a self-contained help page for each command; to view the full help page, click on the command name next to its description below.


Symbolic Display of Kernel Text or Data

The following commands typically take full advantage of the power of gdb to display kernel data structures symbolically.
Command
 
Description
struct
 
Displays a formatted kernel data structure type located at a given address, or at an address referred to by a symbol; if no address is specified, the structure definition is displayed. The output can be narrowed down to a singular member of the structure, or to display the offset of every member from the beginning of the structure. A count may be appended to display an array of structures. Its usage is so common that two short-cuts exist such that the user need not enter the "struct" command name:
  1. The "pointer-to" * command below can be substituted.
  2. If a structure name is entered as the first token on a command line, the "struct" command is actually not necessary.
union
 
Same as struct command, but used for kernel data types defined as unions instead of structures..
*
 
"Pointer-to" command wich can be used in lieu of entering struct or union; the gdb module first determines whether the argument is a structure or a union, and then calls the appropriate function.
p
 
Displays the contents of a kernel variable; the arguments are passed on to gdb's print command for proper formatting. Two builtin aliases, px and pd, set the numerical output radix to hexadecimal or decimal for the print operation, temporarily overriding the current default.
whatis
 
Displays all available symbol table information concerning a data type or a data symbol.
sym
 
Translates a kernel symbol name to its kernel virtual address and section, or a kernel virtual address to its symbol name and section. It can also be used to dump the complete list of kernel symbols, or to query the symbol list for all symbols containing a given sub-string.
dis
 
Disassembles the text of complete kernel function, or from a specified address for a given number of instructions, or from the beginning of a function up to a specified address.


System State

The majority of crash commands come from the following set of "kernel-aware" commands, which delve into various kernel subsystems on a system-wide or per-task basis. The task-specific commands are context-sensitive, meaning that they act upon the current context unless a PID or task address is specified as an argument.
Command
 
Description
bt
 
Arguably the most useful crash command, bt displays a task's kernel stack back-trace, including full exception frame dumps. It is context-sensitive, although the -a option will display the stack traces of the active task on each CPU. This command is often used within the foreach wrapper command in order to display the back traces of all tasks with one command.
dev
 
Displays data concerning the character and block device assignments, I/O port usage, I/O memory usage, and PCI device data.
files
 
This context-sensitive command displays the task's current root directory and working directories, and then for each open file descriptor, shows:
  • its file struct address
  • its dentry struct address
  • its inode struct address
  • the file type
  • the file's full pathname
Another option acts upon a specified dentry address, showing:
  • its inode struct address
  • its superblock struct address
  • the file type
  • the file's full pathname
It can be called from the foreach wrapper command.
fuser
 
Displays a list of tasks that reference a specified filename or inode address as the current root or working directory, an open file descriptor, or which mmap the file.
irq
 
Display data concerning interrupt request numbers and bottom-half handling.
kmem
 
This command has numerous options that delve into the state of several kernel memory subsystems:
  • general memory usage, similar in scope to /proc/meminfo
  • kmalloc slab memory allocator, including an option that lists each slab object and its state, verifying the slab chain
  • display and verification of free page lists
  • vmalloc memory allocator vmlist contents
  • display and verification of the page cache
  • the mem_map page list
  • display NUMA data, if applicable
Also, given an address, this command searches the symbol table, the slab subsystem, the free list, the page_hash_table, the vmlist, and the mem_map array, displaying where it was found.
log
 
Dumps the kernel message buffer chronologically, accounting for any wrap-around.
mach
 
Displays machine and/or processor specific data.
mod
 
Displays the list of currently-loaded kernel modules. More importantly, it loads the debug data from the module object files if they are available, allowing symbolic debugging capability of kernel modules.
mount
 
For each mounted filesystem, or for just a specified filesystem, displays:
  • its vfsmount struct address
  • its super_block struct address
  • its type
  • its device name
  • its mount point
Options exist to dump a list of a specified filesystem's open files or dirty inodes. Filesystems may be specified by vfsmount, super_block, or inode addresses, or by device name or mount point names.
net
 
Displays various network-related data:
  • displays each configured network device's net_device address, its name, and IP address
  • displays the ARP cache
  • context-sensitive display of information concerning the open sockets of a task
  • translates an IP address expressed as a decimal or hexadecimal value into a standard numbers-and-dots notation
It can be called from the foreach wrapper command.
ps
 
Useful process status command, in typical Linux ps command type output, containing:
  • PID number
  • PPID number
  • CPU number
  • task address
  • process state
  • percent of physical memory consumed
  • virtual address size
  • resident set size
  • command name
Also has an option to show a task's parental hierarchy back to the init process, and another to show all children of a task.
pte
 
This command translates the contents of a PTE into its physical page address and page bit settings, or if it references a swap location, the swap device and offset.
runq
 
Displays list of tasks on the run queue.
sig
 
A context-sensitive command which displays a task's signal information, including:
  • whether an unblocked signal is pending
  • the pending and blocked signals
  • the handler data for each signal
  • queued signals, if any
Other options list the signal number/names combination for a processor type, and translate the contents of a sigset_t into the signal names whose bits are set. It can be called from the foreach wrapper command.
swap
 
For each configured swap device, this command displays the same data that is shown by the Linux command swapon -s.
sys
 
Re-displays the same system-related data that is seen during crash initialization:
  • the kernel object filename
  • the dumpfile name
  • the number of CPUS
  • the date
  • system uptime
  • system load average
  • the number of tasks
  • the nodename
  • the kernel release and version data
  • the processor type and speed
  • the amount of memory
  • the panic string
Other options display information concerning the system call table, and one allows the root user to panic a live system.
task
 
This context-sensitive command displays a task's complete task_struct contents, or one or more members of the structure. This command is often used within the foreach wrapper command in order to display task_struct data for all tasks with one command.
timer
 
Displays the timer queue entries in chronological order, listing the target function names, the current value of jiffies, and the expiration time of each entry.
vm
 
This powerful, context-sensitive command displays a wealth of information concerning a task's virtual memory data, including:
  • its mm_struct address
  • its page directory address
  • its resident set size
  • its total virtual memory size
  • each vm_area_struct address, along with its start and ending virtual address, flags, and source file if applicable.
  • optionally, every virtual page referenced by a vm_area_struct can be translated into its physical address, or if not resident, its file and offset.
Other options tranlate the flags of a vm_area_struct, or display the full contents of a task's mm_struct or of each vm_area_struct. It can be called from the foreach wrapper command.
vtop
 
This context-sensitive command translates a user or kernel virtual address to its physical address. Also displayed are:
  • the full PTE translation from page directory through to the page table
  • the vm_area_struct data for user virtual addresses
  • the mem_map page data associated with the physical page
  • the swap location or file location if a user virtual page is not currently mapped
It can be called from the foreach wrapper command.
waitq
 
Lists the tasks linked on a specified kernel wait queue.


Utility Functions

The following commands are a set of useful helper commands serving various purposes, some simple, others quite powerful.
Command
 
Description
ascii
 
Translates a numerical value into its ASCII components; with no arguments, displays an ASCII chart.
btop
 
Translates a byte value (physical address) to its page number.
eval
 
A simple calculator, evaluates an expression and displays the result in hexadecimal, decimal, octal and binary, and optionally showing the bit numbers set in the result.
list
 
Dumps the entries of a linked list of structures. It can handle lists of structures that are singly-linked with simple "next" pointers, or those with embedded list_head structures. The output may be constrained to simply display the address of each structure in the list, or if directed, also dump each complete structure, or just one member of each structure. The gathered list entries are hashed, so a corrupted list that loops back upon itself will be recognized.
ptob
 
translates a page frame number to its byte value (physical address).
ptov
 
Translates a physical address into a kernel virtual address by adding the appropriate PAGE_OFFSET value.
search
 
Searches a range of user or kernel memory space for given value, with an optional "don't care" bit-mask argument.
rd
 
Displays a specified amount of user virtual, kernel virtual, or physical memory in several formats, such as 8, 16, 32 or 64 byte values, hexadecimal or decimal, symbolically, and with ASCII translations. When reading user virtual addresses, the command is context-sensitive.
wr
 
Modifies the contents of memory on a live system. Write permission on /dev/mem is required; this command should obviously be used with great care. The write operation is constrained to one 8, 16, 32 or 64 byte location.


Session Control Commands

The following commands typcally aid in the efficient running of a crash session.
Command
 
Description
alias
 
Creates a single-word alias for a command string. Several aliases are built into crash; user-defined aliases may also be defined in a .crashrc file, or during a crash session by entering it on the command line or reading it from an input file.
exit
 
Shuts down the crash session (same as q).
extend
 
Extend the crash command set by dynamically loading a shared object library containing one or more user-written commands.
foreach
 
Quite often it is helpful, or even necessary, to run the same crash context-sensitive command on a number of tasks by just entering one command. This wrapper command sets off the execution of a given crash command on each of a defined set of tasks, temporarily changing the current context to that of the targeted task before running the command. The set of tasks that are issued the given command can be defined by:
  • one or more PID numbers
  • one or more task numbers
  • one or more command name
  • all user tasks
  • all kernel tasks
  • the active task on each CPU
The identifiers above may be mixed if it makes sense, such as using a combination of PIDs, task addresses, and command names. The context-sensitive commands that can be issued to the selected tasks are: A header containing the PID, task address, CPU and command name will be pre-pended before the command output for each selected task.
gdb
 
This command passes its arguments directly to gdb for processing. This is typically not necessary, but where ambiguities between crash and gdb command names exist, this will force the command to be executed by gdb.
repeat
 
This wrapper command repeats a crash command indefinitely, optionally delaying a given number of seconds between each command execution. Obviously this command is only useful when running on a live system.
set
 
This primary purpose for this command is to set the crash context to a new task, or to display the current context. It can also be used to view or change one of a set of internal crash variables that modify program behavior, such as the default output radix or scrolling behavior. It can be called from the foreach wrapper command for viewing the context data of each task.
q
 
Shuts down the crash session (same as exit).


< Prev Contents Next >