White Paper: Red Hat Crash Utility


< Prev Contents Next >

Invocation

When crash is run on a dumpfile, at least two arguments are always required:

  1. The kernel object filename, often referred to as the kernel namelist. When initially built from the kernel sources, its name is vmlinux; typically it is then copied to the /boot directory, where the operating system release number is appended, as in vmlinux-2.4.21-4.ELsmp.
  2. The dumpfile name. For example, Netdump-generated dumpfiles are named vmcore.
For example:

# crash vmlinux vmcore

When crash is run on a live system, /dev/mem is used as the memory image. Therefore, only the kernel object filename is required:

# crash vmlinux

Furthermore, when crash is run on a live system, the vmlinux argument is not required when the kernel object file is located in any of the following locations:

  • / (root directory)
  • /boot
  • /boot/efi/EFI/redhat
  • any subdirectory of /usr/src
  • /usr/src/redhat/BUILD/kernel-x.x.x/linux
  • /usr/lib/debug/lib/modules/<release><flavor>

When the vmlinux file is not entered on the command line, a search will be made in all of the directories above until a kernel object file is found that contains a version string that matches the running kernel, as indicated by /proc/version. If a matching kernel is found, then crash may be invoked on a live system simply by entering:

# crash

In the examples above, it is presumed that the vmlinux kernel has been built with the -g C flag, which traditionally has not been done by default. To address this requirement, Red Hat Enterprise Linux 3 (RHEL 3) and Red Hat Enterprise Linux 4 (RHEL 4) kernels are now built with -g C flag. The manner of accessing the debug data for RHEL 3 and RHEL 4 kernels is described in the following two sections. Unfortunately, since Red Hat Advanced Server 2.1 kernels are not built with -g, the kernel must be rebuilt; directions for rebuilding Red Hat Advanced Server 2.1 kernels can be found here.

RHEL-3 Kernels

In RHEL 3, the vmlinux kernel debug information is stripped and stored in a separate debuginfo file. The stripped vmlinux file in /boot has an embedded link to its associated debuginfo file in /usr/lib/debug/boot, so that the crash utility (and the built-in gdb module) knows where to find it:

RHEL 3 kernel namelist in /boot RHEL 3 kernel debuginfo file in /usr/lib/debug/boot
vmlinux-<release>.EL
vmlinux-<release>.ELsmp
vmlinux-<release>.ELhugemem
vmlinux-<release>.EL.debug
vmlinux-<release>.ELsmp.debug
vmlinux-<release>.ELhugemem.debug

The debuginfo files for a specific kernel <release> come from a separate RPM that must be installed for the crash utility to work. For example, the i686 RPM for the examples above would be named kernel-debuginfo-<release>.i686.rpm, and would install the debuginfo file for all three of the kernel flavors.

For example, to run crash on a live system, the associated debuginfo package must be installed:

# uname -r
2.4.21-4.ELsmp
# rpm -ivh kernel-debuginfo-2.4.21-4.EL.i686.rpm
Preparing...                ########################################### [100%]
   1:kernel-debuginfo       ########################################### [100%]
# ls /usr/lib/debug/boot
vmlinux-2.4.21-4.EL.debug
vmlinux-2.4.21-4.ELhugemem.debug
vmlinux-2.4.21-4.ELsmp.debug
#

Accordingly, if the running kernel's vmlinux file is in one the search locations above, and its associated debuginfo file is located in the /usr/lib/debug/boot directory or in the current directory from which crash is invoked, no arguments are required to run on a live system:

# crash

However, if the linked debuginfo file is not in either of those locations, it can be added to the crash command line along with the vmlinux filename. So, for example, if the debuginfo file was located in /tmp:

# crash /boot/vmlinux-2.4.21-4.ELsmp /tmp/vmlinux-2.4.21-4.ELsmp.debug

For analyzing dumpfiles however, the vmlinux file name must be on the command line along with the dumpfile name, as in the following examples:

# crash /boot/vmlinux-2.4.21-4.ELsmp vmcore

or if the debuginfo file is not in the standard location:

# crash /boot/vmlinux-2.4.21-4.ELsmp /tmp/vmlinux-2.4.21-4.ELsmp.debug vmcore

RHEL-4 Kernels

The procedure has been made much simpler for RHEL-4 kernels. The kernel is built with the -g flag, and the resultant vmlinux file is stored in the associated debuginfo package. After installing the debuginfo package, the vmlinux file for each kernel flavor of a given RHEL 4 release will be installed in the directory named:

/usr/lib/debug/lib/modules/<release><flavor>/vmlinux

where for i686 kernels, <flavor> can be either hugemem, smp, or nothing (for uniprocessor kernels). For example:

# uname -r
2.6.9-6.39.ELsmp
# rpm -ivh kernel-debuginfo-2.6.9-6.39.EL.i686.rpm
Preparing...                ########################################### [100%]
   1:kernel-debuginfo       ########################################### [100%]
#
# find /usr/lib/debug/lib/modules/2.6.9-6.39.EL* -name vmlinux
/usr/lib/debug/lib/modules/2.6.9-6.39.ELhugemem/vmlinux
/usr/lib/debug/lib/modules/2.6.9-6.39.ELsmp/vmlinux
/usr/lib/debug/lib/modules/2.6.9-6.39.EL/vmlinux
#

Once the debuginfo package is installed, crash can be invoked on the live system with no arguments, because the vmlinux file will be found automatically:

# crash

To run crash on a dumpfile, however, the appropriate vmlinux file and the dumpfile name must both be on the command line, as in:

# crash /usr/lib/debug/lib/modules/2.6.9-6.39.ELsmp/vmlinux vmcore 

Kernels built without -g flag

If the running kernel was not built with the -g C flag, then it is necessary to rebuild a kernel of the same configuration with the -g C flag. The essential change done by this kernel rebuild task is a modification of top-level Makefile of the kernel source tree, such that the CFLAGS definition contains the -g flag. For example, this is the line that must be changed:

CFLAGS := $(CPPFLAGS) -Wall -Wstrict-prototypes -Wno-trigraphs -O2 \
          -fno-strict-aliasing -fno-common

by adding the -g flag:

CFLAGS := $(CPPFLAGS) -Wall -Wstrict-prototypes -Wno-trigraphs -O2 \
          -fno-strict-aliasing -fno-common -g

For example, since Red Hat Advanced Server 2.1 kernels are not built with -g, a kernel rebuild is required. For a detailed example of how to rebuild a Red Hat Advanced Server 2.1 kernel with the -g flag, please refer to these directions.

Upon rebuilding the kernel, a new vmlinux file will be created that contains the debug data required by crash. However, the symbol values will not match those of the running or dumped kernel. To deal with this inequity, the actual symbol values can be gathered from either the original non-debug vmlinux file or from its associated System.map file. That being the case, two arguments must be supplied to crash to fully describe the running/dumped kernel, the newly-created vmlinux file compiled with -g, as well as a source of the real symbol values. So, for example, if the vmlinux file built with -g were renamed to vmlinux.dbg, the invocation line would look like this on a live system:

# crash vmlinux vmlinux.dbg
(or)
# crash /boot/System.map vmlinux.dbg
(or)
# crash -S vmlinux.debug

The -S argument above is simply an alternative to entering the default /boot/System.map string.

Similarly, when looking at a dumpfile, two arguments are required to describe the dumped kernel, along with the vmcore image:

# crash vmlinux vmlinux.dbg vmcore
(or)
# crash /boot/System.map vmlinux.dbg vmcore 
(or)
# crash -S vmlinux.dbg vmcore 

Again, for a detailed example of how to rebuild a Red Hat Advanced Server 2.1 kernel with the -g flag, refer to these directions.

Invocation output

The arguments may be entered in any order. If the file arguments are not in the current directory, absolute pathnames must be used. When in doubt, simply enter crash -h to get an explanation of the command line arguments:

# crash -h

Usage:
  crash [-h [opt]][-v][-s][-i file][-d num] [-S] [mapfile] [namelist] [dumpfile]

  [namelist]
    The [namelist] argument is a pathname to an uncompressed kernel image
    (a vmlinux file) that has been compiled with the "-g" switch, or
    that has an accessible, associated, debuginfo file.  If the [dumpfile]
    argument is entered, then the [namelist] argument must be entered
    If the [namelist] argument is not entered when running on a live
    system, a search will be made in several typical directories for
    for a kernel namelist file that matches the live system.
 
  [dumpfile]
    The [dumpfile] argument is a pathname to a kernel memory core dump
    file.  If the [dumpfile] argument is not entered, the session will be
    invoked on the live system using /dev/mem, which usually requires root
    privileges.
 
  [mapfile]
    If the live system kernel, or the kernel from which the [dumpfile] 
    was derived, was not compiled with the -g switch, then the additional
    [mapfile] argument is required.  The [mapfile] argument may consist
    of either the associated System.map file, or the non-debug kernel
    namelist.  However, if the [mapfile] argument is used, then the
    [namelist] argument must be a kernel namelist of a similar kernel
    version that was built with the -g switch.
 
  [-S]
    Use "/boot/System.map" as the [mapfile].
 
    Examples when running on a live system:
 
      $ crash
      $ crash /usr/tmp/vmlinux
      $ crash /boot/System.map vmlinux.dbg
      $ crash -S vmlinux.dbg
      $ crash vmlinux vmlinux.dbg
 
    Examples when running on a dumpfile:
 
      $ crash vmlinux vmcore
      $ crash /boot/System.map vmlinux.dbg vmcore
      $ crash -S vmlinux.dbg vmcore
      $ crash vmlinux vmlinux.dbg vmcore
 
  [-h [opt]]
    The -h option alone displays this message.  If the [opt] argument is
    a crash command name, the help page for that command is displayed.  If
    the string "input" is entered, a page describing the various crash
    command line input options is displayed.  If the string "output" is
    entered, a page describing command line output options is displayed.
 
  [-v]
    Display the versions of crash and gdb making up this executable.
 
  [-s]
    Do not display any version, GPL, or crash initialization data; proceed
    directly to the "crash>" prompt.
 
  [-i file]
    Execute the crash command(s) in [file] prior to accepting any user
    input from the "crash>" prompt.
 
  [-d num]
    Set crash debug level [num].  The higher the number, the more debug data
    will be printed during crash runtime.

Given that all invocation arguments are in order, here is an example of a successful invocation on a dumpfile, running a kernel that was built with -g, along with a vmcore dump file was created by the Red Hat Netdump facility:

# crash vmlinux-2.4.20-2.1.15.entsmp vmcore

crash 3.7-1
Copyright (C) 2002, 2003  Red Hat, Inc.
Copyright (C) 1998-2003  Hewlett-Packard Co
Copyright (C) 1999, 2002  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.

GNU gdb Red Hat Linux (5.3post-0.20021129.36rh)
Copyright 2003 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...

      KERNEL: vmlinux-2.4.20-2.1.15.entsmp
    DUMPFILE: vmcore
        CPUS: 1
        DATE: Wed Mar 12 10:12:56 2003
      UPTIME: 00:38:25
LOAD AVERAGE: 1.16, 0.74, 0.30
       TASKS: 60
    NODENAME: dhcp64-220.boston.redhat.com
     RELEASE: 2.4.20-2.1.15.entsmp
     VERSION: #1 SMP Tue Mar 11 16:12:22 EST 2003
     MACHINE: i686  (501 Mhz)
      MEMORY: 128 MB
       PANIC: "Oops: 0002" (check log for details)
         PID: 0
     COMMAND: "swapper"
        TASK: c038e000
         CPU: 0
       STATE: TASK_RUNNING (PANIC)

crash>

This next example shows the output when the panicking kernel was not built with -g. In this case, a similar kernel type was built with -g, and the resultant kernel object file was renamed as vmlinux.dbg. Note that there will be a message concerning the patching of gdb data; this indicates that the non-matching symbol values from the vmlinux.dbg are being over-written by the correct symbol values found in the original vmlinux file:

# crash vmlinux vmlinux.dbg vmcore

crash 3.7-1
Copyright (C) 2002, 2003  Red Hat, Inc.
Copyright (C) 1998-2003  Hewlett-Packard Co
Copyright (C) 1999, 2002 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
 
GNU gdb Red Hat Linux (5.3post-0.20021129.36rh)
Copyright 2003 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...

please wait... (patching 16053 gdb minimal_symbol values)

DEBUG KERNEL: vmlinux.dbg
    DUMPFILE: vmcore
        CPUS: 1
        DATE: Wed Mar 27 11:02:31 2002
      UPTIME: 00:07:24
LOAD AVERAGE: 0.43, 0.42, 0.19
       TASKS: 68
    NODENAME: anderson.boston.redhat.com
     RELEASE: 2.4.9-26beta.48enterprise
     VERSION: #1 SMP Thu Mar 21 12:33:05 EST 2002
     MACHINE: i686  (501 Mhz)
      MEMORY: 128 MB
       PANIC: "Oops: 0002" (check log for details)
         PID: 1696
     COMMAND: "insmod"
        TASK: c74de000  
         CPU: 0
       STATE: TASK_RUNNING (PANIC)

crash> 

Invocation on a live system looks essentially the same, except that the DUMPFILE will be indicated as /dev/mem. In the following example, no arguments were entered, because the running RHEL 3 kernel was found in the /boot directory, and its associated debuginfo file in the /usr/lib/debug/boot directory. The debuginfo file is listed next to the DEBUGINFO tag:

# crash

crash 3.7-1
Copyright (C) 2002, 2003  Red Hat, Inc.
Copyright (C) 1998-2003  Hewlett-Packard Co
Copyright (C) 1999, 2002  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.

GNU gdb Red Hat Linux (5.3post-0.20021129.36rh)
Copyright 2003 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...

      KERNEL: /boot/vmlinux-2.4.21-4.ELhugemem
   DEBUGINFO: /usr/lib/debug/boot/vmlinux-2.4.21-4.ELhugemem.debug
    DUMPFILE: /dev/mem
        CPUS: 2
        DATE: Thu Aug 21 11:24:38 2003
      UPTIME: 1 days, 23:14:11
LOAD AVERAGE: 0.14, 0.10, 0.08
       TASKS: 106
    NODENAME: crash.boston.redhat.com
     RELEASE: 2.4.21-4.ELhugemem
     VERSION: #1 SMP Wed Aug 13 21:33:31 EDT 2003
     MACHINE: i686  (1993 Mhz)
      MEMORY: 511.5 MB
         PID: 4757
     COMMAND: "crash"
        TASK: 19b82000
         CPU: 1
       STATE: TASK_RUNNING (ACTIVE)

crash>

Invocation Errors

Invocation errors will cause the crash session to abort upon initialization. Typically they occur as the result of one of the following reasons:

  1. The vmlinux file contains no debug data (i.e., was built without the -g flag), and no additional debug kernel object file name was entered on the command line. The error message will be of the form:

    crash: /boot/vmlinux-2.4.18-14: no debugging data available
    

  2. The vmlinux file does not match the dumpfile. The error message will be of the form:

    crash: vmlinux and tmp/vmcore do not match!
    

  3. The vmlinux file could not be found on a live system. The error message will be of the form:

    crash: cannot find booted kernel -- please enter namelist argument
    

  4. The associated debuginfo file cannot be found. The error message will be of the form:

    crash: /boot/vmlinux-2.4.21-4.ELsmp: no debugging data available
    crash: vmlinux-2.4.21-4.ELsmp.debug: debuginfo file not found
    



< Prev Contents Next >