White Paper: Red Hat Crash Utility


Contents

help page:  kmem


NAME
  kmem - kernel memory

SYNOPSIS
  kmem [-f|-F|-p|-c|-C|-i|-s|-S|-v|-n] [-[l|L][a|i]] [slab-name] [[-P] address]

DESCRIPTION
  This command displays information about the use of kernel memory.

        -f  displays the contents of the system free memory headers.
            also verifies that the page count equals nr_free_pages.
        -F  same as -f, but also dumps all pages linked to that header.
        -p  displays basic information about each page in the system 
            mem_map[] array.
        -c  walks through the page_hash_table and verifies page_cache_size.
        -C  same as -c, but also dumps all pages in the page_hash_table.
        -i  displays general memory usage information
        -s  displays basic kmalloc() slab data.
        -S  displays all kmalloc() slab data, including all slab objects,
            and whether each object is in use or is free.
        -v  displays the vmlist entries.
        -n  display NUMA data (if supported).
       -la  walks through the active_list and verifies nr_active_pages.
       -li  walks through the inactive_list and verifies nr_inactive_pages.
       -La  same as -la, but also dumps each page in the active_list.
       -Li  same as -li, but also dumps each page in the inactive_list.
 slab-name  when used with -s or -S, limits the command to only the slab
            of name "slab-cache".
        -P  declares that the following address argument is a physical address.
   address  when used without any flag, the address can be a kernel virtual,
            or physical address; a search is made through the symbol table,
            the kmalloc() slab subsystem, the free list, the page_hash_table,
            the vmalloc() vmlist subsystem, and the mem_map array. If found
            in any of those areas, the information will be dumped in the
            same manner as if the flags were used.
   address  when used with -s or -S, searches the kmalloc() slab subsystem
            for the slab containing of this virtual address, showing whether
            it is in use or free.
   address  when used with -f, the address can be either a page pointer,
            a physical address, or a kernel virtual address; the free_area
            header containing the page (if any) is displayed.
   address  when used with -p, the address can be either a page pointer, a
            physical address, or a kernel virtual address; its basic mem_map
            page information is displayed.
   address  when used with -c, the address must be a page pointer address;
            the page_hash_table entry containing the page is displayed.
   address  when used with -l, the address must be a page pointer address;
            the page address is displayed if it is contained with the list.
   address  when used with -v, the address can be a mapped kernel virtual
            address or physical address; the vmlist containing the address
            is displayed.

  All address arguments above must be expressed in hexadecimal format.

EXAMPLES
  Display memory usage information:

    crash> kmem -i
                  PAGES        TOTAL      PERCENTAGE
     TOTAL MEM    63602     248.4 MB         ----
          FREE      993       3.9 MB    1% of TOTAL MEM
          USED    62609     244.6 MB   98% of TOTAL MEM
        SHARED    34035     132.9 MB   53% of TOTAL MEM
       BUFFERS    10928      42.7 MB   17% of TOTAL MEM
        CACHED    35249     137.7 MB   55% of TOTAL MEM
          SLAB     2823        11 MB    4% of TOTAL MEM
    
    TOTAL HIGH        0            0    0% of TOTAL MEM
     FREE HIGH        0            0    0% of TOTAL HIGH
     TOTAL LOW    63602     248.4 MB  100% of TOTAL MEM
      FREE LOW      993       3.9 MB    1% of TOTAL LOW
    
    TOTAL SWAP   129792       507 MB         ----
     SWAP USED    14727      57.5 MB   11% of TOTAL SWAP
     SWAP FREE   115065     449.5 MB   88% of TOTAL SWAP
    
    ZONE  NAME      FREE   ACTIVE  INACTIVE_DIRTY  INACTIVE_CLEAN  MIN/LOW/HIGH
      0   DMA        240     1166               7             161  128/256/384 
      1   Normal     753    17009           27834               0  255/510/765 
      2   HighMem      0        0               0               0     0/0/0    
 
  Display and verify free memory data:

    crash> kmem -f
    NODE
      0
    ZONE  NAME        SIZE    FREE  MEM_MAP   START_PADDR  START_MAPNR
      0   DMA         4096    3372  c4000040       0            0     
    AREA  SIZE  FREE_AREA_STRUCT  BLOCKS  PAGES
      0     4k      c02eb004           2      2
      1     8k      c02eb010           3      6
      2    16k      c02eb01c           5     20
      3    32k      c02eb028           4     32
      4    64k      c02eb034           5     80
      5   128k      c02eb040           3     96
      6   256k      c02eb04c           3    192
      7   512k      c02eb058           1    128
      8  1024k      c02eb064           1    256
      9  2048k      c02eb070           5   2560
 
    ZONE  NAME        SIZE    FREE  MEM_MAP   START_PADDR  START_MAPNR
      1   Normal    225280  202269  c4044040    1000000        4096   
    AREA  SIZE  FREE_AREA_STRUCT  BLOCKS  PAGES
      0     4k      c02eb0b8           1      1
      1     8k      c02eb0c4           2      4
      2    16k      c02eb0d0           0      0
      3    32k      c02eb0dc           1      8
      4    64k      c02eb0e8           1     16
      5   128k      c02eb0f4           0      0
      6   256k      c02eb100           0      0
      7   512k      c02eb10c           0      0
      8  1024k      c02eb118           0      0
      9  2048k      c02eb124         395 202240
 
    ZONE  NAME        SIZE    FREE  MEM_MAP   START_PADDR  START_MAPNR
      2   HighMem   819200  748686  c4ee0040    38000000      229376  
    AREA  SIZE  FREE_AREA_STRUCT  BLOCKS  PAGES
      0     4k      c02eb16c          10     10
      1     8k      c02eb178           2      4
      2    16k      c02eb184           0      0
      3    32k      c02eb190           2     16
      4    64k      c02eb19c           1     16
      5   128k      c02eb1a8           1     32
      6   256k      c02eb1b4           1     64
      7   512k      c02eb1c0           0      0
      8  1024k      c02eb1cc           0      0
      9  2048k      c02eb1d8        1462 748544
    
    nr_free_pages: 954327  (verified)
 
  Dump all the base addresses of each free memory area from above:

    crash> kmem -F
    NODE
      0
    ZONE  NAME        SIZE    FREE  MEM_MAP   START_PADDR  START_MAPNR
      0   DMA         4096    3372  c4000040       0            0     
    AREA  SIZE  FREE_AREA_STRUCT
      0     4k      c02eb004      
    c400ded8
    c4042528
    AREA  SIZE  FREE_AREA_STRUCT
      1     8k      c02eb010      
    c400de50
    c400cee8
    c40424a0
    AREA  SIZE  FREE_AREA_STRUCT
      2    16k      c02eb01c      
    c400dd40
    c400cf70
    c40425b0
    c400f7d0
    c40028a0
    AREA  SIZE  FREE_AREA_STRUCT
      3    32k      c02eb028      
    c4042280
    c400f8e0
    c4002680
    c4000260
    AREA  SIZE  FREE_AREA_STRUCT
      4    64k      c02eb034      
    c400d080
    c4041e40
    ...
 
  Dump the mem_map[] array:

    crash> kmem -p
      PAGE    PHYSICAL   INODE     OFFSET  CNT FLAGS
    c0294000         0         0         0  0  DMA,reserved
    c0294028      1000         0         0  0  DMA,reserved
    c0294050      2000         0         0  0  DMA,reserved
    c0294078      3000         0    2cf000  1  DMA,uptodate
    c02940a0      4000         0         0  1  DMA,slab
    c02940c8      5000         0         0  1  DMA,slab
    c02940f0      6000         0         0  1  DMA,slab
    c0294118      7000         0         0  1  DMA,slab
    c0294140      8000         0         0  1  DMA,slab
    c0294168      9000         0         0  0  DMA,slab
    c0294190      a000         0         0  0  DMA,slab
    c02941b8      b000         0         0  0  DMA,slab
    c02941e0      c000         0         0  1  DMA,slab
    c0294208      d000         0         0  0  DMA,slab
    c0294230      e000         0         0  1  DMA,slab
    c0294258      f000         0         0  0  DMA,slab
    c0294280     10000         0     11000  1  DMA,uptodate
    c02942a8     11000         0     10000  1  DMA,uptodate
    c02942d0     12000         0     1e000  1  DMA,uptodate
    c02942f8     13000         0    153000  1  DMA,uptodate
    c0294320     14000  c3243650     4b000  1  DMA,referenced,uptodate
    c0294348     15000  c3243650     4a000  1  DMA,referenced,uptodate
    ...
    
  Use the commands above with a page pointer or a physical address argument:

    crash> kmem -f c40425b0
    NODE 
      0 
    ZONE  NAME        SIZE    FREE  MEM_MAP   START_PADDR  START_MAPNR
      0   DMA         4096    3372  c4000040       0            0     
    AREA  SIZE  FREE_AREA_STRUCT 
      2    16k      c02eb01c      
    c40425b0  (c40425b0 is 1st of 4 pages) 
 
    crash> kmem -p c035de00
      PAGE    PHYSICAL   INODE     OFFSET  CNT FLAGS
    c035de00   50c0000         0    129000  0  uptodate

    crash> kmem -p 50c0000
      PAGE    PHYSICAL   INODE     OFFSET  CNT FLAGS
    c035de00   50c0000         0    129000  0  uptodate

  Display the vmlist entry data:

    crash> kmem -v
    VM_STRUCT      ADDRESS RANGE       SIZE
     c009c560   c8000000 - c8002000    8192
     c009c620   c8002000 - c8004000    8192
     c009cda0   c8004000 - c8016000   73728
     c009cd70   c8016000 - c8019000   12288
     c009cf80   c8019000 - c801b000    8192
     c009cfb0   c801b000 - c801d000    8192
     c009cef0   c801d000 - c802d000   65536
     c3afd060   c802d000 - c8032000   20480
     c3afd090   c8032000 - c8035000   12288
     c3afd0c0   c8035000 - c8037000    8192
     c3afd150   c8037000 - c8039000    8192
     c3afd180   c8039000 - c803b000    8192
     c3afd210   c803b000 - c803d000    8192
     c3afd2a0   c803d000 - c8040000   12288
     c3afd2d0   c8040000 - c8043000   12288
     c3afd300   c8043000 - c8047000   16384
     c3afddb0   c8047000 - c804d000   24576
     c2f8a320   c804d000 - c805c000   61440
     c2f8a380   c805c000 - c8065000   36864
     c2f8a3b0   c8065000 - c806e000   36864
     c2f8aa70   c806e000 - c8095000  159744
     c2f8ab60   c8095000 - c8097000    8192
     c2f519e0   c8097000 - c8099000    8192
 
  Determine (and verify) the page cache size:

    crash> kmem -c
    page_cache_size: 18431 (verified)
 
  Dump all pages in the page_hash_table:

    crash> kmem -C
    page_hash_table[0]
    c0325b40
    c03a0598
    c03b4070
    c0364c28
    c0357690
    c02ef338
    c02d7c60
    c02c11e0
    c02a3d70
    page_hash_table[1]
    c0394ce8
    c03c4218
    c03b4048
    c0364c00
    c0357668
    c02d6e50
    c02d7dc8
    c02c0cb8
    c02db630
    c02ebad0
    page_hash_table[2]
    c037e808
    c034e248
    c03b4020
    c02ec868
    c03baa60
    ...
    page_hash_table[2047]
    c033a798
    c0390b48
    c03b4098
    c0364890
    c03576b8
    c02d2c38
    c02d7c88
    c02de5d8
    
    page_cache_size: 18437 (verified)
    
  Find the page_hash_table entry containing page c03576b8:

    crash> kmem -c c03576b8
    page_hash_table[2047]
    c03576b8
 
  Display kmalloc() slab data:

    crash> kmem -s
    CACHE     NAME                OBJSIZE  ALLOCATED     TOTAL  SLABS  SSIZE
    c02eadc0 kmem_cache               232         58        68      4     4k
    f79c2888 ip_vs_conn               128          0         0      0     4k
    f79c2970 tcp_tw_bucket             96          0         0      0     4k
    f79c2a58 tcp_bind_bucket           32         12       565      5     4k
    f79c2b40 tcp_open_request          64          0        59      1     4k
    f79c2c28 inet_peer_cache           64          1        59      1     4k
    f79c2d10 ip_fib_hash               32         11       339      3     4k
    f79c2df8 ip_dst_cache             160          8       120      5     4k
    f79c2ee0 arp_cache                128          1        30      1     4k
    c8402970 blkdev_requests           96      30208     37800    945     4k
    c8402a58 nfs_read_data            384          0         0      0     4k
    c8402b40 nfs_write_data           384          0         0      0     4k
    c8402c28 nfs_page                  96          0         0      0     4k
    c8402d10 dnotify cache             20          0         0      0     4k
    c8402df8 file lock cache           92          3       336      8     4k
    c8402ee0 fasync cache              16          0         0      0     4k
    c84027a0 uid_cache                 32          3       339      3     4k
    c84026b8 skbuff_head_cache        160        320       624     26     4k
    c84025d0 sock                     832         32       180     20     8k
    c84024e8 sigqueue                 132          0       203      7     4k
    c8402400 cdev_cache                64         19       472      8     4k
    c8402318 bdev_cache                64          8       236      4     4k
    c8402230 mnt_cache                 96         11       120      3     4k
    c8402148 inode_cache              480        817       848    106     4k
    c8402060 dentry_cache             128       1352      1470     49     4k
    c8403ee0 filp                      96        244       440     11     4k
    c8403df8 names_cache             4096          0        12     12     4k
    c8403d10 buffer_head               96      14936     16000    400     4k
    c8403c28 mm_struct                128         25       240      8     4k
    c8403b40 vm_area_struct            64        393      1298     22     4k
    c8403a58 fs_cache                  64         30       472      8     4k
    c8403970 files_cache              416         30       135     15     4k
    c8403888 signal_act              1312         32        99     33     4k
    c84037a0 size-131072(DMA)      131072          0         0      0   128k
    c84036b8 size-131072           131072          1         1      1   128k
    c84035d0 size-65536(DMA)        65536          0         0      0    64k
    c84034e8 size-65536             65536          0         0      0    64k
    c8403400 size-32768(DMA)        32768          0         0      0    32k
    c8403318 size-32768             32768          0         1      1    32k
    c8403230 size-16384(DMA)        16384          0         0      0    16k
    c8403148 size-16384             16384          0         0      0    16k
    c8403060 size-8192(DMA)          8192          0         0      0     8k
    c8401ee0 size-8192               8192          1         2      2     8k
    c8401df8 size-4096(DMA)          4096          0         0      0     4k
    c8401d10 size-4096               4096         30        30     30     4k
    c8401c28 size-2048(DMA)          2048          0         0      0     4k
    c8401b40 size-2048               2048         37       132     66     4k
    c8401a58 size-1024(DMA)          1024          0         0      0     4k
    c8401970 size-1024               1024        301       328     82     4k
    c8401888 size-512(DMA)            512          0         0      0     4k
    c84017a0 size-512                 512        141       168     21     4k
    c84016b8 size-256(DMA)            256          0         0      0     4k
    c84015d0 size-256                 256         80       435     29     4k
    c84014e8 size-128(DMA)            128          0         0      0     4k
    c8401400 size-128                 128        508       840     28     4k
    c8401318 size-64(DMA)              64          0         0      0     4k
    c8401230 size-64                   64        978      1357     23     4k
    c8401148 size-32(DMA)              32          0         0      0     4k
    c8401060 size-32                   32       1244      1808     16     4k
 
  Display all slab data in the "arp_cache" cache:

    crash> kmem -S arp_cache
    CACHE     NAME                OBJSIZE  ALLOCATED     TOTAL  SLABS  SSIZE
    f79c2ee0 arp_cache                128          1        30      1     4k
    SLAB      MEMORY    TOTAL  ALLOCATED  FREE
    f729d000  f729d0a0     30          1    29
    FREE / [ALLOCATED]
       f729d0a0  (cpu 7 cache)
       f729d120  (cpu 7 cache)
       f729d1a0  (cpu 7 cache)
       f729d220  (cpu 7 cache)
       f729d2a0  (cpu 7 cache)
       f729d320  (cpu 7 cache)
       f729d3a0  (cpu 7 cache)
       f729d420  (cpu 7 cache)
       f729d4a0  (cpu 7 cache)
       f729d520  (cpu 7 cache)
       f729d5a0  (cpu 7 cache)
       f729d620  (cpu 7 cache)
       f729d6a0  (cpu 7 cache)
       f729d720  (cpu 7 cache)
       f729d7a0  (cpu 7 cache)
       f729d820  (cpu 7 cache)
       f729d8a0  (cpu 7 cache)
       f729d920  (cpu 7 cache)
       f729d9a0  (cpu 7 cache)
       f729da20  (cpu 7 cache)
       f729daa0  (cpu 7 cache)
       f729db20  (cpu 7 cache)
       f729dba0  (cpu 7 cache)
       f729dc20  (cpu 7 cache)
       f729dca0  (cpu 7 cache)
       f729dd20  (cpu 7 cache)
       f729dda0  (cpu 7 cache)
       f729de20  (cpu 7 cache)
       f729dea0  (cpu 3 cache)
      [f729df20]
 
  Search the kmalloc() slab subsystem for address c3fbdb60:

    crash> kmem -s c3fbdb60
    CACHE     NAME                OBJSIZE  ALLOCATED     TOTAL  SLABS  SSIZE
    c8402970 blkdev_requests           96      30208     37800    945     4k
    SLAB      MEMORY    TOTAL  ALLOCATED  FREE
    c3fbd020  c3fbd0e0     40         40     0
    FREE / [ALLOCATED]
      [c3fbdb60]
 
  Make a generic search (no flags) for the same address c3fbdb60:

    crash> kmem c3fbdb60 
    CACHE     NAME                OBJSIZE  ALLOCATED     TOTAL  SLABS  SSIZE
    c8402970 blkdev_requests           96      30208     37800    945     4k
    SLAB      MEMORY    TOTAL  ALLOCATED  FREE
    c3fbd020  c3fbd0e0     40         40     0 
    FREE / [ALLOCATED]
      [c3fbdb60]
 
      PAGE     PHYSICAL   MAPPING    INDEX CNT FLAGS
    c410ee74    3fbd000         0         0  1 slab
 
  Verify the active and inactive page lists:

    crash> kmem -la -li
    nr_active_pages: 1893 (verified)
    nr_inactive_pages: 2491 (verified)

  Display NUMA data (if supported):

    crash> kmem -n
    NODE    SIZE    PGLIST_DATA   BOOTMEM_DATA   NODE_ZONES
      0    130933     c0332ee0      c0403a44      c0332ee0
                                                  c03333e0
                                                  c03338e0
    MEM_MAP   START_PADDR  START_MAPNR
    c1000030       0            0
    
    ZONE  NAME        SIZE   MEM_MAP  START_PADDR  START_MAPNR
      0   DMA         4096  c1000030            0            0
      1   Normal    126837  c1038030      1000000         4096
      2   HighMem        0         0            0            0


Contents