Page 1  
  Analysis and Correction of Fragmentation Problems
 

Written By: Stan Sieler - Allegro Consultants

0. Tutorial Goals

An understanding of disk and file fragmentation on MPE/iX, and an introduction to the internals of Volumes, Disks, Files, and Databases.

0.1 Tutorial Organization

This tutorial is broken into four sections: definition of terms, exploration, analysis, and correction.

The definition of terms section will define various terms used throughout the tutorial.

The exploration section looks at information about volume sets, volumes, disks, files, and databases using widely available free tools, and some third party products.

The analysis section defines fragmentation on a per disk basis, a per-file basis, and a per-database basis, and uses various tools to explore the status of files and disks.

The correction section demonstrates the use of various tools to control all three types of fragmentation.

1. Definition of terms

This section will define the terms:

Volume Set, Volume Class, Volume Name

Volume

A Volume is a disk drive. Every volume has a name (e.g., "MEMBER2"). Every in-use volume belongs to exactly one Volume Set. The first (original) volume in each volume set is called the Master Volume.

Volume Set

A Volume Set is a set (list) of consisting of one or more volumes (named disks). (The maximum number of volumes in a single volume set is 255.)

Volume Class

A Volume Class is a named subset of a volume set, and consists of one or more volumes. Note that a particular volume may belong to more than one volume class. Since a volume class is a subset of a volume set, it cannot contain volumes belonging to more than one volume set. (The maximum number of volume classes in a volume set is 255.)

Example:

We have three disk drives, currently mounted on ldevs 11, 12, and 13. Ldev 11 is a big drive (4.2 GB), ldev 12 is a small (500 MB) but fast drive, and ldev 13 is a medium size (1GB) drive. Currently, the three disk drives have the following volumes mounted, all part of volume set USERS:

Ldev Volume Volume_Set notes
 11  USER1  USERS      big
 12  USER2  USERS      fast
 13  USER3  USERS
              

Volume Set USERS has three volumes: USER1, USER2, and USER3.

Volume Class DISC has three volumes: USER1, USER2, and USER3.

Volume Class FASTUSER has one volume: USER2

Volume Class BIG has one volume: USER1

Volume Class NOTFAST has two volumes: USER1 and USER3

(Strictly speaking, the volume classes shown above should have a prefix of "USERS:", to indicate that they belong to the volume set USERS.)

A BUILD (or FOPEN) of a new file in a group that is "HOMEVS"'d to the USERS volume set will be built as follows:

DEV=            Volume used
omitted         ("DISC") any (USER1, USER2, USER3)
11              USER1
12              USER2
13              USER3
FASTUSER        USER2
BIG             USER1
NOTFAST  either USER1 or USER3
FOO             error (unknown volume class name)
9000            error (maximum disk ldev is 8192)
1               error (ldev 1 is in the volume set
                       MPEXL_SYSTEM_VOLUME_SET)
              

Note that the above example didn't say "Ldev used", but instead "Volume used". Thus if you changed the ldev numbers of the three disks (say, to 200, 5, and 333), rebooted, and tried again most of the above examples would have the same results (i.e., "BIG" would result in USER1). The only ones that would "change" are the ones where a specific ldev is requested. The ldev is translated to a volume at that time (e.g., originally ldev 11 was translated to USER1, and after the change/reboot, ldev 11 would be an error, and ldev 200 would translate to USER1).

Most users don't change the ldev -to- volume mapping once a volume is created. However, on a system with several removable disks (e.g., 7935 disk drives), it is fairly likely that a given disk pack will be inserted into random drives from time to time. MPE/iX is extremely resilient, in that there is no hardcoded mapping between ldev and volume. At bootup (or volume set open) time, MPE builds a table of ldevs/volumes, corresponding to what it sees actually mounted.

The "system disk" of MPE/iX is actually just another volume set, called MPEXL_SYSTEM_VOLUME_SET. Note that although long volume set names are allowed, they aren't required! At our site, we have two extra volume sets: USERS and TEST.

At bootup, MPE/iX tries to "open" every volume set that is mounted. With a partial exception for MPEXL_SYSTEM_VOLUME_SET, MPE will not open a volume set that doesn't have every volume mounted. Instead, it will wait until all volumes are mounted.

Directory

The "directory" is the set of data structures that keeps track of file names, account names, group names, and user names.

The directory is implemented as a set of hierarchical files, with the entries in each file being kept in alphabetic order.

The top-level file, the "root" of the system directory (known internally as $ROOT (or "/" from the POSIX viewpoint)), contains a list of all account and, as of MPE/iX 4.5, all root-level files and directories (e.g., "/foo" or "/tmp").

There are three kinds of entries in the $ROOT file:

  • POSIX file names (e.g., /foo)
  • POSIX directories (e.g., /tmp)
  • MPE account names (e.g., ALLEGRO, or SYS)

Every account entry has two "pointers". The first of these "pointers" is to a file containing a list of all groups in the account. This file is called a $GROUP_NODE, or (in POSIX), /accountname (e.g., /SYS). The second "pointer" is to a file containing a list of all users in the account. This file is called a $USER_NODE, and currently has no POSIX name.

All other levels of the directory are simply files that contain a list of files and directories at that level. For example, the file /SYS/PUB contains a list of all files in @.PUB.SYS.

The "lists of files and directories" is a sorted, alphabetic order list. Each entry is a pair consisting of a name (e.g., "EDITOR") and a "pointer" to the file label for the file. The "pointer" is actually a UFID (Unique File IDentifi er), which can be thought of as an ldev and a sector offset of the file label, although this is overly simplified.

File

For this tutorial, a "file" means "a disk file", not a printer, or a terminal, or any other kind of non-disk file.

File Label

A file label is a 512 byte data area that is used to keep track of information about a file. This information includes: creation date, record size, creator, file name (usually), and more. In a few cases, the file label will point to "extension labels" that have additional information (e.g., ACD security).

If a file has disk space allocated, the file label will point to the start of a list of records that keep track of disk allocation for the file (see Extents and Extent Blocks below).

HFS

With the release of MPE/iX 4.5 and 5.0, MPE has been enhanced to support a full featured hierarchical file system, like MULTICS or MCP. (Some would say "like UNIX", but UNIX is a latecomer here.)

HFS (Hierarchical File System) refers to the POSIX file names that aren't representable as MPE file names. (E.g., /tmp)

Label Table

A Label Table is a file that stores file labels and extent blocks (see below).

On MPE/iX, a file label is not contiguous with the first record of data of the file. In fact, it is possible for a file label to exist, and no other disk space be associated with the file. (Such a file would show as having 0 sectors in a LISTF,2 command.)

On MPE V, the file label for a file immediately preceded the first record of data for the file on disk. (This also implied that MPE couldn't allocate a file label on disk without also allocating the first extent of the file's data.)

Every volume has a Label Table. Using our "USERS" volume set from above, the volumes USER1, USER2, and USER3 each have a Label Table. If you BUILD/FOPEN a new file on USERS, MPE (or you) picks a volume to put the file on (or, at least, the file label). If USER1 was picked, for whatever reason, the file label for the new file will be stored in the Label Table on USER1. Generally, but not always, the first extent of a file's data will be allocated on the same disk drive as the file's label is on.

When a volume is mounted, MPE "maps" the Label Table into a virtual address range, allowing it (and us) to view the SSM via LOAD/STORE instructions (or via Debug's DV command).

When you do a LISTF/LISTFILE with the -3 option, the virtual address shown for the file label is within a Label Table.

Although not guaranteed, the Label Table for ldev 1 has always started at virtual address $11.0.

UFID

A UFID (Unique File IDentifier), is a "pointer" to a file. In essence, the UFID can be thought of as specifying a volume and an index into the Label Table on that volume where a file label is found. A UFID is 20 bytes of binary data.

The entries in the directory referred to earlier as "pointers" are UFIDs. (example UFID)

Page, Sector

On MPE/iX, disk is allocated in units of a page, where a page is 4,096 bytes.

On MPE V, disk was allocated in units of sectors, where a sector is 256 bytes.

You might encounter disk diagnostic utilities that report that a disk drive has a sector size of 512 bytes instead of 256. This is because some disk drives have a minimum hardware addressable unit of 512 bytes instead of 256. However, this tutorial will always mean "256 bytes" when referring to a "sector". In all cases, every disk page starts on a 4,096 byte boundary on the disk.

Extent

An extent is a contiguous block of disk data allocated to a file.

Extents are always allocated on page boundaries on disk, and always occupy an integral number of pages. Thus, the smallest file that actually has disk data will occupy at least 1 page, or 4,096 bytes, or 16 sectors.

On MPE V, no file could have more than 30 extents, and no file could have less than 1 extent.

On MPE/iX, a file can have 0 or more extents. There is no hard coded limit, the number of extents is stored in a 16 bit field, thus imposing a functional limit of 65,535 extents.

Extent Block

An extent block keeps track of up to 20 extents for a single file. Extent Blocks are stored in the same Label Table as the associated file's file label.

If a file has at least one page of disk allocated to it, it will have at least one extent block in use in the Label Table.

When all 20 entries in an extent block are used, the next request for allocating disk space for the file will cause a new extent block to be allocated. The entries are not sorted in any order whatsoever.

A sample extent entry in an extent block looks like:

EXTENT_SECTOR_ADDR : 3f7ee0
SECTORS_IN_EXTENT  : 800
VOL_SET_INDEX      : 1
FILE_SECTOR_OFFSET : b98c00
              

The VOL_SET_INDEX specified which volume of a volume set the extent is on ... which means that it isn't necessarily on the same disk that the extent block entry is on.

The EXTENT_SECTOR_ADDR specifies where on the disk drive the file is located. Note: this is always in units of 256 byte sectors, and is always a multiple of 16 sectors (4,096 bytes).

The SECTORS_IN_EXTENT records the size of the extent. It is always a multiple of 16 sectors (4,096 bytes).

The FILE_SECTOR_OFFSET specifies what part of the file the extent is associated with. The extent corresponding with the first record of a file would have FILE_SECTOR_OFFSET of 0.

SSM

The SSM (Secondary Storage Manager) bitmap is a data structure on every volume that keeps track of which pages of the disk are free, and which are in use.

The main portion of the SSM bitmap for a disk is a large bitmap, with one entry per page, recording whether or not the page is free.

When a volume is mounted, MPE "maps" the SSM bitmap into a virtual address range, allowing it (and us) to view the SSM bitmap via LOAD/STORE instructions (or via Debug's DV command).
(example SSM)

GUFD

A GUFD (Global Unique File Descriptor) is a virtual memory data structure (record) allocated (usually) when a file is first opened.

If a single process has a particular file open, that file has a GUFD. If two thousands processes open the same file at the same time, that file still has a single GUFD.

When the last user of a file closes it, MPE generally does not deallocate the GUFD. Instead, it puts it on a list of recently "closed" GUFDs. It also doesn't deallocate the virtual space address range associated with the file, and it leaves in memory any pages associated with the file (presumably after making sure that any dirty ones are written to disk).

Later, when a process tries to FOPEN the file again, MPE checks to see if it is on the recently closed list ... if it is, then that GUFD is taken off the list and used again for the same file. When this happens, the same virtual address range is assigned to the file, which means that if pages of it were still in memory, the user may benefit (performance wise).

If the list gets too large, the oldest entries on the list will be deallocated and forgotten.

If the file isn't on the recently closed list, a fresh GUFD will be allocated, along with a fresh virtual address range.
(example GUFD)

Extent B-Tree

An Extent B-Tree is a virtual memory data structure that is used to map from a "file-relative" byte offset back to the equivalent page of disk storage. There is one entry in an extent B-tree for each extent of a file.

Each open file has its own extent B-tree.

Because there is one entry in an extent B-tree for each extent of a file, a 1 MB file with 100 extents will require a much larger extent B-tree than a 1 MB file with a single extent.

When a file is first opened, a GUFD is assigned and the file's extent blocks are read from the Label Table (where the file label is stored), and the extent B-tree is created.

Database

For this tutorial, "database" refers only to IMAGE/SQL databases.

FOS

FOS (Fundamental Operating System) is the set of software that is bundled as part of MPE.

2. Exploration

Now that we've defined terms, let's explore: volume sets, disks, label tables, file labels, open files and databases.

Exploring: Volume Sets

MPE appears to lack a command to display all defined volume sets, although it does have one that will tell us what volume sets are currently mounted.

The DSTAT command, part of FOS, will quickly show all mounted volume sets (and volumes):

:DSTAT ALL
LDEV-TYPE    STATUS      VOLUME (VOLUME SET - GEN)
---------    --------    --------------------------
1-C2474S     MASTER      MEMBER1 .....(MPEXL_SYSTEM_VOLUME_SET-0)
2- 022030    MASTER      MASTER       (USERS-0)
3- 022030    MEMBER      USER2        (USERS-0)
4- 022030    MEMBER      USER3        (USERS-0)
              

This shows us that two volume sets are currently mounted:

MPEXL_SYSTEM_VOLUME_SET and USERS. Note that the master volume for USERS is called "MASTER" ... a distinctly uninformative name. I'm at fault here, and if I were to rebuild this volume set again, I'd call the master volume USER1.

Note: some users name their volumes after the ldev they were initially created on. This may not be desirable, for a variety of reasons, including the fact that the ldev to volume name mapping may change if the disks are moved around or the ldevs are changed.

De-Frag/X also has a DSTAT command, which provides more internal information:

:defragx
dstat all
Ldev mvid  Volume Set Name          :Volume Name      Physical Path
---- ----  ---------------------------- ---------------- -------------
1    1  MPEXL_SYSTEM_VOLUME_SET     :MEMBER1          52.6.0
2    2  USERS                       :MASTER           48.0.2
3    3  USERS                       :USER2            48.0.3
4    4  USERS                       :USER3            48.0.4
16  (not "mounted")
     S
     y
     s
Ldev ? MVT entry LM Area   SSMdevinf SSM map    #Pages  #MBs  #SSMupdate
---- - --------- --------- --------- ---------  ------ -----  ----------
   1 Y $c3fd8e00 $c3fd8f84 $c3fd8eac $d2c00000 330,884 1,292  1,591,869
   2   $c3fd9000 $c3fd9184 $c3fd90ac $d5288000 163,736   639     78,701
   3   $c3fd9200 $c3fd9384 $c3fd92ac $d52a8000 163,736   639     71,248
   4   $c3fd9400 $c3fd9584 $c3fd94ac $84bb8000 163,736   639     55,544
  16   $00000000  (not fully mounted)
Ldev  LT GUFD   LT Header LT EOF    Sectrs  Version  SecSize
----  --------- --------- --------- ------  -------- -------
   1  $ca000054 $011.$300 $00900f00  1,760  A.00.00      512
   2  $ca00b43c $125.$300 $00900f00  1,440  A.00.00      256
   3  $ca00b764 $129.$300 $00900f00  1,440  A.00.00      256
   4  $ca00b8f8 $12c.$300 $00900f00  1,440  A.00.00      256
  16  (not "mounted")
              

Note the "#SSMupdate" column. This is the approximate number of disk space allocations and deallocations (combined) that have occurred on the disk drive. MPE may drop this number to zero from time to time, although I haven't noticed it yet.

The DISCFREE command, part of FOS, will also give us an insight into volume sets volumes:

:discfree c
DISCFREE A.50.01 Copyright (C) Hewlett-Packard 1992.  All rights
reserved.                          FRI, AUG 11, 1995,  6:46 PM
ALL MEASUREMENTS ARE IN SECTORS.
ALL PERCENTAGES ARE RELATIVE TO THE DEVICE SIZE.
           |    Configured     |      In Use       |     Available     |
-----------+-------------------+-------------------+-------------------+
LDEV :     1 -- (MPEXL_SYSTEM_VOLUME_SET:MEMBER1)
 Device    |    5294144        |    4483344 ( 85%) |     810800 ( 15%) |
 Permanent |    4923552 ( 93%) |    4095248 ( 77%) |     810800 ( 15%) |
 Transient |    5294144 (100%) |     388096 (  7%) |     810800 ( 15%) |
LDEV :     2 -- (USERS:MASTER)
 Device    |    2619776        |    2551488 ( 97%) |      68288 (  3%) |
 Permanent |    2619776 (100%) |    2551488 ( 97%) |      68288 (  3%) |
 Transient |    2619776 (100%) |          0 (  0%) |      68288 (  3%) |
...
TOTALS :
 Device    |   13153472        |   12096208 ( 92%) |    1057264 (  8%) |
 Permanent |   12756688 ( 97%) |   11708112 ( 89%) |    1031072 (  8%) |
 Transient |   13101072 (100%) |     388096 (  3%) |    1057264 (  8%) |
              

DISCFREE has five different formats for its output, if you run it without specifying an output format, it will prompt you for one.

De-Frag/X also has a form of the DISCFREE command:

:defragx
discfree
MPEXL_SYSTEM_VOLUME_SET :
          Avail   Avail
Ldev      Pages     MBs  %Avail Volume Name
----  ---------  ------  -----  ----------------
   1     50,776     198    15%  MEMBER1
total MBs available:   198  ( 50,776 pages;    812,416 sectors)
---------------------------------------------------------------
USERS :
          Avail   Avail
Ldev      Pages     MBs  %Avail Volume Name
----  ---------  ------  -----  ----------------
   2      4,268      16     2%  MASTER
   3      5,417      21     3%+ USER2
   4      4,082      15     2%  USER3
total MBs available:    52  ( 13,767 pages;    220,272 sectors)
---------------------------------------------------------------
Note: disks flagged with "+" after their %Avail have a VOLUTIL
configured "Maximum Permanent Space" of less than 100%.
              

De-Frag/X's DISCFREE defaults to grouping the ldevs by volume set, but this can be overridden.

Once we know what volume sets are mounted the VOLUTIL utility, a part of FOS, can be used for further exploration:

:VOLUTIL
Mirvutil  A.01.00, (C) Hewlett-Packard Co., 1991. All Rights Reserved.
volutil: showset users
Volume-set name: USERS
Creation date: SUN, MAY 28, 1995,  7:55 PM
Generation number: 0
Number of volumes in set: 3
Number of classes in set: 1
volutil: showset users;info=struct

Volumes in set: USERS
   MASTER
   USER2
   USER3
Classes in set: USERS
   DISC
Volumes in class: USERS:DISC
   MASTER
   USER2
   USER3
              

VOLUTIL will also show some internals information:

:volutil
showset users; info=labels
Volume name: USERS:MASTER
Initialization date: SUN, MAY 28, 1995,  7:55 PM      Volume type: 2
Member number: 1     Number in set: 3
Label Table Address: $000005A0         MVT Address: $00000000
Free Space Map Address: $00000090      Cold Load ID: $00000000
Logical Volume-id: $055F0001 2513DCD4
Physical Volume-id: $055F0001 2513DCD4
...
              

Exploring: Disk Layout

The first few sectors of every volume has information about which volume set the disk belongs to.

We can use Debug, carefully, to explore this information:

:debug
dsec 1.$100, 40, s, 8
              

The above will display the second sector of ldev 1, as ASCII text, 8 32-bit words per line:

SEC $1.100      "....MPEXL_SYSTEM_VOLUME_SET     "
SEC $1.120      "    ng../..ML.........._..%..."
SEC $1.140      "....J..D.q..._..%.......J..D.q.."
SEC $1.160      "........._..%..................."
SEC $1.180      "... ...............7._..%.....w."
SEC $1.1a0      "J..@.y@.........._..%.....h.J..a"
SEC $1.1c0      ".t.i.MEMBER1         ...._..%..."
SEC $1.1e0      "................................"
              

We see that this disk is MPEXL_SYSTEM_VOLUME_SET:MEMBER1.

Note: sector 0 generally contains information in a format called "LIF" (Logical Interchange Format). One benefit of LIF is that it would, in theory, allow HP-UX to recognize an MPE/iX disk.

Here is sector 0 of ldev 1 on one machine:

:debug
dsec 1.0,40,s,8
SEC $1.0        "..HPESYS........................"
SEC $1.20       "................................"
SEC $1.40       "................................"
SEC $1.60       "................................"
SEC $1.80       "................................"
SEC $1.a0       "................................"
SEC $1.c0       "................................"
SEC $1.e0       "..................0...`........."
              

NOTE: when exploring disks with Debug's dsec ("Display SECondary storage") command, be sure that the ldev number ("1" in the examples above) you use always corresponds to a mounted, spinning disk drive. If you do a "dsec" on a disk drive that is configured (via SYSGEN), but not available, your Debug session will hang until you reboot.

Exploring: SSM Bitmap

The SSM (Secondary Storage Manager) bitmap is a data structure on every volume that keeps track of which pages of the disk are free, and which are in use. If you get the virtual address of the SSM for a disk drive, Debug can be use to explore it. Here's one method:

:defragx
dstat 1 detail
...
     s
Ldev ? MVT entry LM Area   SSMdevinf SSM map       #Pages
---- - --------- --------- --------- ---------  ---------  ...
   1 Y $c3fd8e00 $c3fd8f84 $c3fd8eac $d2c00000    330,884
...
:debug
dv $d2c00000, 20
              

Note that the first $200 bytes of the SSM bitmap are a "header".

You can format the SSM bitmap by doing:

:debug
symopen symos.osb79.telesup      (or appropriate osXXX group)
fv $d2c00000  "sec_storage_map_type"
              

To see the status of a particular page (e.g., page #12345), do:

fv $d2c00000  "sec_storage_map_type.map [#12345]"
CRUNCHED RECORD
PAGE_STATE      : 2
STORAGE_OPTIONS : [ ]
END
              

The PAGE_STATE of 2 means "permanent".

Other page state values are:

   0 free
   1 transient (e.g., stack, heap, MPE data structures)
   2 permanent
              

De-Frag/X has an "SSM" command that allows exploring the state of pages in the SSM bitmap:

:defragx

ssm 1 12345

Page 12,345 : permanent (info @ $d2c01a1c)

Exploring: Label Table

The label table for a volume can be explored with FSCHECK. While not an official part of FOS, FSCHECK seems to generally be bundled with MPE/iX. FSCHECK can be found in FSCHECK.MPEXL.TELESUP.

:fcheck.mpexl.telesup
FSCHECK, A.05.00. (C) Hewlett-Packard Co., 1987. All rights reserved.
fscheck: displaylabel 1
File labels allocated on volume: MPEXL_SYSTEM_VOLUME_SET:MEMBER1
MMSAVE          .MPEXL           .SYS              - $0000F300
MPEXLDIR        .PUB             .SYS              - $0000F600
ISL             .MPEXL           .SYS              - $0000F900
START           .MPEXL           .SYS              - $0000FC00
...
              

Note: be prepared to "Break/ABORT", since FSCHECK doesn't seem to react to control-Y.

De-Frag/X also has the ability to list all, or part of, a Label Table:

:defragx
listlt 1 all
              

A UFID (Unique File IDentifier) acts as a "pointer" into a label table.

Here's the UFID for EDITOR.PUB.SYS on one machine ... note that it will vary from machine to machine:

$055f0001 $251006f5 $002385be $4a053886 $047f54c0
 Volume              Index
              

The above UFID can be decoded as:

VOLUME_ID      :
   SPLIT     : 0
   FILL      : 5
   VSCTS0    : 5f
   PHIL      : 0
   VOL_SET_X : 1           <--- first volume in volume set
   VSCTS1    :
         DAY    : 94
         HOUR   : 10
         MINUTE : 1
         SECOND : 2f
         TENTHS : 5
LABEL_OFFSET   : 2385 <--- index into Label Table
FCTS0          : be
FCTS1          :
   DAY    : 128
   HOUR   : 5
      MINUTE : e
      SECOND : 8
      TENTHS : 6
INTERVAL_TIMER : 47f54c0
              

The extra fields are various forms of timestamps, which help determine which volume set the UFID refers to.

Exploring: File Label

You can see a file label, in hexadecimal, via the LISTF or LISTFILE commands, using the "-1" option (which requires you to have SM capability). Unfortunately, this displays only the first 256 bytes, not the entire 512.

Example:

:listf editor.pub.sys, -1   (output shown slightly edited)
F = EDITOR
00000000 44495343 20202020 20202020 20202020 20202020 ....DISC
20202020 20202020 20202020 20310000 45444954 4F522020         1..EDITOR
20202020 20202020 50554220 20202020 20202020 20202020         PUB
00000000 53595320 20202020 20202020 20202020 00000000 ....SYS
20202020 20202020 20202020 20202020 4D414E41 47455220            MANAGER
53595320 20202020 00000000 FC000000 045F0001 251006F5 SYS     ....ü...._
00010401 00000000 00000300 0002D69A 28509A60 0002D923 ..............Ö.(P
150C5664 0002DEDE 63736820 0002D69A 28509A60 00014300 ..Vd..ÞÞcsh ..
00000150 00000143 00000143 00000142 00000000 00000000 ...P...C...C...B..
00014300 00000000 00000000 00000000 00000100 00000100 ..C...............
00010000 01440001 01440405 00000000                   .....D...D......
              

You can see the entire file label by using Debug, if you know the virtual address of the file label. Fortunately, LISTF and LISTFILE's -3 option displays this information:

Example:

:listf editor.pub.sys, -3
FILE: EDITOR.PUB.SYS
...
MAX LABELS: 0                MODIFIED: WED, APR 26, 1995,  2:45 PM
DISC DEV #: 1                ACCESSED: WED, AUG  9, 1995,  7:31 PM
SEC OFFSET: 0                LABEL ADDR: $00000011.$00238520
VOLCLASS  : MPEXL_SYSTEM_VOLUME_SET:DISC
              

Now that we have the virtual address for the file label, $11.$238520, we can display it in hex via Debug:

:debug
dv $11.$238520, $80, b
c
              

A file's file label can be found via LISTF/LISTFILE, and vie wed via Debug:

:listf editor.pub.sys, -3
...
SEC OFFSET: 0             LABEL ADDR: $00000011.$00238520
...
:debug
dv $11.$00238520, 40, b
              

If you tell Debug to open the SYMOS file corresponding to your release of MPE/iX, you can see the file label in much more d etail.

The SYMOS version can be determined by doing a ":SHOWME" command, as follows:

:showme
RELEASE: C.50.00   MPE/iX HP31900 B.79.06   USER VERSION: C.50.00
              

The underlined area above shows that the appropriate SYMOS file is in the group OSB79.TELESUP. (It's always OSxxx.TELESUP.)

:debug
symopen symos.osb79.telesup
fv $11.$00238520 "flab_t"
              

Here's a partial sample from the above "fv":

RECORD
VERSION               : 0
FILE_FLAGS            :
   TEMP_FILE     : FALSE
   RELEASED      : FALSE
   IGNORE_PATH   : FALSE
   NO_BACKUP     : FALSE
   RESTORE       : FALSE
   STORE         : FALSE
   PURGE_PENDING : FALSE
   PROTECTED     : FALSE
LANG                  : 0
ASCII_EXO_RESTRICTION : 'DISC                             1'
PRIV_IN               : 0
PRIV_OUT              : 0
FILE_NAME             : 'EDITOR          '
GROUP1                :
      NAME : 'PUB             '
      MASK :
...
LOCKWORD              : '                '
CREATOR               :
      NAME : 'MANAGER SYS     '
...
FOPTIONS              :
      FILL_BITS     : 0
      REC_FORMATX   : 0
      FILE_TYPE     : 0
      FILE_EQ       : 1
      LABELED       : 0
      CONTROL       : 0
      REC_FORMAT    : 0
      DEFAULT_DESIG : 0
      ASCII         : 0
      DOMAIN        : 1
FILE_DESC             :
   DEVICE_TYPE     : 0
   DEVICE_SUB_TYPE : 0
   FILE_TYPE       : 0
   RECORD_TYPE     : 0
   ACCESS_METHOD   : 0
FILLER2               : 0
PRIV_LEVEL            : 3
FILLER3               : 0
TIME_STAMPS           :
   CREATED   : 2d69a28509a60
   ALLOCATED : 2d923150c5664
   ACCESSED  : 2df0516fc1a51
   WRITTEN   : 2d69a28509a60
   EOF_OFFSET              : 14300
   SECTORS_IN_FILE         : 150
   LOGICAL_FILE_REC_LIMIT  : 143
   LOGICAL_END_OF_DATA_REC : 143
   END_OF_FILE_BLOCK       : 142
   MSG_OPEN_REC_CNT        : 0
   MSG_FIRST_BLK_NUM       : 0
   FILE_LIMIT              : 14300
   USER_LABEL_EOF          : 0
   USER_LABEL_CNT          : 0
   SOF_OFFSET              : 0
   SONR_OFFSET             : 0
   REC_SIZE                : 100
   BLOCK_SIZE              : 100
   BLOCK_FACTOR            : 1
   FIRST_BLOCK_OFFSET      : 0
   EXTENT_SIZE             : 144
   EXTENTS_IN_FILE         : 1
   LAST_EXTENT_SIZE        : 144
   FILE_CODE               : 405
...
              

De-Frag/X also shows a list of extents for a file:

:defragx
de sl.pub.sys
Ldev   Disk Page#   # Pages  File Page #
----   ----------   -------  -----------
 1      44,273        16            0 ,
 1      44,289        16           16 ,
 1      44,305        32           32 ,
 1      44,337        64           64 ,
 1      44,401       128          128 ,
 1      44,529       128          256 ,
 1      44,657       128          384 ,
 1      44,785       128          512 ,
 1      44,913       128          640 ,
 1      45,041       128          768 ,
 1      45,169       128          896 ,
 1      45,297       128        1,024 ,
...
 1      52,081       128        7,808
# extents in file:           65
# pages in file:   7,936 pages; 31.0 MBs
Space savable by TRIM:  132 pages;  0.5 MBs
% fragmented:    0.0+
              

Exploring: Open Files

Open files can be explored in several manners. The most straight forward method is to determine the GUFD for a file of interest. This can be done by opening the file, and then calling FFILEINFO to obtain the GUFD address (a 32-bit integer).

Alternatively, De-Frag/X can be used to search the list of open files on the system:

:defragx
findfile all file gufd
VSOD      GUFD      Virtual Address #DskPgs FileLabelAddr Filename
--------- --------- --------------- ------- ------------- --------
$ca000000 $ca000054 $0011.$00000000   1,536 $0011.$000620 $LABEL_TABLE
$ca000194 $ca0001e8 $0014.$00000000     110 $0011.$001220 $SYSTEM_VSIT
$ca000328 $ca00037c $0015.$00000000   3,258 $0011.$000c20 $xm system log
$ca0004bc $ca000510 $0018.$00000000      12 $0011.$004820 /SYS/PUB
$ca000650 $ca0006a4 $00c8.$00000000      10 $0125.$4d6720 UDC2.MISC.SIELER
$ca0007e4 $ca000838 $08d1.$00000000       2 $0011.$1deb20 /SYS/LIB
$ca000978 $ca0009cc $001d.$00000000       6 $0011.$001820 $ROOT
...
              

Every open file has a GUFD (Global Unique File iDentifier). This controls global access to the file, including for FLOCK/FUNLOCK.

Once you have the virtual address of a GUFD, you can use Debug to display the GUFD.

Here's an example, looking at the GUFD for /SYS/PUB (we got the GUFD from the output above):

:debug
symopen symos.osb79.telesup
fv ca000510 "gufd_t"
RECORD
   HASH_LINK                         : 0
   LRU_LINK                          : 0
   PREV_LRU_LINK                     : 0
   UFID                              :
         ALL : ... (binary data shown as ASCII)
   GUFD_COUNT                        : c9
   FILE_OPEN_CNT                     : 47
   FILE_LABEL_PTR                    : 11.1820
   LPTR_FILE_TYPE                    : TRUE
   VERIFY_W_FLAG                     : TRUE
   FLOCK_SPECIFIED                   : FALSE
   UNPROTECTED                       : FALSE
   ...
   FLOD_INVALID                      : FALSE
   FLOD_PTR                          : 0.0
   SEMAPHORE                         :
   ...
   FLOCK_SEMAPHORE                   :
         ...
            SEM_OWNER : 7ffd               <-- $7ffd = unowned
         SEM_SPIN_STATE_REC   :
   ...
   OPEN_SEMAPHORE                    :
         ...
            SEM_OWNER : 7ffd
   ...
   FILE_VIR_ADDR                     : 18.0
   GDPD_PTR                          : d4880220
   ...
   TOT_READER                        : 47
   TOT_WRITER                        : 0
   FILE_SHARING_MODE                 :
      ...
      FLAB_DIRTY   : FALSE
      EXCLUSIVE    : FALSE
      WRITE_OPTION : NORMAL_WRITE
      SEMI_COUNT   : 0
      LOAD_BIT     : FALSE
   TM_EOF_OFFSET                     : 0
   REFUSE_ACCESS                     : FALSE
   SHADOW_LOG                        : FALSE
   XM_POST_IN_PROGRESS               : FALSE
   LRU_FILE_VIR_ADDR_FROM_LAST_CLOSE : FALSE
   FCLOSE_DISP                       : 0
   NEW_FILE                          : FALSE
   GDPD_COUNT                        : 1
   CCTL                              : FALSE
   ASCII_FILE                        : FALSE
   ...
   EOF_OFFSET                        : 15a8
   SECTORS_IN_FILE                   : 0
   LOGICAL_FILE_REC_LIMIT            : 0
   CURR_NUM_REC                      : 0
   END_OF_FILE_BLOCK                 : 0
   MSG_OPEN_REC_CNT                  : 0
   MSG_FIRST_BLK_NUM                 : 0
   FILE_SIZE                         : 1000000
   USER_LABEL_EOF                    : 0
   MAP_OUT_MUST_POST                 : TRUE
   PROTECTION_METHOD                 : 0
   PROTECTION_CNT                    : 0
   PID                               : 6
   SOF_OFFSET                        : 0
   XM_PTR                            : 817046dc
   XM_SPAN_LS                        : 0
   ...
   STORE_ACTIVE              : FALSE
   ...
END
              

As you can see, there are a number of interestingly named fields in the GUFD!

In addition to a GUFD, every open file has another data structure associated with it, the VSOD (Virtual Space Object Descriptor). Every non-file virtual address range allocated by MPE, also has a VSOD associated with it. (Example: your NM stack has a VSOD, but no GUFD because it isn't a file.)

De-Frag/X reports the currently in-use VSODs with the FINDSID command (shown above). However, if you know the GUFD for a file (perhaps from calling the FFILEINFO intrinsic), you can determine the address of the GUFD by subtracting hex $54 from the GUFD address.

Here's a sample VSOD, for the open file /SYS/PUB:

:debug
fv ca0004bc fum "vs_od_type"
RECORD
   SEMAPHORE              :
      SEM_INFO_WORD :
         ...
         SEM_OWNER : 7ffd
      ...
   BASE_VA                : VA_TYPE( 18.0  )
   ENDING_VBA             : ffffff
   CURRENT_SIZE           : 1000000
   OPTIONS                : [ 0, 3, 4, 5, 7, 8, 9, 13]
   VIRTUAL_LOCATION       : OBJ_OWN_SPACE
   ACCESS_RIGHTS          :
      PAGE_USAGE : 1
      PL1        : 2
      PL2        : 2
      FILLER     : 0
   PID                    : 0
   NEW_OPTIONS            : [ ]
   INIT_BYTE              : 0
   OBJECT_CLASS           : 7
   SHARED_VS_UNIT_DESC_ID : 0
      CURRENT_SEC_STORAGE_PAGES : 2
      MAX_SEC_STORAGE_PAGES     : 1002
      B_TREE_ROOT               :
         ROOT_PTR    : cc000e58
         NODE_COUNT  : 1
         ENTRY_COUNT : 1
      DEV_RESTRICTION           :
         ...
      VS_LL_HEADER_NUM          : 18
         FIRST_SHR_PAGE_DESC_PTR : 0
         LAST_SHR_PAGE_DESC_PTR  : 0
END
              

Note that like most MPE internal data structures, the VSOD has a lock area within it (the "SEMAPHORE" field). The SEM_OWNER value of $7ffd means that no one owns the semaphore at present.

The "B_TREE_ROOT" section has a field called "ENTRY_COUNT". This is the number of extents the file has.

Here's a look at the VSOD for NL.PUB.SYS:

De-Frag/X> findsid $a file

VSOD      Virtual Address #DskPgs FileLabelAddr Filename
--------- --------------- ------- ------------- --------
$ca000978 $000a.$00000000   7,040 $0011.$015c20 NL.PUB.SYS

(Examined 1,340 objects)
# disk pages occupied by listed objects:     7,040

De-Frag/X> :debug
HPDEBUG Intrinsic at: a.009ea6a8 hxdebug+$e4
$a ($39) nmdebug > var foo ca000978
$b ($39) nmdebug > fv foo "vs_od_type"
RECORD
   SEMAPHORE              :
      ...
         SEM_OWNER : 7ffd
       ...
   BASE_VA                : VA_TYPE( a.0  )
   ENDING_VBA             : 3e7fffff
   CURRENT_SIZE           : 3e800000
   OPTIONS                : [ 0, 4, 5, 6, 7, 8, 9, 1c, 1e]
   VIRTUAL_LOCATION       : OBJ_SR4
   ACCESS_RIGHTS          :
      PAGE_USAGE : 0
      PL1        : 0
      PL2        : 0
      FILLER     : 0
   PID                    : 0
   NEW_OPTIONS            : [ ]
   INIT_BYTE              : 0
   OBJECT_CLASS           : 8
   SHARED_VS_UNIT_DESC_ID : 0
      CURRENT_SEC_STORAGE_PAGES : 1b80
      MAX_SEC_STORAGE_PAGES     : 100000
      B_TREE_ROOT               :
         ROOT_PTR    : ce000410
         NODE_COUNT  : 1d
         ENTRY_COUNT : dd
      DEV_RESTRICTION           :
            EXO_SPLIT       : 0
            EXO_RESTRICTION : 1
            EXO_TS0         : 0
            EXO_INDEX       : 25
            EXO_TS1         :
                  DAY    : 135
                  HOUR   : 21
                  MINUTE : 1a
                  SECOND : 36
                  TENTHS : 5
      VS_LL_HEADER_NUM          : 1c
         FIRST_SHR_PAGE_DESC_PTR : d1ff8000
         LAST_SHR_PAGE_DESC_PTR  : 0
END
              

Note that the B_TREE_ROOT.ENTRY_COUNT field has the value $dd (decimal 221). This indicates that NL.PUB.SYS has 231 extents.

Of course, we could have found out how many extents NL.PUB.SYS has by simply

doing:

listf nl.pub.sys, 2
              

but that wouldn't have been as much fun!

The Extent B-Tree is the data structure that keeps track of those 221 extents for NL.PUB.SYS. If you try to access address $a.$34000, and the page isn't in memory, a page fault occurs. The VSOD is consulted to see if $a.$34000 is a legal address within the file. (It is, since the VSOD (above) says that the ending address is $a.$3e7fffff.)

The B_TREE_ROOT points to the start of the Extent B-Tree. Each of the 221 entries in the Extent B-Tree will describe a single extent of the file NL.PUB.SYS.

Clearly, even an efficient search of a B-Tree with 221 entries must take more time than a search through a B-Tree with only 1 entry...and that's one area where file fragmentation can affect performance. In this case, the amount of extra CPU time required to search the extra B-Tree entries is fairly small ... probably on the order of a hundred microseconds. There is, however, a second penalty to file fragmentation: the memory space occupied by the B-Tree increases with every 20 (or so) extents. Thus, a file with a large number of extents will have a larger B-Tree, using more memory than a defragmented file would. Again, this isn't a terribly large amount of memory .. for an individual file, and the data isn't even marked as "memory resident". However, this means that there is an increasing possibility that the B-Tree for your file won't be in memory when it's needed to process a page fault, which will result in one (or more) subsequent page faults that must be resolved before your page fault is completely resolved.

3. Analysis

This section discusses analysis of the three types of fragmentation: disk, file, and database.

Analyzing Fragmentation: Disk

A rough picture of disk fragmentation can be seen with the FOS tool DISCFREE. A DISCFREE A displays a summary of how many free areas exist, grouped by size. If a large number of small free areas exist, and the largest free area isn't very large, then you can conclude that your disk is fragmented.

:discfree a

DISCFREE A.50.01 Copyright (C) Hewlett-Packard 1992.  All rights reserved.
                         WED, AUG 16, 1995,  5:39 PM

-------------------------------------------------------------------------------
LDEV :     1 -- (MPEXL_SYSTEM_VOLUME_SET:MEMBER1)

LARGEST FREE AREA:      1977664  TOTAL FREE SPACE:    3768624

     0 BLOCK(S) OF      1-     9 CONTIG. SECTORS =          0 FREE SECTORS.  0%
   176 BLOCK(S) OF     10-    99 CONTIG. SECTORS =       6048 FREE SECTORS.  0%
   137 BLOCK(S) OF    100-   999 CONTIG. SECTORS =      44528 FREE SECTORS.  1%
    48 BLOCK(S) OF   1000-  9999 CONTIG. SECTORS =     145264 FREE SECTORS.  4%
     7 BLOCK(S) OF  10000- 99999 CONTIG. SECTORS =     133024 FREE SECTORS.  4%
     2 BLOCK(S) OF 100000-AND UP CONTIG. SECTORS =    3439760 FREE SECTORS. 91%
              

Note that about 300 blocks of less than 1000 sectors are free. The above shows that the disk has been "checkerboarded" (fragmented) into about 360 areas. The final line shows, as a number, the approximate size of the largest contiguous free areas. (The "2 blocks" of 100,000+ sectors implies that the single biggest contiguous free area is at least 1,719,880 sectors in size.) (Don't those commas make the numbers easier to read?!) The initial output gave us the exact number, but again as a simple integer value ... with no obvious correlation to the size of the disk drive. Is that 1,977,64 sectors a significant portion of the disk or not?

Additionally, a complete list of in-use permanent, transient, and free disk areas can be obtained from the VOLUTIL SHOWUSAGE command:

 :volutil
 volutil: showusage 1

 PERM, FREE, TRANS Space on LDEV 1:
 processing ...
 SECTOR ADDRESS        SIZE        SPACE_USED_BY
                   (in sectors)
 --------------    ------------    -------------
             48              96    MMSAVE.MPEXL.SYS
         410752           11136    <transient space>
         436608            1328    <transient space>
         437936              32    MPEXLDIR.PUB.SYS
         437968             240    ISL.MPEXL.SYS
         438208              80    IOMAP.MPEXL.SYS
         438288             320    <free space>
         438608             768    SADPATCH.MPEXL.SYS
         ...
        5846640              32    UTAEPROC.TAE.TELAMON
        5846672         1977664    <free space>
 volutil: exit
              

De-Frag/X's MAP command shows the disk usage in a graphical manner:

   De-Frag/X> map 1
------------------------------------------------------------------------
Ldev    1:   (Each chunk represents 588 pages, or 2.3 MB)
   [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXxXxxXXxxXXXXXXXXXXX]  0
   [XXXXXXXXXXXXxxxxxxxxxxxxxxxxxxxXxxxxxxxxxXXXXXXXXxXXXXXXXXXXXXxx]  1
   [xxXXXpxXX*X*xXXXXxxXXPXXPPPXpPXXXXXXXXXXXXXXXXXXXXXXXXXXXXXxXXXX]  2
   [XXPPxPPPPPPPPPPXXXXXXXXXXXxTTtxpx**XXXXXXXPXXXXXXXXXXXXPPPXXXxXX]  3
   [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXPPXPPPXPPXXXX]  4
   [xXXXPPXXXXXXpPPXXXXXXXXXXXXppPPPppPPPPXXXX***xXpPppppPPPPPPpXPpP]  5
   [ppPpppPPpPppPppPXpxxpppppxppppPPxxPPpPXXppXPPPPPppPPPPPPXPPXXXXP]  6
   [XXXXPPPp**pPpPPp************************************************]  7
   [****************************************************************]  8
   [*******************************************pPp******************]  9
   [****************************************************************] 10
   [****************************************************************] 11
   [****************************************************************] 12
Col[0....+....10...+....20...+....30...+....40...+....50...+....60..] Row
Available Permanent Disk:  808 MB (disk size: 1,910 MB)
MAP_characters:
   Free       *   Permanent  P   Transient  T   unmovable  X
   part Perm  p   part Trans t   same_ldev  x
              

This provides an immediate visual appreciation of the state of disk fragmentation. A map of all disks, one line per disk, can also be obtained:

De-Frag/X> map all
                                                                   Disk Size of
                                                        Available  Size  chunks
Ldev [ (usage map...)                                   ] PermMBs   MBs     MBs
   1 [XXXXXXXXXXXXXXXXXXXXXXXpXXXX*********p************]    808   1,910   38.2
   4 [PPPp*p**p*****************************************]  3,830   4,095   81.9
  11 [p*tt**t*p**************************pPPp***********]  1,871   2,033   40.7
  12 [p**********************************p**************]  1,989   2,033   40.7
  13 [pp******************************pp****************]  1,954   2,033   40.7
MAP_characters:
   Free       *   Permanent  P   Transient  T   unmovable  X
   part Perm  p   part Trans t   same_ldev  x
              

Note that the "Size of Chunks" column varies, since the disks show are of differing sizes.

Analyzing Fragmentation: File

Files are fragmented when their extents are not contiguous. (Note that a file of a single extent is not fragmented.) However, if a file consists of two extents, we can still differentiate between different degrees of "badness" for the fragmentation. If the two extents are on the same disk, at the opposite ends, then a serial read of the file will require one full-length motion of the disk head when switching from one extent to the other. If the second extent was in the middle of the disk, the "cost" of fragmentation isn't quite as high. If the second extent was on a different disk, then the cost is harder to estimate, since we don't know where the other disk's head is when we finish reading the first extent (on the first disk). However, it appears that this type of fragmentation is generally not quite as bad as having two non-contiguous extents on the same drive, even when they are close together, as an MPE "prefetch" could be scheduled to simultaneously fetch the two extents if they are on different disks (and could not do so if they are on the same disk).

De-Frag/X "scores" the fragmentation of a file, taking the above into consideration, as well as the size of the file. A file of 2 pages and with 2 extents (on the same disk) is 100% fragmented ... it can't get any worse! A file of 200 pages and with 2 extents isn't as badly fragmented, no matter where the two extents are located.

The FOS command LISTF (and LISTFILE) can be used to quickly determine whether or not a file is fragmented:

:listf sl.pub.sys,2
ACCOUNT=  SYS         GROUP=  PUB
FILENAME  CODE  ------------LOGICAL RECORD-----------  ----SPACE----
                  SIZE  TYP        EOF      LIMIT R/B  SECTORS #X MX
SL      * SL      128W  FB      125488     320000   1   126976 65  *
              

The above shows that SL.PUB.SYS has 65 extents, so it is fragmented.

FSCHECK can also be used to look at file fragmentation:

:fscheck.mpexl.telesup
displayextents sl.pub.sys
SECT_ADDR   SECTS_IN_EXTENT  VOL_SET_INDEX  FILE_SECT_OFFSET
---------   ---------------  -------------  ----------------
$000C9710        2048               1          $0001C800
$000C9F10        2048               1          $0001D000
$000CA710        2048               1          $0001D800
$000CAF10        2048               1          $0001E000
$000CB710        2048               1          $0001E800
...
              

Note that the extent information displayed is not in any particular order.

Finally, De-Frag/X shows the extents as well, but in logical (relative to the file) order:

De-Frag/X> displayextents sl.pub.sys

File: /SYS/PUB/SL  (65 extents)

   Ldev   Disk Page#   # Pages  File Page #
   ----   ----------   -------  -----------
       1      29,408        16            0
       1      29,452        16           16
       1      29,479        32           32
       1      31,494        64           64
       1      44,397       128          128 ,
       1      44,525       128          256 ,
       1      44,653       128          384 ,
       1      44,781       128          512 ,
       1      44,909       128          640 ,
       1      45,037       128          768 ,
       1      45,165       128          896 ,
       ...
       1      51,693       128        7,424 ,
       1      51,821       128        7,552 ,
       1      51,949       128        7,680 ,
       1      52,077       128        7,808

   # extents in file:           65
   # pages in file:   7,936 pages; 31.0 MBs
Space savable by TRIM:   93 pages;  0.3 MBs
% fragmented:    0.0+
              

For SL.PUB.SYS, De-Frag/X noticed that the extents are contiguous to each other, which is very unusual for files with multiple extents. (Note the "," after the "File Page#" value...this indicates that the extent with the "," on the line is immediately followed on disk by the extent shown on the next line.) De-Frag/X takes this into consideration when scoring the fragmentation of a file. In this case, the file was large (31 Mbs), but had only 65 extents (most of them contiguous), so the fragmentation score was non-zero, but barely so ... so small that it would appear as 0.0 if we formatted it with a ##.# format!

Note: De-Frag/X will show the same data in "sectors" instead of "pages", if desired. I chose pages to be similar to FSCHECK's output. Also, both FSCHECK and De-Frag/X accept "DE" as a synonym for "DISPLAYEXTENTS".

De-Frag/X has a simple command to report just the fragmentation score for a file (or a fileset):

analyze @.pub.sys  min 10
Filename                     Frag%  #Pages       #MBs  #Extents
---------------------------- -----  ---------  ------  ---------
DTCCNF02.PUB.SYS              100%          2                  2
DTCSWB02.PUB.SYS               20%          6                  2
DTCSWE04.PUB.SYS               20%          6                  2
DTCSWG04.PUB.SYS               20%          6                  2
DTCSWH04.PUB.SYS               20%          6                  2
DTCSWJ04.PUB.SYS               50%          3                  2
DTCSWK04.PUB.SYS               20%          6                  2
HPOPTMGR.PUB.SYS               11%         10                  2
IODFAULT.PUB.SYS               10%         11                  2
IODFDATA.PUB.SYS               12%          9                  2
JSMAIN.PUB.SYS                 10%         11                  2
VERSION.PUB.SYS                11%         10                  2

Fragmentation by fileset:    Frag%   #Pages       #MBs  #Extents   #Files
             files shown:      17%         86                 24       12
             files examined:    0%     66,647     260      1,911      831
   Space savable by TRIM: 1,755 pages; 6.8 MBs
              

The final line, "Space savable by TRIM" indicates that some of the files examined have disk space allocated beyond their EOFs. A "TRIM" command will tell MPE to discard this wasted space, without affecting the EOF or the LIMIT of the file.

Analyzing Fragmentation: Database

Database fragmentation is when the physical ordering (within a file) of entries in a dataset does not correspond to the most-desired logical ordering. "Most-desired" is mentioned because a dataset can generally be viewed in many ways. For this tutorial, we assume that the "primary path" is the most-desired logical ordering. Thus, if records with the key value of RED and the key value of BLUE (where COLOR is the primary path) occur in the order: RED, BLUE, RED, BLUE in the dataset, the dataset is fragmented. If they occur in the order: RED, RED, BLUE, BLUE (and RED comes before BLUE in the master dataset), then it is not fragmented.

Database fragmentation can be checked by running HOWMESSY (or the newer version, DBLOADNG). (Both programs are generally free of charge.)

Here is a HOWMESSY report on a sample database before any repacking:

HowMessy/XL (Version 2.0) Data Base: LOCAL.PUB.DEFRAG Run on: FRI, AUG 11, 1
995

for IMAGE/3000 databases By Robelle Consulting Ltd. Page: 2
...
Data Set                 Type    Inefficient Pointers%
ITEM-STRUCTURE             Det             21.8%
ITEM-CONTENT               Det             70.5%
              

Note: the output is heavily edited...I couldn't convince Microsoft Word with Internet Assistant to accept my fixed-field 132 column input.

The "Ineff Ptrs" (Inefficient Pointers) column is the one that indicates, roughly, the logical fragmentation of the dataset. For this tutorial, we will be looking at the datasets ITEM-HEADER and ITEM-CONTENT. Note that the "Ineff Ptrs" for those two are 21% and 70%, respectively. A value of 100% would mean that for a given chain, no two logically contiguous records are ever physically contiguous.

4. Correcting Fragmentation

This section discusses correction of the three types of fragmentation: disk, file, and database.

Correcting Fragmentation: Disk

FOS provides two mechanisms to defragment a disk drive:

STORE/RESTORE
              

and

VOLUTIL: CONTIGVOL
              

The STORE/RESTORE solution is the slowest, but has been available (free) since MPE was released. In this, a STORE of all files is done, followed by a "PURGE" of most files (I generally avoid purging files in @.PUB.SYS), followed by a RESTORE of all files. This results in disk drives with reasonably compact in-use areas, with most of the free space at the end of the drives. Of course, this is a tedious process, at best.

With the release of MPE/iX 5.0, FOS now includes the CONTIGVOL command in the VOLUTIL utility. According to HP, CONTIGVOL is not a disk "condense" utility ... its purpose in life is to create a large contiguous free space on ldev 1.

Here's an example of using VOLUTIL's CONTIGVOL to make a larger contiguous free area on ldev 1:

volutil: contigvol 1
*WARNING: *** Running CONTIGVOL on a busy system may cause *** WARNING*
*WARNING: *** "OUT OF DISK SPACE" errors temporarily on    *** WARNING
*
*WARNING: *** specified LDEV/Volume.                       *** WARNING*
*Verify: 1977664 contiguous sectors available on ldev 1. Continue [Y/N] ?yt
Processing Labels on Ldev 13 ......
Processing Labels on Ldev 12 ......
Processing Labels on Ldev 11 ......
Processing Labels on Ldev 1 ......
Percent Complete 10
Percent Complete 20
Percent Complete 30
Percent Complete 40
Percent Complete 50
Percent Complete 60
Percent Complete 70
Percent Complete 80
Percent Complete 90
Number of Extents Moved 1036
Maximum Contiguous Sectors Free 3584816
              

De-Frag/X has a "CONDENSE" command, which acts a little differently. Instead of simply trying to create a single large hole, the condense command attempts to squeeze out all of the "holes" (free areas) between chunks of allocated disk space (similar to the MS-DOS DEFRAG command). The desired goal is to have the first X% of the disk fully occupied with permanent files (with no "holes), and the rest of the disk free.

Both De-Frag/X and CONTIGVOL will refuse to move in-use files, since that could result in major problems. Also, both programs refuse to move files off an ldev if they were specifically placed on that ldev. For example, the command:

build foo; dev=1 ...
              

means that you want the file's disk to be on ldev 1, no matter what!

In addition to CONDENSE, De-Frag/X has a MAKEROOM command, which allows you to move a specified number of megabytes of files off a particular ldev (and onto a specified list, or simply onto any other disks in the volume set). This command, like the CONTIGVOL command, can be used to make space available on ldev 1 for an UPDATE, if necessary.

Finally, De-Frag/X has a BALANCE command, which is useful when a new disk is added to a volume set. BALANCE will move some files from heavily used disks onto less used disks, aiming at balancing the load (and therefore, hopefully, the I/O traffic) across all of the disks in a volume set.

Correcting Fragmentation: File

A file can be defragmented via FOS tools by simply "copying" it:

:rename myfile, foo
:copy foo, myfile
:purge foo
              

The COPY command will generally copy the file into a single extent. File equates can be used to control which ldev (or volume class) the file is placed onto.

Note: to make a copy of in-use CM program files or CM SL files, you will need to issue a file equate and do the COPY slightly differently. Here's an example for copying an in-use SL:

:file myfile; lock
:copy *myfile, foo
:reset myfile
   then, at a later time:
:purge myfile
:rename foo, myfile
              

If the POSIX shell is used (SH.HPBIN.SYS), it is possible to rename in-use files (via the "mv" command).

De-Frag/X can also defragment a single file:

De-Frag/X>  defragment myfile
              

or a fileset:

De-Frag/X>  defragment myfi@.@
              

Correcting Fragmentation: Database

Database fragmentation can be corrected with the FOS tools DBUNLOAD/DBLOAD, by physically copying all of the database data to tape, erasing the database, and reloading it from tape. This is a time consuming, error prone task.

Several products exist to repack a database. The original and most powerful is Adager, which has a "repack" command.

The following report was run after repacking the sets "item-structure" and "item-content" (of the database shown before) along their primary paths.

HowMessy/XL (Version 2.0) Data Base: LOCAL.PUB.DEFRAG Run on: FRI, AUG 11, 1
995

for IMAGE/3000 databases By Robelle Consulting Ltd. Page: 1
Data Set              Type            Inefficient Pointers %
ITEM-STRUCTURE    Det                     1.0%
ITEM-CONTENT                Det                     39.5%
              

Note that the "Inefficient Pointers" for the primary paths for the two sets went from 21.8 and 70.5% to 1.0 and 39.5%, respectively. Note: HOWMESSY and DBLOADNG don't take into account some realities of life under MPE/iX, partially betraying their origin as tools written for MPE V (and earlier!). In particular, on MPE/iX, it isn't bad to have a chain cross a "block" (a file system record). Instead, the real penalty is if a chain crosses a page boundary , because then it may cause a page fault. HOWMESSY/DBLOADNG currently don't take this into consideration. Nevertheless, the result shows that the Adager repack of the two sets put the detail's records into a physical order (within the file) that more closely matches the logical order. I.e., if an application serially reads the associated master datasets, and for each entry reads the entire chain from the detail, the detail entries are in 100% optimum order (as a result of repacking) ... or, to say it somewhat differently, the dataset has been logically defragmented.

5. Conclusion

The three types of fragmentation: disk, file, and database, can be examined, and analyzed with various tools. Fragmentation can be controlled, as desired. The primary measurable performance impact fragmentation causes is hard to measure...database fragmentation's effect on performance is relatively straightforward to measure, file fragmentation's effect harder, and disk fragmentation's effect (except for the obvious impact on UPDATEs) is the hardest to measure.

I'll be reporting on ongoing efforts to characterize the full performance effect of the various kinds of fragmentation ... stay tuned to HP3000-L!

 
     
  www.resource3000.com   Resource 3000 - Technical Paper