Recovery from LED 552, 554, or 556 in AIX Versions 4 and 5

Contents

About this document

Causes of an LED 552, 554, or 556

Recovery procedure

1. Boot your system into a limited function maintenance shell (Service or Maintenance mode) from bootable AIX media to use this recovery procedure.

   Please refer to your system user's or installation and service guide for specific IPL procedures related to your type and model of hardware. You can also refer to the document titled "Booting in Service Mode", available at http:// techsupport.services.ibm.com/rs6k/techbrowse for more information.

2. With bootable media of the same version and level as the system, boot the system into Service mode. The bootable media can be any ONE of the following:
   • Bootable CD-ROM
   • mksysb
   • Bootable Install Tape

   Follow the screen prompts to the Welcome to Base OS menu.

3. Choose Start Maintenance Mode for System Recovery (Option 3). The next screen displays prompts for the Maintenance menu.
   • Choose Access a Root Volume Group (Option 1).

     The next screen displays a warning that indicates you will not be able to return to the Base OS menu without rebooting.

   • Choose 0 continue.

     The next screen displays information about all volume groups on the system.

   • Select the root volume group by number. The logical volumes in rootvg will be displayed with two options below.

   • Choose Access this volume group and start a shell before mounting the file systems (Option 2).

   If you receive errors from the preceding option, do not continue with the rest of this procedure. Correct the problem causing the error. If you need assistance correcting the problem causing the error, contact one of the following:

   • local branch office
   • your point of sale
   • your AIX support center

4. Run the following commands to check and repair file systems.

      fsck /dev/hd4 
      fsck /dev/hd2 
      fsck /dev/hd9var 
      fsck /dev/hd3
      fsck /dev/hd1 
   

   NOTE: The -y option gives the fsck command permission to repair file system corruption when necessary. This flag can be used to avoid having to manually answer multiple confirmation prompts, however, use of this flag can cause permanent data loss in some situations.

   If any of the following conditions occur, proceed accordingly.

   • If fsck indicates that block 8 could not be read, the file system is probably unrecoverable. See step 5 for information on unrecoverable file systems.

   • If fsck indicates that block 8 could be read, but one of the following errors is given,

        "fsck:  Not an AIXV3 file system."
        "fsck:  Not a recognized file system type."
     

     then go to step 6.

   • If fsck indicates that a file system has an unknown log record type, or if fsck fails in the logredo process, then go to step 7.

   • If the file system checks were successful, skip to step 9.

5. The easiest way to fix an unrecoverable file system is to recreate it. This involves deleting it from the system and restoring it from a very current system backup. Note that hd2, hd9var and hd3 can be recreated but hd4 cannot be recreated. If hd4 is unrecoverable, AIX must be reinstalled. For assistance with unrecoverable file systems, contact your local branch office, point of sale, or AIX support center. Do not follow the rest of the steps in this document.

6. Attempt to repair the file system with this command:

      fsck -p /dev/[hd#]
   

   Replace [hd#] with the appropriate file system.

   Now skip to step 8.

7. A corruption of the JFS log logical volume has been detected. Use the logform command to reformat it.

      /usr/sbin/logform /dev/hd8
   

   Answer yes when asked if you want to destroy the log.

8. Repeat step 4 for all file systems that did not successfully complete fsck the first time. If step 4 fails a second time, the file system is almost always unrecoverable. See step 5 for an explanation of the options at this point. In most cases, step 4 will be successful. If step 4 is successful, continue to step 9.

9. With the key in Normal position, run the following commands to reboot the system:

      exit
      sync;sync;sync;reboot
   

   As you reboot in Normal mode, notice how many times LED 551 appears. If LED 551 appears twice, fsck is probably failing because of a bad fshelper file. If this is the case and you are running AFS, see step 12.

The majority of instances of LED 552, 554, and 556 will be resolved at this point. If you still have an LED 552, 554, or 556, you may try the following steps.

NOTE: The following procedure will overwrite your Object Data Manager (ODM) database files with a very primitive, minimal ODM database. Due to the potential loss of user configuration data caused by this procedure, it should only be used as a last resort effort to gain access to your system to attempt to back up any data that you can. It is NOT recommended to use the following procedure in lieu of restoring from a system backup.

10. Repeat step 1 through step 3.

11. Run the following commands, which remove much of the system's configuration and save it to a backup directory.

       mount /dev/hd4 /mnt
       mount /dev/hd2 /mnt/usr
       mkdir /mnt/etc/objrepos/bak
       cp /mnt/etc/objrepos/Cu* /mnt/etc/objrepos/bak
       cp /etc/objrepos/Cu* /mnt/etc/objrepos
       umount /dev/hd2
       umount /dev/hd4
       exit
    

    Determine which disk is the boot disk with the lslv command. The boot disk will be shown in the PV1 column of the lslv output.

       lslv -m hd5
    

    Save the clean ODM database to the boot logical volume. (# is the number of the fixed disk, determined with the previous command.)

       savebase -d /dev/[hdisk#]
    

    If you are running AFS, go to step 12; otherwise, go to step 13.

12. If you are running the Andrew File System (AFS), use the following commands to find out whether you have more than one version of the v3fshelper file.

       cd /sbin/helpers
       ls -l v3fshelper*
    

    If you have only one version of the v3fshelper file (for example, v3fshelper), proceed to step 13.

    If there is a version of v3fshelper marked as original (for example, v3fshelper.orig), run the following commands:

       cp v3fshelper v3fshelper.afs
       cp v3fshelper.orig v3fshelper
    

13. WARNING: Do not proceed further if the system is a /usr client, diskless client, or dataless client.

    Recreate the boot image (hdisk# is the fixed disk determined in step 11):

       bosboot -a -d /dev/[hdisk#]
    

14. If you copied files in step 12, copy the AFS file system helper back to v3fshelper:

       cp v3fshelper.afs v3fshelper
    

15. Turn the key to the Normal position and run

       shutdown -Fr
    

If after following all of the preceding steps, the system still stops at an LED 552, 554, or 556 during a reboot in Normal mode, reinstalling AIX from a recent system backup is recommended. The time and effort involved attempting to isolate and correct the cause of the failure would generally be extensive at this point. This may not be cost-effective in your operating environment.

If you wish to pursue further system recovery, you may be able to obtain assistence from one of the following:

• local branch office
• your point of sale
• your AIX support center

[ Doc Ref: 90605189214746     Publish Date: Jan. 11, 2002]