5.1 maintenance level 3 update breaks libc

5.1 maintenance level 3 update breaks libc

Post by Daniel Packm » Wed, 01 Jan 2003 06:35:39



On a 640-B50 pseries at 5100-02, downloaded the maintenance
package (volume one) to go to 5100-03. Created the toc and
let smitty update_all go. It seemed to proceed properly until
it replaced libc. Then it complains about almost all commands:
----------------
Could not load program lslpp:
Symbol resolution failed for /usr/lib/libc.a(shr.o) because:
        Symbol getvtid (number 258) is not exported from dependent
          module /unix.
Examine .loader section symbols with the 'dump -Tv' command.
---------------
The page at

http://techsupport.services.ibm.com/server/mlfixes/51/03/02to03.html

mentions checking for IY32749 for potential jfs2 problems.
This was already installed prior to trying this update.

--
Daniel Packman
NCAR/ACD

 
 
 

5.1 maintenance level 3 update breaks libc

Post by Daniel Packm » Sat, 04 Jan 2003 02:23:20





>> On a 640-B50 pseries at 5100-02, downloaded the maintenance
>> package (volume one) to go to 5100-03. Created the toc and
>> let smitty update_all go. It seemed to proceed properly until
>> it replaced libc. Then it complains about almost all commands:
>> ----------------
>> Could not load program lslpp:
>> Symbol resolution failed for /usr/lib/libc.a(shr.o) because:
>>         Symbol getvtid (number 258) is not exported from dependent
>>           module /unix.
>> Examine .loader section symbols with the 'dump -Tv' command.

>Looks like it's a new symbol, and perhaps the update process
>didn't take into account the change to the kernel and libc.
>If you only applied and didn't commit, there should be a libc.a
>in /usr/lpp/bos/bos.rte.libc/<some level>.  You could use LIBPATH
>to point to this version of libc to run commands.  Make sure
>your boot list lists the cd first (before hdisk1) and reboot.
>The system will likely come up fine with the new kernel and libc,
>but just in case, you need to be in a position to boot into maintenance
>mode and do surgery (although how, I'm not sure...you'll need to
>get an old libc onto the machine, maybe).  Did the updates succeed?

.....

The updates suceeded until the libc was replaced. No commands
work, all giving the "could not load..." error. Rebooting fails
as soon as the rootvg is varied on.

The cd-rom is indeed in the bootlist. I can boot from it, but
I'm unsure where to go from here. I'd like to get into a shell
with access to the disk, then I could apply the update by hand
or put a previous version of the library in place.
When I boot from CD, I can

 1. do a preservation install (clearly a final possibility)
 2. maintenance mode.

In maintenance mode, if I start a shell and acess the filesytems,
it tries to do an fsck and gets locked into the same "symbol
resolution failed" errors. This is prior to any attempt to
mount filesystems.

--
Daniel Packman
NCAR/ACD


 
 
 

5.1 maintenance level 3 update breaks libc

Post by Bela Gazd » Sat, 04 Jan 2003 21:36:19





>>> On a 640-B50 pseries at 5100-02, downloaded the maintenance
>>> package (volume one) to go to 5100-03. Created the toc and
>>> let smitty update_all go. It seemed to proceed properly until
>>> it replaced libc. Then it complains about almost all commands:
>>> ----------------
>>> Could not load program lslpp:
>>> Symbol resolution failed for /usr/lib/libc.a(shr.o) because:
>>>         Symbol getvtid (number 258) is not exported from dependent
>>>           module /unix.
>>> Examine .loader section symbols with the 'dump -Tv' command.

>>Looks like it's a new symbol, and perhaps the update process
>>didn't take into account the change to the kernel and libc.
>>If you only applied and didn't commit, there should be a libc.a
>>in /usr/lpp/bos/bos.rte.libc/<some level>.  You could use LIBPATH
>>to point to this version of libc to run commands.  Make sure
>>your boot list lists the cd first (before hdisk1) and reboot.
>>The system will likely come up fine with the new kernel and libc,
>>but just in case, you need to be in a position to boot into maintenance
>>mode and do surgery (although how, I'm not sure...you'll need to
>>get an old libc onto the machine, maybe).  Did the updates succeed?

DP> .....

DP> The updates suceeded until the libc was replaced. No commands
DP> work, all giving the "could not load..." error. Rebooting fails
DP> as soon as the rootvg is varied on.

DP> The cd-rom is indeed in the bootlist. I can boot from it, but
DP> I'm unsure where to go from here. I'd like to get into a shell
DP> with access to the disk, then I could apply the update by hand
DP> or put a previous version of the library in place.
DP> When I boot from CD, I can

DP>  1. do a preservation install (clearly a final possibility)
DP>  2. maintenance mode.

DP> In maintenance mode, if I start a shell and acess the filesytems,
DP> it tries to do an fsck and gets locked into the same "symbol
DP> resolution failed" errors. This is prior to any attempt to
DP> mount filesystems.

Daniel,

Here's what IBM sent me about recovering from the libc.a malaise
It says it's for AIX 4.x, but works with 5.1 as well.

The point is to access the root volume group without mounting the
filesystems, which is a smitty menu item in maintenanace mode. You may
not need to jump through all the hoops if you find a good copy of libc.a
somewhere in /usr/ccs/lib. Just remember that now the commands are in
/mnt/usr/bin...

-Bela

(cut-n-paste):
-------------------------------------------------------------------------------

Contents

About this document
Recovering a deleted symbolic link in /usr/lib
Recovering a deleted /usr/ccs/lib/libc.a system library file
-------------------------------------------------------------------------------

About this document

This procedure documents how to recover the libc.a system library if the link
in /usr/lib or file in /usr/ccs/lib has been accidentally removed. This
document applies to AIX Versions 4.x.
-------------------------------------------------------------------------------
Recovering a deleted symbolic link in /usr/lib

If the symbolic link from /usr/lib/libc.a to /usr/ccs/lib/libc.a has been
removed and a system shell prompt exists, the following steps can be used to
recover the symbolic link.

  1. Set the environment variable LIBPATH to point to the /usr/ccs/lib
     directory. Enter:

           LIBPATH=/usr/ccs/lib:/usr/lib
           export LIBPATH

  2. You should be able to execute system commands at this point. Run the
     following command to re-link /usr/lib/libc.a to /usr/ccs/lib/libc.a and
     /lib to /usr/lib.

           ln -s /usr/ccs/lib/libc.a /usr/lib/libc.a
           ln -s /usr/lib /lib

  3. Unset the LIBPATH environment variable. At this point, commands should run
     as before. Enter:

           unset LIBPATH

     If you don't have a shell accessible to execute this command, follow the
     instructions in the next section.

-------------------------------------------------------------------------------

Recovering a deleted /usr/ccs/lib/libc.a system library file

To recover from a deleted /usr/ccs/lib/libc.a system file, the system must be
booted off of external media, since no commands can run in the machine's
current state.

The machine should be booted from a CD-ROM or NON_AUTOINSTALL mksysb tape of
the same operating system level as the system. See the document "Booting in
Service Mode" if there are any questions about booting your particular machine
into Service mode. Access the rootvg volume group without mounting the file
systems. Use a recent mksysb of the system to retrieve the libc.a file.
Afterward, proceed through the following steps.

  1. Execute the following sequence of commands.

          mount /dev/hd4 /mnt
          mount /dev/hd2 /mnt/usr
          cd /mnt

  2. If the symbolic link from /usr/lib/libc.a to /usr/ccs/lib/libc.a needs to
     be restored, run the following command and skip to step 5.

          ln -s /usr/ccs/lib/libc.a /mnt/usr/lib/libc.a

  3. Insert the mksysb tape in the tape drive and issue the following commands.
     Replace X with the appropriate number for the tape drive device.

     First you will need to set the block size of the tape drive. By default,
     the device driver in Maintenance mode will be set to a block size 512.

       1. tctl -f /dev/rmtX rewind
       2. tctl -f /dev/rmtX.Y fsf 1
       3. restbyname -xvqf /dev/rmtX.Y ./tapeblksz
       4. cat tapeblksz
       5. If the value from step d is not equal to 512, execute: /etc/methods
          /chggen -l rmtX -a block_size=[value from d]
       6. tctl -f /dev/rmtX rewind
       7. tctl -f /dev/rmtX.1 fsf 3

  4. Now use the restbyname command to restore the missing file. Replace X with
     the appropriate number for the tape drive device.

       1. Execute tctl -f /dev/rmtX rewind

       2. Execute tctl -f /dev/rmtX.1 fsf 3

       3. IF YOU NEED TO RESTORE /usr/ccs/lib/libc.a, execute:

                           restbyname -xvqf/dev/rmtX.Y ./usr/ccs/lib/libc.a

       4. IF YOU NEED TO RESTORE THE /usr/ccs/lib DIRECTORY, execute:

                           restbyname -xvqf/dev/rmtX.Y ./usr/ccs/lib

       5. IF YOU NEED TO RESTORE THE /usr/ccs/bin DIRECTORY, execute:

                           restbyname -xvqf/dev/rmtX.Y ./usr/ccs/bin

  5. Flush the data to disk. Enter:

            cd /mnt/usr/sbin
            ./sync;./sync;./sync

  6. Unmount the /usr file system. Enter:

            cd /
            umount /dev/hd2

     If that commands fails, turn the key to Normal mode, if applicable, and
     reboot.

  7. Unmount the / (root) file system. Enter:

            umount /dev/hd4

     If that commands fails, turn the key to Normal mode, if applicable, and
     reboot.

  8. Boot the system in Normal mode. Turn the key to Normal position, if
     applicable, and power cycle the machine.

--
Bela Gazdy, EUCLID/AIX Systems Support

"...who shook his family tree, and a bunch of NUTS fell out."

 
 
 

5.1 maintenance level 3 update breaks libc

Post by Aamir Kh » Tue, 14 Jan 2003 23:28:39


Daniel,

I get the same issues when, I create a NIM Master RML03 and try and
restore a RML03 Client from another NIM Master also at RML03.

The Mksysb completes but when it goes to Over Mounting /. 84% complete
... I think I get all the same symptoms as yourself.

Any Resolutions or work arounds that have sorted this one out would be
much appreciated.

kind regards

Aamir Khan

4u2c Consulting Ltd
http://www.4u2c.uk.com

 
 
 

5.1 maintenance level 3 update breaks libc

Post by Daniel Packm » Wed, 15 Jan 2003 02:31:28




>I get the same issues when, I create a NIM Master RML03 and try and
>restore a RML03 Client from another NIM Master also at RML03.

>The Mksysb completes but when it goes to Over Mounting /. 84% complete
>... I think I get all the same symptoms as yourself.

I went back to the 5.1 installation media, did a preservation install,
and then applied the single update to maintenance release 3. I don't
know if there is a more sophisticated way to resolve this.

--
Daniel Packman
NCAR/ACD