Very Computer

Hello all.... sorry for the cross-post, but it applies everywhere....

It has been a rough week.  I have some questions, to which I cannot
test the answers to (until Monday), and though I'd come seeking some
help ... my Googling hasn't come up too well.

Several months ago we bought a system from Monarch Computer, equipped
with a Gigabyte GA-7DPXDW-P, which has a FastTrack RAID controller on
it.  I installed the controller, and dual 80GB hard drives, which
after some trouble, I settled on the native ataraid driver, running RH
7.3.

It's been running fine, until this morning, when I totally hosed it.
Lack of sleep, no attention to detail, etc... and I went ahead and did
a system upgrade to it.  To the latest GLIBC, from RH 9.  Most of our
other systems are RH9, I completely forgot this one wasn't, and when
the package I was installing requested the new GLIBC, it didn't
register.  Bad, bad, bad sysadmin.

Anyway, RPM got hosed as per the
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=89135  and by the
time I had decided on a "downgrade path", my SSH connection got
dropped when a disk failed in our firewall.

Long and short of it ... GLIBC 2.3.X uses different POSIX threads (or
POSIX threads in general?) and aren't compatible with a lot of
programs not written for POSIX use.  Including our primary database,
the SSH installed on it, mount, you name it.  It won't boot much past
grub now.

We spent a couple of hours rebuilding the firewall (not bad
considering it was 1800+ miles away and the tech re-installing the OS
has never used Linux), and are proceeding to the main system, which
now that no one can log-in, is completely hosed.

We rely on some programs that must stay on RH 7.3, so it cannot be
upgraded, but I don't have a nice boot/rescue disk that will do the
ATA RAID.  Booting from the stock RH 7.3 ISO discs, RH finds no hard
drives, and has no way to do ataraid.  The hard drives are mirrored,
and the / partitions are ext3, with the other partitions on LVM.

I have the weekend to locate an acceptable "downgrade" solution, or,
at worst, a complete reformat and load from tape.  However, it would
appear from the solution at
http://www.linuxquestions.org/questions/archive/5/2003/09/1/57607 that
there is a possible solution.

My question:

a.  If I boot from a RH9 disk 1 (ISO) instead of RH7.3, will it find
the /dev/ataraid/d0 and /dev/ataraid/d1?  If not, can I drop to a
shell and load up the kernel modules to find them?

b.  Will the RH9 ISO locate the LVM partitions?  If not, can I somehow
locate them?

c.  Assuming I've managed to find the drives / partitions, is it then
feasible to load up the RH 7.3 disk's in cdrom1, and perform a rpm
-ivh --force --root /mnt/sysimage/ glibc*.rpm binutils*.rpm
tcp_wrapper*.rpm

d.  If these aren't valid, is there a standard distrib / kernel I can
build / setup an ISO to burn for them to locate the drives and do the
downgrade?

The goal is to bring up the system asap on Monday morning, so we can
all get back to work.

Thanks for your feedback.  I'll be posting results (I can't be the
only person with a ATARAID drive running LVM who needs a rescue disk).

You bought a Promise RAID card. That was a *serious* mistake. Promise
not only sucks in Linux driver support, they suck period in quality
control and in ever actually answering letters or telephone calls. They
only get business because their chips come with great *published*, not
real, specifications, and they're slightly cheaper.

Do not use them. Replace the system with a 3Ware controller or SIIG
controller ASAPl.

Quote:> It's been running fine, until this morning, when I totally hosed it.
> Lack of sleep, no attention to detail, etc... and I went ahead and did
> a system upgrade to it.  To the latest GLIBC, from RH 9.  Most of our
> other systems are RH9, I completely forgot this one wasn't, and when
> the package I was installing requested the new GLIBC, it didn't
> register.  Bad, bad, bad sysadmin.

Bad. No biscuit. You badly need to either do a forklift upgrade to RH 9,
or get at this machine's console with a set of RedHat 8 or 9 CD's and
boot into rescue mode, and use the network or CD drive to download and
install the older glibc packages.

Quote:> We rely on some programs that must stay on RH 7.3, so it cannot be
> upgraded, but I don't have a nice boot/rescue disk that will do the
> ATA RAID.  Booting from the stock RH 7.3 ISO discs, RH finds no hard
> drives, and has no way to do ataraid.  The hard drives are mirrored,
> and the / partitions are ext3, with the other partitions on LVM.

> I have the weekend to locate an acceptable "downgrade" solution, or,
> at worst, a complete reformat and load from tape.  However, it would
> appear from the solution at
> http://www.linuxquestions.org/questions/archive/5/2003/09/1/57607 that
> there is a possible solution.

Ouch. You mean you put your boot OS on the Promise RAID controller? Of
course, Promise will not tell you which chipet the particular Fastrack
controller uses. They seem to pretend that they don't use chips, because
there is literally *NO WAY* to get them to tell you what chipset a
device uses without buying one and dismantling it to read the chips or
having a kernel that already knows about it.

In the meantime: can you get someone onsite to plug in a RedHat 9 boot
CD and see if you can boot to rescue mode with it? Since RH 9 uses a
nice 2.4.x kernel with a lot of drivers backported to it from newer
kernels, you may be able to boot with it enough to get into rescue mode
and salvage the mess.

Quote:> My question:

> a.  If I boot from a RH9 disk 1 (ISO) instead of RH7.3, will it find
> the /dev/ataraid/d0 and /dev/ataraid/d1?  If not, can I drop to a
> shell and load up the kernel modules to find them?

See above. *MAYBE*. The Promise controller chipset has to be included in
the kernel. Did you have to compile and install a custom kernel to see
these RAIDsets?

Quote:> b.  Will the RH9 ISO locate the LVM partitions?  If not, can I somehow
> locate them?

See above. *MAYBE*. And why are you wasting time and CPU with LVM on a
server with a hardware RAID controller? Do you have somewhere on backup
or offsite a tape with your old /etc/fstab and LVM settings, for reference?

Quote:> c.  Assuming I've managed to find the drives / partitions, is it then
> feasible to load up the RH 7.3 disk's in cdrom1, and perform a rpm
> -ivh --force --root /mnt/sysimage/ glibc*.rpm binutils*.rpm
> tcp_wrapper*.rpm

*FIRST*, try  this:

        cp glibc*.rpm binutils*.rpm tcp_wrapper*.rpm /mnt/sysimage/tmp
        chroot /mnt/sysimage
        rpm -Uvvh --replacepkgs --oldpackage {packages.rpm}

I've never trusted the rpm "--root" command.

Quote:> d.  If these aren't valid, is there a standard distrib / kernel I can
> build / setup an ISO to burn for them to locate the drives and do the
> downgrade?

See above. Promise is *VERY, VERY, VERY* bad about Linux support. They
say their stuff is compatible, and then they publish drivers that do
stupidities like auto-renumber your IDE controllers so the Promise
controllers are *always* first. That drove me nuts when I did a kernel
change and had to rewrite fstab depending on which kernel I booted with,
because a manager refused to permit us to use partition labels.

Quote:> The goal is to bring up the system asap on Monday morning, so we can
> all get back to work.

Good freaking luck. Sounds like you need a kernel/hardware consultant,
*NOW*. Is your contact address on the posting valid? And what part of
the country is the target machine in? I may be able to recommend any of
several people in different parts of New England, the CA kernel people
don't know an IDE connector from a CVS command.

Quote:> Thanks for your feedback.  I'll be posting results (I can't be the
> only person with a ATARAID drive running LVM who needs a rescue disk).

Promise RAID ... yes, after I installed it completely, I started
reading about the horror stories.  If I can't get this machine up
Monday morning, I may just have it overnighted to me, and completely
redo it from tape backups w/o the ataraid (just use Linux software
RAID).

Quote:> > It's been running fine, until this morning, when I totally hosed it.
> > Lack of sleep, no attention to detail, etc... and I went ahead and did
> > a system upgrade to it.  To the latest GLIBC, from RH 9.  Most of our
> > other systems are RH9, I completely forgot this one wasn't, and when
> > the package I was installing requested the new GLIBC, it didn't
> > register.  Bad, bad, bad sysadmin.

> Bad. No biscuit. You badly need to either do a forklift upgrade to RH 9,
> or get at this machine's console with a set of RedHat 8 or 9 CD's and
> boot into rescue mode, and use the network or CD drive to download and
> install the older glibc packages.

Cannot do a forklift upgrade to RH9 w/o losing my database (older
Informix), and that would be very bad (although we're migrating off of
it).

Quote:> > We rely on some programs that must stay on RH 7.3, so it cannot be
> > upgraded, but I don't have a nice boot/rescue disk that will do the
> > ATA RAID.  Booting from the stock RH 7.3 ISO discs, RH finds no hard
> > drives, and has no way to do ataraid.  The hard drives are mirrored,
> > and the / partitions are ext3, with the other partitions on LVM.

> > I have the weekend to locate an acceptable "downgrade" solution, or,
> > at worst, a complete reformat and load from tape.  However, it would
> > appear from the solution at
> > http://www.linuxquestions.org/questions/archive/5/2003/09/1/57607 that
> > there is a possible solution.

> Ouch. You mean you put your boot OS on the Promise RAID controller? Of
> course, Promise will not tell you which chipet the particular Fastrack
> controller uses. They seem to pretend that they don't use chips, because
> there is literally *NO WAY* to get them to tell you what chipset a
> device uses without buying one and dismantling it to read the chips or
> having a kernel that already knows about it.

> In the meantime: can you get someone onsite to plug in a RedHat 9 boot
> CD and see if you can boot to rescue mode with it? Since RH 9 uses a
> nice 2.4.x kernel with a lot of drivers backported to it from newer
> kernels, you may be able to boot with it enough to get into rescue mode
> and salvage the mess.

This is my hope, it has dual cdrom's, so perhaps I can get the machine
to boot with RH9 ISO's, get to the boot, load up the ataraid module
(if not done initially -- does anyone know if RH9 will autodetect
ataraid?)

I have a 7.3 machine here with me, which I have VMware'd, and I did
the same thing I did to the machine in question.  Rebooted, it runs
fine.  I can login, SSH to the machine, you name it.  Only difference
is that my test machine has no ATARAID.  Joyful.  I still get the
errors according to rpm (ie, it seg. faults if I run it), but
otherwise it is FINE.  I've double and triple checked, the only thing
different is the damned card.  Consider my hand slapped, HARD.

Quote:> > a.  If I boot from a RH9 disk 1 (ISO) instead of RH7.3, will it find
> > the /dev/ataraid/d0 and /dev/ataraid/d1?  If not, can I drop to a
> > shell and load up the kernel modules to find them?

> See above. *MAYBE*. The Promise controller chipset has to be included in
> the kernel. Did you have to compile and install a custom kernel to see
> these RAIDsets?

I took the RH7.3 kernel, and compiled in support for the ATARAID
drivers as per the advice on the ATARAID HOWTO
http://www.murty.net/ataraid/nativeraid.html -- and that was that.
What I don't know is if RH9 can find the modules for the Gigabyte
board.

Quote:> > b.  Will the RH9 ISO locate the LVM partitions?  If not, can I somehow
> > locate them?

> See above. *MAYBE*. And why are you wasting time and CPU with LVM on a
> server with a hardware RAID controller? Do you have somewhere on backup
> or offsite a tape with your old /etc/fstab and LVM settings, for reference?

Err ... nope, the backups are onsite.  What are you alluding to with
LVM and hardware RAID controller?

Quote:> > c.  Assuming I've managed to find the drives / partitions, is it then
> > feasible to load up the RH 7.3 disk's in cdrom1, and perform a rpm
> > -ivh --force --root /mnt/sysimage/ glibc*.rpm binutils*.rpm
> > tcp_wrapper*.rpm

> *FIRST*, try  this:

>    cp glibc*.rpm binutils*.rpm tcp_wrapper*.rpm /mnt/sysimage/tmp
>    chroot /mnt/sysimage
>    rpm -Uvvh --replacepkgs --oldpackage {packages.rpm}

> I've never trusted the rpm "--root" command.

> > d.  If these aren't valid, is there a standard distrib / kernel I can
> > build / setup an ISO to burn for them to locate the drives and do the
> > downgrade?

> See above. Promise is *VERY, VERY, VERY* bad about Linux support. They
> say their stuff is compatible, and then they publish drivers that do
> stupidities like auto-renumber your IDE controllers so the Promise
> controllers are *always* first. That drove me nuts when I did a kernel
> change and had to rewrite fstab depending on which kernel I booted with,
> because a manager refused to permit us to use partition labels.

Huh ... strange.  Nope, I agree, and had I known what I knew now, some
8 or so months ago, I wouldn't have implemented the hardware RAID,
would've stuck with software (which I know much better).

Quote:> Good freaking luck. Sounds like you need a kernel/hardware consultant,
> *NOW*. Is your contact address on the posting valid? And what part of
> the country is the target machine in? I may be able to recommend any of
> several people in different parts of New England, the CA kernel people
> don't know an IDE connector from a CVS command.

The contract address is valid, and the machine is in NY.  I have some
good people in VA, but that's too far to be going to do what I need.
May be possible to have it overnight'd, with morning service to me on
Monday so that I can work on it Tuesday and have it back Thursday
morning.  That is, if my current plan doesn't work (VERY reliant on
Redhat having some built-in ATARAID support ... any ideas about
Knoppix, or others?).

--Anthony

Oh, I know for SURE that the chip is a PDC20276 from Promise .... does
Redhat 9 support it out of the box (ie, from booting off of disc 1)
... or do I need to create a custom kernel that I can (possibly) plug
into the RH 9 ISO?

--Anthony

I was unclear. With a RAID controller, you *should* be able to merge a
large number of disks into a single RAIDset and have that occur in
hardware, saving cycles from doing other sophisticated tasks (like LVM).

What do you expect to gain from using LVM?

Quote:>>>c.  Assuming I've managed to find the drives / partitions, is it then
>>>feasible to load up the RH 7.3 disk's in cdrom1, and perform a rpm
>>>-ivh --force --root /mnt/sysimage/ glibc*.rpm binutils*.rpm
>>>tcp_wrapper*.rpm

>>*FIRST*, try  this:

>>        cp glibc*.rpm binutils*.rpm tcp_wrapper*.rpm /mnt/sysimage/tmp
>>        chroot /mnt/sysimage
>>        rpm -Uvvh --replacepkgs --oldpackage {packages.rpm}

>>I've never trusted the rpm "--root" command.

>>>d.  If these aren't valid, is there a standard distrib / kernel I can
>>>build / setup an ISO to burn for them to locate the drives and do the
>>>downgrade?

>>See above. Promise is *VERY, VERY, VERY* bad about Linux support. They
>>say their stuff is compatible, and then they publish drivers that do
>>stupidities like auto-renumber your IDE controllers so the Promise
>>controllers are *always* first. That drove me nuts when I did a kernel
>>change and had to rewrite fstab depending on which kernel I booted with,
>>because a manager refused to permit us to use partition labels.

> Huh ... strange.  Nope, I agree, and had I known what I knew now, some
> 8 or so months ago, I wouldn't have implemented the hardware RAID,
> would've stuck with software (which I know much better).

That wouldn't have helped against *THIS* problem. You wouldn't have been
able to use the Promise chipsets *at all*, even for non-RAID use.

Quote:>>Good freaking luck. Sounds like you need a kernel/hardware consultant,
>>*NOW*. Is your contact address on the posting valid? And what part of
>>the country is the target machine in? I may be able to recommend any of
>>several people in different parts of New England, the CA kernel people
>>don't know an IDE connector from a CVS command.

> The contract address is valid, and the machine is in NY.  I have some
> good people in VA, but that's too far to be going to do what I need.
> May be possible to have it overnight'd, with morning service to me on
> Monday so that I can work on it Tuesday and have it back Thursday
> morning.  That is, if my current plan doesn't work (VERY reliant on
> Redhat having some built-in ATARAID support ... any ideas about
> Knoppix, or others?).

Various forms of hardware and software RAID have been in the Linux
kernels for ages. ATARAID is not the problem. It's the Promise chipset
and drivers for it: be *very* clear on what you need.

Do you know the chipset of the FasTrak system? Can you have someone boot
the damn thing and write down what it says? And what is your disk layout
like?

It was a harrowing day, but I wanted to list what happened for future
seekers of the path to being a superior being.

I have VMware on a machine, running RH7.3 (same config as the hosed
machine, but with no Promise chip).  I booted it up, and did the same
thing (over-wrote glibc), which killed rpm (seg fault when trying to
run), but it rebooted fine, came up, and I could log in.  Very
confusing.

After checking around, I built a custom RH9 Boot ISO (with ATARAID,
which the default BOOT kernel doesn't have), as well as prepared a
KNOPPIX boot CD for emergency use.  My plan was:

--Attempt to boot the hosed machine and login.  Upon failing....
--Attempt to boot into single-user mode.  Upon failing....
--Attempt rescue with RH9 stock ISO.  Upon failing....
--Attempt rescue with custom RH9.  Upon failing....
--Attempt rescue with KNOPPX CD (which DOES include ATARAID, Promise
20276, LVM, etc... from BOOT).  Upon failing....
--Break the RAID, attempt to access w/o the RAID.  Upon failing....
--Scream, ship the box to TX for repairs.

That being said, Monday morning came around and we were ready to walk
through on the phone, having done the restores about 30 times over the
weekend.  They came into work, booted it up (step 1), and it said it
could not find any hard drives (after grub and the kernel loaded).  We
rebooted, and went into single-user mode, at which point it found the
hard drives, and booted perfectly.  Curious, I thought, so we rebooted
and tried step 1 again -- it worked fine.  Furthermore, I could login
with SSH, etc...  Everything was fine.  RPM didn't run, but rpm2cpio
did -- so I restored everything using rpm2cpio * | cpio -iudv and then
updated /etc/ld.so.conf.  I then did a rpm -e of the glibc packages I
had screwed up, and then reinstalled the RH7.3, without a hitch.

Which makes no sense.  Friday, I could not login, could not startup
Informix, could not get networked, etc... and today it worked fine,
even with GLIBC 2.3.2 running on RH7.3 -- how WEIRD.

If anyone can explain, please do.  I was expecting a day of hell, and
instead, at 8am, it worked fine.

Quote:> > Err ... nope, the backups are onsite.  What are you alluding to with
> > LVM and hardware RAID controller?

> I was unclear. With a RAID controller, you *should* be able to merge a
> large number of disks into a single RAIDset and have that occur in
> hardware, saving cycles from doing other sophisticated tasks (like LVM).

> What do you expect to gain from using LVM?

Resizable partitions that I can allocate based on the owner's desire
to launch new programs (ie, customer websites or a new database) w/o
running into as many growing pains.

Quote:> > Huh ... strange.  Nope, I agree, and had I known what I knew now, some
> > 8 or so months ago, I wouldn't have implemented the hardware RAID,
> > would've stuck with software (which I know much better).

> That wouldn't have helped against *THIS* problem. You wouldn't have been
> able to use the Promise chipsets *at all*, even for non-RAID use.

Well, I'm still not sure what *THIS* problem was, because by my own,
yours, and everything I've read -- I was not in a good situation.  I
couldn't have written a simpler "restore" solution.

Quote:> Various forms of hardware and software RAID have been in the Linux
> kernels for ages. ATARAID is not the problem. It's the Promise chipset
> and drivers for it: be *very* clear on what you need.

> Do you know the chipset of the FasTrak system? Can you have someone boot
> the damn thing and write down what it says? And what is your disk layout
> like?

Yup -- it's back up, so these are less important.  But, around X-Mas
we will be moving from ATARAID to software RAID, with a good restore
solution* around.