Kernel panics on raw I/O stress test

Kernel panics on raw I/O stress test

Post by Takanori Kawan » Fri, 20 Apr 2001 21:10:07



When I ran raw I/O SCSI read/write test with 2.4.1 kernel
on our IA64 8way SMP box, kernel paniced and following
message was displayed.

  Aiee, killing interrupt handler!

No stack trace and register dump are displayed.

Then I analyze FSB traces around the panic, and found that
following functions are called before panic().

   CPU0:                              CPU1:

      ?                                  ?
      ?                                  ?
      ?                                rw_raw_dev()
      ?                                  ?
      ?                                  ?
      ?                                 brw_kiovec()
      ?                                  ?
      ?                                  ?
      ?                                 free_kiovec()
      ?                                  ?
      ?                                  ?
    end_kio_request()
     __wake_up()
     ia64_do_page_fault()
      do_exit()
       panic()

I suppose that free_kiobuf() is called on CPU1 before
end_kio_request() is called on CPU0 for the same kiobuf
and resulted in the panic.
In 2.4.1 source code, I think there is no assurance
that free_kiovec() in rw_raw_dev() is called after
end_kio_request() is done.

I tried following two workarounds.

(1) Wait in rw_raw_dev() while io_count is positive.

--- drivers/char/raw.c        Mon Oct  2 12:35:15 2000

                        break;
        }

+       while(atomic_read(&iobuf->io_count)) {
+         set_task_state(current, TASK_UNINTERRUPTIBLE);
+         schedule();
+       }
+
        free_kiovec(1, &iobuf);

        if (transferred) {

(2) Keep buffer lock until end_kio_request() is done.

--- fs/buffer.c       Tue Jan 16 05:42:32 2001

        mark_buffer_uptodate(bh, uptodate);

        kiobuf = bh->b_private;
-       unlock_buffer(bh);
        end_kio_request(kiobuf, uptodate);
+       unlock_buffer(bh);
 }

Both of them worked well for our raw I/O testing,
but I'm not sure they are right.

Does anybody have comments?

regards,

---
Takanori Kawano
Hitachi Ltd,
Internet Systems Platform Division

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

Kernel panics on raw I/O stress test

Post by Alan Co » Fri, 20 Apr 2001 22:00:07


Quote:> When I ran raw I/O SCSI read/write test with 2.4.1 kernel
> on our IA64 8way SMP box, kernel paniced and following
> message was displayed.

>   Aiee, killing interrupt handler!

> (1) Wait in rw_raw_dev() while io_count is positive.

Stephen submitted a chunk of raw i/o fixes which are in recent -ac kernels.
I don't know if Linus has merged them offhand. But 2.4.1 raw is definitely
not watertight

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

Kernel panics on raw I/O stress test

Post by Andrea Arcangel » Sat, 21 Apr 2001 02:20:07



> When I ran raw I/O SCSI read/write test with 2.4.1 kernel
> on our IA64 8way SMP box, kernel paniced and following
> message was displayed.

Could you try again with 2.4.4pre4 plus the below patch?

        ftp://ftp.us.kernel.org/pub/linux/kernel/people/andrea/patches/v2.4/2...

You should experience also a quite noticeable improvement on both CPU usage and
disk I/O (also depends on the max size of a I/O request for your hardware disk
controller).

Andrea
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

Kernel panics on raw I/O stress test

Post by Takanori Kawan » Sat, 21 Apr 2001 20:50:10


Quote:> Could you try again with 2.4.4pre4 plus the below patch?

>    ftp://ftp.us.kernel.org/pub/linux/kernel/people/andrea/patches/v2.4/2...

I suppose that 2.4.4-pre4 + rawio-3 patch still has SMP-unsafe
raw i/o code and can cause the same panic I reported.

I think the following scenario is possible if there are 3 or more CPUs.

(1) CPU0 enter rw_raw_dev()
(2) CPU0 execute alloc_kiovec(1, &iobuf)    // drivers/char/raw.c line 309
(3) CPU0 enter brw_kiovec(rw, 1, &iobuf,..) // drivers/char/raw.c line 362
(4) CPU0 enter __wait_on_buffer()
(5) CPU0 execute run_task_queue() and wait
    while buffer_locked(bh) is true.        // fs/buffer.c line 152-158
(6) CPU1 enter end_buffer_io_kiobuf() with
     iobuf allocated at (2)
(7) CPU1 execute unlock_buffer()            // fs/buffer.c line 1994
(8) CPU0 exit __wait_on_buffer()
(9) CPU0 exit brw_kiovec(rw, 1, &iobuf,..)
(10) CPU0 execute free_kiovec(1, &iobuf)     // drivers/char/raw.c line 388
(11) The task on CPU2 reused the area freed
     at (10).
(12) CPU1 enter end_kio_request() and touch
     the corrupted iobuf, then panic.

---
Takanori Kawano
Hitachi Ltd,
Internet Systems Platform Division

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

Kernel panics on raw I/O stress test

Post by Andrea Arcangel » Sat, 21 Apr 2001 23:00:09



> > Could you try again with 2.4.4pre4 plus the below patch?

> >       ftp://ftp.us.kernel.org/pub/linux/kernel/people/andrea/patches/v2.4/2...

> I suppose that 2.4.4-pre4 + rawio-3 patch still has SMP-unsafe
> raw i/o code and can cause the same panic I reported.

I just fixed that as well last week and 2.4.4-pre4 + rawio-3 should be just SMP
safe, faster and my patch is racommended for integration.

Quote:> I think the following scenario is possible if there are 3 or more CPUs.

> (1) CPU0 enter rw_raw_dev()
> (2) CPU0 execute alloc_kiovec(1, &iobuf)    // drivers/char/raw.c line 309
> (3) CPU0 enter brw_kiovec(rw, 1, &iobuf,..) // drivers/char/raw.c line 362
> (4) CPU0 enter __wait_on_buffer()

With my patch applied the kernel doesn't execute wait_on_buffer from wait_kio
here, it first executes kiobuf_wait_for_io and that is also a performance
optimization because kiobuf_wait_for_io will sleep only once and it will
get only 1 wakeup once the whole kiobuf I/O completed.

Quote:> (5) CPU0 execute run_task_queue() and wait
>     while buffer_locked(bh) is true.        // fs/buffer.c line 152-158
> (6) CPU1 enter end_buffer_io_kiobuf() with
>      iobuf allocated at (2)
> (7) CPU1 execute unlock_buffer()            // fs/buffer.c line 1994
> (8) CPU0 exit __wait_on_buffer()
> (9) CPU0 exit brw_kiovec(rw, 1, &iobuf,..)
> (10) CPU0 execute free_kiovec(1, &iobuf)     // drivers/char/raw.c line 388
> (11) The task on CPU2 reused the area freed
>      at (10).
> (12) CPU1 enter end_kio_request() and touch
>      the corrupted iobuf, then panic.

The end_kio_request in CPU1 with my patch applied is executed before CPU0 can
execute wait_kio so it cannot race the above way.

Thanks for your comments (and yes you are right that the above race can happen
in all 2.4 kernels out there except in the aa latest ones).

Andrea
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

1. Kernel stress testing coverage

I'm looking for some advice from all of you that know and understand the
Linux kernel so well.  I'm not a kernel developer, but I want to do some
verification work on it, namely stress testing to begin with.  I'm working
on putting together a suite of tests to test the linux kernels under stress
loads for extended runs.  I'll be doing these tests on a mixture of
machines, but most of my focus will be on 2-way, 4-way and 8-way smp
machines.
We've been using some open source tools such as bonnie (for filesystem
stress), but I welcome suggestions for others that will provide good
coverage.  To fill in the gaps, we'll probably be developing our own
testcases.  We have no problems generating mixed loads, and even maxing out
8-way SMP servers for long periods of time, but to be useful, I want to
make sure that whatever tests we are running are getting good coverage of
the kernel code.
Is there any way to see what pieces of the kernel or even percentage we are
hitting with any given test?  I've heard of tools like gcov for doing this
with applications, but the kernel itself seems like it might require
something more.  Are there any ideas you can suggest for writing tests that
will hit as much code as possible in any given section of the kernel like
FS, MM, Scheduler, IPC?  What major sections (like those I previously
mentioned) should I be focused on trying to hit?
All the tests we are writing will be open source of course, and I welcome
any input you may provide.

Thanks,
Paul Larson

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

2. Help with NIC installation..

3. test test test test test test test

4. Canon BJ-200 printcap..

5. Stress testing a web server

6. 3ware discontinuing the Escalade Series

7. Stress testing a new server.

8. Seattle Programmer Needed

9. jdk/servlet container stress tests?

10. How to stress test webserver?

11. NFS performance tuning for stress test...

12. Solaris 7 Stress Testing

13. Stress testing an Apache-based webserver?