lockup on Athlon systems, kernel race condition?

lockup on Athlon systems, kernel race condition?

Post by Terence Ripperd » Sun, 01 Sep 2002 06:00:13



I'm working on a problem where leaving viewperf running overnight with our drivers on a dual Athlon system locks up. The same problem is not reproducible on a dual P3 or dual P4.

Using kdb, I was able to break in and take a look at things once they hung. It appears to be a core kernel race condition. I'm not 100% sure, so I wanted to lay the information out for you to see if the diagnosis looks correct.

The de* breaks into cpu 1, with the following backtrace:

Entering kdb (current=0xe1eea000, pid 2103) on processor 1 due to Keyboard Entry
[1]kdb> bt
EBP        EIP        Function (args)
0xe1eeb910 0xc0115739 smp_call_function+0x99 (0xc0115550, 0x0, 0x1, 0x1, 0x400000)
                               kernel .text 0xc0100000 0xc01156a0 0xc0115790
0xe1eeb92c 0xc01155b4 flush_tlb_all+0x14 (0x1000, 0xf9b82000, 0x2d29f000, 0xc0101f9c, 0x73)
                               kernel .text 0xc0100000 0xc01155a0 0xc0115600
0xe1eeb96c 0xc011883f remap_area_pages+0x1ef (0xf9c00000, 0x2d29f000, 0x1000, 0x10, 0x1000)
                               kernel .text 0xc0100000 0xc0118650 0xc0118850
0xe1eeb994 0xc011890c __ioremap+0xbc (0x2d29f000, 0x1000, 0x10, 0xf891eb7f, 0x2d29f000)
                               kernel .text 0xc0100000 0xc0118850 0xc0118940
0xe1eeb9c4 0xf890cf3c [NVdriver]os_map_kernel_space+0x68 (0x2d29f000, 0x1000, 0x1, 0xf891da57, 0xf9b82000)
                               NVdriver .text 0xf8909060 0xf890ced4 0xf890cf80
... nvidia agp functions ...
0xe1eebf94 0xf890a7a4 [NVdriver]nv_kern_ioctl+0x2d0 (0xf5099380, 0xeb01ccc0, 0xc0284627, 0xbfffe810, 0xe1eea000)
                               NVdriver .text 0xf8909060 0xf890a4d4 0xf890a7c4
0xe1eebfbc 0xc0154d29 sys_ioctl+0x209 (0x6, 0xc0284627, 0xbfffe810, 0x2100, 0x8147048)
                               kernel .text 0xc0100000 0xc0154b20 0xc0154d90
           0xc0109415 system_call+0x55
                               kernel .text 0xc0100000 0xc01093c0 0xc010941c

the basic idea is that when handling agp allocations, we're mapping and unmapping pages, which is triggering tlb flushes. (this lockup occurs with nvagp or agpgart). I originally suspected we were causing a race condition by holding a lock, then calling a blocking kernel function, but that is not the case. We are not holding any locks here.

checking out the other cpu reveals:

[1]kdb> cpu 0

Entering kdb (current=0xf1612000, pid 1661) on processor 0 due to cpu switch
[0]kdb> bt
EBP        EIP        Function (args)
           0xc0154d97 .text.lock.ioctl+0x7
                               kernel .text 0xc0100000 0xc0154d90 0xc0154dc0
0xf1613fbc 0xc0154b8a sys_ioctl+0x6a (0x4, 0x541b, 0xbfffe15c, 0x805b050, 0x805ab10)
                               kernel .text 0xc0100000 0xc0154b20 0xc0154d90
           0xc0109415 system_call+0x55
                               kernel .text 0xc0100000 0xc01093c0 0xc010941c

disassembling the code at cpu 0's EIP shows:

0xc0154b8a <sys_ioctl+106>:     lock decb 0xc03cae40

looking up 0xc03cae40 in the System.map files shows:

c03cae40 D kernel_flag_cacheline

This code appears to be the following code from fs/ioctl.c (grabbing the big kernel lock):

asmlinkage long sys_ioctl(unsigned int fd, unsigned int cmd, unsigned long arg)
{      
        struct file * filp;
        unsigned int flag;
        int on, error = -EBADF;

        filp = fget(fd);
        if (!filp)
                goto out;
        error = 0;
        lock_kernel();    <====
        switch (cmd) {

Recall that the earlier backtrace on cpu 1 was also handling an ioctl, which suggests cpu 1 has the big kernel lock (by nature of being in an ioctl), and cpu 0 is attempting to grab the big kernel lock. While handling the ioctl, cpu 1 is mapping/unmapping pages, which causes an IPI to flush all cpu's tlbs. cpu 0 cannot respond to the IPI, due to being in an atomic operation grabbing the big kernel lock. But cpu 1 has the lock and won't surrender the lock until cpu 0 finishes the IPI.

I believe this is a core kernel race condition. I would also suspect that the same problem could happen with other drivers when run through such a stress test. Perhaps the same problem could be reproduced with another graphics card (calling to the kernel to allocate (agp) memory) left running viewperf overnight.

Thanks,
Terence
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://www.veryComputer.com/
Please read the FAQ at  http://www.veryComputer.com/

 
 
 

lockup on Athlon systems, kernel race condition?

Post by Andrew Morto » Sun, 01 Sep 2002 06:20:10



> ...

> asmlinkage long sys_ioctl(unsigned int fd, unsigned int cmd, unsigned long arg)
> {
>         struct file * filp;
>         unsigned int flag;
>         int on, error = -EBADF;

>         filp = fget(fd);
>         if (!filp)
>                 goto out;
>         error = 0;
>         lock_kernel();    <====
>         switch (cmd) {

This CPU is spinning, waiting for kernel_flag.  It will take the IPI
and the other CPU's smp_call_function() will succeed.

Possibly the IPI has got lost - seems that this is a popular failure mode
for flakey chipsets/motherboards.

Or someone has called sys_ioctl() with interrupts disabled.  That's very
doubtful.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

lockup on Athlon systems, kernel race condition?

Post by Alan Co » Sun, 01 Sep 2002 09:40:07


Duplicate the problem without NVdriver having ever been loaded that
boot, then it might be interesting

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

lockup on Athlon systems, kernel race condition?

Post by Terence Ripperd » Thu, 05 Sep 2002 03:40:11



> This CPU is spinning, waiting for kernel_flag.  It will take the IPI
> and the other CPU's smp_call_function() will succeed.

thank you for the clarification, Andrew. I was unclear whether the 'lock' command would block out the IPI or not.

Quote:> Possibly the IPI has got lost - seems that this is a popular failure mode
> for flakey chipsets/motherboards.

this sounds like the most likely candidate. I'm working on tracking down documentation for further study. Is there an easy way to determine this as the cause?

Quote:> Or someone has called sys_ioctl() with interrupts disabled.  That's very
> doubtful.

I agree, that's highly doubtful.

one of our QA guys tried to reproduce this issue on an ATI card, but was unable to get through viewperf long enough to repro this problem before hitting other system lockups. I'm also installing an ATI card to try and see if I can reproduce the problem.

Terence
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

lockup on Athlon systems, kernel race condition?

Post by Manfred Sprau » Thu, 05 Sep 2002 06:50:07



>> ...

>> asmlinkage long sys_ioctl(unsigned int fd, unsigned int cmd, unsigned long arg)
>> {
>>         struct file * filp;
>>         unsigned int flag;
>>         int on, error = -EBADF;

>>         filp = fget(fd);
>>         if (!filp)
>>                 goto out;
>>         error = 0;
>>         lock_kernel();    <====

Which compiler to you use, and which kernel? Which additional patches?

With my 2.4.20-pre4-ac1 kernel, the lock_kernel is at offset +3a,
according to your dump it's at +6a.

Quote:>>         switch (cmd) {

> This CPU is spinning, waiting for kernel_flag.  It will take the IPI
> and the other CPU's smp_call_function() will succeed.

> Possibly the IPI has got lost - seems that this is a popular failure mode
> for flakey chipsets/motherboards.

> Or someone has called sys_ioctl() with interrupts disabled.  That's very
> doubtful.

Is it possible to display the cpu registers with kdb? Could you check
that the interrupts are enabled?

I'd add a quick test into sys_ioctl() or lock_kernel: save_flags, and
check that bit 9 is always enabled. Check __global_cli for sample code.
The X server probably runs with enough priveledges to disable the
interrupts, perhaps it's doing something stupid.

--
        Manfred

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

lockup on Athlon systems, kernel race condition?

Post by Terence Ripperd » Thu, 05 Sep 2002 07:10:08



> Which compiler to you use, and which kernel? Which additional patches?

> With my 2.4.20-pre4-ac1 kernel, the lock_kernel is at offset +3a,
> according to your dump it's at +6a.

I'm using gcc 2.95.4 (on debian). The kernel I'm currently using is 2.4.19, with the kdb patch and apic_routing patches applied. Neither of these is required to reproduce the problem, they're just to help debug the problem.

that reminds me. when I originally tried to use the nmi watchdog to break into the de* when the hang occured, I got the following at bootup:

Sep  3 14:56:49 localhost kernel: activating NMI Watchdog ... done.
Sep  3 14:56:49 localhost kernel: testing NMI watchdog ... CPU#0: NMI appears to be stuck!

Quote:> Is it possible to display the cpu registers with kdb? Could you check
> that the interrupts are enabled?

I don't have things caught in the de* currently, but I did check the registers at the time. bit 9 of eflags was enabled on both cpus.

Terence
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://www.veryComputer.com/
Please read the FAQ at  http://www.veryComputer.com/

 
 
 

lockup on Athlon systems, kernel race condition?

Post by Manfred Sprau » Fri, 06 Sep 2002 00:10:05



> I don't have things caught in the de* currently, but I did check the registers at the time. bit 9 of eflags was enabled on both cpus.

That rules out a disabled interrupt flag.

What was cpu1 doing? There are 3 loops in smp_call_function:
spin_lock(&call_lock), and 2 while loops with barrier() [first one for
IPI arrival, second one for synchroneous ipi's. TLB flush is sync].

--
        Manfred

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://www.veryComputer.com/
Please read the FAQ at  http://www.veryComputer.com/

 
 
 

lockup on Athlon systems, kernel race condition?

Post by Terence Ripperd » Fri, 06 Sep 2002 01:20:08



> What was cpu1 doing? There are 3 loops in smp_call_function:
> spin_lock(&call_lock), and 2 while loops with barrier() [first one for
> IPI arrival, second one for synchroneous ipi's. TLB flush is sync].

it was spinning on the first barrier loop, waiting for verification of the IPI arrival:

        /* Wait for response */
        while (atomic_read(&data.started) != cpus)
                barrier();

Terence
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

1. 2.0.14 problems: assuming a kernel race condition

Hi,

Using 2.0.x Kernels (currently 2.0.14) I can reliably crash my system.
I first noticed the problem somewhere during 1.3.7x, but didn't bother
because it only happens during heavy disk usage and when I try to
backup to tape.

The kernel seems to trash its file system cache every once in a while
during heavy disk IO.  Using 1.2.13+aic7xxx patch, the machine runs
rock solid, and I'm quite sure the hardware is ok.  (I already had
the problems a few times with my previous 486-based system).

System:  ASUS TP4/XE (Triton/256k PB Cache), 120Mhz Pentium, 64 MB Ram
         aha-2940 with 2x IBM DPES 31080 and HP-35480 DAT,
         currently Quantum Fireball TM 2110 on IDE #1
         and 5xSpeed CDrom on IDE #2

Using 2.0.11 and 2 smaller IDE drives, I got frequent crashes and I
even got a Kernel panics (OOPS) logged a few times.
With 2.0.13 the system takes _much_ longer to crash, but then it locks
up tight leaving no traces in the log files (booting from/logging to
IDE).

From those recent crashes with 2.0.11 where I saw a kernel OOPS
(usually I run X...), it happened at >>> in linux/fs/buffer.c:

    static struct buffer_head * get_unused_buffer_head(void)
    {
        struct buffer_head * bh;

        recover_reusable_buffer_heads();
        get_more_buffer_heads();
        if (!unused_list)
                return NULL;
        bh = unused_list;
        bh->b_next_free = NULL;
        bh->b_data = NULL;
        bh->b_size = 0;
        bh->b_state = 0;
        return bh;
   }

When I replaced my 2 old IDE drives with a single 2G drive, I cleaned
the old drives with "dd if=/dev/zero of=/dev/hdb bs=8192" using 2.0.13.
The first run (on a 540 MB drive) went through just fine, but the
second run locked the machine up tight after ~1 minutes, consistently
and reproducible.  Running on the Console, I even didn't get a Kernel
panic message on the screen...

Using 1.2.13 I can dd to the disk as often as I like, everythings
just fine.

Btw. I am not using the "Triton chipset support".  I tried this once
with ~2.0.4, and got quite many kernel panics crying "inode changed
from under us", e.g. an fsck on an IDE partition will panic the
machine in a few seconds.

Without Triton chipset support, 2.0.x usually copes with (sequential)
fscks and finds, but it always crashes during a backup to DAT tape.
If I run other heavy process during backup, it again crashes within
seconds when reading the IDE drives.  Using 1.2.13, heavy
processes parallel to backup at most prevents the tape from streaming
continously, but the machine crunches away happily and has never
crashed or paniced so far.

-Martin

2. kernel 2.5.40 fails to compile

3. lockup on dual athlon system.

4. XFree86/Acer Aspire Help

5. Kernel lockups on dual-Athlon board -- help wanted

6. How to best upgrade kernel under debian release...

7. Weird lockups on boot, dual athlon and kernel 2.4.20

8. Decipher "core" File

9. lockup in KDE2 and unable to load athlon kernel

10. Kernel 2.2.21 aic7xxx lockup or kernel lockup

11. In/Output buffer race conditions

12. Race condition with MOD_INC_USE_COUNT?

13. Wait queues and a race condition on 2.2.x