2.5.32: DEBUG_SLAB and PREEMPT = constant oops in schedule()

2.5.32: DEBUG_SLAB and PREEMPT = constant oops in schedule()

Post by Burton Windl » Thu, 29 Aug 2002 23:20:08



When running the Linux Test Project's process_stress test on 2.5.32 that
is compiled with DEBUG_SLAB and PREEMPT, I am able to easily reproduce
this oops. Undef either of the above kernel options, and the system is
stable again.

Unable to handle kernel paging request at virtual address 5a5a5ace
c010ecaf
*pde = 00000000
Oops: 0002
CPU:    0
EIP:    0060:[<c010ecaf>]    Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010817
eax: c9a92000   ebx: c9fd3760   ecx: c9b81288   edx: 5a5a5a5a
esi: c9b863ec   edi: c9fd31e0   ebp: c9a93ee0   esp: c9a93ec8
ds: 0068   es: 0068   ss: 0068
Stack: 7fffffff c9b863ec c9a93f60 c9fd31e0 00000282 c9a92000 c405d000 c0117e10
       00000008 c9b863ec 080dc050 00000246 c405d980 c9b863ec c01b51e4 c01b5234
       c405d000 c9b863ec c9d8c5ec c9b8640c c405d980 c405dbfc 00000000 7fffffff
Call Trace: [<c0117e10>] [<c01b51e4>] [<c01b5234>] [<c010ee2c>] [<c010ee2c>]
   [<c01b1294>] [<c0132a1b>] [<c0132bf6>] [<c010725f>]
Code: 0f ba 6a 74 00 8b 42 0c 05 00 00 00 40 0f 22 d8 8b 82 98 00

Quote:>>EIP; c010ecaf <schedule+18f/2a8>   <=====

Trace; c0117e10 <schedule_timeout+14/a4>
Trace; c01b51e4 <read_chan+354/74c>
Trace; c01b5234 <read_chan+3a4/74c>
Trace; c010ee2c <default_wake_function+0/34>
Trace; c010ee2c <default_wake_function+0/34>
Trace; c01b1294 <tty_read+d0/128>
Trace; c0132a1b <vfs_read+b7/138>
Trace; c0132bf6 <sys_read+2a/3c>
Trace; c010725f <syscall_call+7/b>

Code;  c010ecaf <schedule+18f/2a8>
00000000 <_EIP>:
Code;  c010ecaf <schedule+18f/2a8>   <=====
   0:   0f ba 6a 74 00            btsl   $0x0,0x74(%edx)   <=====
Code;  c010ecb4 <schedule+194/2a8>
   5:   8b 42 0c                  mov    0xc(%edx),%eax
Code;  c010ecb7 <schedule+197/2a8>
   8:   05 00 00 00 40            add    $0x40000000,%eax
Code;  c010ecbc <schedule+19c/2a8>
   d:   0f 22 d8                  mov    %eax,%cr3
Code;  c010ecbf <schedule+19f/2a8>
  10:   8b 82 98 00 00 00         mov    0x98(%edx),%eax

Linux razor 2.5.32 #5 Wed Aug 28 09:21:53 EST 2002 i686 unknown unknown GNU/Linux

Gnu C                  2.95.4
Gnu make               3.79.1
util-linux             2.11n
mount                  2.11n
modutils               2.4.15
e2fsprogs              1.27
Linux C Library        2.2.5
Dynamic linker (ldd)   2.2.5
Procps                 2.0.7
Net-tools              1.60
Console-tools          0.2.3
Sh-utils               2.0.12

Config:
CONFIG_X86=y
CONFIG_ISA=y
CONFIG_UID16=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_EXPERIMENTAL=y
CONFIG_NET=y
CONFIG_SYSVIPC=y
CONFIG_BSD_PROCESS_ACCT=y
CONFIG_SYSCTL=y
CONFIG_M686=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INVLPG=y
CONFIG_X86_CMPXCHG=y
CONFIG_X86_XADD=y
CONFIG_X86_BSWAP=y
CONFIG_X86_POPAD_OK=y
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
CONFIG_X86_L1_CACHE_SHIFT=5
CONFIG_X86_TSC=y
CONFIG_X86_GOOD_APIC=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
CONFIG_X86_PPRO_FENCE=y
CONFIG_PREEMPT=y
CONFIG_X86_MCE=y
CONFIG_X86_MCE_NONFATAL=y
CONFIG_NOHIGHMEM=y
CONFIG_HAVE_DEC_LOCK=y
CONFIG_PM=y
CONFIG_PCI=y
CONFIG_PCI_GOANY=y
CONFIG_PCI_BIOS=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_NAMES=y
CONFIG_KCORE_ELF=y
CONFIG_BINFMT_ELF=y
CONFIG_PNP=y
CONFIG_ISAPNP=y
CONFIG_BLK_DEV_FD=y
CONFIG_IDE=y
CONFIG_BLK_DEV_IDE=y
CONFIG_BLK_DEV_IDEDISK=y
CONFIG_IDEDISK_MULTI_MODE=y
CONFIG_BLK_DEV_IDECD=y
CONFIG_BLK_DEV_IDEPCI=y
CONFIG_IDEPCI_SHARE_IRQ=y
CONFIG_BLK_DEV_IDEDMA_PCI=y
CONFIG_IDEDMA_PCI_AUTO=y
CONFIG_BLK_DEV_IDEDMA=y
CONFIG_BLK_DEV_PIIX=y
CONFIG_PIIX_TUNING=y
CONFIG_IDEDMA_AUTO=y
CONFIG_BLK_DEV_IDE_MODES=y
CONFIG_PACKET=y
CONFIG_UNIX=y
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_NETDEVICES=y
CONFIG_NET_ETHERNET=y
CONFIG_NET_VENDOR_3COM=y
CONFIG_VORTEX=y
CONFIG_INPUT=y
CONFIG_SOUND_GAMEPORT=y
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_I8042_REG_BASE=60
CONFIG_I8042_KBD_IRQ=1
CONFIG_I8042_AUX_IRQ=12
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y
CONFIG_VT=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
CONFIG_UNIX98_PTYS=y
CONFIG_UNIX98_PTY_COUNT=256
CONFIG_AGP=y
CONFIG_AGP_INTEL=y
CONFIG_DRM=y
CONFIG_DRM_R128=y
CONFIG_DRM_RADEON=y
CONFIG_AUTOFS4_FS=y
CONFIG_EXT3_FS=y
CONFIG_JBD=y
CONFIG_JBD_DEBUG=y
CONFIG_TMPFS=y
CONFIG_RAMFS=y
CONFIG_ISO9660_FS=y
CONFIG_PROC_FS=y
CONFIG_DEVPTS_FS=y
CONFIG_EXT2_FS=y
CONFIG_NFS_FS=y
CONFIG_NFS_V3=y
CONFIG_NFSD=y
CONFIG_NFSD_V3=y
CONFIG_SUNRPC=y
CONFIG_LOCKD=y
CONFIG_LOCKD_V4=y
CONFIG_EXPORTFS=y
CONFIG_SMB_FS=y
CONFIG_MSDOS_PARTITION=y
CONFIG_SMB_NLS=y
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="iso8859-1"
CONFIG_VGA_CONSOLE=y
CONFIG_DEBUG_KERNEL=y
CONFIG_DEBUG_SLAB=y
CONFIG_DEBUG_IOVIRT=y
CONFIG_MAGIC_SYSRQ=y
CONFIG_DEBUG_SPINLOCK=y
CONFIG_SECURITY_CAPABILITIES=y

--

Linux: the "grim reaper of innocent orphaned children."
          from /usr/src/linux-2.4.18/init/main.c:461

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

2.5.32: DEBUG_SLAB and PREEMPT = constant oops in schedule()

Post by Christian Ehrhard » Fri, 30 Aug 2002 18:10:08


Hi,


> When running the Linux Test Project's process_stress test on 2.5.32 that
> is compiled with DEBUG_SLAB and PREEMPT, I am able to easily reproduce
> this oops. Undef either of the above kernel options, and the system is
> stable again.

This looks like the following is happening:

Task 1 (do_exit)                  Task 2 (parent of task 1, sys_wait4)
set task state to TASK_ZOMBIE
preempt before the final schedule
                                  Picks up exit status and finally calls
                                  release_task this effectivly frees the
                                  task structure of task 1 (while it
                                  is still on the runqueue.

                                  schedule (); tries to switch back to
                                  task 1 -> BOOM

Note that this is not preempt-only, it can happen on plain SMP as
well.

There are other paths that do suspicious things with task struct
accounting, e.g.

sys_sched_setaffinity, lines:
   1571         get_task_struct(p);
   1572         read_unlock(&tasklist_lock);
   1573
   1574         retval = -EPERM;
   1575         if ((current->euid != p->euid) && (current->euid != p->uid) &&
   1576                         !capable(CAP_SYS_NICE))
   1577                 goto out_unlock;
   1578
   1579         retval = 0;
   1580         set_cpus_allowed(p, new_mask);
   1581
   1582 out_unlock:
   1583         put_task_struct(p);

Line 1571 calls get_task_struct because the task might exit during
the syscall. Suppose that this is exactly what happens. This means
that line 1583 will effectivly free the task. But set_cpus_allowed
stuffed a pointer to the task into a request struct without incrementing
the usage count of the task struct.

Quote:> Unable to handle kernel paging request at virtual address 5a5a5ace
> c010ecaf
> *pde = 00000000
> Oops: 0002
> CPU:    0
> EIP:    0060:[<c010ecaf>]    Not tainted
> Using defaults from ksymoops -t elf32-i386 -a i386
> EFLAGS: 00010817
> eax: c9a92000   ebx: c9fd3760   ecx: c9b81288   edx: 5a5a5a5a
> esi: c9b863ec   edi: c9fd31e0   ebp: c9a93ee0   esp: c9a93ec8
> ds: 0068   es: 0068   ss: 0068
> Stack: 7fffffff c9b863ec c9a93f60 c9fd31e0 00000282 c9a92000 c405d000 c0117e10
>        00000008 c9b863ec 080dc050 00000246 c405d980 c9b863ec c01b51e4 c01b5234
>        c405d000 c9b863ec c9d8c5ec c9b8640c c405d980 c405dbfc 00000000 7fffffff
> Call Trace: [<c0117e10>] [<c01b51e4>] [<c01b5234>] [<c010ee2c>] [<c010ee2c>]
>    [<c01b1294>] [<c0132a1b>] [<c0132bf6>] [<c010725f>]
> Code: 0f ba 6a 74 00 8b 42 0c 05 00 00 00 40 0f 22 d8 8b 82 98 00

> >>EIP; c010ecaf <schedule+18f/2a8>   <=====

> Trace; c0117e10 <schedule_timeout+14/a4>
> Trace; c01b51e4 <read_chan+354/74c>
> Trace; c01b5234 <read_chan+3a4/74c>
> Trace; c010ee2c <default_wake_function+0/34>
> Trace; c010ee2c <default_wake_function+0/34>
> Trace; c01b1294 <tty_read+d0/128>
> Trace; c0132a1b <vfs_read+b7/138>
> Trace; c0132bf6 <sys_read+2a/3c>
> Trace; c010725f <syscall_call+7/b>

> Code;  c010ecaf <schedule+18f/2a8>
> 00000000 <_EIP>:
> Code;  c010ecaf <schedule+18f/2a8>   <=====
>    0:   0f ba 6a 74 00            btsl   $0x0,0x74(%edx)   <=====
> Code;  c010ecb4 <schedule+194/2a8>
>    5:   8b 42 0c                  mov    0xc(%edx),%eax
> Code;  c010ecb7 <schedule+197/2a8>
>    8:   05 00 00 00 40            add    $0x40000000,%eax
> Code;  c010ecbc <schedule+19c/2a8>
>    d:   0f 22 d8                  mov    %eax,%cr3
> Code;  c010ecbf <schedule+19f/2a8>
>   10:   8b 82 98 00 00 00         mov    0x98(%edx),%eax

> Linux razor 2.5.32 #5 Wed Aug 28 09:21:53 EST 2002 i686 unknown unknown GNU/Linux

    regards   Christian

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

2.5.32: DEBUG_SLAB and PREEMPT = constant oops in schedule()

Post by Ingo Molna » Sat, 31 Aug 2002 16:10:07



> sys_sched_setaffinity, lines:
>    1571         get_task_struct(p);
>    1572         read_unlock(&tasklist_lock);
> Line 1571 calls get_task_struct because the task might exit during the
> syscall. Suppose that this is exactly what happens. This means that line
> 1583 will effectivly free the task. But set_cpus_allowed stuffed a
> pointer to the task into a request struct without incrementing the usage
> count of the task struct.

note that the scenario you describe is not possible, because
set_cpus_allowed() will wait for the migration thread to do its work - so
the put_task_struct can never come before the last use of the task
structure. See the 'down(&req.sem)' in set_cpus_allowed(), and the
up(&req->sem) in migration_thread().

        Ingo

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

2.5.32: DEBUG_SLAB and PREEMPT = constant oops in schedule()

Post by Christian Ehrhard » Mon, 02 Sep 2002 02:40:05




> > sys_sched_setaffinity, lines:
> >    1571         get_task_struct(p);
> >    1572         read_unlock(&tasklist_lock);

> > Line 1571 calls get_task_struct because the task might exit during the
> > syscall. Suppose that this is exactly what happens. This means that line
> > 1583 will effectivly free the task. But set_cpus_allowed stuffed a
> > pointer to the task into a request struct without incrementing the usage
> > count of the task struct.

> note that the scenario you describe is not possible, because
> set_cpus_allowed() will wait for the migration thread to do its work - so
> the put_task_struct can never come before the last use of the task
> structure. See the 'down(&req.sem)' in set_cpus_allowed(), and the
> up(&req->sem) in migration_thread().

Agreed. Is there anything that prevents the other scenario described
in the original mail, i.e. sys_wait frees the task struct before the
task finally scheduled away? There used to be a lock_kernel in sys_wait
and sys_exit to make sure this doesn't happen but it is gone in 2.5.

    regards  Christian

--
THAT'S ALL FOLKS!
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

1. 2.5.32 oops during reboot

2.5.32 oopsed during shutdown, terminating the reboot process.
The machine wouldn't die this way.

Of course there were no logs, so I wrote down
some numbers and looked them up later:

Unable to handle kernel paging request at d0c65498
eip: c019bb38  tty_register_devfs

call trace:
c0197077 disable_irq_nosync
c011f78d sys_reboot
c014gf40 posix_lock_file
c024be45 devinet_ioctl
c02175c7 sock_destroy_inode

process reboot exited with preempt count 2

UP kernel, with preempt & devfs

Helge Hafting
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

2. FTPD under SuSE 7.2

3. 2.5.32 oops on unmount of ext2 filesystems

4. view certain font in linux

5. prevent oops in xprt_lock_write, against 2.5.32

6. CAN'T ACCESS DEVICE

7. 2.5.27: PREEMPT + DEBUG_SLAB = 100% reproducable oops

8. Anyone holding their breath waiting for LSB?

9. 2.5.30: LTP process_stress causes oops with DEBUG_SLAB and PREEMPT

10. sigio/sigurg cleanup for 2.5.32 (fwd)

11. [2.5.32] CPUfreq Documentation (4/4)

12. software suspend in 2.5.32 test reporter.

13. 2.5.32-smph