Very Computer

Apr 19 20:10:59 stvmm01 unix: [ID 836849 kern.notice]
Apr 19 20:10:59 stvmm01 ^Mpanic[cpu0]/thread=30000f766e0:
Apr 19 20:11:00 stvmm01 unix: [ID 799565 kern.notice] BAD TRAP:
type=10 rp=2a10058ba50 addr=10035648 mmu_fsr=0
Apr 19 20:11:00 stvmm01 unix: [ID 100000 kern.notice]
Apr 19 20:11:00 stvmm01 unix: [ID 839527 kern.notice] mibiisa:
Apr 19 20:11:00 stvmm01 unix: [ID 901337 kern.notice] illegal
instruction fault:
Apr 19 20:11:00 stvmm01 unix: [ID 381800 kern.notice] addr=0x10035648
Apr 19 20:11:00 stvmm01 unix: [ID 101969 kern.notice] pid=324,
pc=0x10035648, sp=0x2a10058b2f1, tstate=0x4400001601, context=0x1080
Apr 19 20:11:00 stvmm01 unix: [ID 743441 kern.notice] g1-g7: 100a93b0,
0, 7f603f, 0, 0, 30b742b9, 30000f766e0
Apr 19 20:11:00 stvmm01 unix: [ID 100000 kern.notice]
Apr 19 20:11:00 stvmm01 genunix: [ID 723222 kern.notice]
000002a10058b780 unix:die+a4 (10, 2a10058ba50, 10035648, 0,
2a10058ba50, 10
41c808)
Apr 19 20:11:00 stvmm01 genunix: [ID 179002 kern.notice]   %l0-3:
00000000100bd64c 0000000000000000 0000000000000040 000000000000000
0
Apr 19 20:11:00 stvmm01   %l4-7: 0000000000000000 0000000000000000
0000000000000000 0000000000000000
Apr 19 20:11:00 stvmm01 genunix: [ID 723222 kern.notice]
000002a10058b860 unix:trap+1220 (20, 0, 0, 10000, 2a10058ba50, 0)
Apr 19 20:11:01 stvmm01 genunix: [ID 179002 kern.notice]   %l0-3:
0000000010441568 0000000010441570 000003000102b538 000000000000000
0
Apr 19 20:11:01 stvmm01   %l4-7: 0000000000000010 0000030001019880
0000000000010000 0000000000000000
Apr 19 20:11:01 stvmm01 genunix: [ID 723222 kern.notice]
000002a10058b9a0 unix:prom_rtt+0 (0, 0, 30000f766e0, fec07e60, 0, 2)
Apr 19 20:11:01 stvmm01 genunix: [ID 179002 kern.notice]   %l0-3:
0000000000000002 0000000000001400 0000004400001601 000000001002b6a
4
Apr 19 20:11:01 stvmm01   %l4-7: 0000000000000000 0000000000000000
0000000000000000 000002a10058ba50
Apr 19 20:11:01 stvmm01 unix: [ID 100000 kern.notice]
Apr 19 20:11:01 stvmm01 genunix: [ID 672855 kern.notice] syncing file
systems...
Apr 19 20:11:01 stvmm01 genunix: [ID 904073 kern.notice]  done
Apr 19 20:11:02 stvmm01 genunix: [ID 353387 kern.notice] dumping to
/dev/dsk/c0t0d0s1, offset 107479040
Apr 19 20:11:06 stvmm01 genunix: [ID 409368 kern.notice] ^M100% done:
4386 pages dumped, compression ratio 3.48,
Apr 19 20:11:06 stvmm01 genunix: [ID 851671 kern.notice] dump
succeeded
Apr 20 14:37:07 stvmm01 genunix: [ID 540533 kern.notice] ^MSunOS
Release 5.8 Version Generic_108528-24 64-bit
Apr 20 14:37:07 stvmm01 genunix: [ID 913632 kern.notice] Copyright
1983-2003 Sun Microsystems, Inc.  All rights reserved.
Apr 20 14:37:07 stvmm01 genunix: [ID 678236 kern.info] Ethernet
address = 8:0:20:f9:47:26
Apr 20 14:37:07 stvmm01 unix: [ID 389951 kern.info] mem = 262144K
(0x10000000)
Apr 20 14:37:07 stvmm01 unix: [ID 930857 kern.info] avail mem =
251371520
Apr 20 14:37:07 stvmm01 rootnex: [ID 466748 kern.info] root nexus =
Sun Ultra 5/10 UPA/PCI (UltraSPARC-IIi 440MHz

Is it a 400Mhz by any chance? If so, its a known problem with the
external CPU cache.

More info here:
http://aa11.cjb.net/sun_managers/2000/09/msg00109.html

If you don't need snmp services (which mibiisa belongs too) I'd
stop them by running "/etc/init.d/init.snmpdx stop" (you can
make this permanent by renaming /etc/rc3.d/S76snmpdx to e.g.
/etc/rc3.d/s76snmpdx).

mp.
--
Systems Administrator | Institute for Software Science | Univ. of Vienna

Quote:> Apr 19 20:11:00 stvmm01 unix: [ID 100000 kern.notice]
> Apr 19 20:11:00 stvmm01 unix: [ID 839527 kern.notice] mibiisa:
> Apr 19 20:11:00 stvmm01 unix: [ID 901337 kern.notice] illegal
> instruction fault:

If other failures all show mibiisa being affected then this would
appear to be some software failure that could be induced by
some behaviour in mibiisa.  This is unlikely at the best of times,
but particularly so for bad instructions (on the odd occasion
that there have been bugs like that it's not been a case of the
kernel being induced into executing bad instructions - the instructions
the kernel executes are fixed and not influenced by userland behaviour).

Quote:> Apr 19 20:11:00 stvmm01 unix: [ID 381800 kern.notice] addr=0x10035648
> Apr 19 20:11:00 stvmm01 unix: [ID 101969 kern.notice] pid=324,
> pc=0x10035648, sp=0x2a10058b2f1, tstate=0x4400001601, context=0x1080

 > 0x10035648/i
sun4`flush_user_windows_to_stack+0x1c:
                 call      -0x2e69c      <sun4`flush_user_windows>

The tstate value (some preserved processor state at the point we trapped)
indicates that we were privileged - so mibiisa had already trapped
into the kernel (I can't tell what for from the info we have;
the crash dump would likely tell us).

But the point is that this is a kernel instruction that has been corrupted.
flush_user_windows_to_stack is called pretty often for all sorts of
reasons, so we can be pretty certain that this is not the first time
we've executed the instruction.  It could have been corrupted in
all sorts of ways, eg bad memory, ecache error (we'd have expected
an error report for those - maybe one is buried in the crash dump), fried cpu,
kernel memory corruption by a wayward driver (usually not easy
to corrupt instructions) ...

A look at the other failures you've had could probably help decide.
eg, if it's always mibiisa, always the same %pc, always an
illegal instruction trap etc.  If the symptoms are pretty random
(eg, different trap types at different addresses) then hardware
would shoot right up the suspect list (and it's quite high already).

Gavin

Quote:> If you don't need snmp services (which mibiisa belongs too) I'd
> stop them by running "/etc/init.d/init.snmpdx stop" (you can
> make this permanent by renaming /etc/rc3.d/S76snmpdx to e.g.
> /etc/rc3.d/s76snmpdx).

Gavin

Quote:> although we have had Ultra 10's virtually
> rebuilt and still having this type of problem

Quote:> Is it a 400Mhz by any chance? If so, its a known problem with the
> external CPU cache.
> More info here:
> http://aa11.cjb.net/sun_managers/2000/09/msg00109.html

I would defintely not class the observed failures as anything
to do with ecache (unless there are UE/EDP/LDP etc messages
that the OP has not mentioned).

Gavin

----

Bye,
Roland

--
  __ .  . __

  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 2426 901568 FAX +49 2426 901569
 (;O/ \/ \O;)

Quote:> The original poster mentioned no async error reports such
> as UE errors (as mentioned in the link).  The issue
> mentioned there is that a parity error detected in the
> external cache will panic the system.  These were more
> likely on the faster cpus with bigger ecaches, but this
> is an ultra10 and the ecache on those was pretty small
> (I think 512K or similar vs 8MB).

Newer UltraSPARC-IIi versions have smaller-but-onchip 2ndLevel caches.

----

Bye,
Roland

--
  __ .  . __

  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 2426 901568 FAX +49 2426 901569
 (;O/ \/ \O;)