On a quest for knowledge about APIC...no, really!

On a quest for knowledge about APIC...no, really!

Post by Brent Busb » Sat, 21 Jun 2003 23:54:08



I've even read the kernel source trying to find an answer.
And this is the second time I've asked on Usenet.

My question is about APIC.  In the kernel LINT file, there are three
options listed together in the same section: SMP, APIC_IO, and HTT.  SMP
is obvious.  The other two are for APIC and HyperThreading.  The
Handbook tells you you need SMP and APIC_IO for multiprocessor:

======================================================================
# To make an SMP kernel, the next two are needed                            
    #options        SMP              # Symmetric MultiProcessor Kernel
    #options        APIC_IO          # Symmetric (APIC) I/O              

   The above are both required for SMP support.
======================================================================

And that's it.  Problem is, have you taken a look at the single
processor motherboards that are coming out on the market recently?  I'm
talking about SINGLE processor motherboards.  They have APIC.  One of
the nice side-effects of APIC, aside from allowing you to implement more
than one CPU, is that it gives you extra IRQ's, usually increasing the
total from the usual 0-15 to a very nice 0-23, though in theory it could
give even more than that if they wanted it to.  I've heard people talk
in hardware forums about prospects of having as many as 128 IRQ's.

Current motherboards are wisely taking advantage of this even when
there's only a single processor.  It also has the advantage of providing
a slight performance benefit, since the APIC chip is now providing the
bus interrupts on its own with no need for work from the CPU(s).  It's
gotten so prevalent that I haven't been able to find a recent
motherboard that doesn't do this, even amongst single-processor ones.  
Asus seems to have taken recently to a habit of making their on-board
integrated stuff (IDE, Ethernet, and so on) hardwired to particular IRQ
numbers, and has forsaken the practice of giving you switches in the
BIOS setup to turn stuff like that off.  Why?  Because since you're
obviously using APIC (!!!), you're up to your knees in available IRQ's!

Here's where my question comes in for FreeBSD:  **ARE** we using APIC?  
No one ever talks about this.  From digging through apic.h, apic_ipl.h,
and apic_vector.s in the kernel source tree, I've determined that
FreeBSD does indeed gain an understanding of 24 interrupts when APIC_IO
is being used.  That answers my own question, right?  Not quite.  Nobody
ever talks about whether you can use this option by itself.

In the Handbook, it says you enable it along with SMP in a
multiprocessor system.  On Usenet, they always say enable it if you have
two processors.  What if you have a single processor board that uses
APIC?  This might seem an unimportant question if you've got a system
that's over a year or two old, but it's going to get awfully important
here real quick, as it seems APIC has already become a norm for one CPU.

So...are we supposed to be turning this on by itself (e.g., APIC_IO but
*no* SMP)?  Is that safe?  And if so, considering the way motherboards
are headed, should this be in the GENERIC kernel in the future?

Thanks in advance.  I've searched the whole Internet on this, and the
only information I've come up with regards Linux's APIC support.  Almost
no one seems to know about APIC_IO on FreeBSD.

--
|-+{ Brent A. Busby  }  +-| "I've been on the Internet a long time, and I'm   |

|-+  Normal,  IL  (USA) |-|  like Rambo.   I'm tough like a freezer-burned    |
|-*  FreeBSD/i386 4.8   *-|  turkey that's been oven-reheated over and over." |

 
 
 

On a quest for knowledge about APIC...no, really!

Post by Bill Vermilli » Sun, 22 Jun 2003 01:54:54




Quote:>I've even read the kernel source trying to find an answer.
>And this is the second time I've asked on Usenet.
>My question is about APIC. In the kernel LINT file, there
>are three options listed together in the same section: SMP,
>APIC_IO, and HTT. SMP is obvious. The other two are for APIC and
>HyperThreading. The Handbook tells you you need SMP and APIC_IO
>for multiprocessor:

....

Quote:>And that's it. Problem is, have you taken a look at the single
>processor motherboards that are coming out on the market
>recently? I'm talking about SINGLE processor motherboards. They
>have APIC.

You'd probably have to talk to a board designed but there is a
possbility that they just put the APIC chips on all boards for
economy of scale and inventory.   Perhaps there is information
on using it in single CPU systems on the docs for APIC

I see info on it at intel under their
http://www.intel.com/design/chipsets/datashts/290566.htm page.
There is a link there to updates also.

Since I'm currently on a text only machine I can't look at the PDF
there or it's update.
--


 
 
 

On a quest for knowledge about APIC...no, really!

Post by Brent Busb » Sun, 22 Jun 2003 03:25:52


: You'd probably have to talk to a board designed but there is a
: possbility that they just put the APIC chips on all boards for
: economy of scale and inventory.

I'm sure that helped the decision, but people in the Windows and Linux
world are actually using it to get IRQ's with numbers as high as 23.  It
does this in the BIOS without even asking you if you want to.  The
question is, what does that mean for a FreeBSD user?  The apic.h and
other files in the kernel source actually seem to understand it, judging
from this code snippet in apic_ipl.h:

#define NHWI            24              /* number of h/w interrupts */
#define HWI_MASK        0x00ffffff      /* bits for h/w interrupts */

But does that mean we can just enable APIC_IO without SMP to use it?  
And what does that mean for users doing new installs of FreeBSD?  Are
they now in the same boat as 386 users (as if there were any ;-), having
now to compile a custom kernel with APIC_IO on another machine because
the GENERIC kernel won't work with their interrupts in a kosher way out
of the box?

Of course, the boards I've used that have APIC have an option in the
BIOS setup to turn it off, but this would seem a waste.  It gives you
extra interrupts and speeds up your machine.  (I don't want to turn it
off, do I?)  I'm also not positive that all machines have this option to
turn it off; just ones I've seen do.

: Perhaps there is information
: on using it in single CPU systems on the docs for APIC

I haven't seen any FreeBSD docs for APIC, though another code snippet I
ran into in the kernel source made mention of some without telling where
it is.  (That didn't help much...)  <g>

/*
 * Handle "spurious INTerrupts".
 * Notes:
 *  This is different than the "spurious INTerrupt" generated by an
 *   8259 PIC for missing INTs.  See the APIC documentation for details.
 *  This routine should NOT do an 'EOI' cycle.
 */

*What* APIC documentation?!

--
|-+{ Brent A. Busby  }  +-| "I've been on the Internet a long time, and I'm   |

|-+  Normal,  IL  (USA) |-|  like Rambo.   I'm tough like a freezer-burned    |
|-*  FreeBSD/i386 4.8   *-|  turkey that's been oven-reheated over and over." |

 
 
 

On a quest for knowledge about APIC...no, really!

Post by Erik Nygre » Sun, 22 Jun 2003 04:26:33


...

Quote:> So...are we supposed to be turning this on by itself (e.g., APIC_IO but
> *no* SMP)?  Is that safe?  And if so, considering the way motherboards
> are headed, should this be in the GENERIC kernel in the future?

It's in 5.1-RELEASE already. Don't know about 4.x

--
Erik Nygren
e r i k { a t } s w i p { d o t } n e t
Linux - If you hate Microsoft, FreeBSD - If you love Unix

 
 
 

On a quest for knowledge about APIC...no, really!

Post by Michel Tal » Mon, 23 Jun 2003 19:51:32




> : You'd probably have to talk to a board designed but there is a
> : possbility that they just put the APIC chips on all boards for
> : economy of scale and inventory.

> I'm sure that helped the decision, but people in the Windows and Linux
> world are actually using it to get IRQ's with numbers as high as 23.  It
> does this in the BIOS without even asking you if you want to.  The
> question is, what does that mean for a FreeBSD user?  The apic.h and
> other files in the kernel source actually seem to understand it, judging
> from this code snippet in apic_ipl.h:

By the way, why not try to experiment with that? Try to compile a kernel
with IO_APIC without SMP and see if
- config accepts that and it compiles
- in this case if it boots and doesn't panic
- finally if interrupts work as you expect with this kernel. This would
answer most effectively your question :-)

--

Michel TALON

 
 
 

On a quest for knowledge about APIC...no, really!

Post by Ivan Vora » Tue, 24 Jun 2003 22:42:06



> By the way, why not try to experiment with that? Try to compile a
> kernel with IO_APIC without SMP and see if
> - config accepts that and it compiles

it does not:

cc -c -x
assembler-with-cpp -DLOCORE -O -pipe -march=pentium3 -Wall -Wredundant-decls
 -Wnested-externs -Wstrict-prototypes  -Wmissing-prototypes -Wpointer-arith
-Winline -Wcast-qual  -fformat-extensions -std=c99  -nostdinc -I-  -I. -I../
../.. -I../../../dev -I../../../contrib/dev/acpica -I../../../contrib/ipfilt
er -D_KERNEL -include
opt_global.h -fno-common  -mno-align-long-strings -mpreferred-stack-boundary
=2 -ffreestanding -Werror ../../../i386/i386/exception.s
{standard input}: Assembler messages:
{standard input}:384: Error: undefined symbol `lapic' in operation
{standard input}:384: Error: undefined symbol `LA_EOI' in operation

(config passes, make depend passes, make kernel fails with above error).

--
--
Ashes to ashes, DOS to DOS

 
 
 

On a quest for knowledge about APIC...no, really!

Post by Michel Tal » Tue, 24 Jun 2003 22:57:04




>> By the way, why not try to experiment with that? Try to compile a
>> kernel with IO_APIC without SMP and see if
>> - config accepts that and it compiles

> it does not:

> cc -c -x
> assembler-with-cpp -DLOCORE -O -pipe -march=pentium3 -Wall -Wredundant-decls
> -Wnested-externs -Wstrict-prototypes  -Wmissing-prototypes -Wpointer-arith
> -Winline -Wcast-qual  -fformat-extensions -std=c99  -nostdinc -I-  -I. -I../
> ../.. -I../../../dev -I../../../contrib/dev/acpica -I../../../contrib/ipfilt
> er -D_KERNEL -include
> opt_global.h -fno-common  -mno-align-long-strings -mpreferred-stack-boundary
> =2 -ffreestanding -Werror ../../../i386/i386/exception.s
> {standard input}: Assembler messages:
> {standard input}:384: Error: undefined symbol `lapic' in operation
> {standard input}:384: Error: undefined symbol `LA_EOI' in operation

> (config passes, make depend passes, make kernel fails with above error).

So i suppose this answers the question :-)

--

Michel TALON

 
 
 

On a quest for knowledge about APIC...no, really!

Post by Brent Busb » Wed, 25 Jun 2003 02:59:54



:>
:>> By the way, why not try to experiment with that? Try to compile a
:>> kernel with IO_APIC without SMP and see if
:>> - config accepts that and it compiles
:>
:> it does not:
:>
:> cc -c -x
:> assembler-with-cpp -DLOCORE -O -pipe -march=pentium3 -Wall -Wredundant-decls
:> -Wnested-externs -Wstrict-prototypes  -Wmissing-prototypes -Wpointer-arith
:> -Winline -Wcast-qual  -fformat-extensions -std=c99  -nostdinc -I-  -I. -I../
:> ../.. -I../../../dev -I../../../contrib/dev/acpica -I../../../contrib/ipfilt
:> er -D_KERNEL -include
:> opt_global.h -fno-common  -mno-align-long-strings -mpreferred-stack-boundary
:> =2 -ffreestanding -Werror ../../../i386/i386/exception.s
:> {standard input}: Assembler messages:
:> {standard input}:384: Error: undefined symbol `lapic' in operation
:> {standard input}:384: Error: undefined symbol `LA_EOI' in operation
:>
:>
:> (config passes, make depend passes, make kernel fails with above error).
:>
:
: So i suppose this answers the question :-)

Yes, certainly does.  Thank you, Mr. Voras, by the way for the
information.  I was going to try that, but you beat me to it.

So that means for right now, single-CPU machine with APIC (which seems
to be just about all the new ones) will need to have APIC disabled in
the BIOS if possible, and run in regular PIC mode.

The thing that disturbs me is that with all the recent assumptions that
you're using it that manufacturers seem to be making (e.g., the way they
no longer let you turn off things like IDE because they assume you're
loaded with free IRQ's), assuming that nobody uses APIC on FreeBSD with
single-CPU may become increasingly difficult as time goes on.  I'm not
even sure all boards have a BIOS option to turn it off.  Oh well, mine
does at least, so I can just run in PIC mode with the regular 16 IRQ's.

--
|-+{ Brent A. Busby  }  +-| "I've been on the Internet a long time, and I'm   |

|-+  Normal,  IL  (USA) |-|  like Rambo.   I'm tough like a freezer-burned    |
|-*  FreeBSD/i386 4.8   *-|  turkey that's been oven-reheated over and over." |

 
 
 

On a quest for knowledge about APIC...no, really!

Post by Reko Turj » Wed, 25 Jun 2003 03:05:04




> > By the way, why not try to experiment with that? Try to compile a
> > kernel with IO_APIC without SMP and see if
> > - config accepts that and it compiles

> it does not:

--snip

How about compiling a SMP kernel with APIC support? At least WinNT  runs
with single processor with SMP kernel quite happily since NT4, so I'd guess
FBSD could do the same.

-Reko

 
 
 

On a quest for knowledge about APIC...no, really!

Post by Michel Tal » Wed, 25 Jun 2003 17:13:48






>> > By the way, why not try to experiment with that? Try to compile a
>> > kernel with IO_APIC without SMP and see if
>> > - config accepts that and it compiles

>> it does not:

> --snip

> How about compiling a SMP kernel with APIC support? At least WinNT  runs
> with single processor with SMP kernel quite happily since NT4, so I'd guess
> FBSD could do the same.

It may:
-pure and simple not work.
-even if it works it may have disastrous effects on performance, since
on SMP machine code path is more complicated to deal with locking
issues.

Quote:

> -Reko

--

Michel TALON

 
 
 

On a quest for knowledge about APIC...no, really!

Post by Michel Tal » Wed, 25 Jun 2003 17:11:24



> The thing that disturbs me is that with all the recent assumptions that
> you're using it that manufacturers seem to be making (e.g., the way they
> no longer let you turn off things like IDE because they assume you're
> loaded with free IRQ's), assuming that nobody uses APIC on FreeBSD with
> single-CPU may become increasingly difficult as time goes on.  I'm not
> even sure all boards have a BIOS option to turn it off.  Oh well, mine
> does at least, so I can just run in PIC mode with the regular 16 IRQ's.

First most of the additional cards you put in the box are PCI cards and
happy sharing interrupts. I haven't seen any interrupt sharing problems
since at least 2 or 3 years here. Second, if it appears that FreeBSD
does not work with new mobos, you may be sure that developers will hurry
implementing APIC_IO for monoprocs :-)

--

Michel TALON

 
 
 

On a quest for knowledge about APIC...no, really!

Post by Thierry Herbelo » Thu, 26 Jun 2003 04:16:14



> How about compiling a SMP kernel with APIC support? At least WinNT  runs
> with single processor with SMP kernel quite happily since NT4, so I'd
> guess FBSD could do the same.

how about getting down to the code ? at least, FreeBSD is delivered with all
its source code and the makefiles to compile enverything in one command !

please, stop whining and UTSL !
(find /usr/src/sys -type f -a -exec vi {} \;) for a start

        TfH

Quote:

> -Reko

 
 
 

On a quest for knowledge about APIC...no, really!

Post by Brent Busb » Thu, 26 Jun 2003 08:04:52


: First most of the additional cards you put in the box are PCI cards and
: happy sharing interrupts. I haven't seen any interrupt sharing problems
: since at least 2 or 3 years here.

That's actually something I wasn't sure about, whether FreeBSD could
handle regular PCI interrupt sharing.  I've always tried really hard to
get every device on a unique IRQ just because I wasn't sure about that.

--
|-+{ Brent A. Busby  }  +-| "I've been on the Internet a long time, and I'm   |

|-+  Normal,  IL  (USA) |-|  like Rambo.   I'm tough like a freezer-burned    |
|-*  FreeBSD/i386 4.8   *-|  turkey that's been oven-reheated over and over." |

 
 
 

On a quest for knowledge about APIC...no, really!

Post by Michel Tal » Thu, 26 Jun 2003 17:29:31




> : First most of the additional cards you put in the box are PCI cards and
> : happy sharing interrupts. I haven't seen any interrupt sharing problems
> : since at least 2 or 3 years here.

> That's actually something I wasn't sure about, whether FreeBSD could
> handle regular PCI interrupt sharing.  I've always tried really hard to
> get every device on a unique IRQ just because I wasn't sure about that.

Yes, no problem, i have almost all my cards and mobo chips
on the same IRQ without any problem.

--

Michel TALON

 
 
 

1. i810 I/O APIC (was Sticky IO-APIC problem)

D'oh!  Well, I still haven't gotten the APIC to work... after *several* unclean reboots (and repeatedly checking my hex math), I'm back where I started.  After tinkering inside the configuration space of the I/O Controller Hub, no change is evident in /proc/interrupts.  The next time I start up XFree86, it freezes at a blank screen.  No vterm switches, Magic SysRq keys, or three-fingered salutes rescue me from it; the kernel doesn't even sing a verse of "Aiee, Killing My Interrupt Handler Softly".  Nothin' + bupkus.  Nothing.  Well, it's obviously not going to work from user space. 8-(  Any thoughts from the gurus out there?

     -- Colin

On the first day, man created the computer.  On the second day, God proclaimed from the heavens, "F0 0F C7 C8".

------------------------------------------------------------
The CompNerd Network: http://www.compnerd.com/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

2. Why is my kernel so HUGE?

3. IO-APIC says: unexpected IO-APIC was found

4. stuck with MOD3 on Linux NEED HELP PLEASE

5. DMI/APIC updates 4 of 4: dmi-apic-fixups

6. How to redirect X window outputs?

7. APIC error on CPU0 (UP APIC kernel)

8. CATIA CAD-System on rs6000/aix

9. Switch APIC to driver model (and make S3 sleep with APIC on)

10. Knoppix is *really* *really* cool!

11. Really, really slow connection to local server via PPP.

12. Not Available these Signal nos on Linux