PCI init issues

PCI init issues

Post by jama » Mon, 03 Mar 2003 19:50:13



Seems like Linux 2.4.x has PCI initialization issues when you start
having funky PCI bus setups. Now before you point a finger at
the BIOs please read on...
In other words Linux needs a lot of help from the bios and if it doesnt
get it things get messed up. When the bios initializes _all_ the mp table
or acpi stuff things work great.
For example Linux doesnt seem to be very smart about things like second
level pci bridges...

Example:
------

-[00]-+-00.0  Intel Corp. e7500 DRAM Controller
      +-00.1  Intel Corp. e7500 DRAM Controller Error Reporting
      +-02.0-[01-04]--+-1c.0  Intel Corp. 82870P2 P64H2 I/OxAPIC
      |               +-1d.0-[02]--
      |               +-1e.0  Intel Corp. 82870P2 P64H2 I/OxAPIC
      |               \-1f.0-[03-04]----01.0-[04]--+-04.0  Digital
Equipment Corporation DECchip 21142/43
      |                                            +-05.0  Digital
Equipment Corporation DECchip 21142/43
      |                                            +-06.0  Digital
Equipment Corporation DECchip 21142/43
      |                                            \-07.0  Digital
Equipment Corporation DECchip 21142/43
      +-03.0-[05-07]--+-1c.0  Intel Corp. 82870P2 P64H2 I/OxAPIC
      |               +-1d.0-[06]--+-01.0  Intel Corp.: Unknown device
100f
      |               |            +-02.0  Intel Corp.: Unknown device
1010
      |               |            \-02.1  Intel Corp.: Unknown device
1010
      |               +-1e.0  Intel Corp. 82870P2 P64H2 I/OxAPIC
      |               \-1f.0-[07]--
      +-1d.0  Intel Corp. 82801CA/CAM USB (Hub
      +-1d.1  Intel Corp. 82801CA/CAM USB (Hub
      +-1d.2  Intel Corp. 82801CA/CAM USB (Hub
      +-1e.0-[08]--+-01.0  ATI Technologies Inc Rage XL
      |            +-02.0  Intel Corp. 82557/8/9 [Ethernet Pro 100]
      |            \-03.0  Intel Corp. 82557/8/9 [Ethernet Pro 100]
      +-1f.0  Intel Corp. 82801CA ISA Bridge (LPC)
      +-1f.1  Intel Corp. 82801CA IDE U100
      \-1f.3  Intel Corp. 82801CA/CAM SMBus
-------------

See those DECchip 21142/43 devices? They are about 3 layers down
from the root and theyll never work properly unless the BIOS is capable
of initializing everything. The first of the 4 devices will receive
interupts correctly, the rest dont.

-------

           CPU0       CPU1       CPU2       CPU3
  0:     441198     441826     442361     439749    IO-APIC-edge  timer
  1:          1          0          0          1    IO-APIC-edge  keyboard
  2:          0          0          0          0          XT-PIC  cascade
  4:         75         78         82         52    IO-APIC-edge  serial
  8:          1          0          0          0    IO-APIC-edge  rtc
 14:       2000       1787       1960       1535    IO-APIC-edge  ide0
 15:       1860       1750       2021       1603    IO-APIC-edge  ide1
 17:       3396       3440       3400       3415   IO-APIC-level  eth0
 18:          6          2          2          3   IO-APIC-level  eth1
 24:          5          3          5          3   IO-APIC-level  eth5,
eth6, eth7, eth8
 96:        426        424        425        424   IO-APIC-level  eth2
100:        427        424        424        425   IO-APIC-level  eth3
101:        425        424        425        423   IO-APIC-level  eth4
NMI:          0          0          0          0
LOC:    1762876    1762850    1762971    1762970
ERR:          0
MIS:          0
--------------

Interupt routing as can be seen above is really messed for that device.

The workaround is to move the board so the the DECchip 21142/43 devices
are at a depth not to exceed 3.

Seems this has been an ongoing problem on Linux.
http://www.tux.org/hypermail/linux-tulip/2002-Aug/0000.html

The above is the same motherboard as mentioned in the thread above except
the two processors have hyperthreading capability.

Q1: How do we resolve this other than moving around boards?
Q2: Should we really be dependent on bios this bad?
According to that thread things dont work on the BSDs either but work fine
on WinXP (shudder). Does this mean winxp doesnt depend on BIOS?
Should we really be dependent on the BIOS this much?
Shouldnt things like second level pci bridge be part of Linux discovery?

cheers,
jamal
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

PCI init issues

Post by Ivan Kokshaysk » Tue, 04 Mar 2003 14:20:11



> Interupt routing as can be seen above is really messed for that device.

Indeed. Quad tulip cards usually have irq pin A of each chip routed
to pins A-D on the connector, so they cannot have the same irq unless
all four irq pins are shared in the parent slot, which is unlikely in
this case.

Quote:> Q1: How do we resolve this other than moving around boards?

Here's [completely untested] patch that attempts to validate BIOS irq
assignments. This may break "good" irqs for other devices, but it's
interesting to see if it changes something.

Quote:> Q2: Should we really be dependent on bios this bad?

We certainly shouldn't wrt PCI IO and MMIO allocations, but irq routing
appears to be so horribly overcomplexed on x86 that dependence on
BIOS seems almost unavoidable here...
Other architectures don't have such kind of problems, BTW.

Ivan.

--- 2.4/arch/i386/kernel/pci-irq.c      Fri Feb 28 16:46:28 2003

                 */
                if (io_apic_assign_pci_irqs)
                {
-                       int irq;
+                       int irq, irq1 = -1;

                        if (pin) {

         * parent slot, and pin number. The SMP code detects such bridged
         * busses itself so we should get into this branch reliably.
         */
-                               if (irq < 0 && dev->bus->parent) { /* go back to the bridge */
+                               if (dev->bus->parent) { /* go back to the bridge */
                                        struct pci_dev * bridge = dev->bus->self;

                                        pin = (pin + PCI_SLOT(dev->devfn)) % 4;
-                                       irq = IO_APIC_get_PCI_irq_vector(bridge->bus->number,
+                                       irq1 = IO_APIC_get_PCI_irq_vector(bridge->bus->number,
                                                        PCI_SLOT(bridge->devfn), pin);
-                                       if (irq >= 0)
-                                               printk(KERN_WARNING "PCI: using PPB(B%d,I%d,P%d) to get irq %d\n",
-                                                       bridge->bus->number, PCI_SLOT(bridge->devfn), pin, irq);
+                               }
+                               if (irq1 >= 0 && irq1 != irq) {
+                                       /* IRQ listed in MP-table doesn't match one for parent slot.
+                                          Use that we've found instead. */
+                                       irq = irq1;
+                                       printk(KERN_WARNING "PCI: using PPB(B%d,I%d,P%d) to get irq %d\n",
+                                               bridge->bus->number, PCI_SLOT(bridge->devfn), pin, irq);
                                }
                                if (irq >= 0) {
                                        printk(KERN_INFO "PCI->APIC IRQ transform: (B%d,I%d,P%d) -> %d\n",
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

PCI init issues

Post by David S. Mille » Tue, 04 Mar 2003 17:20:24



> > Q2: Should we really be dependent on bios this bad?

> We certainly shouldn't wrt PCI IO and MMIO allocations, but irq routing
> appears to be so horribly overcomplexed on x86 that dependence on
> BIOS seems almost unavoidable here...
> Other architectures don't have such kind of problems, BTW.

Actually, for issues like that one Jamal is pointing out, I believe
there is something we can do.

What varies so much from motherboard to motherboard on x86 is how
the interrupts are wired up.  So for example, this tells us that
PCI slot X maps to x86 IRQ Y.  If we know that, we can figure out
where his deeply bridged tulips will send IRQs.

If the mp tables don't give exactly this kind of information,
then Ivan is right :(

Anyone know if FreeBSD fares better in situations like this?
What do they do if so?

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

PCI init issues

Post by jama » Wed, 05 Mar 2003 04:00:05


Ivan,

Patch was applied; no luck at all. Below is dmesg output with
debug turned on in the pci-irq.c file.

--------
Mar  3 18:01:39 localhost kernel: PCI: PCI BIOS revision 2.10 entry at
0xfd875last bus=8
Mar  3 18:01:39 localhost kernel: PCI: Using configuration type 1
Mar  3 18:01:39 localhost kernel: PCI: Probing PCI hardware
Mar  3 18:01:39 localhost kernel: PCI: Probing PCI hardware
Mar  3 18:01:39 localhost kernel: Transparent bridge - Intel Corp.
82801BA/CA/ PCI Bridge
Mar  3 18:01:39 localhost kernel: PCI: Discovered primary peer bus 10
[IRQ]
Mar  3 18:01:39 localhost kernel: PCI: Discovered primary peer bus 11
[IRQ]
Mar  3 18:01:39 localhost kernel: PCI: Discovered primary peer bus 12
[IRQ]
Mar  3 18:01:39 localhost kernel: PCI: Using IRQ router PIIX [8086/2480]
at 00f.0
Mar  3 18:01:39 localhost kernel: PCI: IRQ fixup
Mar  3 18:01:39 localhost kernel: PCI->APIC IRQ transform: (B0,I29,P0) ->
16
Mar  3 18:01:39 localhost kernel: PCI->APIC IRQ transform: (B0,I29,P1) ->
19
Mar  3 18:01:39 localhost kernel: PCI->APIC IRQ transform: (B0,I29,P2) ->
18
Mar  3 18:01:39 localhost kernel: PCI->APIC IRQ transform: (B2,I1,P1) ->
48
Mar  3 18:01:39 localhost kernel: PCI->APIC IRQ transform: (B3,I1,P1) ->
24
Mar  3 18:01:39 localhost kernel: PCI->APIC IRQ transform: (B3,I1,P2) ->
25
Mar  3 18:01:39 localhost kernel: PCI->APIC IRQ transform: (B6,I4,P0) ->
96
Mar  3 18:01:39 localhost kernel: PCI->APIC IRQ transform: (B6,I5,P1) ->
96
Mar  3 18:01:39 localhost kernel: PCI->APIC IRQ transform: (B6,I6,P2) ->
96
Mar  3 18:01:39 localhost kernel: PCI->APIC IRQ transform: (B6,I7,P3) ->
96
Mar  3 18:01:39 localhost kernel: PCI->APIC IRQ transform: (B8,I1,P1) ->
16
Mar  3 18:01:39 localhost kernel: PCI->APIC IRQ transform: (B8,I2,P2) ->
17
Mar  3 18:01:39 localhost kernel: PCI->APIC IRQ transform: (B8,I3,P3) ->
18
---------------

BIOS irq assignments i suppose are validated.

I have a feeling the reason it works in windows is because of a
functional ACPI.

What next? ;->

cheers,
jamal
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

PCI init issues

Post by jama » Wed, 05 Mar 2003 04:00:10



> Anyone know if FreeBSD fares better in situations like this?

All BSDs apparently have this problem. Ive heard rumors of windows XP
working just fine with the same setup on these mboards.
Unfortunately up north we have igloos so i cant verify these rumors ;->

cheers,
jamal
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

PCI init issues

Post by Ivan Kokshaysk » Wed, 05 Mar 2003 17:40:08



> Mar  3 18:01:39 localhost kernel: PCI->APIC IRQ transform: (B6,I4,P0) ->
> 96
> Mar  3 18:01:39 localhost kernel: PCI->APIC IRQ transform: (B6,I5,P1) ->
> 96
> Mar  3 18:01:39 localhost kernel: PCI->APIC IRQ transform: (B6,I6,P2) ->
> 96
> Mar  3 18:01:39 localhost kernel: PCI->APIC IRQ transform: (B6,I7,P3) ->
> 96
...
> BIOS irq assignments i suppose are validated.

No, this means that mp tables are broken in a way I didn't expect...
Also, it's unclear whether or not the IOxAPIC routing is broken as well.
Probably full dmesg output (especially things related to IO_APICs setup
and irq<->pin mapping) could shed some light on this.

Quote:> I have a feeling the reason it works in windows is because of a
> functional ACPI.

Perhaps.

Ivan.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

PCI init issues

Post by Linus Torvald » Wed, 05 Mar 2003 18:50:12



Quote:

>    Here is the full messages output for a system with this problem.   I had
> only provided part of the
> messages file to him earlier.   This is the entire output.

Well, the interesting parts are missing, because your klogd output has
been truncated due to the huge output from the md layer etc. However, I
suspect strongly that the issue is in IO_APIC_get_PCI_irq_vector() in
arch/i386/kernel/io_apic.c, and I would further guess that it's the fact
that your MP tables don't have the full mapping or we're somehow screwing
it up.

Can you add a printk() to the case where we return "best_guess" at the end
of that function? We have that fuzzy match thing where if we cannot find
the exact entry in the MP table we instead use the first entry that
matches in everything but the <pin> number, and it would be interesting to
hear if that's the logic that triggers.

It's also quite possible that your MP tables just aren't right, and the
reason XP works on the machine is that XP uses the ACPI irq lookup stuff.  
You could try enabling ACPI bootup support (I've never used it on 2.4.x
myself, but it's required for at least one of my machines and works fine
on 2.5.x).

                Linus

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

PCI init issues

Post by Donald Becke » Wed, 05 Mar 2003 20:30:15




> > Interupt routing as can be seen above is really messed for that device.

> Indeed. Quad tulip cards usually have irq pin A of each chip routed
> to pins A-D on the connector, so they cannot have the same irq unless

Incorrect.
Most quad Tulip boards have the bus bridge wired so that all interrupts
are sent on the INTA output of the board.

The tulip.c driver has worked around the BIOS bug on x86 machines for
years.  It should really be handled in the generic PCI quirks code, but
has not yet been correctly handled in the kernel's code.

--

Scyld Computing Corporation             http://www.scyld.com
410 Severn Ave. Suite 210               Scyld Beowulf cluster system
Annapolis MD 21403                      410-990-9993

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

PCI init issues

Post by Ivan Kokshaysk » Wed, 05 Mar 2003 20:30:17



> It may not be complete, so I also included part of /var/log/messages.

It's complete, thanks.

Quote:> IO APIC #4......
...
> .... IRQ redirection table:
>  NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect:  
>  00 00F 0F  1    1    0   1   0    1    1    C1
>  01 000 00  1    0    0   0   0    0    0    00
>  02 000 00  1    0    0   0   0    0    0    00
>  03 000 00  1    0    0   0   0    0    0    00
...
> IRQ to pin mappings:
...
> IRQ48 -> 2:0

Indeed, looks like only pin 0 (INT_A of that card) is connected. :-(

Ivan.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

PCI init issues

Post by Linus Torvald » Wed, 05 Mar 2003 21:00:24



> Indeed, looks like only pin 0 (INT_A of that card) is connected. :-(

Well, I'd say it looks like the MP table _claims_ that only pin0 is
connected. Remember: the claim was that this machine worked on WinXP.

So there are at least two potential reasons for that:

 - The MP table is simply wrong, and WinXP gets the routing information
   from somewhere else (ie most likely ACPI)

 - The MP table is right, and only pin0 is connected, and WinXP only uses
   pin0 (ie it puts the card in some state where all irqs are shared
   across all of the four tulip chips).

Maybe somebody can come up with other schenarios.

It would be interesting to hear what "Device Manager" (or whatever it is
called) unde WinXP claims the interrupts are on this machine... Are they
all on irq 48 on XP too? Or has XP gotten magic knowledge somewhere
(ACPI?) and they are on different irq's?

                Linus

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

PCI init issues

Post by Ivan Kokshaysk » Thu, 06 Mar 2003 00:40:08



> Incorrect.
> Most quad Tulip boards have the bus bridge wired so that all interrupts
> are sent on the INTA output of the board.

This can be true for older cards, but post-1998 hardware must follow
the spec.
PCI-to-PCI Bridge Architecture Specification, Rev 1.1, Dec 18, 1998,
pp 113-114:

 "... The interrupt binding defined in this table is mandatory for
  expansion boards utilizing a bridge.

  [table skipped]

  Device 0 on a secondary bus will have its INTA# line connected to
  the INTA# line of the connector. Device 1 will have its INTA# line
  connected to INTB# of the connector. This sequence continues and
  then wraps around once INTD# has been assigned."

I know for a fact that at least D-Link card mentioned in some reports
utilizes all four INT# lines, because I have one.

Ivan.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

PCI init issues

Post by Ivan Kokshaysk » Thu, 06 Mar 2003 00:50:11



> Well, I'd say it looks like the MP table _claims_ that only pin0 is
> connected. Remember: the claim was that this machine worked on WinXP.

It seems that "IRQ redirection table" is gathered directly from
IO_APIC hardware (I may be wrong though), and it conforms to other
data...

Quote:> So there are at least two potential reasons for that:

>  - The MP table is simply wrong, and WinXP gets the routing information
>    from somewhere else (ie most likely ACPI)

>  - The MP table is right, and only pin0 is connected, and WinXP only uses
>    pin0 (ie it puts the card in some state where all irqs are shared
>    across all of the four tulip chips).

> Maybe somebody can come up with other schenarios.

  - XP is able to reprogramm the IO_APIC so that all four pins are
    routed properly.

Sounds a bit heretical, I know. :-)

Ivan.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

PCI init issues

Post by Zwane Mwaikamb » Thu, 06 Mar 2003 01:00:11



> > So there are at least two potential reasons for that:

> >  - The MP table is simply wrong, and WinXP gets the routing information
> >    from somewhere else (ie most likely ACPI)

> >  - The MP table is right, and only pin0 is connected, and WinXP only uses
> >    pin0 (ie it puts the card in some state where all irqs are shared
> >    across all of the four tulip chips).

> > Maybe somebody can come up with other schenarios.

> > It would be interesting to hear what "Device Manager" (or whatever it is
> > called) unde WinXP claims the interrupts are on this machine... Are they
> > all on irq 48 on XP too? Or has XP gotten magic knowledge somewhere
> > (ACPI?) and they are on different irq's?

> >               Linus

How about using 'mptable' (Authored by Pete Zaitcev) to view the
bus,pin and ioapic layout? It showed all the INTA,B,C,D pins on a
troublesome system of mine and helped track down the problem.

http://people.redhat.com/zaitcev/linux/mptable-2.0.15a-1.i386.rpm
http://people.redhat.com/zaitcev/linux/mptable-2.0.15a-1.src.rpm

Regards,
        Zwane
--
function.linuxpower.ca
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

PCI init issues

Post by kuz.. » Thu, 06 Mar 2003 01:10:09


Hello!

Quote:> I know for a fact that at least D-Link card mentioned in some reports
> utilizes all four INT# lines, because I have one.

I confirm, my Znyx card (pci subsys 110d:0029) is wired correctly to INTA/B/C/D
and works nicely with plain flat motherboards.

Look at Jamal's mail: the cards work well when attached to primary
bus. They stop to work when connected via one more bridge (which is
common for server motherboards).

Alexey
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

PCI init issues

Post by kuz.. » Thu, 06 Mar 2003 01:30:09


Hello!

Quote:>   - XP is able to reprogramm the IO_APIC so that all four pins are
>     routed properly.

> Sounds a bit heretical, I know. :-)

Let me to cite message from Znyx engineer.

Quote:>>The question is important. Looking into our implemenation, I see
>>that it is strongly bound to correctness of mp table.

>We had a similar problem in about 1994, when we designed
>the first multi-port adapters with a bridge. Most BIOSes at that
>point did not initialize PPBs, or they did not do it correctly. We
>decided to build the required support into our drivers. That meant
>that some vendors with broken BIOS would point to us and say 'see
>it works'. Our advantage in selling these people adapters, but
>maybe it took a bit longer for the market to force vendors to fix
>BIOSes.

So, maybe the workaround is in znyx driver for XP. :-)
BIOSes were fixed to understand the first level bridges,
but I guess none of them construct correct tables
for second level bridges.

Alexey
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

1. PCI init issues

Yes, it would be interesting to test this on similar motherboards
(with E7500 chipset) which do work without that patch, just to make
sure it doesn't break something.

Why not, but I'd like to clean it up first (will do in a next
couple of days).

Ivan.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

2. Mail delivery backups

3. Disabling RAMDISK when installing

4. module-init-tools 0.9.3 -- "missing" issue

5. Multi homing an interface on RedHat

6. SMB setup/init issues

7. Linux for Asus P4G8Deluxe

8. init.d script shutdown issues

9. 2.5.25 - drivers/net/rrunner DMA mapping + pci init update

10. PCI bios init crashes the kernel.

11. Trust Communicator 128 PCI and "getting no interrupts during init 1"

12. Missing init.h in drivers/pci/power.c