pcibus_to_node() addition to topology infrastructure

pcibus_to_node() addition to topology infrastructure

Post by Matthew Dobso » Fri, 01 Nov 2002 01:50:15



Linus,
        Here's a patch that adds PCI busses to the list of basic topology
elements (incl. CPUs, MemBlks, & Nodes).

pcibus_to_node-2.5.44.patch

This patch adds a new topology macro: pcibus_to_node().  This will be
useful to allow I/O bound processes to bind themselves to CPUs/Nodes
close to the PCI busses they are communicating over.

1) Adds pcibus_to_node() macro to asm-generic/topology.h
2) Makes small modifications to NUMA-Q PCI code, mostly modifying macros.
3) Uses the macros from #2 to implement pcibus_to_node() in
asm-i386/topology.h


  arch/i386/pci/numa.c           |   33 +++++++++++++++------------------
  include/asm-generic/topology.h |    3 +++
  include/asm-i386/topology.h    |    3 +++
  3 files changed, 21 insertions(+), 18 deletions(-)

Cheers!

-Matt

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

pcibus_to_node() addition to topology infrastructure

Post by Matthew Dobso » Fri, 01 Nov 2002 02:00:09


Whoops!  A patch would be nice...

Cheers!

-Matt


> Linus,
>     Here's a patch that adds PCI busses to the list of basic topology
> elements (incl. CPUs, MemBlks, & Nodes).

> pcibus_to_node-2.5.44.patch

> This patch adds a new topology macro: pcibus_to_node().  This will be
> useful to allow I/O bound processes to bind themselves to CPUs/Nodes
> close to the PCI busses they are communicating over.

> 1) Adds pcibus_to_node() macro to asm-generic/topology.h
> 2) Makes small modifications to NUMA-Q PCI code, mostly modifying macros.
> 3) Uses the macros from #2 to implement pcibus_to_node() in
> asm-i386/topology.h


>  arch/i386/pci/numa.c           |   33 +++++++++++++++------------------
>  include/asm-generic/topology.h |    3 +++
>  include/asm-i386/topology.h    |    3 +++
>  3 files changed, 21 insertions(+), 18 deletions(-)

> Cheers!

> -Matt

[ pcibus_to_node-2.5.44.patch 4K ]
diff -Nur --exclude-from=/usr/src/.dontdiff linux-2.5.44-vanilla/arch/i386/pci/numa.c linux-2.5.44-pcibus_to_node/arch/i386/pci/numa.c
--- linux-2.5.44-vanilla/arch/i386/pci/numa.c   Fri Oct 18 21:01:17 2002

 #include <linux/pci.h>
 #include <linux/init.h>
 #include "pci.h"
-
-#define BUS2QUAD(global) (mp_bus_id_to_node[global])
-#define BUS2LOCAL(global) (mp_bus_id_to_local[global])
-#define QUADLOCAL2BUS(quad,local) (quad_local_to_mp_bus_id[quad][local])
+#include <asm/topology.h>

 #define PCI_CONF1_MQ_ADDRESS(bus, dev, fn, reg) \
-       (0x80000000 | (BUS2LOCAL(bus) << 16) | (dev << 11) | (fn << 8) | (reg & ~3))
+       (0x80000000 | mp_bus_id_to_local[bus] << 16) | (dev << 11) | (fn << 8) | (reg & ~3))

 static int __pci_conf1_mq_read (int seg, int bus, int dev, int fn, int reg, int len, u32 *value)

        spin_lock_irqsave(&pci_config_lock, flags);

-       outl_quad(PCI_CONF1_MQ_ADDRESS(bus, dev, fn, reg), 0xCF8, BUS2QUAD(bus));
+       outl_quad(PCI_CONF1_MQ_ADDRESS(bus, dev, fn, reg), 0xCF8, __pcibus_to_node(bus));

        switch (len) {
        case 1:
-               *value = inb_quad(0xCFC + (reg & 3), BUS2QUAD(bus));
+               *value = inb_quad(0xCFC + (reg & 3), __pcibus_to_node(bus));
                break;
        case 2:
-               *value = inw_quad(0xCFC + (reg & 2), BUS2QUAD(bus));
+               *value = inw_quad(0xCFC + (reg & 2), __pcibus_to_node(bus));
                break;
        case 4:
-               *value = inl_quad(0xCFC, BUS2QUAD(bus));
+               *value = inl_quad(0xCFC, __pcibus_to_node(bus));
                break;
        }

        spin_lock_irqsave(&pci_config_lock, flags);

-       outl_quad(PCI_CONF1_MQ_ADDRESS(bus, dev, fn, reg), 0xCF8, BUS2QUAD(bus));
+       outl_quad(PCI_CONF1_MQ_ADDRESS(bus, dev, fn, reg), 0xCF8, __pcibus_to_node(bus));

        switch (len) {
        case 1:
-               outb_quad((u8)value, 0xCFC + (reg & 3), BUS2QUAD(bus));
+               outb_quad((u8)value, 0xCFC + (reg & 3), __pcibus_to_node(bus));
                break;
        case 2:
-               outw_quad((u16)value, 0xCFC + (reg & 2), BUS2QUAD(bus));
+               outw_quad((u16)value, 0xCFC + (reg & 2), __pcibus_to_node(bus));
                break;
        case 4:
-               outl_quad((u32)value, 0xCFC, BUS2QUAD(bus));
+               outl_quad((u32)value, 0xCFC, __pcibus_to_node(bus));
                break;
        }

         */
        int pxb, reg;
        u8 busno, suba, subb;
-       int quad = BUS2QUAD(d->bus->number);
+       int quad = __pcibus_to_node(d->bus->number);

        printk("PCI: Searching for i450NX host bridges on %s\n", d->slot_name);

                pci_read_config_byte(d, reg++, &subb);
                DBG("i450NX PXB %d: %02x/%02x/%02x\n", pxb, busno, suba, subb);
                if (busno)
-                       pci_scan_bus(QUADLOCAL2BUS(quad,busno), pci_root_ops, NULL);    /* Bus A */
+                       pci_scan_bus(quad_local_to_mp_bus_id[quad][busno], pci_root_ops, NULL); /* Bus A */
                if (suba < subb)
-                       pci_scan_bus(QUADLOCAL2BUS(quad,suba+1), pci_root_ops, NULL);   /* Bus B */
+                       pci_scan_bus(quad_local_to_mp_bus_id[quad][suba+1], pci_root_ops, NULL);        /* Bus B */
        }
        pcibios_last_bus = -1;

        if (clustered_apic_mode && (numnodes > 1)) {
                for (quad = 1; quad < numnodes; ++quad) {
                        printk("Scanning PCI bus %d for quad %d\n",
-                               QUADLOCAL2BUS(quad,0), quad);
-                       pci_scan_bus(QUADLOCAL2BUS(quad,0),
+                               quad_local_to_mp_bus_id[quad][0], quad);
+                       pci_scan_bus(quad_local_to_mp_bus_id[quad][0],
                                pci_root_ops, NULL);
                }
        }
diff -Nur --exclude-from=/usr/src/.dontdiff linux-2.5.44-vanilla/include/asm-generic/topology.h linux-2.5.44-pcibus_to_node/include/asm-generic/topology.h
--- linux-2.5.44-vanilla/include/asm-generic/topology.h Fri Oct 18 21:01:11 2002

 #ifndef __node_to_memblk
 #define __node_to_memblk(node)         (0)
 #endif
+#ifndef __pcibus_to_node
+#define __pcibus_to_node(bus)          (0)
+#endif

 #endif /* _ASM_GENERIC_TOPOLOGY_H */
diff -Nur --exclude-from=/usr/src/.dontdiff linux-2.5.44-vanilla/include/asm-i386/topology.h linux-2.5.44-pcibus_to_node/include/asm-i386/topology.h
--- linux-2.5.44-vanilla/include/asm-i386/topology.h    Fri Oct 18 21:01:16 2002

 /* Returns the number of the first MemBlk on Node 'node' */
 #define __node_to_memblk(node) (node)

+/* Returns the number of the node containing PCI bus 'bus' */
+#define __pcibus_to_node(bus) (mp_bus_id_to_node[bus])
+
 #else /* !CONFIG_X86_NUMAQ */
 /*
  * Other i386 platforms should define their own version of the

 
 
 

pcibus_to_node() addition to topology infrastructure

Post by Jesse Barne » Fri, 01 Nov 2002 02:10:12


This is a nice addition, but just FYI there are SGI systems that have
PCI busses attached to more than one node.  I guess for now we can just
round-robin through the attached nodes for the return value.

Thanks,
Jesse


> Linus,
>    Here's a patch that adds PCI busses to the list of basic topology
> elements (incl. CPUs, MemBlks, & Nodes).

> pcibus_to_node-2.5.44.patch

> This patch adds a new topology macro: pcibus_to_node().  This will be
> useful to allow I/O bound processes to bind themselves to CPUs/Nodes
> close to the PCI busses they are communicating over.

> 1) Adds pcibus_to_node() macro to asm-generic/topology.h
> 2) Makes small modifications to NUMA-Q PCI code, mostly modifying macros.
> 3) Uses the macros from #2 to implement pcibus_to_node() in
> asm-i386/topology.h


>  arch/i386/pci/numa.c           |   33 +++++++++++++++------------------
>  include/asm-generic/topology.h |    3 +++
>  include/asm-i386/topology.h    |    3 +++
>  3 files changed, 21 insertions(+), 18 deletions(-)

> Cheers!

> -Matt

> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/
 
 
 

pcibus_to_node() addition to topology infrastructure

Post by Matthew Dobso » Fri, 01 Nov 2002 02:40:05



> This is a nice addition, but just FYI there are SGI systems that have
> PCI busses attached to more than one node.  I guess for now we can just
> round-robin through the attached nodes for the return value.

> Thanks,
> Jesse

Ah, yes...  The p-bricks, i-bricks, etc. right?

Yes, I suppose a round-robin return for the SGI version of the macro
would work...  Certainly not ideal, but it would work.  The problem is
if you assume that binding several processes to the node PCI bus X is on
will ensure that those processes are on the same node.  In this case
(SGI), it would be a false assumption.  I suppose that's what you get
for assuming, though? ;)

Cheers!

-Matt

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

pcibus_to_node() addition to topology infrastructure

Post by Jesse Barne » Fri, 01 Nov 2002 03:10:08



> Ah, yes...  The p-bricks, i-bricks, etc. right?

Yup.

Quote:> Yes, I suppose a round-robin return for the SGI version of the macro
> would work...  Certainly not ideal, but it would work.  The problem is

Can you think of any better way to do it?  Perhaps make pcibus_to_node
return a list of nodes?

Thanks,
Jesse
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

pcibus_to_node() addition to topology infrastructure

Post by Matthew Dobso » Fri, 01 Nov 2002 03:30:05




>>Ah, yes...  The p-bricks, i-bricks, etc. right?

> Yup.

>>Yes, I suppose a round-robin return for the SGI version of the macro
>>would work...  Certainly not ideal, but it would work.  The problem is

> Can you think of any better way to do it?  Perhaps make pcibus_to_node
> return a list of nodes?

Yeah... I was thinking about having it return a bitmask...  It makes it
a bit of a pain to iterate through the mask, only to find that in most
cases there is only one bit set in the mask. :(

It's a bit late in the day... I'll mull this over and see if I can come
up with something good.

Cheers!

-Matt

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

1. network topology/infrastructure management

We are managing a several-thousand subnet infrastructure and have been using
m$ excel spreadsheets (ack).

A lot of time were put into the spreadsheets and they act as we would expect
them to, but it's just a really clunky way to manage our subnets.

You guys have any recommendations?  It'd really be great if the tool would
allow us to store additional per-node and per-subnet information (like
attach files, pki keys, etc...)

Thanks!

2. telnet

3. Fix asm-alpha/topology.h & asm-ppc64/topology.h

4. PPP/mgetty question

5. National Infrastructure Protection Center.

6. Mail folder parcer - please advise

7. atomic copy_*_user() infrastructure

8. XP can't see Samba

9. Anyone have a good Solaris/UNIX infrastructure design doc...

10. REQ: Triarch Infrastructure Docs---If available--TIA!

11. SUNWami Sun Authentication Management Infrastructure

12. Distributed Software Development Infrastructure Challenge

13. 2.5 and DRM/3D infrastructure?