Messages in this thread | | | Subject | Re: [PATCH] PCI: Warn about host bridge device when its numa node is NO_NODE | From | Robin Murphy <> | Date | Thu, 24 Oct 2019 11:16:41 +0100 |
| |
On 2019-10-23 6:10 pm, Bjorn Helgaas wrote: > On Wed, Oct 23, 2019 at 04:22:43PM +0800, Yunsheng Lin wrote: >> On 2019/10/23 5:04, Bjorn Helgaas wrote: >>> On Sat, Oct 19, 2019 at 02:45:43PM +0800, Yunsheng Lin wrote: > >>> I think the underlying problem you're addressing is that: >>> >>> - NUMA_NO_NODE == -1, >>> - dev_to_node(dev) may return NUMA_NO_NODE, >>> - kmalloc(dev) relies on cpumask_of_node(dev_to_node(dev)), and >>> - cpumask_of_node(NUMA_NO_NODE) makes an invalid array reference >>> >>> For example, on arm64, mips loongson, s390, and x86, >>> cpumask_of_node(node) returns "node_to_cpumask_map[node]", and -1 is >>> an invalid array index. >> >> The invalid array index of -1 is the underlying problem here when >> cpumask_of_node(dev_to_node(dev)) is called and cpumask_of_node() >> is not NUMA_NO_NODE aware yet. >> >> In the "numa: make node_to_cpumask_map() NUMA_NO_NODE aware" thread >> disscusion, it is requested that it is better to warn about the pcie >> device without a node assigned by the firmware before making the >> cpumask_of_node() NUMA_NO_NODE aware, so that the system with pci >> devices of "NUMA_NO_NODE" node can be fixed by their vendor. >> >> See: https://lore.kernel.org/lkml/20191011111539.GX2311@hirez.programming.kicks-ass.net/ > > Right. We should warn if the NUMA node number would help us but DT or > the firmware didn't give us one. > > But we can do that independently of any cpumask_of_node() changes. > There's no need to do one patch before the other. Even if you make > cpumask_of_node() tolerate NUMA_NO_NODE, we'll still get the warning > because we're not actually changing any node assignments. > >> So maybe change the warning to below: >> >> if (nr_node_ids > 1 && pcibus_to_node(bus) == NUMA_NO_NODE) >> dev_err(&bus->dev, FW_BUG "No node assigned on NUMA capable HW. Please contact your vendor for updates.\n"); > > I think this is perfect and I don't see the need for the refinement > below: > >> And it seems a pci device's parent will always set to the bridge >> device in pci_setup_device(), and device_add() which will set the >> node to its parent's when the child device' node is NUMA_NO_NODE, >> maybe we can add the bridge device' node checking to make sure >> the pci device really does not have a node assigned, as below: >> >> if (nr_node_ids > 1 && pcibus_to_node(bus) == NUMA_NO_NODE && >> dev_to_node(bus->bridge) == NUMA_NO_NODE) >> dev_err(&bus->dev, FW_BUG "No node assigned on NUMA capable HW. Please contact your vendor for updates.\n"); > > Anyway, would the attached patch work for you? I have it tentatively > queued up on pci/enumeration for v5.5. > >>>> It is possible to >>>> have a PCI bridge shared between two nodes, such that the PCI >>>> devices have equidistance. But the moment you scale this out, you >>>> either get devices that are 'local' to a package while having >>>> multiple packages, or if you maintain a single bridge in a big >>>> system, things become so slow it all doesn't matter anyway. >>>> Assigning a node (one of the shared) is, in the generic ase of >>>> multiple packages, the better solution over assigning all nodes. >>>> >>>> As pci_device_add() will assign the pci device' node according to >>>> the bus the device is on, which is decided by pcibus_to_node(). >>>> Currently different arch may implement the pcibus_to_node() based >>>> on bus->sysdata or bus device' node, which has the same node as >>>> the bridge device. >>>> >>>> And for devices behind another bridge case, the child bus device >>>> is setup with proper parent bus device and inherit its parent' >>>> sysdata in pci_alloc_child_bus(), so the pcie device under the >>>> child bus should have the same node as the parent bridge when >>>> device_add() is called, which will set the node to its parent's >>>> node when the child device' node is NUMA_NO_NODE. >>>> >>>> So this patch only warns about the case when a host bridge device >>>> is registered with a node of NO_NODE in pci_register_host_bridge(). >>>> And it only warns about that when there are more than one numa >>>> nodes in the system. >>> >>> >>>> [1] https://lore.kernel.org/lkml/1568724534-146242-1-git-send-email-linyunsheng@huawei.com/ >>>> >>>> Signed-off-by: Yunsheng Lin <linyunsheng@huawei.com> >>>> --- >>>> drivers/pci/probe.c | 3 +++ >>>> 1 file changed, 3 insertions(+) >>>> >>>> diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c >>>> index 3d5271a..22be96a 100644 >>>> --- a/drivers/pci/probe.c >>>> +++ b/drivers/pci/probe.c >>>> @@ -927,6 +927,9 @@ static int pci_register_host_bridge(struct pci_host_bridge *bridge) >>>> list_add_tail(&bus->node, &pci_root_buses); >>>> up_write(&pci_bus_sem); >>>> >>>> + if (nr_node_ids > 1 && dev_to_node(bus->bridge) == NUMA_NO_NODE) >>>> + dev_err(bus->bridge, FW_BUG "No node assigned on NUMA capable HW by BIOS. Please contact your vendor for updates.\n"); >>>> + >>>> return 0; >>>> >>>> unregister: > > commit 8f8cf239c4f1 > Author: Yunsheng Lin <linyunsheng@huawei.com> > Date: Sat Oct 19 14:45:43 2019 +0800 > > PCI: Warn if no host bridge NUMA node info > > In pci_call_probe(), we try to run driver probe functions on the node where > the device is attached. If we don't know which node the device is attached > to, the driver will likely run on the wrong node. This will still work, > but performance will not be as good as it could be.
Is it guaranteed to be purely a performance issue? In other words, is there definitely no way a physical node could be disabled via idle/hotplug/etc. such that unattributed devices can silently disappear while still in use?
> > On NUMA systems, warn if we don't know which node a PCI host bridge is > attached to. This is likely an indication that ACPI didn't supply a _PXM > method or the DT didn't supply a "numa-node-id" property. > > [bhelgaas: commit log, check bus node] > Link: https://lore.kernel.org/r/1571467543-26125-1-git-send-email-linyunsheng@huawei.com > Signed-off-by: Yunsheng Lin <linyunsheng@huawei.com> > Signed-off-by: Bjorn Helgaas <bhelgaas@google.com> > > diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c > index 3d5271a7a849..40259c38d66a 100644 > --- a/drivers/pci/probe.c > +++ b/drivers/pci/probe.c > @@ -897,6 +897,9 @@ static int pci_register_host_bridge(struct pci_host_bridge *bridge) > else > pr_info("PCI host bridge to bus %s\n", name); > > + if (nr_node_ids > 1 && pcibus_to_node(bus) == NUMA_NO_NODE) > + dev_warn(&bus->dev, "Unknown NUMA node; performance will be reduced\n");
I think this still deserves the FW_BUG prefix.
Robin.
> + > /* Add initial resources to the bus */ > resource_list_for_each_entry_safe(window, n, &resources) { > list_move_tail(&window->node, &bridge->windows); >
| |