Messages in this thread | | | Date | Sun, 5 Nov 2023 03:18:32 +0100 | Subject | Re: [PATCH net-next v6 0/5] Coalesce mac ocp write/modify calls to reduce spinlock contention | From | Mirsad Todorovac <> |
| |
On 11/4/23 23:37, Heiner Kallweit wrote: > On 04.11.2023 23:15, Mirsad Goran Todorovac wrote: >> The motivation for these helpers was the locking overhead of 130 consecutive >> r8168_mac_ocp_write() calls in the RTL8411b reset after the NIC gets confused >> if the PHY is powered-down. >> >> To quote Heiner: >> >> On RTL8411b the RX unit gets confused if the PHY is powered-down. >> This was reported in [0] and confirmed by Realtek. Realtek provided >> a sequence to fix the RX unit after PHY wakeup. >> >> A series of about 130 r8168_mac_ocp_write() calls is performed to program the >> RTL registers for recovery, each doing an expensive spin_lock_irqsave() and >> spin_unlock_irqrestore(). >> >> Each mac ocp write is made of: >> >> static void __r8168_mac_ocp_write(struct rtl8169_private *tp, u32 reg, >> u32 data) >> { >> if (rtl_ocp_reg_failure(reg)) >> return; >> >> RTL_W32(tp, OCPDR, OCPAR_FLAG | (reg << 15) | data); >> } >> >> static void r8168_mac_ocp_write(struct rtl8169_private *tp, u32 reg, >> u32 data) >> { >> unsigned long flags; >> >> raw_spin_lock_irqsave(&tp->mac_ocp_lock, flags); >> __r8168_mac_ocp_write(tp, reg, data); >> raw_spin_unlock_irqrestore(&tp->mac_ocp_lock, flags); >> } >> >> Register programming is done through RTL_W32() macro which expands into >> >> #define RTL_W32(tp, reg, val32) writel((val32), tp->mmio_addr + (reg)) >> >> which is further (on Alpha): >> >> extern inline void writel(u32 b, volatile void __iomem *addr) >> { >> mb(); >> __raw_writel(b, addr); >> } >> >> or on i386/x86_64: >> >> #define build_mmio_write(name, size, type, reg, barrier) \ >> static inline void name(type val, volatile void __iomem *addr) \ >> { asm volatile("mov" size " %0,%1": :reg (val), \ >> "m" (*(volatile type __force *)addr) barrier); } >> >> build_mmio_write(writel, "l", unsigned int, "r", :"memory") >> >> This obviously involves iat least a compiler barrier. >> >> mb() expands into something like this i.e. on x86_64: >> >> #define mb() asm volatile("lock; addl $0,0(%%esp)" ::: "memory") >> >> This means a whole lot of memory bus stalls: for spin_lock_irqsave(), >> memory barrier, writel(), and spin_unlock_irqrestore(). >> >> With about 130 of these sequential calls to r8168_mac_ocp_write() this looks like >> a lock storm that will stall all of the cores and CPUs on the same memory controller >> for certain time I/O takes to finish. >> >> In a sequential case of RTL register programming, the writes to RTL registers >> can be coalesced under a same raw spinlock. This can dramatically decrease the >> number of bus stalls in a multicore or multi-CPU system. >> >> Macro helpers r8168_mac_ocp_write_seq() and r8168_mac_ocp_modify_seq() are >> provided to reduce lock contention: >> >> static void rtl_hw_start_8411_2(struct rtl8169_private *tp) >> { >> >> ... >> >> /* The following Realtek-provided magic fixes an issue with the RX unit >> * getting confused after the PHY having been powered-down. >> */ >> >> static const struct recover_8411b_info init_zero_seq[] = { >> { 0xFC28, 0x0000 }, { 0xFC2A, 0x0000 }, { 0xFC2C, 0x0000 }, >> ... >> }; >> >> ... >> >> r8168_mac_ocp_write_seq(tp, init_zero_seq); >> >> ... >> >> } >> >> The hex data is preserved intact through s/r8168_mac_ocp_write[(]tp,/{ / and s/[)];/ },/ >> functions that only changed the function names and the ending of the line, so the actual >> hex data is unchanged. >> >> To repeat, the reason for the introduction of the original commit >> was to enable recovery of the RX unit on the RTL8411b which was confused by the >> powered-down PHY. This sequence of r8168_mac_ocp_write() calls amplifies the problem >> into a series of about 500+ memory bus locks, most waiting for the main memory read, >> modify and write under a LOCK. The memory barrier in RTL_W32 should suffice for >> the programming sequence to reach RTL NIC registers. >> >> [0] https://bugzilla.redhat.com/show_bug.cgi?id=1692075 >> >> v6: >> proceeded according to Jacob Keller's suggestions by creating a cover page and reducing >> the text within the commits. Applying to the net-next tree as Heiner Kallweit requested. >> >> v5: >> attempted some new optimisations, which were rejected, but not all and not completely. >> >> v4: >> fixed complaints as advised by Heiner and checkpatch.pl. >> split the patch into five sections to be more easily manipulated and reviewed >> introduced r8168_mac_ocp_write_seq() >> applied coalescing of mac ocp writes/modifies for 8168H, 8125 and 8125B >> >> v3: >> removed register/mask pair array sentinels, so using ARRAY_SIZE(). >> avoided duplication of RTL_W32() call code as advised by Heiner. >> >> Mirsad Goran Todorovac (5): >> r8169: Coalesce r8169_mac_ocp_write/modify calls to reduce spinlock >> stalls >> r8169: Coalesce RTL8411b PHY power-down recovery calls to reduce >> spinlock stalls >> r8169: Coalesce mac ocp write and modify for 8168H start to reduce >> spinlocks >> r8169: Coalesce mac ocp commands for 8125 and 8125B start to reduce >> spinlock contention >> r8169: Coalesce mac ocp commands for rtl_hw_init_8125 to reduce >> spinlocks >> >> drivers/net/ethernet/realtek/r8169_main.c | 304 +++++++++++----------- >> 1 file changed, 150 insertions(+), 154 deletions(-) >> > > You still write: > "a lock storm that will stall all of the cores and CPUs on the same memory controller" > even though you were informed that that's not the case. > There's no actual problem, therefore your Fixes tags are incorrect. > Also net-next is closed at the moment. > In patches 3-5 I see no benefit. And I have doubts whether the small benefit in > patch 2 is worth adding all the helpers in patch 1.
After some thought, I would like to have a consensus on these patches, rather than someone feels defeated or outvoted.
So I will try to reach some common ground, if you think the cause is worth it.
Why is adding six lines of a helper a problem worse than removing 130 lines of callers?
I would hate to think that the Linux kernel developer community became the place where Authority has higher weight than Reason and Logic.
I have no personal gain from improving these drivers other than the Galactic credits.
One thing I wouldn't like and do not like is the Windows drivers being better because their programmers are more innovative.
Best regards, Mirsad Todorovac
| |