lkml.org 
[lkml]   [2008]   [Apr]   [15]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRe: Bad network performance over 2Gbps
    Willy Tarreau wrote:
    > On Tue, Apr 15, 2008 at 09:06:44PM +0300, Anton Titov wrote:
    >> I use Linux for serving a huge amount of static web on few servers. When
    >> network traffic goes above 2Gbit/sec ksoftirqd/5 (not every time 5, but
    >> every time just one) starts using exactly 100% CPU time and packet
    >> packet loss starts preventing traffic from going up. When the network
    >> traffic is lower than 1.9Gbit ksoftirqds use 0% CPU according to top.
    >>
    >> Uplink is 6 gigabit Intel cards bonded together using 802.3ad algorithm
    >> with xmit_hash_policy set to layer3+4. On the other side is Cisco 2960
    >> switch. Machine is with two quad core Intel Xeons @2.33GHz.
    >>
    >> Here goes a screen snapshot of "top" command. The described behavior
    >> have nothing to do with 13% io-wait. It happens even if it is 0%
    >> io-wait.
    >> http://www.titov.net/misc/top-snap.png
    >>
    >> kernel configuration:
    >> http://www.titov.net/misc/config.gz
    >>
    >> /proc/interrupts, lspci, dmesg (nothing intresting there), ifconfig,
    >> uname -a:
    >> http://www.titov.net/misc/misc.txt.gz
    >>
    >> Is it a Linux bug or some hardware limitation?
    >
    > possibly some missing parameters when loading your e1000 drivers.
    > e1000 NICs support interrupt rate limitation, which proves very
    > efficient in cases such as yours. I'm used to limit them to about
    > 5k ints/s. Do a "modinfo e1000" to get the parameter name, I don't
    > have it quite right in mind.
    >
    > Also, I've CCed linux-net.

    # cat /proc/interrupts
    CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6
    CPU7
    0: 342 261 258 278 271 253 264
    283 IO-APIC-edge timer
    1: 0 0 1 0 1 0 0
    0 IO-APIC-edge i8042
    6: 0 1 0 1 0 0 1
    0 IO-APIC-edge floppy
    9: 0 0 0 0 0 0 0
    0 IO-APIC-fasteoi acpi
    12: 1 1 0 0 0 1 1
    0 IO-APIC-edge i8042
    17: 180 190 178 183 182 186 186
    188 IO-APIC-fasteoi uhci_hcd:usb1, ehci_hcd:usb4
    18: 843504 842514 843653 842033 842416 842742 841903
    842960 IO-APIC-fasteoi 3w-9xxx, uhci_hcd:usb3
    19: 0 0 0 0 0 0 0
    0 IO-APIC-fasteoi uhci_hcd:usb2
    498: 534642903 534635899 534726883 534732377 534701710 534708588 534730550
    534742730 PCI-MSI-edge eth5
    499: 531832274 531846609 531917849 531942676 531855140 531850692 531885565
    531863468 PCI-MSI-edge eth4
    500: 487251627 487279206 487248030 487220044 487239637 487231454 487281672
    487227202 PCI-MSI-edge eth3
    501: 486083953 486062203 486109925 486075793 486036977 486035152 486097551
    486117164 PCI-MSI-edge eth2
    502: 528889380 528863624 528760188 528798619 528891886 528890760 528807939
    528822746 PCI-MSI-edge eth1
    503: 529043135 529056706 528980250 528975209 529018995 529027386 528941583
    528970472 PCI-MSI-edge eth0
    NMI: 0 0 0 0 0 0 0
    0 Non-maskable interrupts
    LOC: 62893699 62809502 62744208 62746035 62708815 62709055 62739182
    62620363 Local timer interrupts
    RES: 15454866 15827970 16235695 15386970 15761053 16097167 16190851
    16159843 Rescheduling interrupts
    CAL: 85 98 85 84 98 93 94
    91 function call interrupts
    TLB: 3565361 3561798 3570271 3566272 3556996 3555866 3578257
    3564557 TLB shootdowns
    TRM: 0 0 0 0 0 0 0
    0 Thermal event interrupts
    THR: 0 0 0 0 0 0 0
    0 Threshold APIC interrupts
    SPU: 0 0 0 0 0 0 0
    0 Spurious interrupts


    Yikes! all wrong!

    the network irq's are being ping-ponged around all the cores! bad!

    1) turn the in-kernel IRQBALANCE option off !
    2) use either the userspace `irqbalance` daemon or
    3) set smp_affinity manually

    Auke

    >
    > Regards,
    > Willy
    >
    > --
    > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    > the body of a message to majordomo@vger.kernel.org
    > More majordomo info at http://vger.kernel.org/majordomo-info.html
    > Please read the FAQ at http://www.tux.org/lkml/



    \
     
     \ /
      Last update: 2008-04-15 22:43    [W:2.712 / U:0.868 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site