Messages in this thread | | | Subject | Re: OOM when adding ipv6 route: How to make available more per-cpu memory? | From | Eric Dumazet <> | Date | Fri, 05 Nov 2010 19:06:01 +0100 |
| |
Le vendredi 05 novembre 2010 à 10:19 -0700, Ben Greear a écrit : > We are testing 500 mac-vlans with IPv6 addresses on a 32-bit kernel > (2.6.36 + ubuntu patches + our patches) We have one > routing table per interface. It seems that some of them cannot > add routes due to lack of memory. > > root@lanforge-ubuntu:/home/lanforge# ip route show table 29 > root@lanforge-ubuntu:/home/lanforge# ./local/sbin/ip -6 route add default via 2001:98::1 dev eth12#15 table 29 > RTNETLINK answers: Cannot allocate memory > > I see errors such as this in the logs: > > [507107.846864] PERCPU: allocation failed, size=2048 align=4, failed to allocate new chunk > [507107.846867] Pid: 3246, comm: ip Tainted: P 2.6.36-1-ct #7 > [507107.846869] Call Trace: > [507107.846874] [<c05d4996>] ? printk+0x2d/0x2f > [507107.846878] [<c021405e>] pcpu_alloc+0x32e/0x360 > [507107.846880] [<c02140bf>] __alloc_percpu+0xf/0x20 > [507107.846883] [<c0558d6d>] snmp_mib_init+0x3d/0x70 > [507107.846885] [<c0585692>] ipv6_add_dev+0x132/0x350 > [507107.846887] [<c0557f39>] ? inetdev_init+0xb9/0x180 > [507107.846889] [<c05895ff>] addrconf_notify+0x3f/0x490 > [507107.846891] [<c055884f>] ? inetdev_event+0x20f/0x280 > [507107.846894] [<c05daef3>] notifier_call_chain+0x43/0x60 > [507107.846897] [<c016df4f>] raw_notifier_call_chain+0x1f/0x30 > [507107.846899] [<c05008ec>] call_netdevice_notifiers+0x2c/0x60 > [507107.846901] [<c0502ecc>] register_netdevice+0x23c/0x380 > [507107.846907] [<f80aecf7>] macvlan_common_newlink+0x187/0x2a0 [macvlan] > [507107.846910] [<f80ae540>] ? macvlan_setup+0x0/0x20 [macvlan] > [507107.846912] [<f80aee37>] macvlan_newlink+0x27/0x30 [macvlan] > [507107.846914] [<c0503060>] ? netif_rx+0x0/0x120 > [507107.846915] [<c05033c0>] ? dev_forward_skb+0x0/0xf0 > [507107.846917] [<f80aee10>] ? macvlan_newlink+0x0/0x30 [macvlan] > [507107.846920] [<c050ea94>] rtnl_newlink+0x464/0x590 > [507107.846921] [<c050e7e6>] ? rtnl_newlink+0x1b6/0x590 > [507107.846927] [<c050cd2d>] rtnetlink_rcv_msg+0x13d/0x230 > [507107.846929] [<c05d721f>] ? _raw_spin_lock_irqsave+0x2f/0x50 > [507107.846931] [<c050e630>] ? rtnl_newlink+0x0/0x590 > [507107.846933] [<c050cbf0>] ? rtnetlink_rcv_msg+0x0/0x230 > [507107.846935] [<c0522276>] netlink_rcv_skb+0x86/0xb0 > [507107.846936] [<c050cbdc>] rtnetlink_rcv+0x1c/0x30 > [507107.846938] [<c0521f69>] netlink_unicast+0x259/0x280 > [507107.846940] [<c0522ba8>] netlink_sendmsg+0x1f8/0x300 > [507107.846943] [<c04f0b79>] sock_sendmsg+0xd9/0x100 > [507107.846945] [<c013e0dd>] ? finish_task_switch+0x3d/0xc0 > [507107.846947] [<c04f0b79>] ? sock_sendmsg+0xd9/0x100 > [507107.846950] [<c013361d>] ? kmap_atomic_prot+0xcd/0xf0 > [507107.846957] [<f88bd559>] ? h_d_revalidate+0xe9/0x240 [aufs] > [507107.846958] [<c013361d>] ? kmap_atomic_prot+0xcd/0xf0 > [507107.846962] [<c0364c7d>] ? _copy_from_user+0x3d/0x130 > [507107.846965] [<c04f947a>] ? verify_iovec+0x5a/0xa0 > [507107.846966] [<c04f112d>] sys_sendmsg+0x15d/0x290 > [507107.846972] [<f88c05f1>] ? aufs_fault+0xf1/0x110 [aufs] > [507107.846974] [<c013361d>] ? kmap_atomic_prot+0xcd/0xf0 > [507107.846977] [<c01f8726>] ? handle_mm_fault+0x146/0x400 > [507107.846979] [<c05dac0d>] ? do_page_fault+0x1cd/0x470 > [507107.846981] [<c023419d>] ? alloc_fd+0xbd/0xf0 > [507107.846983] [<c0364c7d>] ? _copy_from_user+0x3d/0x130 > [507107.846986] [<c04f183b>] sys_socketcall+0xeb/0x2a0 > [507107.846989] [<c0151d99>] ? irq_exit+0x39/0x70 > [507107.846991] [<c021bb2e>] ? sys_open+0x2e/0x40 > [507107.846993] [<c05d77a4>] syscall_call+0x7/0xb > > > But, it appears to me that we have plenty of memory available. > > top - 10:09:13 up 6 days, 15:56, 9 users, load average: 0.37, 0.39, 0.45 > Tasks: 211 total, 2 running, 209 sleeping, 0 stopped, 0 zombie > Cpu(s): 0.7%us, 2.1%sy, 0.0%ni, 97.2%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st > Mem: 3604332k total, 1510252k used, 2094080k free, 84928k buffers > Swap: 154620k total, 0k used, 154620k free, 1045912k cached > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > > > root@lanforge-ubuntu:/home/lanforge# cat /proc/meminfo > MemTotal: 3604332 kB > MemFree: 2094568 kB > Buffers: 84928 kB > Cached: 1045920 kB > SwapCached: 0 kB > Active: 466784 kB > Inactive: 805632 kB > Active(anon): 148988 kB > Inactive(anon): 6788 kB > Active(file): 317796 kB > Inactive(file): 798844 kB > Unevictable: 0 kB > Mlocked: 0 kB > HighTotal: 2752008 kB > HighFree: 1512380 kB > LowTotal: 852324 kB > LowFree: 582188 kB > SwapTotal: 154620 kB > SwapFree: 154620 kB > Dirty: 492 kB > Writeback: 0 kB > AnonPages: 141316 kB > Mapped: 39428 kB > Shmem: 14208 kB > Slab: 170788 kB > SReclaimable: 81904 kB > SUnreclaim: 88884 kB > KernelStack: 2520 kB > PageTables: 4516 kB > NFS_Unstable: 0 kB > Bounce: 0 kB > WritebackTmp: 0 kB > CommitLimit: 1956784 kB > Committed_AS: 582604 kB > VmallocTotal: 122880 kB > VmallocUsed: 48592 kB > VmallocChunk: 24280 kB > HardwareCorrupted: 0 kB > HugePages_Total: 0 > HugePages_Free: 0 > HugePages_Rsvd: 0 > HugePages_Surp: 0 > Hugepagesize: 4096 kB > DirectMap4k: 12280 kB > DirectMap4M: 897024 kB > > > Is there any way to tune the system so that it has more memory available > for per-cpu data structures?
CC linux-kernel and Tejun Heo
How many possible cpus do you have ? head -1 /proc/interrupts
and please post : cat /proc/vmallocinfo
Thanks
-- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
| |