Eric Dumazet 79640a4ca6 net: add additional lock to qdisc to increase throughput ... When many cpus compete for sending frames on a given qdisc, the qdisc spinlock suffers from very high contention. The cpu owning __QDISC_STATE_RUNNING bit has same priority to acquire the lock, and cannot dequeue packets fast enough, since it must wait for this lock for each dequeued packet. One solution to this problem is to force all cpus spinning on a second lock before trying to get the main lock, when/if they see __QDISC_STATE_RUNNING already set. The owning cpu then compete with at most one other cpu for the main lock, allowing for higher dequeueing rate. Based on a previous patch from Alexander Duyck. I added the heuristic to avoid the atomic in fast path, and put the new lock far away from the cache line used by the dequeue worker. Also try to release the busylock lock as late as possible. Tests with following script gave a boost from ~50.000 pps to ~600.000 pps on a dual quad core machine (E5450 @3.00GHz), tg3 driver. (A single netperf flow can reach ~800.000 pps on this platform) for j in `seq 0 3`; do for i in `seq 0 7`; do netperf -H 192.168.0.1 -t UDP_STREAM -l 60 -N -T $i -- -m 6 & done done Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com> Acked-by: Alexander Duyck <alexander.h.duyck@intel.com> Signed-off-by: David S. Miller <davem@davemloft.net>		2010-06-02 05:09:29 -07:00
..
act_api.c	net sched: printk message severity	2010-05-17 23:23:12 -07:00
act_gact.c	net sched: printk message severity	2010-05-17 23:23:12 -07:00
act_ipt.c	net sched: printk message severity	2010-05-17 23:23:12 -07:00
act_mirred.c	net sched: printk message severity	2010-05-17 23:23:12 -07:00
act_nat.c	tc: propogate errors from tcf_hash_create	2008-11-25 21:13:25 -08:00
act_pedit.c	net sched: printk message severity	2010-05-17 23:23:12 -07:00
act_police.c	include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h	2010-03-30 22:02:32 +09:00
act_simple.c	net sched: printk message severity	2010-05-17 23:23:12 -07:00
act_skbedit.c	pkt_sched: skbedit add support for setting mark	2009-10-22 21:56:42 -07:00
cls_api.c	Merge branch 'master' of master.kernel.org:/pub/scm/linux/kernel/git/davem/net-2.6	2010-04-11 14:53:53 -07:00
cls_basic.c	include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h	2010-03-30 22:02:32 +09:00
cls_cgroup.c	cls_cgroup: Store classid in struct sock	2010-05-24 00:12:34 -07:00
cls_flow.c	net: Remove unnecessary returns from void function()s	2010-05-17 23:23:14 -07:00
cls_fw.c	include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h	2010-03-30 22:02:32 +09:00
cls_route.c	include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h	2010-03-30 22:02:32 +09:00
cls_rsvp6.c	[NET_SCHED]: Remove unnecessary includes	2007-07-10 22:16:41 -07:00
cls_rsvp.c	[NET_SCHED]: Remove unnecessary includes	2007-07-10 22:16:41 -07:00
cls_rsvp.h	net: Move && and || to end of previous line	2009-11-29 16:55:45 -08:00
cls_tcindex.c	include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h	2010-03-30 22:02:32 +09:00
cls_u32.c	net sched: printk message severity	2010-05-17 23:23:12 -07:00
em_cmp.c	net: em_cmp.c use unaligned access helpers	2008-09-22 19:20:51 -07:00
em_meta.c	include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h	2010-03-30 22:02:32 +09:00
em_nbyte.c	include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h	2010-03-30 22:02:32 +09:00
em_text.c	include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h	2010-03-30 22:02:32 +09:00
em_u32.c	[PKT_SCHED]: annotate cls_u32	2008-03-17 22:46:46 -07:00
ematch.c	net sched: printk message severity	2010-05-17 23:23:12 -07:00
Kconfig	cgroups: net_cls as module	2010-03-23 13:06:14 -07:00
Makefile	net_sched: add classful multiqueue dummy scheduler	2009-09-06 02:07:05 -07:00
sch_api.c	net_sched: Fix qdisc_notify()	2010-05-23 23:11:07 -07:00
sch_atm.c	include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h	2010-03-30 22:02:32 +09:00
sch_blackhole.c	pkt_sched: Add qdisc->ops->peek() implementation.	2008-10-31 00:45:55 -07:00
sch_cbq.c	include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h	2010-03-30 22:02:32 +09:00
sch_drr.c	include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h	2010-03-30 22:02:32 +09:00
sch_dsmark.c	include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h	2010-03-30 22:02:32 +09:00
sch_fifo.c	include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h	2010-03-30 22:02:32 +09:00
sch_generic.c	net: add additional lock to qdisc to increase throughput	2010-06-02 05:09:29 -07:00
sch_gred.c	include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h	2010-03-30 22:02:32 +09:00
sch_hfsc.c	net: Remove unnecessary returns from void function()s	2010-05-17 23:23:14 -07:00
sch_htb.c	include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h	2010-03-30 22:02:32 +09:00
sch_ingress.c	net: Remove unnecessary returns from void function()s	2010-05-17 23:23:14 -07:00
sch_mq.c	net: Remove unnecessary returns from void function()s	2010-05-17 23:23:14 -07:00
sch_multiq.c	net: Remove unnecessary returns from void function()s	2010-05-17 23:23:14 -07:00
sch_netem.c	include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h	2010-03-30 22:02:32 +09:00
sch_prio.c	net: Remove unnecessary returns from void function()s	2010-05-17 23:23:14 -07:00
sch_red.c	net: Remove unnecessary returns from void function()s	2010-05-17 23:23:14 -07:00
sch_sfq.c	net: Fix various endianness glitches	2010-04-20 19:06:52 -07:00
sch_tbf.c	tbf: stop wanton destruction of children (v2)	2010-05-17 17:44:35 -07:00
sch_teql.c	include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h	2010-03-30 22:02:32 +09:00