linux/include
Eric Dumazet 34f01cc1f5 FUTEX: new PRIVATE futexes
Analysis of current linux futex code :
  --------------------------------------

A central hash table futex_queues[] holds all contexts (futex_q) of waiting
threads.

Each futex_wait()/futex_wait() has to obtain a spinlock on a hash slot to
perform lookups or insert/deletion of a futex_q.

When a futex_wait() is done, calling thread has to :

1) - Obtain a read lock on mmap_sem to be able to validate the user pointer
     (calling find_vma()). This validation tells us if the futex uses
     an inode based store (mapped file), or mm based store (anonymous mem)

2) - compute a hash key

3) - Atomic increment of reference counter on an inode or a mm_struct

4) - lock part of futex_queues[] hash table

5) - perform the test on value of futex.
	(rollback is value != expected_value, returns EWOULDBLOCK)
	(various loops if test triggers mm faults)

6) queue the context into hash table, release the lock got in 4)

7) - release the read_lock on mmap_sem

   <block>

8) Eventually unqueue the context (but rarely, as this part  may be done
   by the futex_wake())

Futexes were designed to improve scalability but current implementation has
various problems :

- Central hashtable :

  This means scalability problems if many processes/threads want to use
  futexes at the same time.
  This means NUMA unbalance because this hashtable is located on one node.

- Using mmap_sem on every futex() syscall :

  Even if mmap_sem is a rw_semaphore, up_read()/down_read() are doing atomic
  ops on mmap_sem, dirtying cache line :
    - lot of cache line ping pongs on SMP configurations.

  mmap_sem is also extensively used by mm code (page faults, mmap()/munmap())
  Highly threaded processes might suffer from mmap_sem contention.

  mmap_sem is also used by oprofile code. Enabling oprofile hurts threaded
  programs because of contention on the mmap_sem cache line.

- Using an atomic_inc()/atomic_dec() on inode ref counter or mm ref counter:
  It's also a cache line ping pong on SMP. It also increases mmap_sem hold time
  because of cache misses.

Most of these scalability problems come from the fact that futexes are in
one global namespace.  As we use a central hash table, we must make sure
they are all using the same reference (given by the mm subsystem).  We
chose to force all futexes be 'shared'.  This has a cost.

But fact is POSIX defined PRIVATE and SHARED, allowing clear separation,
and optimal performance if carefuly implemented.  Time has come for linux
to have better threading performance.

The goal is to permit new futex commands to avoid :
 - Taking the mmap_sem semaphore, conflicting with other subsystems.
 - Modifying a ref_count on mm or an inode, still conflicting with mm or fs.

This is possible because, for one process using PTHREAD_PROCESS_PRIVATE
futexes, we only need to distinguish futexes by their virtual address, no
matter the underlying mm storage is.

If glibc wants to exploit this new infrastructure, it should use new
_PRIVATE futex subcommands for PTHREAD_PROCESS_PRIVATE futexes.  And be
prepared to fallback on old subcommands for old kernels.  Using one global
variable with the FUTEX_PRIVATE_FLAG or 0 value should be OK.

PTHREAD_PROCESS_SHARED futexes should still use the old subcommands.

Compatibility with old applications is preserved, they still hit the
scalability problems, but new applications can fly :)

Note : the same SHARED futex (mapped on a file) can be used by old binaries
*and* new binaries, because both binaries will use the old subcommands.

Note : Vast majority of futexes should be using PROCESS_PRIVATE semantic,
as this is the default semantic. Almost all applications should benefit
of this changes (new kernel and updated libc)

Some bench results on a Pentium M 1.6 GHz (SMP kernel on a UP machine)

/* calling futex_wait(addr, value) with value != *addr */
433 cycles per futex(FUTEX_WAIT) call (mixing 2 futexes)
424 cycles per futex(FUTEX_WAIT) call (using one futex)
334 cycles per futex(FUTEX_WAIT_PRIVATE) call (mixing 2 futexes)
334 cycles per futex(FUTEX_WAIT_PRIVATE) call (using one futex)
For reference :
187 cycles per getppid() call
188 cycles per umask() call
181 cycles per ni_syscall() call

Signed-off-by: Eric Dumazet <dada1@cosmosbay.com>
Pierre Peiffer <pierre.peiffer@bull.net>
Cc: "Ulrich Drepper" <drepper@gmail.com>
Cc: "Nick Piggin" <nickpiggin@yahoo.com.au>
Cc: "Ingo Molnar" <mingo@elte.hu>
Cc: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-05-09 12:30:55 -07:00
..
acpi PNPACPI sets pnpdev->dev.archdata 2007-05-08 11:15:08 -07:00
asm-alpha Remove hardcoding of hard_smp_processor_id on UP systems 2007-05-09 12:30:48 -07:00
asm-arm Use common cpu_is_xxx() macros on AT91 and AVR32 2007-05-09 12:30:50 -07:00
asm-arm26 Remove tas() 2007-05-08 11:15:20 -07:00
asm-avr32 Use common cpu_is_xxx() macros on AT91 and AVR32 2007-05-09 12:30:50 -07:00
asm-blackfin Remove tas() 2007-05-08 11:15:20 -07:00
asm-cris move die notifier handling to common code 2007-05-08 11:15:04 -07:00
asm-frv FRV: Replace pgd management via slabs through quicklists 2007-05-09 12:30:46 -07:00
asm-generic local_t: architecture independent extension 2007-05-08 11:15:20 -07:00
asm-h8300 Remove tas() 2007-05-08 11:15:20 -07:00
asm-i386 Use the APIC to determine the hardware processor id - i386 2007-05-09 12:30:48 -07:00
asm-ia64 Always ask the hardware to obtain hardware processor id - ia64 2007-05-09 12:30:49 -07:00
asm-m32r Remove hardcoding of hard_smp_processor_id on UP systems 2007-05-09 12:30:48 -07:00
asm-m68k m68k: <asm/scatterlist.h> needs <linux/types.h> 2007-05-08 20:41:14 -07:00
asm-m68knommu Remove tas() 2007-05-08 11:15:20 -07:00
asm-mips Remove tas() 2007-05-08 11:15:20 -07:00
asm-parisc local_t: parisc cleanup 2007-05-08 11:15:20 -07:00
asm-powerpc Remove hardcoding of hard_smp_processor_id on UP systems 2007-05-09 12:30:48 -07:00
asm-ppc Remove tas() 2007-05-08 11:15:20 -07:00
asm-s390 Remove hardcoding of hard_smp_processor_id on UP systems 2007-05-09 12:30:48 -07:00
asm-sh Remove tas() 2007-05-08 11:15:20 -07:00
asm-sh64 Remove tas() 2007-05-08 11:15:20 -07:00
asm-sparc Remove hardcoding of hard_smp_processor_id on UP systems 2007-05-09 12:30:48 -07:00
asm-sparc64 Remove hardcoding of hard_smp_processor_id on UP systems 2007-05-09 12:30:48 -07:00
asm-um Remove hardcoding of hard_smp_processor_id on UP systems 2007-05-09 12:30:48 -07:00
asm-v850 Remove tas() 2007-05-08 11:15:20 -07:00
asm-x86_64 Use the APIC to determine the hardware processor id - x86_64 2007-05-09 12:30:48 -07:00
asm-xtensa Remove tas() 2007-05-08 11:15:20 -07:00
crypto [CRYPTO] cryptd: Add software async crypto daemon 2007-05-02 14:38:32 +10:00
keys [AF_RXRPC]: Provide secure RxRPC sockets for use by userspace and kernel both 2007-04-26 15:48:28 -07:00
linux FUTEX: new PRIVATE futexes 2007-05-09 12:30:55 -07:00
math-emu Delete unused header file math-emu/extended.h 2007-05-08 11:15:05 -07:00
media i2c: Cleanup the includes of <linux/i2c.h> 2007-05-01 23:26:29 +02:00
mtd UBI: Unsorted Block Images 2007-04-27 14:23:33 +03:00
net cleanup compat ioctl handling 2007-05-08 11:15:09 -07:00
pcmcia add new_id to PCMCIA drivers 2007-05-07 12:12:50 -07:00
rdma Merge branch 'for-linus' of master.kernel.org:/pub/scm/linux/kernel/git/roland/infiniband 2007-05-07 12:18:21 -07:00
rxrpc [AF_RXRPC]: Delete the old RxRPC code. 2007-04-26 15:55:48 -07:00
scsi [SCSI] sas_scsi_host: Convert to use the kthread API 2007-05-06 09:33:17 -05:00
sound [ALSA] version 1.0.14rc3 2007-03-14 08:25:52 +01:00
video atyfb: halve XCLK with Mobility and 32bit memory 2007-05-08 11:15:32 -07:00
Kbuild