korg/linux
0
0
Fork 0
mirror of https://mirrors.bfsu.edu.cn/git/linux.git synced 2024-11-17 09:14:19 +08:00
Code Issues Packages Projects Releases Wiki Activity
f36391d279
linux/arch/sparc/kernel
History
David S. Miller f36391d279 sparc64: Fix race in TLB batch processing. 
As reported by Dave Kleikamp, when we emit cross calls to do batched
TLB flush processing we have a race because we do not synchronize on
the sibling cpus completing the cross call.

So meanwhile the TLB batch can be reset (tb->tlb_nr set to zero, etc.)
and either flushes are missed or flushes will flush the wrong
addresses.

Fix this by using generic infrastructure to synchonize on the
completion of the cross call.

This first required getting the flush_tlb_pending() call out from
switch_to() which operates with locks held and interrupts disabled.
The problem is that smp_call_function_many() cannot be invoked with
IRQs disabled and this is explicitly checked for with WARN_ON_ONCE().

We get the batch processing outside of locked IRQ disabled sections by
using some ideas from the powerpc port. Namely, we only batch inside
of arch_{enter,leave}_lazy_mmu_mode() calls.  If we're not in such a
region, we flush TLBs synchronously.

1) Get rid of xcall_flush_tlb_pending and per-cpu type
   implementations.

2) Do TLB batch cross calls instead via:

	smp_call_function_many()
		tlb_pending_func()
			__flush_tlb_pending()

3) Batch only in lazy mmu sequences:

	a) Add 'active' member to struct tlb_batch
	b) Define __HAVE_ARCH_ENTER_LAZY_MMU_MODE
	c) Set 'active' in arch_enter_lazy_mmu_mode()
	d) Run batch and clear 'active' in arch_leave_lazy_mmu_mode()
	e) Check 'active' in tlb_batch_add_one() and do a synchronous
           flush if it's clear.

4) Add infrastructure for synchronous TLB page flushes.

	a) Implement __flush_tlb_page and per-cpu variants, patch
	   as needed.
	b) Likewise for xcall_flush_tlb_page.
	c) Implement smp_flush_tlb_page() to invoke the cross-call.
	d) Wire up global_flush_tlb_page() to the right routine based
           upon CONFIG_SMP

5) It turns out that singleton batches are very common, 2 out of every
   3 batch flushes have only a single entry in them.

   The batch flush waiting is very expensive, both because of the poll
   on sibling cpu completeion, as well as because passing the tlb batch
   pointer to the sibling cpus invokes a shared memory dereference.

   Therefore, in flush_tlb_pending(), if there is only one entry in
   the batch perform a completely asynchronous global_flush_tlb_page()
   instead.

Reported-by: Dave Kleikamp <dave.kleikamp@oracle.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
Acked-by: Dave Kleikamp <dave.kleikamp@oracle.com>
2013-04-19 17:26:26 -04:00
..
.gitignore
apc.c sparc idle: rename pm_idle to sparc_idle 2013-02-17 23:36:56 -05:00
asm-offsets.c
audit.c
auxio_32.c sparc32: Remove sun4 and sun4c from enum sparc_cpu. 2012-05-12 00:23:23 -07:00
auxio_64.c sparc: remove __devinit, __devexit annotations 2013-01-12 15:28:45 -08:00
btext.c sparc: remove several unnecessary module.h include instances 2011-10-31 19:30:54 -04:00
central.c sparc: remove __devinit, __devexit annotations 2013-01-12 15:28:45 -08:00
cherrs.S
chmc.c sparc: remove __devinit, __devexit annotations 2013-01-12 15:28:45 -08:00
compat_audit.c
cpu.c sparc64: correctly recognize SPARC64-X chips 2013-03-11 05:06:27 -07:00
cpumap.c sparc: move symbol exporters to use export.h not module.h 2011-10-31 19:30:53 -04:00
cpumap.h
devices.c sparc32: remove sun4c traps 2012-05-11 19:27:46 -07:00
dma.c sparc: remove several unnecessary module.h include instances 2011-10-31 19:30:54 -04:00
ds.c sparc: remove __devinit, __devexit annotations 2013-01-12 15:28:45 -08:00
dtlb_miss.S
dtlb_prot.S
ebus.c sparc: move symbol exporters to use export.h not module.h 2011-10-31 19:30:53 -04:00
entry.h sparc64: Improvde documentation and readability of atomic backoff code. 2012-10-28 13:04:47 -07:00
entry.S sparc: switch to generic sigaltstack 2013-02-03 18:16:17 -05:00
etrap_32.S sparc32: introduce support for run-time patching for all shared assembler code 2012-05-27 23:52:49 -07:00
etrap_64.S sparc64: clear syscall_noerror on the entry to syscall, not on the exit 2012-10-14 19:26:52 -04:00
fpu_traps.S
ftrace.c sparc64: Add function graph tracer support. 2010-04-12 22:37:26 -07:00
getsetcc.S
head_32.S sparc32: drop swapper_pg_dir 2012-07-26 16:46:15 -07:00
head_64.S sparc64: correctly recognize SPARC64-X chips 2013-03-11 05:06:27 -07:00
helpers.S sparc64: Fix perf_arch_get_caller_regs(). 2010-08-08 22:07:36 -07:00
hvapi.c sparc: fix format string argument for prom_printf() 2012-10-02 23:20:34 -04:00
hvcalls.S sparc64: Add hypervisor interfaces for SPARC-T4 perf counter access. 2012-08-18 23:03:53 -07:00
hvtramp.S
idprom.c sparc: Clear out unused asm/machines.h values. 2012-05-11 20:45:18 -07:00
iommu_common.h
iommu.c SPARC: adapt for dma_map_ops changes 2012-03-28 16:36:34 +02:00
ioport.c sparc32: Export leon_dma_ops to modules. 2012-05-28 00:07:03 -07:00
irq_32.c sparc32: handle leon in irq_32.c 2012-05-27 23:52:44 -07:00
irq_64.c sparc32: Trivial removal of sun4c references in comments. 2012-05-12 00:35:45 -07:00
irq.h sparc32: remove runtime btfix support 2012-05-14 14:05:09 -07:00
itlb_miss.S
ivec.S
jump_label.c sparc: Fix even more fallout from system.h split. 2012-03-29 22:40:52 -07:00
kernel.h sparc32: refactor smp boot 2013-02-20 13:36:50 -08:00
kgdb_32.c sparc: explicitly include sched.h to get task_thread_info declaration 2013-02-06 11:04:10 -08:00
kgdb_64.c sparc: Fix even more fallout from system.h split. 2012-03-29 22:40:52 -07:00
kprobes.c hlist: drop the node parameter from iterators 2013-02-27 19:10:24 -08:00
kstack.h sparc64: Run NMIs on the hardirq stack. 2010-04-14 02:04:29 -07:00
ktlb.S sparc64: Support 2GB and 16GB page sizes for kernel linear mappings. 2012-09-06 18:13:58 -07:00
ldc.c hlist: drop the node parameter from iterators 2013-02-27 19:10:24 -08:00
led.c include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h 2010-03-30 22:02:32 +09:00
leon_kernel.c sparc32, leon: Check for existent irq_map entry in leon_handle_ext_irq 2012-11-09 16:30:50 -08:00
leon_pci_grpci2.c sparc,leon: fix GRPCI2 device0 PCI config space access 2013-03-02 22:36:09 -08:00
leon_pci.c sparc: remove __devinit, __devexit annotations 2013-01-12 15:28:45 -08:00
leon_pmc.c sparc idle: rename pm_idle to sparc_idle 2013-02-17 23:36:56 -05:00
leon_smp.c sparc32: refactor smp boot 2013-02-20 13:36:50 -08:00
Makefile sparc32,leon: always include leon_pmc in build 2012-05-27 23:52:53 -07:00
mdesc.c sparc64: Probe cpu page size support more portably. 2012-09-06 19:01:25 -07:00
misctrap.S
module.c modules: don't hand 0 to vmalloc. 2012-12-14 13:06:43 +10:30
nmi.c sparc64: Abstract away the %pcr values used to enable/disable NMI 2012-08-18 23:26:19 -07:00
of_device_32.c sparc32: rename sparc_irq_config to sparc_config 2012-04-15 10:28:49 -07:00
of_device_64.c devicetree: add helper inline for retrieving a node's full name 2012-07-06 07:16:34 -05:00
of_device_common.c sparc: move symbol exporters to use export.h not module.h 2011-10-31 19:30:53 -04:00
of_device_common.h
pci_common.c sparc: Remove unnecessary semicolons 2011-06-07 16:06:34 -07:00
pci_fire.c sparc: remove __devinit, __devexit annotations 2013-01-12 15:28:45 -08:00
pci_impl.h drivers/of: Constify device_node->name and ->path_component_name 2012-11-17 12:05:57 +00:00
pci_msi.c sparc: convert old cpumask API into new one 2011-05-16 13:38:07 -07:00
pci_psycho.c sparc: remove __devinit, __devexit annotations 2013-01-12 15:28:45 -08:00
pci_sabre.c sparc: remove __devinit, __devexit annotations 2013-01-12 15:28:45 -08:00
pci_schizo.c sparc: remove __devinit, __devexit annotations 2013-01-12 15:28:45 -08:00
pci_sun4v_asm.S
pci_sun4v.c sparc: remove __devinit, __devexit annotations 2013-01-12 15:28:45 -08:00
pci_sun4v.h
pci.c sparc: remove __devinit, __devexit annotations 2013-01-12 15:28:45 -08:00
pcic.c sparc: remove __devinit, __devexit annotations 2013-01-12 15:28:45 -08:00
pcr.c sparc64: Add PCR ops for SPARC-T4. 2012-08-18 23:26:19 -07:00
perf_event.c sparc64: Make montmul/montsqr/mpmul usable in 32-bit threads. 2012-10-26 15:18:37 -07:00
pmc.c sparc idle: rename pm_idle to sparc_idle 2013-02-17 23:36:56 -05:00
power.c sparc: remove __devinit, __devexit annotations 2013-01-12 15:28:45 -08:00
process_32.c sparc idle: rename pm_idle to sparc_idle 2013-02-17 23:36:56 -05:00
process_64.c flagday: kill pt_regs argument of do_fork() 2012-11-29 00:01:08 -05:00
prom_32.c sparc: remove several unnecessary module.h include instances 2011-10-31 19:30:54 -04:00
prom_64.c sparc: fix format string argument for prom_printf() 2012-10-02 23:20:34 -04:00
prom_common.c OF: convert devtree lock from rw_lock to raw spinlock 2013-02-08 17:02:40 -06:00
prom_irqtrans.c sparc: Remove unnecessary semicolons 2011-06-07 16:06:34 -07:00
prom.h of/promtree: make drivers/of/pdt.c no longer sparc-only 2010-10-10 21:53:30 -06:00
psycho_common.c sparc: Remove unnecessary semicolons 2011-06-07 16:06:34 -07:00
psycho_common.h sparc: remove references to of_device and to_of_device 2010-07-24 09:58:22 -06:00
ptrace_32.c Disintegrate asm/system.h for Sparc 2012-03-28 18:30:03 +01:00
ptrace_64.c sparc64: Make montmul/montsqr/mpmul usable in 32-bit threads. 2012-10-26 15:18:37 -07:00
reboot.c Disintegrate asm/system.h for Sparc 2012-03-28 18:30:03 +01:00
rtrap_32.S sparc32: introduce support for run-time patching for all shared assembler code 2012-05-27 23:52:49 -07:00
rtrap_64.S sparc64: Remove trap return code which is now unnecessary. 2012-04-13 13:56:46 -07:00
sbus.c sparc: kernel/sbus.c: fix memory leakage 2013-01-21 14:33:00 -08:00
setup_32.c sparc32: centralize all mmu context handling in srmmu.c 2012-07-26 16:46:16 -07:00
setup_64.c taint: add explicit flag to show whether lock dep is still OK. 2013-01-21 17:17:57 +10:30
signal32.c [regression] braino in "sparc: convert to ksignal" 2013-03-02 02:55:16 -05:00
signal_32.c sparc: convert to ksignal 2013-02-14 09:21:16 -05:00
signal_64.c sparc: convert to ksignal 2013-02-14 09:21:16 -05:00
sigutil_32.c Disintegrate asm/system.h for Sparc 2012-03-28 18:30:03 +01:00
sigutil_64.c Disintegrate asm/system.h for Sparc 2012-03-28 18:30:03 +01:00
sigutil.h sparc: Allow handling signals when stack is corrupted. 2011-08-20 17:14:54 -07:00
smp_32.c sparc32: refactor smp boot 2013-02-20 13:36:50 -08:00
smp_64.c sparc64: Fix race in TLB batch processing. 2013-04-19 17:26:26 -04:00
sparc_ksyms_32.c sparc32: Implement hard_smp_processor_id() via instruction patching. 2012-05-14 13:31:38 -07:00
sparc_ksyms_64.c Disintegrate asm/system.h for Sparc 2012-03-28 18:30:03 +01:00
spiterrs.S
sstate.c sparc: Set reboot-cmd using reboot data hypervisor call if available. 2011-08-02 21:28:52 -07:00
stacktrace.c sparc: move symbol exporters to use export.h not module.h 2011-10-31 19:30:53 -04:00
starfire.c sparc: explicitly cast negative phandle checks to s32 2011-01-03 20:02:06 -07:00
sun4d_irq.c sparc32: remove remaining users of btfixup 2012-05-14 14:05:08 -07:00
sun4d_smp.c sparc32: refactor smp boot 2013-02-20 13:36:50 -08:00
sun4m_irq.c sparc32: remove remaining users of btfixup 2012-05-14 14:05:08 -07:00
sun4m_smp.c sparc32: refactor smp boot 2013-02-20 13:36:50 -08:00
sun4v_ivec.S
sun4v_tlb_miss.S sparc64: Support transparent huge pages. 2012-10-09 16:23:06 +09:00
sys32.S fix compat truncate/ftruncate 2013-02-25 09:24:55 -05:00
sys_sparc32.c sparc: kill sign-extending wrappers for native syscalls 2013-02-03 22:43:36 -05:00
sys_sparc_32.c sparc: switch to use of generic old sigaction 2013-02-03 22:43:35 -05:00
sys_sparc_64.c Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/signal 2012-12-12 12:22:13 -08:00
syscalls.S sparc: switch to generic sigaltstack 2013-02-03 18:16:17 -05:00
sysfs.c cpu: convert 'cpu' and 'machinecheck' sysdev_class to a regular subsystem 2011-12-21 14:29:42 -08:00
systbls_32.S sparc: switch to use of generic old sigaction 2013-02-03 22:43:35 -05:00
systbls_64.S fix compat truncate/ftruncate 2013-02-25 09:24:55 -05:00
systbls.h sparc: switch to generic old sigsuspend 2013-02-03 22:44:37 -05:00
tadpole.c of/sparc: convert various prom_* functions to use phandle 2010-10-09 02:33:34 -06:00
time_32.c sparc: remove __devinit, __devexit annotations 2013-01-12 15:28:45 -08:00
time_64.c sparc: remove __devinit, __devexit annotations 2013-01-12 15:28:45 -08:00
trampoline_32.S sparc32: refactor smp boot 2013-02-20 13:36:50 -08:00
trampoline_64.S
traps_32.c taint: add explicit flag to show whether lock dep is still OK. 2013-01-21 17:17:57 +10:30
traps_64.c taint: add explicit flag to show whether lock dep is still OK. 2013-01-21 17:17:57 +10:30
tsb.S sparc64: Fix tsb_grow() in atomic context. 2013-02-20 09:46:08 -08:00
ttable_32.S sparc32: move trap table to a separate file 2012-05-19 23:27:25 -07:00
ttable_64.S sparc64: renamed ttable.S to ttable_64.S 2012-05-19 23:26:41 -07:00
una_asm_32.S sparc32: unaligned memory access (MNA) trap handler bug 2011-02-01 12:39:59 -08:00
una_asm_64.S sparc: Fix .size directive for do_int_load 2011-03-16 18:19:15 -07:00
unaligned_32.c Disintegrate asm/system.h for Sparc 2012-03-28 18:30:03 +01:00
unaligned_64.c sparc64: Make montmul/montsqr/mpmul usable in 32-bit threads. 2012-10-26 15:18:37 -07:00
us2e_cpufreq.c sparc: Remove unnecessary semicolons 2011-06-07 16:06:34 -07:00
us3_cpufreq.c sparc: Remove unnecessary semicolons 2011-06-07 16:06:34 -07:00
utrap.S
vio.c Fix typo in printed messages 2012-06-24 11:03:52 -07:00
viohs.c sparc: move symbol exporters to use export.h not module.h 2011-10-31 19:30:53 -04:00
visemul.c sparc64: Make montmul/montsqr/mpmul usable in 32-bit threads. 2012-10-26 15:18:37 -07:00
vmlinux.lds.S sparc64: Improvde documentation and readability of atomic backoff code. 2012-10-28 13:04:47 -07:00
windows.c BKL: remove extraneous #include <smp_lock.h> 2010-11-17 08:59:32 -08:00
winfixup.S sparc64: Make montmul/montsqr/mpmul usable in 32-bit threads. 2012-10-26 15:18:37 -07:00
wof.S sparc32: introduce support for run-time patching for all shared assembler code 2012-05-27 23:52:49 -07:00
wuf.S sparc32: introduce support for run-time patching for all shared assembler code 2012-05-27 23:52:49 -07:00