Check TLB flush hypervisor calls for errors and report them.
Pass HV_MMU_ALL always for now, we can add back the optimization
to avoid the I-TLB flush later.
Always explicitly page align the virtual address arguments.
Signed-off-by: David S. Miller <davem@davemloft.net>
The context allocation scheme we use depends upon there being a 1<-->1
mapping from cpu to physical TLB for correctness. Chips like Niagara
break this assumption.
So what we do is notify all cpus with a cross call when the context
version number changes, and if necessary this makes them allocate
a valid context for the address space they are running at the time.
Stress tested with make -j1024, make -j2048, and make -j4096 kernel
builds on a 32-strand, 8 core, T2000 with 16GB of ram.
Signed-off-by: David S. Miller <davem@davemloft.net>
Otherwise with too much stuff enabled in the kernel config
we can end up with an unaligned trap table.
Signed-off-by: David S. Miller <davem@davemloft.net>
If we take a window fault, on SUN4V set %gl to zero before we
turn PSTATE_IE back on in %pstate. Otherwise if we take an
interrupt we'll end up with corrupt register state.
Signed-off-by: David S. Miller <davem@davemloft.net>
It can map all of the linear kernel mappings with zero TSB hash
conflicts for systems with 16GB or less ram. In such cases, on
SUN4V, once we load up this TSB the first time with all the
mappings, we never take a linear kernel mapping TLB miss ever
again, the hypervisor handles them all.
Signed-off-by: David S. Miller <davem@davemloft.net>
We use a bitmap, one bit for every 256MB of memory. If the
bit is set we can use a 256MB PTE for linear mappings, else
we have to use a 4MB PTE.
SUN4V support is there, and we can very easily add support
for Panther cpu 256MB PTEs in the future.
Signed-off-by: David S. Miller <davem@davemloft.net>
We have to turn off the "polling nrflag" bit when we sleep
the cpu like this, so that we'll get a cross-cpu interrupt
to wake the processor up from the yield.
We also have to disable PSTATE_IE in %pstate around the yield
call and recheck need_resched() in order to avoid any races.
Signed-off-by: David S. Miller <davem@davemloft.net>
Set, but never used.
We used to use this for dynamic IRQ retargetting, but that
code died a long time ago.
Signed-off-by: David S. Miller <davem@davemloft.net>
It's extremely noisy and causes much grief on slow
consoles with large numbers of cpus.
We'll have to provide this some saner way in order
to re-enable this.
Signed-off-by: David S. Miller <davem@davemloft.net>
We're about to seriously die in these cases so it is important
that the messages make it to the console.
Signed-off-by: David S. Miller <davem@davemloft.net>
Another case where we have to force ourselves into global register
level one. Also make sure the arguments passed to sun4v_do_mna() are
correct.
This area actually needs some more work, for example spill fixup is
not necessarily going to do the right thing for this case.
Signed-off-by: David S. Miller <davem@davemloft.net>
Just like kvmap_dtlb_longpath we have to force the
global register level to one in order to mimick the
PSTATE_MG --> PSTATE_AG trasition done on SUN4U.
Signed-off-by: David S. Miller <davem@davemloft.net>
The SUN4V convention with non-shared TSBs is that the context
bit of the TAG is clear. So we have to choose an "invalid"
bit and initialize new TSBs appropriately. Otherwise a zero
TAG looks "valid".
Make sure, for the window fixup cases, that we use the right
global registers and that we don't potentially trample on
the live global registers in etrap/rtrap handling (%g2 and
%g6) and that we put the missing virtual address properly
in %g5.
Signed-off-by: David S. Miller <davem@davemloft.net>
1) Add error return checking for TLB load hypervisor
calls.
2) Don't fallthru to dtlb tsb miss handler from itlb tsb
miss handler, oops.
3) On window fixups, propagate fault information to fixup
handler correctly.
Signed-off-by: David S. Miller <davem@davemloft.net>
This gives more consistent bogomips and delay() semantics,
especially on sun4v. It gives weird looking values though...
Signed-off-by: David S. Miller <davem@davemloft.net>
We need to use the real hardware processor ID when
targetting interrupts, not the "define to 0" thing
the uniprocessor build gives us.
Also, fill in the Node-ID and Agent-ID fields properly
on sun4u/Safari.
Signed-off-by: David S. Miller <davem@davemloft.net>
If the top-level cnode had multi entries in it's "reg"
property, we'd fail. The buffer wasn't large enough in
such cases.
Signed-off-by: David S. Miller <davem@davemloft.net>
The sibling cpu bringup is extremely fragile. We can only
perform the most basic calls until we take over the trap
table from the firmware/hypervisor on the new cpu.
This means no accesses to %g4, %g5, %g6 since those can't be
TLB translated without our trap handlers.
In order to achieve this:
1) Change sun4v_init_mondo_queues() so that it can operate in
several modes.
It can allocate the queues, or install them in the current
processor, or both.
The boot cpu does both in it's call early on.
Later, the boot cpu allocates the sibling cpu queue, starts
the sibling cpu, then the sibling cpu loads them in.
2) init_cur_cpu_trap() is changed to take the current_thread_info()
as an argument instead of reading %g6 directly on the current
cpu.
3) Create a trampoline stack for the sibling cpus. We do our basic
kernel calls using this stack, which is locked into the kernel
image, then go to our proper thread stack after taking over the
trap table.
4) While we are in this delicate startup state, we put 0xdeadbeef
into %g4/%g5/%g6 in order to catch accidental accesses.
5) On the final prom_set_trap_table*() call, we put &init_thread_union
into %g6. This is a hack to make prom_world(0) work. All that
wants to do is restore the %asi register using
get_thread_current_ds().
Longer term we should just do the OBP calls to set the trap table by
hand just like we do for everything else. This would avoid that silly
prom_world(0) issue, then we can remove the init_thread_union hack.
Signed-off-by: David S. Miller <davem@davemloft.net>
For 32 cpus and a slow console, it just wedges the
machine especially with DETECT_SOFTLOCKUP enabled.
Signed-off-by: David S. Miller <davem@davemloft.net>
The whole algorithm was wrong. What we need to do is:
1) Walk each PCI bus above this device on the path to the
PCI controller nexus, and for each:
a) If interrupt-map exists, apply it, record IRQ controller node
b) Else, swivel interrupt number using PCI_SLOT(), use PCI bus
parent OBP node as controller node
c) Walk up to "controller node" until we hit the first PCI bus
in this domain, or "controller node" is the PCI controller
OBP node
2) If we walked to PCI controller OBP node, we're done.
3) Else, apply PCI controller interrupt-map to interrupt.
There is some stuff that needs to be checked out for ebus and
isa, but the PCI part is good to go.
Signed-off-by: David S. Miller <davem@davemloft.net>
We need to set the global register set _AND_ disable
PSTATE_IE in %pstate. The original patch sequence was
leaving PSTATE_IE enabled when returning to kernel mode,
oops.
This fixes the random register corruption being seen
on SUN4V.
Signed-off-by: David S. Miller <davem@davemloft.net>
Forgot to multiply by 8 * 1024, oops. Correct the size constant when
the virtual-dma arena is 2GB in size, it should bet 256 not 128.
Finally, log some info about the TSB at probe time.
Signed-off-by: David S. Miller <davem@davemloft.net>
For SUN4V, we were clobbering %o5 to do the hypervisor call.
This clobbers the saved %pstate value and we end up writing
garbage into that register as a result. Oops.
Signed-off-by: David S. Miller <davem@davemloft.net>
Use prom_startcpu_cpuid() on SUN4V instead of prom_startcpu().
We should really test for "SUNW,start-cpu-by-cpuid" presence
and use it if present even on SUN4U.
Signed-off-by: David S. Miller <davem@davemloft.net>
When crawling up the PCI bus chain, stop at the first node
that has an interrupt-map property before we hit the root.
Also, if we use a bus interrupt-{map,mask} do not forget to
update the 'intmask' pointer as we do for the 'intmap' pointer.
Signed-off-by: David S. Miller <davem@davemloft.net>
On SUN4V, force IRQ state to idle in enable_irq(). However,
I'm still not sure this is %100 correct.
Call add_interrupt_randomness() on SUN4V too.
Signed-off-by: David S. Miller <davem@davemloft.net>
On the PBM's first bus number, only allow device 0, function 0, to be
poked at with PCI config space accesses.
For some reason, this single device responds to all device numbers.
Also, reduce the verbiage of the debugging log printk's for PCI cfg
space accesses in the SUN4V PCI controller driver, so that it doesn't
overwhelm the slow SUN4V hypervisor console.
Signed-off-by: David S. Miller <davem@davemloft.net>
We should dynamically allocate the per-cpu pglist not use
an in-kernel-image datum, since __pa() does not work on
such addresses.
Also, consistently use "u32" for devhandle.
Signed-off-by: David S. Miller <davem@davemloft.net>
Add udelay to polling console write loop, and increment
the loop limit.
Name the device "ttyHV" and pass that to add_preferred_console()
when we're using hypervisor console.
Kill sunhv_console_setup(), it's empty.
Handle the case where we don't want to use hypervisor console.
(ie. we have a head attached to a sun4v machine)
Signed-off-by: David S. Miller <davem@davemloft.net>
Get bus range from child of PCI controller root nexus.
This is actually a hack, but the PCI-E bridge sitting
at the top of the PCI tree responds to PCI config cycles
for every device number, so best to just ignore it for now.
Preliminary PCI irq routing, needs lots of work.
Signed-off-by: David S. Miller <davem@davemloft.net>
Clear top 8-bits of physical addresses in "ranges" property.
This gives the actual physical address.
Detect PBM-A vs. PBM-B by checking bit 0x40 of the devhandle.
Signed-off-by: David S. Miller <davem@davemloft.net>
PCI cfg space is accessed transparently through the Hypervisor and not
through direct cpu PIO operations.
Signed-off-by: David S. Miller <davem@davemloft.net>
We have to use bootmem during init_IRQ and page alloc
for sibling cpu calls.
Also, fix incorrect hypervisor call return value
checks in the hypervisor SMP cpu mondo send code.
Signed-off-by: David S. Miller <davem@davemloft.net>
Yes, you heard it right, they changed the PTE layout for
SUN4V. Ho hum...
This is the simple and inefficient way to support this.
It'll get optimized, don't worry.
Signed-off-by: David S. Miller <davem@davemloft.net>
Code patching did not sign extend negative branch
offsets correctly.
Kernel TLB miss path needs patching and %g4 register
preservation in order to handle SUN4V correctly.
Signed-off-by: David S. Miller <davem@davemloft.net>
prom_sun4v_name should be "sun4v" not "SUNW,sun4v"
Also, this is too early to make use of the
.sun4v_Xinsn_patch code patching, so just check
things manually.
This gets us at least to prom_init() on Niagara.
Signed-off-by: David S. Miller <davem@davemloft.net>
There was also a bug in sun4v_itlb_miss, it loaded the
MMU Fault Status base into %g3 instead of %g2.
This pointed out a fast path for TSB miss processing,
since we have %g2 with the MMU Fault Status base, we
can use that to quickly load up the PGD phys address.
Signed-off-by: David S. Miller <davem@davemloft.net>
This is where the virtual address of the fault status
area belongs.
To set it up we don't make a hypervisor call, instead
we call OBP's SUNW,set-trap-table with the real address
of the fault status area as the second argument. And
right before that call we write the virtual address into
ASI_SCRATCHPAD vaddr 0x0.
Signed-off-by: David S. Miller <davem@davemloft.net>
Add assembler file for PCI hypervisor calls.
Setup basic skeleton of SUN4V PCI controller driver.
Add 32-bit devhandle to PBM struct, as this is needed for
hypervisor calls.
Signed-off-by: David S. Miller <davem@davemloft.net>
Abstract out IOMMU operations so that we can have a different
set of calls on sun4v, which needs to do things through
hypervisor calls.
Signed-off-by: David S. Miller <davem@davemloft.net>
When we register a TSB with the hypervisor, so that it or hardware can
handle TLB misses and do the TSB walk for us, the hypervisor traps
down to these trap when it incurs a TSB miss.
Processing is simple, we load the missing virtual address and context,
and do a full page table walk.
Signed-off-by: David S. Miller <davem@davemloft.net>
We look for "SUNW,sun4v" in the 'compatible' property
of the root OBP device tree node.
Protect every %ver register access, to make sure it is
not touched on sun4v, as %ver is hyperprivileged there.
Lock kernel TLB entries using hypervisor calls instead of
calls into OBP.
Signed-off-by: David S. Miller <davem@davemloft.net>
Technically the hypervisor call supports sending in a list
of all cpus to get the cross-call, but I only pass in one
cpu at a time for now.
The multi-cpu support is there, just ifdef'd out so it's easy to
enable or delete it later.
Signed-off-by: David S. Miller <davem@davemloft.net>
Sun4v has 4 interrupt queues: cpu, device, resumable errors,
and non-resumable errors. A set of head/tail offset pointers
help maintain a work queue in physical memory. The entries
are 64-bytes in size.
Each queue is allocated then registered with the hypervisor
as we bring cpus up.
The two error queues each get a kernel side buffer that we
use to quickly empty the main interrupt queue before we
call up to C code to log the event and possibly take evasive
action.
Signed-off-by: David S. Miller <davem@davemloft.net>
Happily we have no D-cache aliasing issues on these
chips, so the implementation is very straightforward.
Add a stub in bootup which will be where the patching
calls will be made for niagara/sun4v/hypervisor.
Signed-off-by: David S. Miller <davem@davemloft.net>
Things are a little tricky because, unlike sun4u, we have
to:
1) do a hypervisor trap to do the TLB load.
2) do the TSB lookup calculations by hand
Signed-off-by: David S. Miller <davem@davemloft.net>
If we're just switching between different alternate global
sets, nop it out on sun4v. Also, get rid of all of the
alternate global save/restore in the OBP CIF trampoline code.
Signed-off-by: David S. Miller <davem@davemloft.net>
They are totally unnecessary because:
1) Interrupts are already disabled when switch_to()
runs.
2) We don't use hard-coded alternate globals any longer.
This found a case in rtrap, which still assumed alternate
global %g6 was current_thread_info(), and that is fixed
by this changeset as well.
Signed-off-by: David S. Miller <davem@davemloft.net>
As we save trap state onto the stack, the store buffer fills up
mid-way through and we stall for several cycles as the store buffer
trickles out to the L2 cache. Meanwhile we can do some privileged
register reads and other calculations, essentially for free.
Signed-off-by: David S. Miller <davem@davemloft.net>
And more consistently check cheetah{,_plus} instead
of assuming anything not spitfire is cheetah{,_plus}.
Signed-off-by: David S. Miller <davem@davemloft.net>
When saving and restoing trap state, do the window spill/fill
handling inline so that we never trap deeper than 2 trap levels.
This is important for chips like Niagara.
The window fixup code is massively simplified, and many more
improvements are now possible.
Signed-off-by: David S. Miller <davem@davemloft.net>
On uniprocessor, it's always zero for optimize that.
On SMP, the jmpl to the stub kills the return address stack in the cpu
branch prediction logic, so expand the code sequence inline and use a
code patching section to fix things up. This also always better and
explicit register selection, which will be taken advantage of in a
future changeset.
The hard_smp_processor_id() function is big, so do not inline it.
Fix up tests for Jalapeno to also test for Serrano chips too. These
tests want "jbus Ultra-IIIi" cases to match, so that is what we should
test for.
Signed-off-by: David S. Miller <davem@davemloft.net>
The are distrupting, which by the sparc v9 definition means they
can only occur when interrupts are enabled in the %pstate register.
This never occurs in any of the trap handling code running at
trap levels > 0.
So just mark it as an unexpected trap.
This allows us to kill off the cee_stuff member of struct thread_info.
Signed-off-by: David S. Miller <davem@davemloft.net>
This way we don't need to lock the TSB into the TLB.
The trick is that every TSB load/store is registered into
a special instruction patch section. The default uses
virtual addresses, and the patch instructions use physical
address load/stores.
We can't do this on all chips because only cheetah+ and later
have the physical variant of the atomic quad load.
Signed-off-by: David S. Miller <davem@davemloft.net>
If we are returning back to kernel mode, %g4 could be live
(for example, in the case where we window spill in the etrap
code). So do not change it's value if going back to kernel.
Signed-off-by: David S. Miller <davem@davemloft.net>
Since we use %g5 itself as a temporary, it can get clobbered
if we take an interrupt mid-stream and thus cause end up with
the final %g5 value too early as a result of rtrap processing.
Set %g5 at the very end, atomically, to avoid this problem.
Signed-off-by: David S. Miller <davem@davemloft.net>
%g6 is not necessarily set to current_thread_info()
at sparc64_realfault_common. So store the fault
code and address after we invoke etrap and %g6 is
properly set up.
Signed-off-by: David S. Miller <davem@davemloft.net>
Just flip the bit off of whatever it's currently set to.
PSTATE_IE is guarenteed to be enabled when we get here.
Signed-off-by: David S. Miller <davem@davemloft.net>
It is totally unnecessary complexity. After we take over
the trap table, we handle all PROM tlb misses fully.
Signed-off-by: David S. Miller <davem@davemloft.net>
Some of the trap code was still assuming that alternate
global %g6 was hard coded with current_thread_info().
Let's just consistently flush at KERNBASE when we need
a pipeline synchronization. That's locked into the TLB
and will always work.
Signed-off-by: David S. Miller <davem@davemloft.net>
As the RSS grows, grow the TSB in order to reduce the likelyhood
of hash collisions and thus poor hit rates in the TSB.
This definitely needs some serious tuning.
Signed-off-by: David S. Miller <davem@davemloft.net>
This also cleans up tsb_context_switch(). The assembler
routine is now __tsb_context_switch() and the former is
an inline function that picks out the bits from the mm_struct
and passes it into the assembler code as arguments.
setup_tsb_parms() computes the locked TLB entry to map the
TSB. Later when we support using the physical address quad
load instructions of Cheetah+ and later, we'll simply use
the physical address for the TSB register value and set
the map virtual and PTE both to zero.
Signed-off-by: David S. Miller <davem@davemloft.net>
Move {init_new,destroy}_context() out of line.
Do not put huge pages into the TSB, only base page size translations.
There are some clever things we could do here, but for now let's be
correct instead of fancy.
Signed-off-by: David S. Miller <davem@davemloft.net>
UltraSPARC has special sets of global registers which are switched to
for certain trap types. There is one set for MMU related traps, one
set of Interrupt Vector processing, and another set (called the
Alternate globals) for all other trap types.
For what seems like forever we've hard coded the values in some of
these trap registers. Some examples include:
1) Interrupt Vector global %g6 holds current processors interrupt
work struct where received interrupts are managed for IRQ handler
dispatch.
2) MMU global %g7 holds the base of the page tables of the currently
active address space.
3) Alternate global %g6 held the current_thread_info() value.
Such hardcoding has resulted in some serious issues in many areas.
There are some code sequences where having another register available
would help clean up the implementation. Taking traps such as
cross-calls from the OBP firmware requires some trick code sequences
wherein we have to save away and restore all of the special sets of
global registers when we enter/exit OBP.
We were also using the IMMU TSB register on SMP to hold the per-cpu
area base address, which doesn't work any longer now that we actually
use the TSB facility of the cpu.
The implementation is pretty straight forward. One tricky bit is
getting the current processor ID as that is different on different cpu
variants. We use a stub with a fancy calling convention which we
patch at boot time. The calling convention is that the stub is
branched to and the (PC - 4) to return to is in register %g1. The cpu
number is left in %g6. This stub can be invoked by using the
__GET_CPUID macro.
We use an array of per-cpu trap state to store the current thread and
physical address of the current address space's page tables. The
TRAP_LOAD_THREAD_REG loads %g6 with the current thread from this
table, it uses __GET_CPUID and also clobbers %g1.
TRAP_LOAD_IRQ_WORK is used by the interrupt vector processing to load
the current processor's IRQ software state into %g6. It also uses
__GET_CPUID and clobbers %g1.
Finally, TRAP_LOAD_PGD_PHYS loads the physical address base of the
current address space's page tables into %g7, it clobbers %g1 and uses
__GET_CPUID.
Many refinements are possible, as well as some tuning, with this stuff
in place.
Signed-off-by: David S. Miller <davem@davemloft.net>
Taking a nod from the powerpc port.
With the per-cpu caching of both the page allocator and SLAB, the
pgtable quicklist scheme becomes relatively silly and primitive.
Signed-off-by: David S. Miller <davem@davemloft.net>
We now use the TSB hardware assist features of the UltraSPARC
MMUs.
SMP is currently knowingly broken, we need to find another place
to store the per-cpu base pointers. We hid them away in the TSB
base register, and that obviously will not work any more :-)
Another known broken case is non-8KB base page size.
Also noticed that flush_tlb_all() is not referenced anywhere, only
the internal __flush_tlb_all() (local cpu only) is used by the
sparc64 port, so we can get rid of flush_tlb_all().
The kernel gets it's own 8KB TSB (swapper_tsb) and each address space
gets it's own private 8K TSB. Later we can add code to dynamically
increase the size of per-process TSB as the RSS grows. An 8KB TSB is
good enough for up to about a 4MB RSS, after which the TSB starts to
incur many capacity and conflict misses.
We even accumulate OBP translations into the kernel TSB.
Another area for refinement is large page size support. We could use
a secondary address space TSB to handle those.
Signed-off-by: David S. Miller <davem@davemloft.net>
The patch "[SPARC64]: Get rid of fast IRQ feature"
moved the the code from arch/sparc64/kernel/entry.S:
lduba [%g7] ASI_PHYS_BYPASS_EC_E, %g5
or %g5, AUXIO_AUX1_FTCNT, %g5
stba %g5, [%g7] ASI_PHYS_BYPASS_EC_E
andn %g5, AUXIO_AUX1_FTCNT, %g5
stba %g5, [%g7] ASI_PHYS_BYPASS_EC_E
to arch/sparc64/kernel/irq.c:
val = readb(auxio_register);
val |= AUXIO_AUX1_FTCNT;
writeb(val, auxio_register);
val &= AUXIO_AUX1_FTCNT;
writeb(val, auxio_register);
This looks like it it missing a bitwise not, which is reintroduced
by this patch.
Due to lack of a floppy device, I could not test it, but it looks
evident.
Signed-off-by: Bernhard R Link <brlink@debian.org>
Signed-off-by: David S. Miller <davem@davemloft.net>
We must use the "a" (allocate) attribute every time we
emit an entry into the __ex_table section.
For consistency, use "a" instead of #alloc which is some
Solaris compat cruft GNU as provides on Sparc.
Signed-off-by: David S. Miller <davem@davemloft.net>
The change to kernel/sched.c's init code to use for_each_cpu()
requires that the cpu_possible_map be setup much earlier.
Set it up via setup_arch(), constrained to NR_CPUS, and later
constrain it to max_cpus in smp_prepare_cpus().
This fixes SMP booting on sparc64.
Signed-off-by: David S. Miller <davem@davemloft.net>
The sparc64 64 bit syscall table seems to be broken as it has
compat_sys_newfstatat in its syscall table instead of sys_newfstatat.
Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
Also, the Solaris syscall table is sized differrently,
and does not go beyond entry 255, so trim off the excess
entries.
Signed-off-by: David S. Miller <davem@davemloft.net>
This also includes by necessity _TIF_RESTORE_SIGMASK support,
which actually resulted in a lot of cleanups.
The sparc signal handling code is quite a mess and I should
clean it up some day.
Signed-off-by: David S. Miller <davem@davemloft.net>
