korg/linux
0
0
Fork 0
mirror of https://mirrors.bfsu.edu.cn/git/linux.git synced 2024-11-26 13:44:15 +08:00
Code Issues Packages Projects Releases Wiki Activity

sched/completions/Documentation: Clean up the document some more

Browse Source 
Refresh the document:

 - Remove unnecessary liguistic complexity and improve the clarity of the text

 - Improve the explanations all around

 - Remove unnecessary and stale version info

 - Fix whitespace noise

 - Make pseudo-code match kernel style

 - Fix minor syntax errors in pseudo-code

 - Use consistent denotation

 - Mark multi-CPU sequences more explicitly

 - Unbreak line breaks

 - Use quotes to refer to 'struct completion'

 - Use 'IRQ context' and 'IRQs' consistently

 - Improve grammar

 - etc.

Cc: John Garry <john.garry@huawei.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Nicholas Mc Guire <hofrat@osadl.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: corbet@lwn.net
Cc: linux-doc@vger.kernel.org
Link: http://lkml.kernel.org/r/1539183392-239389-1-git-send-email-john.garry@huawei.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
This commit is contained in:
Ingo Molnar 2018-10-11 10:36:23 +02:00
parent 7b6abce7e1
commit 0c373344b5
1 changed files with 123 additions and 106 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

229
Documentation/scheduler/completion.txt
View File

@ -1,146 +1,161 @@
completions - wait for completion handling Completions - "wait for completion" barrier APIs
========================================== ================================================
This document was originally written based on 3.18.0 (linux-next)
Introduction: Introduction:
------------- -------------
If you have one or more threads of execution that must wait for some process If you have one or more threads that must wait for some kernel activity
to have reached a point or a specific state, completions can provide a to have reached a point or a specific state, completions can provide a
race-free solution to this problem. Semantically they are somewhat like a race-free solution to this problem. Semantically they are somewhat like a
pthread_barrier and have similar use-cases. pthread_barrier() and have similar use-cases.
Completions are a code synchronization mechanism which is preferable to any Completions are a code synchronization mechanism which is preferable to any
misuse of locks. Any time you think of using yield() or some quirky misuse of locks/semaphores and busy-loops. Any time you think of using
msleep(1) loop to allow something else to proceed, you probably want to yield() or some quirky msleep(1) loop to allow something else to proceed,
look into using one of the wait_for_completion*() calls instead. The you probably want to look into using one of the wait_for_completion*()
advantage of using completions is clear intent of the code, but also more calls and complete() instead.
efficient code as both threads can continue until the result is actually
needed.
Completions are built on top of the generic event infrastructure in Linux, The advantage of using completions is that they have a well defined, focused
with the event reduced to a simple flag (appropriately called "done") in purpose which makes it very easy to see the intent of the code, but they
struct completion that tells the waiting threads of execution if they also result in more efficient code as all threads can continue execution
can continue safely. until the result is actually needed, and both the waiting and the signalling
is highly efficient using low level scheduler sleep/wakeup facilities.
As completions are scheduling related, the code is found in Completions are built on top of the waitqueue and wakeup infrastructure of
the Linux scheduler. The event the threads on the waitqueue are waiting for
is reduced to a simple flag in 'struct completion', appropriately called "done".
As completions are scheduling related, the code can be found in
kernel/sched/completion.c. kernel/sched/completion.c.
Usage: Usage:
------ ------
There are three parts to using completions, the initialization of the There are three main parts to using completions:
struct completion, the waiting part through a call to one of the variants of
wait_for_completion() and the signaling side through a call to complete()
or complete_all(). Further there are some helper functions for checking the
state of completions.
To use completions one needs to include <linux/completion.h> and - the initialization of the 'struct completion' synchronization object
create a variable of type struct completion. The structure used for - the waiting part through a call to one of the variants of wait_for_completion(),
handling of completions is: - the signaling side through a call to complete() or complete_all().
There are also some helper functions for checking the state of completions.
Note that while initialization must happen first, the waiting and signaling
part can happen in any order. I.e. it's entirely normal for a thread
to have marked a completion as 'done' before another thread checks whether
it has to wait for it.
To use completions you need to #include <linux/completion.h> and
create a static or dynamic variable of type 'struct completion',
which has only two fields:
struct completion { struct completion {
unsigned int done; unsigned int done;
wait_queue_head_t wait; wait_queue_head_t wait;
}; };
providing the wait queue to place tasks on for waiting and the flag for This provides the ->wait waitqueue to place tasks on for waiting (if any), and
indicating the state of affairs. the ->done completion flag for indicating whether it's completed or not.
Completions should be named to convey the intent of the waiter. A good Completions should be named to refer to the event that is being synchronized on.
example is: A good example is:
wait_for_completion(&early_console_added); wait_for_completion(&early_console_added);
complete(&early_console_added); complete(&early_console_added);
Good naming (as always) helps code readability. Good, intuitive naming (as always) helps code readability. Naming a completion
'complete' is not helpful unless the purpose is super obvious...
Initializing completions: Initializing completions:
------------------------- -------------------------
Initialization of dynamically allocated completions, often embedded in Initialization of dynamically allocated completion objects, often embedded in
other structures, is done with: other structures, is done via a call to init_completion():
void init_completion(&done); init_completion(&dynamic_object->done);
Initialization is accomplished by initializing the wait queue and setting In this call we initialize the waitqueue and set ->done to 0, i.e. "not completed"
the default state to "not available", that is, "done" is set to 0. or "not done".
The re-initialization function, reinit_completion(), simply resets the The re-initialization function, reinit_completion(), simply resets the
done element to "not available", thus again to 0, without touching the ->done field to 0 ("not done"), without touching the waitqueue.
wait queue. Calling init_completion() twice on the same completion object is Callers of this function must make sure that there are no racy
wait_for_completion() calls going on in parallel.
Calling init_completion() on the same completion object twice is
most likely a bug as it re-initializes the queue to an empty queue and most likely a bug as it re-initializes the queue to an empty queue and
enqueued tasks could get "lost" - use reinit_completion() in that case. enqueued tasks could get "lost" - use reinit_completion() in that case,
but be aware of other races.
For static declaration and initialization, macros are available. These are: For static declaration and initialization, macros are available.
static DECLARE_COMPLETION(setup_done) For static (or global) declarations in file scope you can use DECLARE_COMPLETION():
used for static declarations in file scope. Within functions the static static DECLARE_COMPLETION(setup_done);
initialization should always use: DECLARE_COMPLETION(setup_done);
Note that in this case the completion is boot time (or module load time)
initialized to 'not done' and doesn't require an init_completion() call.
When a completion is declared as a local variable within a function,
then the initialization should always use:
DECLARE_COMPLETION_ONSTACK(setup_done) DECLARE_COMPLETION_ONSTACK(setup_done)
suitable for automatic/local variables on the stack and will make lockdep A simple DECLARE_COMPLETION() on the stack makes lockdep unhappy.
happy. Note also that one needs to make *sure* the completion passed to
work threads remains in-scope, and no references remain to on-stack data
when the initiating function returns.
Using on-stack completions for code that calls any of the _timeout or
_interruptible/_killable variants is not advisable as they will require
additional synchronization to prevent the on-stack completion object in
the timeout/signal cases from going out of scope. Consider using dynamically
allocated completions when intending to use the _interruptible/_killable
or _timeout variants of wait_for_completion().
Note that when using completion objects as local variables you must be
aware of the short life time of the function stack: the function must
not return to a calling context until all activities (such as waiting
threads) have ceased and the completion is ... completely unused.
Waiting for completions: Waiting for completions:
------------------------ ------------------------
For a thread of execution to wait for some concurrent work to finish, it For a thread to wait for some concurrent activity to finish, it
calls wait_for_completion() on the initialized completion structure. calls wait_for_completion() on the initialized completion structure:
void wait_for_completion(struct completion *done)
A typical usage scenario is: A typical usage scenario is:
CPU#1 CPU#2
struct completion setup_done; struct completion setup_done;
init_completion(&setup_done); init_completion(&setup_done);
initialize_work(...,&setup_done,...) initialize_work(...,&setup_done,...);
/* run non-dependent code */ /* do setup */ /* run non-dependent code */ /* do setup */
wait_for_completion(&setup_done); complete(setup_done) wait_for_completion(&setup_done); complete(setup_done);
This is not implying any temporal order on wait_for_completion() and the This is not implying any particular order between wait_for_completion() and
call to complete() - if the call to complete() happened before the call the call to complete() - if the call to complete() happened before the call
to wait_for_completion() then the waiting side simply will continue to wait_for_completion() then the waiting side simply will continue
immediately as all dependencies are satisfied; if not, it will block until immediately as all dependencies are satisfied; if not, it will block until
completion is signaled by complete(). completion is signaled by complete().
Note that wait_for_completion() is calling spin_lock_irq()/spin_unlock_irq(), Note that wait_for_completion() is calling spin_lock_irq()/spin_unlock_irq(),
so it can only be called safely when you know that interrupts are enabled. so it can only be called safely when you know that interrupts are enabled.
Calling it from hard-irq or irqs-off atomic contexts will result in Calling it from IRQs-off atomic contexts will result in hard-to-detect
hard-to-detect spurious enabling of interrupts. spurious enabling of interrupts.
wait_for_completion():
void wait_for_completion(struct completion *done):
The default behavior is to wait without a timeout and to mark the task as The default behavior is to wait without a timeout and to mark the task as
uninterruptible. wait_for_completion() and its variants are only safe uninterruptible. wait_for_completion() and its variants are only safe
in process context (as they can sleep) but not in atomic context, in process context (as they can sleep) but not in atomic context,
interrupt context, with disabled irqs, or preemption is disabled - see also interrupt context, with disabled IRQs, or preemption is disabled - see also
try_wait_for_completion() below for handling completion in atomic/interrupt try_wait_for_completion() below for handling completion in atomic/interrupt
context. context.
As all variants of wait_for_completion() can (obviously) block for a long As all variants of wait_for_completion() can (obviously) block for a long
time, you probably don't want to call this with held mutexes. time depending on the nature of the activity they are waiting for, so in
most cases you probably don't want to call this with held mutexes.
Variants available: wait_for_completion*() variants available:
------------------- ------------------------------------------
The below variants all return status and this status should be checked in The below variants all return status and this status should be checked in
most(/all) cases - in cases where the status is deliberately not checked you most(/all) cases - in cases where the status is deliberately not checked you
@ -148,51 +163,53 @@ probably want to make a note explaining this (e.g. see
arch/arm/kernel/smp.c:__cpu_up()). arch/arm/kernel/smp.c:__cpu_up()).
A common problem that occurs is to have unclean assignment of return types, A common problem that occurs is to have unclean assignment of return types,
so care should be taken with assigning return-values to variables of proper so take care to assign return-values to variables of the proper type.
type. Checking for the specific meaning of return values also has been found
to be quite inaccurate e.g. constructs like Checking for the specific meaning of return values also has been found
if (!wait_for_completion_interruptible_timeout(...)) would execute the same to be quite inaccurate, e.g. constructs like:
code path for successful completion and for the interrupted case - which is
probably not what you want. if (!wait_for_completion_interruptible_timeout(...))
... would execute the same code path for successful completion and for the
interrupted case - which is probably not what you want.
int wait_for_completion_interruptible(struct completion *done) int wait_for_completion_interruptible(struct completion *done)
This function marks the task TASK_INTERRUPTIBLE. If a signal was received This function marks the task TASK_INTERRUPTIBLE while it is waiting.
while waiting it will return -ERESTARTSYS; 0 otherwise. If a signal was received while waiting it will return -ERESTARTSYS; 0 otherwise.
unsigned long wait_for_completion_timeout(struct completion *done, unsigned long wait_for_completion_timeout(struct completion *done, unsigned long timeout)
unsigned long timeout)
The task is marked as TASK_UNINTERRUPTIBLE and will wait at most 'timeout' The task is marked as TASK_UNINTERRUPTIBLE and will wait at most 'timeout'
(in jiffies). If timeout occurs it returns 0 else the remaining time in jiffies. If a timeout occurs it returns 0, else the remaining time in
jiffies (but at least 1). Timeouts are preferably calculated with jiffies (but at least 1).
msecs_to_jiffies() or usecs_to_jiffies(). If the returned timeout value is
deliberately ignored a comment should probably explain why (e.g. see
drivers/mfd/wm8350-core.c wm8350_read_auxadc())
long wait_for_completion_interruptible_timeout( Timeouts are preferably calculated with msecs_to_jiffies() or usecs_to_jiffies(),
struct completion *done, unsigned long timeout) to make the code largely HZ-invariant.
If the returned timeout value is deliberately ignored a comment should probably explain
why (e.g. see drivers/mfd/wm8350-core.c wm8350_read_auxadc()).
long wait_for_completion_interruptible_timeout(struct completion *done, unsigned long timeout)
This function passes a timeout in jiffies and marks the task as This function passes a timeout in jiffies and marks the task as
TASK_INTERRUPTIBLE. If a signal was received it will return -ERESTARTSYS; TASK_INTERRUPTIBLE. If a signal was received it will return -ERESTARTSYS;
otherwise it returns 0 if the completion timed out or the remaining time in otherwise it returns 0 if the completion timed out, or the remaining time in
jiffies if completion occurred. jiffies if completion occurred.
Further variants include _killable which uses TASK_KILLABLE as the Further variants include _killable which uses TASK_KILLABLE as the
designated tasks state and will return -ERESTARTSYS if it is interrupted or designated tasks state and will return -ERESTARTSYS if it is interrupted,
else 0 if completion was achieved. There is a _timeout variant as well: or 0 if completion was achieved. There is a _timeout variant as well:
long wait_for_completion_killable(struct completion *done) long wait_for_completion_killable(struct completion *done)
long wait_for_completion_killable_timeout(struct completion *done, long wait_for_completion_killable_timeout(struct completion *done, unsigned long timeout)
unsigned long timeout)
The _io variants wait_for_completion_io() behave the same as the non-_io The _io variants wait_for_completion_io() behave the same as the non-_io
variants, except for accounting waiting time as waiting on IO, which has variants, except for accounting waiting time as 'waiting on IO', which has
an impact on how the task is accounted in scheduling stats. an impact on how the task is accounted in scheduling/IO stats:
void wait_for_completion_io(struct completion *done) void wait_for_completion_io(struct completion *done)
unsigned long wait_for_completion_io_timeout(struct completion *done unsigned long wait_for_completion_io_timeout(struct completion *done, unsigned long timeout)
unsigned long timeout)
Signaling completions: Signaling completions:
@ -200,30 +217,30 @@ Signaling completions:
A thread that wants to signal that the conditions for continuation have been A thread that wants to signal that the conditions for continuation have been
achieved calls complete() to signal exactly one of the waiters that it can achieved calls complete() to signal exactly one of the waiters that it can
continue. continue:
void complete(struct completion *done) void complete(struct completion *done)
or calls complete_all() to signal all current and future waiters. ... or calls complete_all() to signal all current and future waiters:
void complete_all(struct completion *done) void complete_all(struct completion *done)
The signaling will work as expected even if completions are signaled before The signaling will work as expected even if completions are signaled before
a thread starts waiting. This is achieved by the waiter "consuming" a thread starts waiting. This is achieved by the waiter "consuming"
(decrementing) the done element of struct completion. Waiting threads (decrementing) the done field of 'struct completion'. Waiting threads
wakeup order is the same in which they were enqueued (FIFO order). wakeup order is the same in which they were enqueued (FIFO order).
If complete() is called multiple times then this will allow for that number If complete() is called multiple times then this will allow for that number
of waiters to continue - each call to complete() will simply increment the of waiters to continue - each call to complete() will simply increment the
done element. Calling complete_all() multiple times is a bug though. Both done field. Calling complete_all() multiple times is a bug though. Both
complete() and complete_all() can be called in hard-irq/atomic context safely. complete() and complete_all() can be called in IRQ/atomic context safely.
There only can be one thread calling complete() or complete_all() on a There can only be one thread calling complete() or complete_all() on a
particular struct completion at any time - serialized through the wait particular 'struct completion' at any time - serialized through the wait
queue spinlock. Any such concurrent calls to complete() or complete_all() queue spinlock. Any such concurrent calls to complete() or complete_all()
probably are a design bug. probably are a design bug.
Signaling completion from hard-irq context is fine as it will appropriately Signaling completion from IRQ context is fine as it will appropriately
lock with spin_lock_irqsave()/spin_unlock_irqrestore() and it will never sleep. lock with spin_lock_irqsave()/spin_unlock_irqrestore() and it will never sleep.
@ -236,7 +253,7 @@ else it consumes one posted completion and returns true.
bool try_wait_for_completion(struct completion *done) bool try_wait_for_completion(struct completion *done)
Finally, to check the state of a completion without changing it in any way, Finally, to check the state of a completion without changing it in any way,
call completion_done(), which returns false if there are no posted call completion_done(), which returns false if there are no posted
completions that were not yet consumed by waiters (implying that there are completions that were not yet consumed by waiters (implying that there are
waiters) and true otherwise; waiters) and true otherwise;
@ -244,4 +261,4 @@ waiters) and true otherwise;
bool completion_done(struct completion *done) bool completion_done(struct completion *done)
Both try_wait_for_completion() and completion_done() are safe to be called in Both try_wait_for_completion() and completion_done() are safe to be called in
hard-irq or atomic context. IRQ or atomic context.