2018-05-05 03:34:32 +08:00
|
|
|
// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
|
2007-09-11 01:50:12 +08:00
|
|
|
/*
|
2017-10-31 04:22:14 +08:00
|
|
|
* Copyright (c) 2014-2017 Oracle. All rights reserved.
|
2007-09-11 01:51:18 +08:00
|
|
|
* Copyright (c) 2003-2007 Network Appliance, Inc. All rights reserved.
|
|
|
|
|
*
|
|
|
|
|
* This software is available to you under a choice of one of two
|
|
|
|
|
* licenses. You may choose to be licensed under the terms of the GNU
|
|
|
|
|
* General Public License (GPL) Version 2, available from the file
|
|
|
|
|
* COPYING in the main directory of this source tree, or the BSD-type
|
|
|
|
|
* license below:
|
|
|
|
|
*
|
|
|
|
|
* Redistribution and use in source and binary forms, with or without
|
|
|
|
|
* modification, are permitted provided that the following conditions
|
|
|
|
|
* are met:
|
|
|
|
|
*
|
|
|
|
|
* Redistributions of source code must retain the above copyright
|
|
|
|
|
* notice, this list of conditions and the following disclaimer.
|
|
|
|
|
*
|
|
|
|
|
* Redistributions in binary form must reproduce the above
|
|
|
|
|
* copyright notice, this list of conditions and the following
|
|
|
|
|
* disclaimer in the documentation and/or other materials provided
|
|
|
|
|
* with the distribution.
|
|
|
|
|
*
|
|
|
|
|
* Neither the name of the Network Appliance, Inc. nor the names of
|
|
|
|
|
* its contributors may be used to endorse or promote products
|
|
|
|
|
* derived from this software without specific prior written
|
|
|
|
|
* permission.
|
|
|
|
|
*
|
|
|
|
|
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
|
|
|
|
|
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
|
|
|
|
|
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
|
|
|
|
|
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
|
|
|
|
|
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
|
|
|
|
|
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
|
|
|
|
|
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
|
|
|
|
|
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
|
|
|
|
|
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
|
|
|
|
|
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
|
|
|
|
|
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
2007-09-11 01:50:12 +08:00
|
|
|
*/
|
|
|
|
|
|
2007-09-11 01:51:18 +08:00
|
|
|
/*
|
|
|
|
|
* verbs.c
|
|
|
|
|
*
|
|
|
|
|
* Encapsulates the major functions managing:
|
|
|
|
|
* o adapters
|
|
|
|
|
* o endpoints
|
|
|
|
|
* o connections
|
|
|
|
|
* o buffer memory
|
|
|
|
|
*/
|
|
|
|
|
|
2011-06-06 18:43:46 +08:00
|
|
|
#include <linux/interrupt.h>
|
include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h
percpu.h is included by sched.h and module.h and thus ends up being
included when building most .c files. percpu.h includes slab.h which
in turn includes gfp.h making everything defined by the two files
universally available and complicating inclusion dependencies.
percpu.h -> slab.h dependency is about to be removed. Prepare for
this change by updating users of gfp and slab facilities include those
headers directly instead of assuming availability. As this conversion
needs to touch large number of source files, the following script is
used as the basis of conversion.
http://userweb.kernel.org/~tj/misc/slabh-sweep.py
The script does the followings.
* Scan files for gfp and slab usages and update includes such that
only the necessary includes are there. ie. if only gfp is used,
gfp.h, if slab is used, slab.h.
* When the script inserts a new include, it looks at the include
blocks and try to put the new include such that its order conforms
to its surrounding. It's put in the include block which contains
core kernel includes, in the same order that the rest are ordered -
alphabetical, Christmas tree, rev-Xmas-tree or at the end if there
doesn't seem to be any matching order.
* If the script can't find a place to put a new include (mostly
because the file doesn't have fitting include block), it prints out
an error message indicating which .h file needs to be added to the
file.
The conversion was done in the following steps.
1. The initial automatic conversion of all .c files updated slightly
over 4000 files, deleting around 700 includes and adding ~480 gfp.h
and ~3000 slab.h inclusions. The script emitted errors for ~400
files.
2. Each error was manually checked. Some didn't need the inclusion,
some needed manual addition while adding it to implementation .h or
embedding .c file was more appropriate for others. This step added
inclusions to around 150 files.
3. The script was run again and the output was compared to the edits
from #2 to make sure no file was left behind.
4. Several build tests were done and a couple of problems were fixed.
e.g. lib/decompress_*.c used malloc/free() wrappers around slab
APIs requiring slab.h to be added manually.
5. The script was run on all .h files but without automatically
editing them as sprinkling gfp.h and slab.h inclusions around .h
files could easily lead to inclusion dependency hell. Most gfp.h
inclusion directives were ignored as stuff from gfp.h was usually
wildly available and often used in preprocessor macros. Each
slab.h inclusion directive was examined and added manually as
necessary.
6. percpu.h was updated not to include slab.h.
7. Build test were done on the following configurations and failures
were fixed. CONFIG_GCOV_KERNEL was turned off for all tests (as my
distributed build env didn't work with gcov compiles) and a few
more options had to be turned off depending on archs to make things
build (like ipr on powerpc/64 which failed due to missing writeq).
* x86 and x86_64 UP and SMP allmodconfig and a custom test config.
* powerpc and powerpc64 SMP allmodconfig
* sparc and sparc64 SMP allmodconfig
* ia64 SMP allmodconfig
* s390 SMP allmodconfig
* alpha SMP allmodconfig
* um on x86_64 SMP allmodconfig
8. percpu.h modifications were reverted so that it could be applied as
a separate patch and serve as bisection point.
Given the fact that I had only a couple of failures from tests on step
6, I'm fairly confident about the coverage of this conversion patch.
If there is a breakage, it's likely to be something in one of the arch
headers which should be easily discoverable easily on most builds of
the specific arch.
Signed-off-by: Tejun Heo <tj@kernel.org>
Guess-its-ok-by: Christoph Lameter <cl@linux-foundation.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Lee Schermerhorn <Lee.Schermerhorn@hp.com>
2010-03-24 16:04:11 +08:00
|
|
|
#include <linux/slab.h>
|
2015-03-31 02:33:43 +08:00
|
|
|
#include <linux/sunrpc/addr.h>
|
xprtrdma: Fix receive buffer accounting
An RPC can terminate before its reply arrives, if a credential
problem or a soft timeout occurs. After this happens, xprtrdma
reports it is out of Receive buffers.
A Receive buffer is posted before each RPC is sent, and returned to
the buffer pool when a reply is received. If no reply is received
for an RPC, that Receive buffer remains posted. But xprtrdma tries
to post another when the next RPC is sent.
If this happens a few dozen times, there are no receive buffers left
to be posted at send time. I don't see a way for a transport
connection to recover at that point, and it will spit warnings and
unnecessarily delay RPCs on occasion for its remaining lifetime.
Commit 1e465fd4ff47 ("xprtrdma: Replace send and receive arrays")
removed a little bit of logic to detect this case and not provide
a Receive buffer so no more buffers are posted, and then transport
operation continues correctly. We didn't understand what that logic
did, and it wasn't commented, so it was removed as part of the
overhaul to support backchannel requests.
Restore it, but be wary of the need to keep extra Receives posted
to deal with backchannel requests.
Fixes: 1e465fd4ff47 ("xprtrdma: Replace send and receive arrays")
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
Reviewed-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
Signed-off-by: Trond Myklebust <trond.myklebust@primarydata.com>
2016-09-06 23:22:58 +08:00
|
|
|
#include <linux/sunrpc/svc_rdma.h>
|
xprtrdma: Add data structure to manage RDMA Send arguments
Problem statement:
Recently Sagi Grimberg <sagi@grimberg.me> observed that kernel RDMA-
enabled storage initiators don't handle delayed Send completion
correctly. If Send completion is delayed beyond the end of a ULP
transaction, the ULP may release resources that are still being used
by the HCA to complete a long-running Send operation.
This is a common design trait amongst our initiators. Most Send
operations are faster than the ULP transaction they are part of.
Waiting for a completion for these is typically unnecessary.
Infrequently, a network partition or some other problem crops up
where an ordering problem can occur. In NFS parlance, the RPC Reply
arrives and completes the RPC, but the HCA is still retrying the
Send WR that conveyed the RPC Call. In this case, the HCA can try
to use memory that has been invalidated or DMA unmapped, and the
connection is lost. If that memory has been re-used for something
else (possibly not related to NFS), and the Send retransmission
exposes that data on the wire.
Thus we cannot assume that it is safe to release Send-related
resources just because a ULP reply has arrived.
After some analysis, we have determined that the completion
housekeeping will not be difficult for xprtrdma:
- Inline Send buffers are registered via the local DMA key, and
are already left DMA mapped for the lifetime of a transport
connection, thus no additional handling is necessary for those
- Gathered Sends involving page cache pages _will_ need to
DMA unmap those pages after the Send completes. But like
inline send buffers, they are registered via the local DMA key,
and thus will not need to be invalidated
In addition, RPC completion will need to wait for Send completion
in the latter case. However, nearly always, the Send that conveys
the RPC Call will have completed long before the RPC Reply
arrives, and thus no additional latency will be accrued.
Design notes:
In this patch, the rpcrdma_sendctx object is introduced, and a
lock-free circular queue is added to manage a set of them per
transport.
The RPC client's send path already prevents sending more than one
RPC Call at the same time. This allows us to treat the consumer
side of the queue (rpcrdma_sendctx_get_locked) as if there is a
single consumer thread.
The producer side of the queue (rpcrdma_sendctx_put_locked) is
invoked only from the Send completion handler, which is a single
thread of execution (soft IRQ).
The only care that needs to be taken is with the tail index, which
is shared between the producer and consumer. Only the producer
updates the tail index. The consumer compares the head with the
tail to ensure that the a sendctx that is in use is never handed
out again (or, expressed more conventionally, the queue is empty).
When the sendctx queue empties completely, there are enough Sends
outstanding that posting more Send operations can result in a Send
Queue overflow. In this case, the ULP is told to wait and try again.
This introduces strong Send Queue accounting to xprtrdma.
As a final touch, Jason Gunthorpe <jgunthorpe@obsidianresearch.com>
suggested a mechanism that does not require signaling every Send.
We signal once every N Sends, and perform SGE unmapping of N Send
operations during that one completion.
Reported-by: Sagi Grimberg <sagi@grimberg.me>
Suggested-by: Jason Gunthorpe <jgunthorpe@obsidianresearch.com>
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
2017-10-20 22:48:12 +08:00
|
|
|
|
|
|
|
|
#include <asm-generic/barrier.h>
|
xprtrdma: Reduce the number of hardway buffer allocations
While marshaling an RPC/RDMA request, the inline_{rsize,wsize}
settings determine whether an inline request is used, or whether
read or write chunks lists are built. The current default value of
these settings is 1024. Any RPC request smaller than 1024 bytes is
sent to the NFS server completely inline.
rpcrdma_buffer_create() allocates and pre-registers a set of RPC
buffers for each transport instance, also based on the inline rsize
and wsize settings.
RPC/RDMA requests and replies are built in these buffers. However,
if an RPC/RDMA request is expected to be larger than 1024, a buffer
has to be allocated and registered for that RPC, and deregistered
and released when the RPC is complete. This is known has a
"hardway allocation."
Since the introduction of NFSv4, the size of RPC requests has become
larger, and hardway allocations are thus more frequent. Hardway
allocations are significant overhead, and they waste the existing
RPC buffers pre-allocated by rpcrdma_buffer_create().
We'd like fewer hardway allocations.
Increasing the size of the pre-registered buffers is the most direct
way to do this. However, a blanket increase of the inline thresholds
has interoperability consequences.
On my 64-bit system, rpcrdma_buffer_create() requests roughly 7000
bytes for each RPC request buffer, using kmalloc(). Due to internal
fragmentation, this wastes nearly 1200 bytes because kmalloc()
already returns an 8192-byte piece of memory for a 7000-byte
allocation request, though the extra space remains unused.
So let's round up the size of the pre-allocated buffers, and make
use of the unused space in the kmalloc'd memory.
This change reduces the amount of hardway allocated memory for an
NFSv4 general connectathon run from 1322092 to 9472 bytes (99%).
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
Tested-by: Steve Wise <swise@opengridcomputing.com>
Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
2014-05-28 22:33:59 +08:00
|
|
|
#include <asm/bitops.h>
|
2017-04-12 01:23:34 +08:00
|
|
|
|
2017-02-09 06:00:35 +08:00
|
|
|
#include <rdma/ib_cm.h>
|
2007-09-11 01:51:18 +08:00
|
|
|
|
2007-09-11 01:50:12 +08:00
|
|
|
#include "xprt_rdma.h"
|
2018-05-08 03:27:05 +08:00
|
|
|
#include <trace/events/rpcrdma.h>
|
2007-09-11 01:50:12 +08:00
|
|
|
|
2007-09-11 01:51:18 +08:00
|
|
|
/*
|
|
|
|
|
* Globals/Macros
|
|
|
|
|
*/
|
|
|
|
|
|
2014-11-18 05:58:04 +08:00
|
|
|
#if IS_ENABLED(CONFIG_SUNRPC_DEBUG)
|
2007-09-11 01:51:18 +08:00
|
|
|
# define RPCDBG_FACILITY RPCDBG_TRANS
|
|
|
|
|
#endif
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* internal functions
|
|
|
|
|
*/
|
2018-05-05 03:35:41 +08:00
|
|
|
static void rpcrdma_sendctx_put_locked(struct rpcrdma_sendctx *sc);
|
2017-12-15 09:57:55 +08:00
|
|
|
static void rpcrdma_mrs_create(struct rpcrdma_xprt *r_xprt);
|
|
|
|
|
static void rpcrdma_mrs_destroy(struct rpcrdma_buffer *buf);
|
2018-05-05 03:35:20 +08:00
|
|
|
static int rpcrdma_create_rep(struct rpcrdma_xprt *r_xprt, bool temp);
|
2017-04-12 01:23:10 +08:00
|
|
|
static void rpcrdma_dma_unmap_regbuf(struct rpcrdma_regbuf *rb);
|
2018-12-19 23:58:24 +08:00
|
|
|
static void rpcrdma_post_recvs(struct rpcrdma_xprt *r_xprt, bool temp);
|
2007-09-11 01:51:18 +08:00
|
|
|
|
2018-12-19 23:58:29 +08:00
|
|
|
/* Wait for outstanding transport work to finish.
|
|
|
|
|
*/
|
|
|
|
|
static void rpcrdma_xprt_drain(struct rpcrdma_xprt *r_xprt)
|
2007-09-11 01:51:18 +08:00
|
|
|
{
|
2018-12-19 23:58:29 +08:00
|
|
|
struct rpcrdma_buffer *buf = &r_xprt->rx_buf;
|
|
|
|
|
struct rpcrdma_ia *ia = &r_xprt->rx_ia;
|
2007-09-11 01:51:18 +08:00
|
|
|
|
2018-12-19 23:58:29 +08:00
|
|
|
/* Flush Receives, then wait for deferred Reply work
|
|
|
|
|
* to complete.
|
|
|
|
|
*/
|
|
|
|
|
ib_drain_qp(ia->ri_id->qp);
|
|
|
|
|
drain_workqueue(buf->rb_completion_wq);
|
2014-11-09 09:14:37 +08:00
|
|
|
|
2018-12-19 23:58:29 +08:00
|
|
|
/* Deferred Reply processing might have scheduled
|
|
|
|
|
* local invalidations.
|
|
|
|
|
*/
|
|
|
|
|
ib_drain_sq(ia->ri_id->qp);
|
2014-11-09 09:14:37 +08:00
|
|
|
}
|
|
|
|
|
|
2018-10-02 02:26:13 +08:00
|
|
|
/**
|
|
|
|
|
* rpcrdma_qp_event_handler - Handle one QP event (error notification)
|
|
|
|
|
* @event: details of the event
|
|
|
|
|
* @context: ep that owns QP where event occurred
|
|
|
|
|
*
|
|
|
|
|
* Called from the RDMA provider (device driver) possibly in an interrupt
|
|
|
|
|
* context.
|
|
|
|
|
*/
|
2007-09-11 01:51:18 +08:00
|
|
|
static void
|
2018-10-02 02:26:13 +08:00
|
|
|
rpcrdma_qp_event_handler(struct ib_event *event, void *context)
|
2007-09-11 01:51:18 +08:00
|
|
|
{
|
|
|
|
|
struct rpcrdma_ep *ep = context;
|
2017-12-21 05:31:45 +08:00
|
|
|
struct rpcrdma_xprt *r_xprt = container_of(ep, struct rpcrdma_xprt,
|
|
|
|
|
rx_ep);
|
2007-09-11 01:51:18 +08:00
|
|
|
|
2018-10-02 02:26:13 +08:00
|
|
|
trace_xprtrdma_qp_event(r_xprt, event);
|
2007-09-11 01:51:18 +08:00
|
|
|
}
|
|
|
|
|
|
2016-03-05 00:28:53 +08:00
|
|
|
/**
|
|
|
|
|
* rpcrdma_wc_send - Invoked by RDMA provider for each polled Send WC
|
|
|
|
|
* @cq: completion queue (ignored)
|
|
|
|
|
* @wc: completed WR
|
|
|
|
|
*
|
2014-05-28 22:33:25 +08:00
|
|
|
*/
|
|
|
|
|
static void
|
2016-03-05 00:28:53 +08:00
|
|
|
rpcrdma_wc_send(struct ib_cq *cq, struct ib_wc *wc)
|
2014-05-28 22:33:25 +08:00
|
|
|
{
|
xprtrdma: Add data structure to manage RDMA Send arguments
Problem statement:
Recently Sagi Grimberg <sagi@grimberg.me> observed that kernel RDMA-
enabled storage initiators don't handle delayed Send completion
correctly. If Send completion is delayed beyond the end of a ULP
transaction, the ULP may release resources that are still being used
by the HCA to complete a long-running Send operation.
This is a common design trait amongst our initiators. Most Send
operations are faster than the ULP transaction they are part of.
Waiting for a completion for these is typically unnecessary.
Infrequently, a network partition or some other problem crops up
where an ordering problem can occur. In NFS parlance, the RPC Reply
arrives and completes the RPC, but the HCA is still retrying the
Send WR that conveyed the RPC Call. In this case, the HCA can try
to use memory that has been invalidated or DMA unmapped, and the
connection is lost. If that memory has been re-used for something
else (possibly not related to NFS), and the Send retransmission
exposes that data on the wire.
Thus we cannot assume that it is safe to release Send-related
resources just because a ULP reply has arrived.
After some analysis, we have determined that the completion
housekeeping will not be difficult for xprtrdma:
- Inline Send buffers are registered via the local DMA key, and
are already left DMA mapped for the lifetime of a transport
connection, thus no additional handling is necessary for those
- Gathered Sends involving page cache pages _will_ need to
DMA unmap those pages after the Send completes. But like
inline send buffers, they are registered via the local DMA key,
and thus will not need to be invalidated
In addition, RPC completion will need to wait for Send completion
in the latter case. However, nearly always, the Send that conveys
the RPC Call will have completed long before the RPC Reply
arrives, and thus no additional latency will be accrued.
Design notes:
In this patch, the rpcrdma_sendctx object is introduced, and a
lock-free circular queue is added to manage a set of them per
transport.
The RPC client's send path already prevents sending more than one
RPC Call at the same time. This allows us to treat the consumer
side of the queue (rpcrdma_sendctx_get_locked) as if there is a
single consumer thread.
The producer side of the queue (rpcrdma_sendctx_put_locked) is
invoked only from the Send completion handler, which is a single
thread of execution (soft IRQ).
The only care that needs to be taken is with the tail index, which
is shared between the producer and consumer. Only the producer
updates the tail index. The consumer compares the head with the
tail to ensure that the a sendctx that is in use is never handed
out again (or, expressed more conventionally, the queue is empty).
When the sendctx queue empties completely, there are enough Sends
outstanding that posting more Send operations can result in a Send
Queue overflow. In this case, the ULP is told to wait and try again.
This introduces strong Send Queue accounting to xprtrdma.
As a final touch, Jason Gunthorpe <jgunthorpe@obsidianresearch.com>
suggested a mechanism that does not require signaling every Send.
We signal once every N Sends, and perform SGE unmapping of N Send
operations during that one completion.
Reported-by: Sagi Grimberg <sagi@grimberg.me>
Suggested-by: Jason Gunthorpe <jgunthorpe@obsidianresearch.com>
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
2017-10-20 22:48:12 +08:00
|
|
|
struct ib_cqe *cqe = wc->wr_cqe;
|
|
|
|
|
struct rpcrdma_sendctx *sc =
|
|
|
|
|
container_of(cqe, struct rpcrdma_sendctx, sc_cqe);
|
|
|
|
|
|
2016-03-05 00:28:53 +08:00
|
|
|
/* WARNING: Only wr_cqe and status are reliable at this point */
|
2017-12-21 05:30:40 +08:00
|
|
|
trace_xprtrdma_wc_send(sc, wc);
|
2016-03-05 00:28:53 +08:00
|
|
|
if (wc->status != IB_WC_SUCCESS && wc->status != IB_WC_WR_FLUSH_ERR)
|
|
|
|
|
pr_err("rpcrdma: Send: %s (%u/0x%x)\n",
|
|
|
|
|
ib_wc_status_msg(wc->status),
|
|
|
|
|
wc->status, wc->vendor_err);
|
xprtrdma: Add data structure to manage RDMA Send arguments
Problem statement:
Recently Sagi Grimberg <sagi@grimberg.me> observed that kernel RDMA-
enabled storage initiators don't handle delayed Send completion
correctly. If Send completion is delayed beyond the end of a ULP
transaction, the ULP may release resources that are still being used
by the HCA to complete a long-running Send operation.
This is a common design trait amongst our initiators. Most Send
operations are faster than the ULP transaction they are part of.
Waiting for a completion for these is typically unnecessary.
Infrequently, a network partition or some other problem crops up
where an ordering problem can occur. In NFS parlance, the RPC Reply
arrives and completes the RPC, but the HCA is still retrying the
Send WR that conveyed the RPC Call. In this case, the HCA can try
to use memory that has been invalidated or DMA unmapped, and the
connection is lost. If that memory has been re-used for something
else (possibly not related to NFS), and the Send retransmission
exposes that data on the wire.
Thus we cannot assume that it is safe to release Send-related
resources just because a ULP reply has arrived.
After some analysis, we have determined that the completion
housekeeping will not be difficult for xprtrdma:
- Inline Send buffers are registered via the local DMA key, and
are already left DMA mapped for the lifetime of a transport
connection, thus no additional handling is necessary for those
- Gathered Sends involving page cache pages _will_ need to
DMA unmap those pages after the Send completes. But like
inline send buffers, they are registered via the local DMA key,
and thus will not need to be invalidated
In addition, RPC completion will need to wait for Send completion
in the latter case. However, nearly always, the Send that conveys
the RPC Call will have completed long before the RPC Reply
arrives, and thus no additional latency will be accrued.
Design notes:
In this patch, the rpcrdma_sendctx object is introduced, and a
lock-free circular queue is added to manage a set of them per
transport.
The RPC client's send path already prevents sending more than one
RPC Call at the same time. This allows us to treat the consumer
side of the queue (rpcrdma_sendctx_get_locked) as if there is a
single consumer thread.
The producer side of the queue (rpcrdma_sendctx_put_locked) is
invoked only from the Send completion handler, which is a single
thread of execution (soft IRQ).
The only care that needs to be taken is with the tail index, which
is shared between the producer and consumer. Only the producer
updates the tail index. The consumer compares the head with the
tail to ensure that the a sendctx that is in use is never handed
out again (or, expressed more conventionally, the queue is empty).
When the sendctx queue empties completely, there are enough Sends
outstanding that posting more Send operations can result in a Send
Queue overflow. In this case, the ULP is told to wait and try again.
This introduces strong Send Queue accounting to xprtrdma.
As a final touch, Jason Gunthorpe <jgunthorpe@obsidianresearch.com>
suggested a mechanism that does not require signaling every Send.
We signal once every N Sends, and perform SGE unmapping of N Send
operations during that one completion.
Reported-by: Sagi Grimberg <sagi@grimberg.me>
Suggested-by: Jason Gunthorpe <jgunthorpe@obsidianresearch.com>
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
2017-10-20 22:48:12 +08:00
|
|
|
|
|
|
|
|
rpcrdma_sendctx_put_locked(sc);
|
2014-05-28 22:33:25 +08:00
|
|
|
}
|
2007-09-11 01:51:18 +08:00
|
|
|
|
2016-03-05 00:28:36 +08:00
|
|
|
/**
|
2016-09-15 22:57:49 +08:00
|
|
|
* rpcrdma_wc_receive - Invoked by RDMA provider for each polled Receive WC
|
2016-03-05 00:28:36 +08:00
|
|
|
* @cq: completion queue (ignored)
|
|
|
|
|
* @wc: completed WR
|
|
|
|
|
*
|
|
|
|
|
*/
|
2014-05-28 22:33:25 +08:00
|
|
|
static void
|
2016-09-15 22:57:49 +08:00
|
|
|
rpcrdma_wc_receive(struct ib_cq *cq, struct ib_wc *wc)
|
2014-05-28 22:33:25 +08:00
|
|
|
{
|
2016-03-05 00:28:36 +08:00
|
|
|
struct ib_cqe *cqe = wc->wr_cqe;
|
|
|
|
|
struct rpcrdma_rep *rep = container_of(cqe, struct rpcrdma_rep,
|
|
|
|
|
rr_cqe);
|
2018-12-19 23:58:24 +08:00
|
|
|
struct rpcrdma_xprt *r_xprt = rep->rr_rxprt;
|
2014-05-28 22:33:25 +08:00
|
|
|
|
2018-12-19 23:58:24 +08:00
|
|
|
/* WARNING: Only wr_cqe and status are reliable at this point */
|
2018-05-05 03:35:14 +08:00
|
|
|
trace_xprtrdma_wc_receive(wc);
|
2018-12-19 23:58:24 +08:00
|
|
|
--r_xprt->rx_ep.rep_receive_count;
|
2015-01-22 00:02:04 +08:00
|
|
|
if (wc->status != IB_WC_SUCCESS)
|
2018-12-19 23:58:24 +08:00
|
|
|
goto out_flushed;
|
2014-05-28 22:33:25 +08:00
|
|
|
|
2015-01-22 00:02:04 +08:00
|
|
|
/* status == SUCCESS means all fields in wc are trustworthy */
|
2017-08-04 02:30:03 +08:00
|
|
|
rpcrdma_set_xdrlen(&rep->rr_hdrbuf, wc->byte_len);
|
2016-09-15 22:57:16 +08:00
|
|
|
rep->rr_wc_flags = wc->wc_flags;
|
|
|
|
|
rep->rr_inv_rkey = wc->ex.invalidate_rkey;
|
|
|
|
|
|
2017-04-12 01:23:02 +08:00
|
|
|
ib_dma_sync_single_for_cpu(rdmab_device(rep->rr_rdmabuf),
|
2015-01-22 00:04:25 +08:00
|
|
|
rdmab_addr(rep->rr_rdmabuf),
|
2017-08-04 02:30:44 +08:00
|
|
|
wc->byte_len, DMA_FROM_DEVICE);
|
2016-03-05 00:28:27 +08:00
|
|
|
|
2018-12-19 23:58:24 +08:00
|
|
|
rpcrdma_post_recvs(r_xprt, false);
|
2017-10-17 03:01:30 +08:00
|
|
|
rpcrdma_reply_handler(rep);
|
2015-01-22 00:02:04 +08:00
|
|
|
return;
|
2015-10-25 05:27:10 +08:00
|
|
|
|
2018-12-19 23:58:24 +08:00
|
|
|
out_flushed:
|
2015-01-22 00:02:04 +08:00
|
|
|
if (wc->status != IB_WC_WR_FLUSH_ERR)
|
2016-03-05 00:28:36 +08:00
|
|
|
pr_err("rpcrdma: Recv: %s (%u/0x%x)\n",
|
|
|
|
|
ib_wc_status_msg(wc->status),
|
|
|
|
|
wc->status, wc->vendor_err);
|
2018-12-19 23:58:24 +08:00
|
|
|
rpcrdma_recv_buffer_put(rep);
|
2014-05-28 22:33:25 +08:00
|
|
|
}
|
|
|
|
|
|
2016-09-15 22:57:07 +08:00
|
|
|
static void
|
|
|
|
|
rpcrdma_update_connect_private(struct rpcrdma_xprt *r_xprt,
|
|
|
|
|
struct rdma_conn_param *param)
|
|
|
|
|
{
|
|
|
|
|
struct rpcrdma_create_data_internal *cdata = &r_xprt->rx_data;
|
|
|
|
|
const struct rpcrdma_connect_private *pmsg = param->private_data;
|
|
|
|
|
unsigned int rsize, wsize;
|
|
|
|
|
|
2016-09-15 22:57:16 +08:00
|
|
|
/* Default settings for RPC-over-RDMA Version One */
|
2017-02-09 05:59:54 +08:00
|
|
|
r_xprt->rx_ia.ri_implicit_roundup = xprt_rdma_pad_optimize;
|
2016-09-15 22:57:07 +08:00
|
|
|
rsize = RPCRDMA_V1_DEF_INLINE_SIZE;
|
|
|
|
|
wsize = RPCRDMA_V1_DEF_INLINE_SIZE;
|
|
|
|
|
|
|
|
|
|
if (pmsg &&
|
|
|
|
|
pmsg->cp_magic == rpcrdma_cmp_magic &&
|
|
|
|
|
pmsg->cp_version == RPCRDMA_CMP_VERSION) {
|
2017-02-09 06:00:02 +08:00
|
|
|
r_xprt->rx_ia.ri_implicit_roundup = true;
|
2016-09-15 22:57:07 +08:00
|
|
|
rsize = rpcrdma_decode_buffer_size(pmsg->cp_send_size);
|
|
|
|
|
wsize = rpcrdma_decode_buffer_size(pmsg->cp_recv_size);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
if (rsize < cdata->inline_rsize)
|
|
|
|
|
cdata->inline_rsize = rsize;
|
|
|
|
|
if (wsize < cdata->inline_wsize)
|
|
|
|
|
cdata->inline_wsize = wsize;
|
2016-11-29 23:53:13 +08:00
|
|
|
dprintk("RPC: %s: max send %u, max recv %u\n",
|
|
|
|
|
__func__, cdata->inline_wsize, cdata->inline_rsize);
|
2016-09-15 22:57:07 +08:00
|
|
|
rpcrdma_set_max_header_sizes(r_xprt);
|
|
|
|
|
}
|
|
|
|
|
|
2018-10-02 02:25:47 +08:00
|
|
|
/**
|
|
|
|
|
* rpcrdma_cm_event_handler - Handle RDMA CM events
|
|
|
|
|
* @id: rdma_cm_id on which an event has occurred
|
|
|
|
|
* @event: details of the event
|
|
|
|
|
*
|
|
|
|
|
* Called with @id's mutex held. Returns 1 if caller should
|
|
|
|
|
* destroy @id, otherwise 0.
|
|
|
|
|
*/
|
2007-09-11 01:51:18 +08:00
|
|
|
static int
|
2018-10-02 02:25:47 +08:00
|
|
|
rpcrdma_cm_event_handler(struct rdma_cm_id *id, struct rdma_cm_event *event)
|
2007-09-11 01:51:18 +08:00
|
|
|
{
|
2018-10-02 02:25:52 +08:00
|
|
|
struct rpcrdma_xprt *r_xprt = id->context;
|
|
|
|
|
struct rpcrdma_ia *ia = &r_xprt->rx_ia;
|
|
|
|
|
struct rpcrdma_ep *ep = &r_xprt->rx_ep;
|
|
|
|
|
struct rpc_xprt *xprt = &r_xprt->rx_xprt;
|
2007-09-11 01:51:18 +08:00
|
|
|
|
2018-10-02 02:25:47 +08:00
|
|
|
might_sleep();
|
|
|
|
|
|
2018-10-02 02:25:52 +08:00
|
|
|
trace_xprtrdma_cm_event(r_xprt, event);
|
2007-09-11 01:51:18 +08:00
|
|
|
switch (event->event) {
|
|
|
|
|
case RDMA_CM_EVENT_ADDR_RESOLVED:
|
|
|
|
|
case RDMA_CM_EVENT_ROUTE_RESOLVED:
|
2008-10-10 03:01:41 +08:00
|
|
|
ia->ri_async_rc = 0;
|
2007-09-11 01:51:18 +08:00
|
|
|
complete(&ia->ri_done);
|
2018-10-02 02:26:03 +08:00
|
|
|
return 0;
|
2007-09-11 01:51:18 +08:00
|
|
|
case RDMA_CM_EVENT_ADDR_ERROR:
|
2018-05-05 03:34:37 +08:00
|
|
|
ia->ri_async_rc = -EPROTO;
|
2007-09-11 01:51:18 +08:00
|
|
|
complete(&ia->ri_done);
|
2018-10-02 02:26:03 +08:00
|
|
|
return 0;
|
2007-09-11 01:51:18 +08:00
|
|
|
case RDMA_CM_EVENT_ROUTE_ERROR:
|
|
|
|
|
ia->ri_async_rc = -ENETUNREACH;
|
|
|
|
|
complete(&ia->ri_done);
|
2018-10-02 02:26:03 +08:00
|
|
|
return 0;
|
2017-04-12 01:23:10 +08:00
|
|
|
case RDMA_CM_EVENT_DEVICE_REMOVAL:
|
|
|
|
|
#if IS_ENABLED(CONFIG_SUNRPC_DEBUG)
|
2017-12-15 09:56:50 +08:00
|
|
|
pr_info("rpcrdma: removing device %s for %s:%s\n",
|
2017-06-08 23:53:00 +08:00
|
|
|
ia->ri_device->name,
|
2018-10-02 02:25:52 +08:00
|
|
|
rpcrdma_addrstr(r_xprt), rpcrdma_portstr(r_xprt));
|
2017-04-12 01:23:10 +08:00
|
|
|
#endif
|
|
|
|
|
set_bit(RPCRDMA_IAF_REMOVING, &ia->ri_flags);
|
|
|
|
|
ep->rep_connected = -ENODEV;
|
2018-10-02 02:25:52 +08:00
|
|
|
xprt_force_disconnect(xprt);
|
2017-04-12 01:23:10 +08:00
|
|
|
wait_for_completion(&ia->ri_remove_done);
|
|
|
|
|
|
|
|
|
|
ia->ri_id = NULL;
|
|
|
|
|
ia->ri_device = NULL;
|
|
|
|
|
/* Return 1 to ensure the core destroys the id. */
|
|
|
|
|
return 1;
|
2007-09-11 01:51:18 +08:00
|
|
|
case RDMA_CM_EVENT_ESTABLISHED:
|
2018-10-02 02:25:52 +08:00
|
|
|
++xprt->connect_cookie;
|
2018-10-02 02:25:57 +08:00
|
|
|
ep->rep_connected = 1;
|
2018-10-02 02:25:52 +08:00
|
|
|
rpcrdma_update_connect_private(r_xprt, &event->param.conn);
|
2018-10-02 02:26:08 +08:00
|
|
|
wake_up_all(&ep->rep_connect_wait);
|
|
|
|
|
break;
|
2007-09-11 01:51:18 +08:00
|
|
|
case RDMA_CM_EVENT_CONNECT_ERROR:
|
2018-10-02 02:25:57 +08:00
|
|
|
ep->rep_connected = -ENOTCONN;
|
2018-10-02 02:26:08 +08:00
|
|
|
goto disconnected;
|
2007-09-11 01:51:18 +08:00
|
|
|
case RDMA_CM_EVENT_UNREACHABLE:
|
2018-10-02 02:25:57 +08:00
|
|
|
ep->rep_connected = -ENETUNREACH;
|
2018-10-02 02:26:08 +08:00
|
|
|
goto disconnected;
|
2007-09-11 01:51:18 +08:00
|
|
|
case RDMA_CM_EVENT_REJECTED:
|
2017-12-15 09:56:50 +08:00
|
|
|
dprintk("rpcrdma: connection to %s:%s rejected: %s\n",
|
2018-10-02 02:25:52 +08:00
|
|
|
rpcrdma_addrstr(r_xprt), rpcrdma_portstr(r_xprt),
|
2017-02-09 06:00:35 +08:00
|
|
|
rdma_reject_msg(id, event->status));
|
2018-10-02 02:25:57 +08:00
|
|
|
ep->rep_connected = -ECONNREFUSED;
|
2017-02-09 06:00:35 +08:00
|
|
|
if (event->status == IB_CM_REJ_STALE_CONN)
|
2018-10-02 02:25:57 +08:00
|
|
|
ep->rep_connected = -EAGAIN;
|
2018-10-02 02:26:08 +08:00
|
|
|
goto disconnected;
|
2007-09-11 01:51:18 +08:00
|
|
|
case RDMA_CM_EVENT_DISCONNECTED:
|
2018-10-02 02:25:57 +08:00
|
|
|
ep->rep_connected = -ECONNABORTED;
|
2018-10-02 02:26:08 +08:00
|
|
|
disconnected:
|
|
|
|
|
xprt_force_disconnect(xprt);
|
2007-09-11 01:51:18 +08:00
|
|
|
wake_up_all(&ep->rep_connect_wait);
|
2018-10-02 02:26:03 +08:00
|
|
|
break;
|
2007-09-11 01:51:18 +08:00
|
|
|
default:
|
|
|
|
|
break;
|
|
|
|
|
}
|
|
|
|
|
|
2018-12-19 23:59:01 +08:00
|
|
|
dprintk("RPC: %s: %s:%s on %s/frwr: %s\n", __func__,
|
2018-10-02 02:26:03 +08:00
|
|
|
rpcrdma_addrstr(r_xprt), rpcrdma_portstr(r_xprt),
|
2018-12-19 23:59:01 +08:00
|
|
|
ia->ri_device->name, rdma_event_msg(event->event));
|
2007-09-11 01:51:18 +08:00
|
|
|
return 0;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static struct rdma_cm_id *
|
2017-12-15 09:56:58 +08:00
|
|
|
rpcrdma_create_id(struct rpcrdma_xprt *xprt, struct rpcrdma_ia *ia)
|
2007-09-11 01:51:18 +08:00
|
|
|
{
|
2016-11-29 23:52:40 +08:00
|
|
|
unsigned long wtimeout = msecs_to_jiffies(RDMA_RESOLVE_TIMEOUT) + 1;
|
2007-09-11 01:51:18 +08:00
|
|
|
struct rdma_cm_id *id;
|
|
|
|
|
int rc;
|
|
|
|
|
|
2017-12-21 05:31:29 +08:00
|
|
|
trace_xprtrdma_conn_start(xprt);
|
|
|
|
|
|
2008-10-10 03:01:31 +08:00
|
|
|
init_completion(&ia->ri_done);
|
2017-04-12 01:23:10 +08:00
|
|
|
init_completion(&ia->ri_remove_done);
|
2008-10-10 03:01:31 +08:00
|
|
|
|
2018-10-02 02:25:47 +08:00
|
|
|
id = rdma_create_id(xprt->rx_xprt.xprt_net, rpcrdma_cm_event_handler,
|
2018-05-05 03:34:42 +08:00
|
|
|
xprt, RDMA_PS_TCP, IB_QPT_RC);
|
2018-12-19 23:59:39 +08:00
|
|
|
if (IS_ERR(id))
|
2007-09-11 01:51:18 +08:00
|
|
|
return id;
|
|
|
|
|
|
2008-10-10 03:01:41 +08:00
|
|
|
ia->ri_async_rc = -ETIMEDOUT;
|
2017-12-15 09:56:58 +08:00
|
|
|
rc = rdma_resolve_addr(id, NULL,
|
|
|
|
|
(struct sockaddr *)&xprt->rx_xprt.addr,
|
|
|
|
|
RDMA_RESOLVE_TIMEOUT);
|
2018-12-19 23:59:39 +08:00
|
|
|
if (rc)
|
2007-09-11 01:51:18 +08:00
|
|
|
goto out;
|
2016-11-29 23:52:40 +08:00
|
|
|
rc = wait_for_completion_interruptible_timeout(&ia->ri_done, wtimeout);
|
|
|
|
|
if (rc < 0) {
|
2017-12-21 05:31:29 +08:00
|
|
|
trace_xprtrdma_conn_tout(xprt);
|
2016-11-29 23:52:40 +08:00
|
|
|
goto out;
|
|
|
|
|
}
|
2015-08-04 01:05:04 +08:00
|
|
|
|
2007-09-11 01:51:18 +08:00
|
|
|
rc = ia->ri_async_rc;
|
|
|
|
|
if (rc)
|
|
|
|
|
goto out;
|
|
|
|
|
|
2008-10-10 03:01:41 +08:00
|
|
|
ia->ri_async_rc = -ETIMEDOUT;
|
2007-09-11 01:51:18 +08:00
|
|
|
rc = rdma_resolve_route(id, RDMA_RESOLVE_TIMEOUT);
|
2018-12-19 23:59:39 +08:00
|
|
|
if (rc)
|
2017-04-12 01:23:34 +08:00
|
|
|
goto out;
|
2016-11-29 23:52:40 +08:00
|
|
|
rc = wait_for_completion_interruptible_timeout(&ia->ri_done, wtimeout);
|
|
|
|
|
if (rc < 0) {
|
2017-12-21 05:31:29 +08:00
|
|
|
trace_xprtrdma_conn_tout(xprt);
|
2017-04-12 01:23:34 +08:00
|
|
|
goto out;
|
2016-11-29 23:52:40 +08:00
|
|
|
}
|
2007-09-11 01:51:18 +08:00
|
|
|
rc = ia->ri_async_rc;
|
|
|
|
|
if (rc)
|
2017-04-12 01:23:34 +08:00
|
|
|
goto out;
|
2007-09-11 01:51:18 +08:00
|
|
|
|
|
|
|
|
return id;
|
2017-04-12 01:23:34 +08:00
|
|
|
|
2007-09-11 01:51:18 +08:00
|
|
|
out:
|
|
|
|
|
rdma_destroy_id(id);
|
|
|
|
|
return ERR_PTR(rc);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Exported functions.
|
|
|
|
|
*/
|
|
|
|
|
|
2017-04-12 01:22:54 +08:00
|
|
|
/**
|
|
|
|
|
* rpcrdma_ia_open - Open and initialize an Interface Adapter.
|
2017-12-15 09:56:58 +08:00
|
|
|
* @xprt: transport with IA to (re)initialize
|
2017-04-12 01:22:54 +08:00
|
|
|
*
|
|
|
|
|
* Returns 0 on success, negative errno if an appropriate
|
|
|
|
|
* Interface Adapter could not be found and opened.
|
2007-09-11 01:51:18 +08:00
|
|
|
*/
|
|
|
|
|
int
|
2017-12-15 09:56:58 +08:00
|
|
|
rpcrdma_ia_open(struct rpcrdma_xprt *xprt)
|
2007-09-11 01:51:18 +08:00
|
|
|
{
|
|
|
|
|
struct rpcrdma_ia *ia = &xprt->rx_ia;
|
2015-08-04 01:03:30 +08:00
|
|
|
int rc;
|
|
|
|
|
|
2017-12-15 09:56:58 +08:00
|
|
|
ia->ri_id = rpcrdma_create_id(xprt, ia);
|
2007-09-11 01:51:18 +08:00
|
|
|
if (IS_ERR(ia->ri_id)) {
|
|
|
|
|
rc = PTR_ERR(ia->ri_id);
|
2017-04-12 01:22:54 +08:00
|
|
|
goto out_err;
|
2007-09-11 01:51:18 +08:00
|
|
|
}
|
2015-05-26 23:51:56 +08:00
|
|
|
ia->ri_device = ia->ri_id->device;
|
2007-09-11 01:51:18 +08:00
|
|
|
|
2016-09-05 18:56:17 +08:00
|
|
|
ia->ri_pd = ib_alloc_pd(ia->ri_device, 0);
|
2007-09-11 01:51:18 +08:00
|
|
|
if (IS_ERR(ia->ri_pd)) {
|
|
|
|
|
rc = PTR_ERR(ia->ri_pd);
|
2016-06-30 01:53:27 +08:00
|
|
|
pr_err("rpcrdma: ib_alloc_pd() returned %d\n", rc);
|
2017-04-12 01:22:54 +08:00
|
|
|
goto out_err;
|
2007-09-11 01:51:18 +08:00
|
|
|
}
|
|
|
|
|
|
2017-04-12 01:22:54 +08:00
|
|
|
switch (xprt_rdma_memreg_strategy) {
|
2017-12-15 09:57:47 +08:00
|
|
|
case RPCRDMA_FRWR:
|
2018-12-19 23:59:01 +08:00
|
|
|
if (frwr_is_supported(ia))
|
2016-06-30 01:53:27 +08:00
|
|
|
break;
|
|
|
|
|
/*FALLTHROUGH*/
|
2008-10-10 03:00:09 +08:00
|
|
|
default:
|
2017-04-12 01:22:54 +08:00
|
|
|
pr_err("rpcrdma: Device %s does not support memreg mode %d\n",
|
|
|
|
|
ia->ri_device->name, xprt_rdma_memreg_strategy);
|
2016-06-30 01:53:27 +08:00
|
|
|
rc = -EINVAL;
|
2017-04-12 01:22:54 +08:00
|
|
|
goto out_err;
|
2007-09-11 01:51:18 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
return 0;
|
2015-01-22 00:03:19 +08:00
|
|
|
|
2017-04-12 01:22:54 +08:00
|
|
|
out_err:
|
|
|
|
|
rpcrdma_ia_close(ia);
|
2007-09-11 01:51:18 +08:00
|
|
|
return rc;
|
|
|
|
|
}
|
|
|
|
|
|
2017-04-12 01:23:10 +08:00
|
|
|
/**
|
|
|
|
|
* rpcrdma_ia_remove - Handle device driver unload
|
|
|
|
|
* @ia: interface adapter being removed
|
|
|
|
|
*
|
|
|
|
|
* Divest transport H/W resources associated with this adapter,
|
|
|
|
|
* but allow it to be restored later.
|
|
|
|
|
*/
|
|
|
|
|
void
|
|
|
|
|
rpcrdma_ia_remove(struct rpcrdma_ia *ia)
|
|
|
|
|
{
|
|
|
|
|
struct rpcrdma_xprt *r_xprt = container_of(ia, struct rpcrdma_xprt,
|
|
|
|
|
rx_ia);
|
|
|
|
|
struct rpcrdma_ep *ep = &r_xprt->rx_ep;
|
|
|
|
|
struct rpcrdma_buffer *buf = &r_xprt->rx_buf;
|
|
|
|
|
struct rpcrdma_req *req;
|
|
|
|
|
struct rpcrdma_rep *rep;
|
|
|
|
|
|
|
|
|
|
cancel_delayed_work_sync(&buf->rb_refresh_worker);
|
|
|
|
|
|
|
|
|
|
/* This is similar to rpcrdma_ep_destroy, but:
|
|
|
|
|
* - Don't cancel the connect worker.
|
|
|
|
|
* - Don't call rpcrdma_ep_disconnect, which waits
|
|
|
|
|
* for another conn upcall, which will deadlock.
|
|
|
|
|
* - rdma_disconnect is unneeded, the underlying
|
|
|
|
|
* connection is already gone.
|
|
|
|
|
*/
|
|
|
|
|
if (ia->ri_id->qp) {
|
2018-12-19 23:58:29 +08:00
|
|
|
rpcrdma_xprt_drain(r_xprt);
|
2017-04-12 01:23:10 +08:00
|
|
|
rdma_destroy_qp(ia->ri_id);
|
|
|
|
|
ia->ri_id->qp = NULL;
|
|
|
|
|
}
|
|
|
|
|
ib_free_cq(ep->rep_attr.recv_cq);
|
2018-03-20 02:23:16 +08:00
|
|
|
ep->rep_attr.recv_cq = NULL;
|
2017-04-12 01:23:10 +08:00
|
|
|
ib_free_cq(ep->rep_attr.send_cq);
|
2018-03-20 02:23:16 +08:00
|
|
|
ep->rep_attr.send_cq = NULL;
|
2017-04-12 01:23:10 +08:00
|
|
|
|
|
|
|
|
/* The ULP is responsible for ensuring all DMA
|
|
|
|
|
* mappings and MRs are gone.
|
|
|
|
|
*/
|
|
|
|
|
list_for_each_entry(rep, &buf->rb_recv_bufs, rr_list)
|
|
|
|
|
rpcrdma_dma_unmap_regbuf(rep->rr_rdmabuf);
|
|
|
|
|
list_for_each_entry(req, &buf->rb_allreqs, rl_all) {
|
|
|
|
|
rpcrdma_dma_unmap_regbuf(req->rl_rdmabuf);
|
|
|
|
|
rpcrdma_dma_unmap_regbuf(req->rl_sendbuf);
|
|
|
|
|
rpcrdma_dma_unmap_regbuf(req->rl_recvbuf);
|
|
|
|
|
}
|
2017-12-15 09:57:55 +08:00
|
|
|
rpcrdma_mrs_destroy(buf);
|
2018-03-20 02:23:16 +08:00
|
|
|
ib_dealloc_pd(ia->ri_pd);
|
|
|
|
|
ia->ri_pd = NULL;
|
2017-04-12 01:23:10 +08:00
|
|
|
|
|
|
|
|
/* Allow waiters to continue */
|
|
|
|
|
complete(&ia->ri_remove_done);
|
2017-12-21 05:31:29 +08:00
|
|
|
|
|
|
|
|
trace_xprtrdma_remove(r_xprt);
|
2017-04-12 01:23:10 +08:00
|
|
|
}
|
|
|
|
|
|
2017-04-12 01:22:54 +08:00
|
|
|
/**
|
|
|
|
|
* rpcrdma_ia_close - Clean up/close an IA.
|
|
|
|
|
* @ia: interface adapter to close
|
|
|
|
|
*
|
2007-09-11 01:51:18 +08:00
|
|
|
*/
|
|
|
|
|
void
|
|
|
|
|
rpcrdma_ia_close(struct rpcrdma_ia *ia)
|
|
|
|
|
{
|
2008-10-10 03:01:00 +08:00
|
|
|
if (ia->ri_id != NULL && !IS_ERR(ia->ri_id)) {
|
|
|
|
|
if (ia->ri_id->qp)
|
|
|
|
|
rdma_destroy_qp(ia->ri_id);
|
2017-04-12 01:23:34 +08:00
|
|
|
rdma_destroy_id(ia->ri_id);
|
2008-10-10 03:01:00 +08:00
|
|
|
}
|
2017-04-12 01:22:54 +08:00
|
|
|
ia->ri_id = NULL;
|
|
|
|
|
ia->ri_device = NULL;
|
2015-05-26 23:51:27 +08:00
|
|
|
|
|
|
|
|
/* If the pd is still busy, xprtrdma missed freeing a resource */
|
|
|
|
|
if (ia->ri_pd && !IS_ERR(ia->ri_pd))
|
2015-08-06 04:34:31 +08:00
|
|
|
ib_dealloc_pd(ia->ri_pd);
|
2017-04-12 01:22:54 +08:00
|
|
|
ia->ri_pd = NULL;
|
2007-09-11 01:51:18 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Create unconnected endpoint.
|
|
|
|
|
*/
|
|
|
|
|
int
|
|
|
|
|
rpcrdma_ep_create(struct rpcrdma_ep *ep, struct rpcrdma_ia *ia,
|
2017-02-09 06:00:10 +08:00
|
|
|
struct rpcrdma_create_data_internal *cdata)
|
2007-09-11 01:51:18 +08:00
|
|
|
{
|
2016-09-15 22:57:07 +08:00
|
|
|
struct rpcrdma_connect_private *pmsg = &ep->rep_cm_private;
|
2014-05-28 22:33:25 +08:00
|
|
|
struct ib_cq *sendcq, *recvcq;
|
2018-05-05 03:34:48 +08:00
|
|
|
unsigned int max_sge;
|
2016-03-05 00:28:53 +08:00
|
|
|
int rc;
|
2007-09-11 01:51:18 +08:00
|
|
|
|
2018-06-18 23:05:26 +08:00
|
|
|
max_sge = min_t(unsigned int, ia->ri_device->attrs.max_send_sge,
|
2017-03-12 04:52:47 +08:00
|
|
|
RPCRDMA_MAX_SEND_SGES);
|
2017-02-09 06:00:10 +08:00
|
|
|
if (max_sge < RPCRDMA_MIN_SEND_SGES) {
|
|
|
|
|
pr_warn("rpcrdma: HCA provides only %d send SGEs\n", max_sge);
|
2015-08-04 01:03:39 +08:00
|
|
|
return -ENOMEM;
|
|
|
|
|
}
|
xprtrdma: Fix calculation of ri_max_send_sges
Commit 16f906d66cd7 ("xprtrdma: Reduce required number of send
SGEs") introduced the rpcrdma_ia::ri_max_send_sges field. This fixes
a problem where xprtrdma would not work if the device's max_sge
capability was small (low single digits).
At least RPCRDMA_MIN_SEND_SGES are needed for the inline parts of
each RPC. ri_max_send_sges is set to this value:
ia->ri_max_send_sges = max_sge - RPCRDMA_MIN_SEND_SGES;
Then when marshaling each RPC, rpcrdma_args_inline uses that value
to determine whether the device has enough Send SGEs to convey an
NFS WRITE payload inline, or whether instead a Read chunk is
required.
More recently, commit ae72950abf99 ("xprtrdma: Add data structure to
manage RDMA Send arguments") used the ri_max_send_sges value to
calculate the size of an array, but that commit erroneously assumed
ri_max_send_sges contains a value similar to the device's max_sge,
and not one that was reduced by the minimum SGE count.
This assumption results in the calculated size of the sendctx's
Send SGE array to be too small. When the array is used to marshal
an RPC, the code can write Send SGEs into the following sendctx
element in that array, corrupting it. When the device's max_sge is
large, this issue is entirely harmless; but it results in an oops
in the provider's post_send method, if dev.attrs.max_sge is small.
So let's straighten this out: ri_max_send_sges will now contain a
value with the same meaning as dev.attrs.max_sge, which makes
the code easier to understand, and enables rpcrdma_sendctx_create
to calculate the size of the SGE array correctly.
Reported-by: Michal Kalderon <Michal.Kalderon@cavium.com>
Fixes: 16f906d66cd7 ("xprtrdma: Reduce required number of send SGEs")
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
Tested-by: Michal Kalderon <Michal.Kalderon@cavium.com>
Cc: stable@vger.kernel.org # v4.10+
Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
2018-02-01 01:34:05 +08:00
|
|
|
ia->ri_max_send_sges = max_sge;
|
2015-08-04 01:03:39 +08:00
|
|
|
|
2018-12-19 23:59:01 +08:00
|
|
|
rc = frwr_open(ia, ep, cdata);
|
2018-05-05 03:34:48 +08:00
|
|
|
if (rc)
|
|
|
|
|
return rc;
|
2007-09-11 01:51:18 +08:00
|
|
|
|
2018-10-02 02:26:13 +08:00
|
|
|
ep->rep_attr.event_handler = rpcrdma_qp_event_handler;
|
2007-09-11 01:51:18 +08:00
|
|
|
ep->rep_attr.qp_context = ep;
|
|
|
|
|
ep->rep_attr.srq = NULL;
|
2017-02-09 06:00:10 +08:00
|
|
|
ep->rep_attr.cap.max_send_sge = max_sge;
|
2007-09-11 01:51:18 +08:00
|
|
|
ep->rep_attr.cap.max_recv_sge = 1;
|
|
|
|
|
ep->rep_attr.cap.max_inline_data = 0;
|
|
|
|
|
ep->rep_attr.sq_sig_type = IB_SIGNAL_REQ_WR;
|
|
|
|
|
ep->rep_attr.qp_type = IB_QPT_RC;
|
|
|
|
|
ep->rep_attr.port_num = ~0;
|
|
|
|
|
|
|
|
|
|
dprintk("RPC: %s: requested max: dtos: send %d recv %d; "
|
|
|
|
|
"iovs: send %d recv %d\n",
|
|
|
|
|
__func__,
|
|
|
|
|
ep->rep_attr.cap.max_send_wr,
|
|
|
|
|
ep->rep_attr.cap.max_recv_wr,
|
|
|
|
|
ep->rep_attr.cap.max_send_sge,
|
|
|
|
|
ep->rep_attr.cap.max_recv_sge);
|
|
|
|
|
|
|
|
|
|
/* set trigger for requesting send completion */
|
xprtrdma: Add data structure to manage RDMA Send arguments
Problem statement:
Recently Sagi Grimberg <sagi@grimberg.me> observed that kernel RDMA-
enabled storage initiators don't handle delayed Send completion
correctly. If Send completion is delayed beyond the end of a ULP
transaction, the ULP may release resources that are still being used
by the HCA to complete a long-running Send operation.
This is a common design trait amongst our initiators. Most Send
operations are faster than the ULP transaction they are part of.
Waiting for a completion for these is typically unnecessary.
Infrequently, a network partition or some other problem crops up
where an ordering problem can occur. In NFS parlance, the RPC Reply
arrives and completes the RPC, but the HCA is still retrying the
Send WR that conveyed the RPC Call. In this case, the HCA can try
to use memory that has been invalidated or DMA unmapped, and the
connection is lost. If that memory has been re-used for something
else (possibly not related to NFS), and the Send retransmission
exposes that data on the wire.
Thus we cannot assume that it is safe to release Send-related
resources just because a ULP reply has arrived.
After some analysis, we have determined that the completion
housekeeping will not be difficult for xprtrdma:
- Inline Send buffers are registered via the local DMA key, and
are already left DMA mapped for the lifetime of a transport
connection, thus no additional handling is necessary for those
- Gathered Sends involving page cache pages _will_ need to
DMA unmap those pages after the Send completes. But like
inline send buffers, they are registered via the local DMA key,
and thus will not need to be invalidated
In addition, RPC completion will need to wait for Send completion
in the latter case. However, nearly always, the Send that conveys
the RPC Call will have completed long before the RPC Reply
arrives, and thus no additional latency will be accrued.
Design notes:
In this patch, the rpcrdma_sendctx object is introduced, and a
lock-free circular queue is added to manage a set of them per
transport.
The RPC client's send path already prevents sending more than one
RPC Call at the same time. This allows us to treat the consumer
side of the queue (rpcrdma_sendctx_get_locked) as if there is a
single consumer thread.
The producer side of the queue (rpcrdma_sendctx_put_locked) is
invoked only from the Send completion handler, which is a single
thread of execution (soft IRQ).
The only care that needs to be taken is with the tail index, which
is shared between the producer and consumer. Only the producer
updates the tail index. The consumer compares the head with the
tail to ensure that the a sendctx that is in use is never handed
out again (or, expressed more conventionally, the queue is empty).
When the sendctx queue empties completely, there are enough Sends
outstanding that posting more Send operations can result in a Send
Queue overflow. In this case, the ULP is told to wait and try again.
This introduces strong Send Queue accounting to xprtrdma.
As a final touch, Jason Gunthorpe <jgunthorpe@obsidianresearch.com>
suggested a mechanism that does not require signaling every Send.
We signal once every N Sends, and perform SGE unmapping of N Send
operations during that one completion.
Reported-by: Sagi Grimberg <sagi@grimberg.me>
Suggested-by: Jason Gunthorpe <jgunthorpe@obsidianresearch.com>
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
2017-10-20 22:48:12 +08:00
|
|
|
ep->rep_send_batch = min_t(unsigned int, RPCRDMA_MAX_SEND_BATCH,
|
|
|
|
|
cdata->max_requests >> 2);
|
|
|
|
|
ep->rep_send_count = ep->rep_send_batch;
|
2007-09-11 01:51:18 +08:00
|
|
|
init_waitqueue_head(&ep->rep_connect_wait);
|
2018-12-19 23:58:24 +08:00
|
|
|
ep->rep_receive_count = 0;
|
2007-09-11 01:51:18 +08:00
|
|
|
|
2016-03-05 00:28:53 +08:00
|
|
|
sendcq = ib_alloc_cq(ia->ri_device, NULL,
|
|
|
|
|
ep->rep_attr.cap.max_send_wr + 1,
|
2017-10-31 04:21:49 +08:00
|
|
|
1, IB_POLL_WORKQUEUE);
|
2014-05-28 22:33:25 +08:00
|
|
|
if (IS_ERR(sendcq)) {
|
|
|
|
|
rc = PTR_ERR(sendcq);
|
2007-09-11 01:51:18 +08:00
|
|
|
goto out1;
|
|
|
|
|
}
|
|
|
|
|
|
2016-03-05 00:28:36 +08:00
|
|
|
recvcq = ib_alloc_cq(ia->ri_device, NULL,
|
|
|
|
|
ep->rep_attr.cap.max_recv_wr + 1,
|
2017-10-17 03:01:30 +08:00
|
|
|
0, IB_POLL_WORKQUEUE);
|
2014-05-28 22:33:25 +08:00
|
|
|
if (IS_ERR(recvcq)) {
|
|
|
|
|
rc = PTR_ERR(recvcq);
|
|
|
|
|
goto out2;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
ep->rep_attr.send_cq = sendcq;
|
|
|
|
|
ep->rep_attr.recv_cq = recvcq;
|
2007-09-11 01:51:18 +08:00
|
|
|
|
|
|
|
|
/* Initialize cma parameters */
|
2016-05-03 02:43:03 +08:00
|
|
|
memset(&ep->rep_remote_cma, 0, sizeof(ep->rep_remote_cma));
|
2007-09-11 01:51:18 +08:00
|
|
|
|
2016-09-15 22:57:07 +08:00
|
|
|
/* Prepare RDMA-CM private message */
|
|
|
|
|
pmsg->cp_magic = rpcrdma_cmp_magic;
|
|
|
|
|
pmsg->cp_version = RPCRDMA_CMP_VERSION;
|
2018-12-19 23:59:01 +08:00
|
|
|
pmsg->cp_flags |= RPCRDMA_CMP_F_SND_W_INV_OK;
|
2016-09-15 22:57:07 +08:00
|
|
|
pmsg->cp_send_size = rpcrdma_encode_buffer_size(cdata->inline_wsize);
|
|
|
|
|
pmsg->cp_recv_size = rpcrdma_encode_buffer_size(cdata->inline_rsize);
|
|
|
|
|
ep->rep_remote_cma.private_data = pmsg;
|
|
|
|
|
ep->rep_remote_cma.private_data_len = sizeof(*pmsg);
|
2007-09-11 01:51:18 +08:00
|
|
|
|
|
|
|
|
/* Client offers RDMA Read but does not initiate */
|
2008-10-10 03:00:30 +08:00
|
|
|
ep->rep_remote_cma.initiator_depth = 0;
|
2018-03-01 04:30:33 +08:00
|
|
|
ep->rep_remote_cma.responder_resources =
|
|
|
|
|
min_t(int, U8_MAX, ia->ri_device->attrs.max_qp_rd_atom);
|
2007-09-11 01:51:18 +08:00
|
|
|
|
2016-05-03 02:43:03 +08:00
|
|
|
/* Limit transport retries so client can detect server
|
|
|
|
|
* GID changes quickly. RPC layer handles re-establishing
|
|
|
|
|
* transport connection and retransmission.
|
|
|
|
|
*/
|
|
|
|
|
ep->rep_remote_cma.retry_count = 6;
|
|
|
|
|
|
|
|
|
|
/* RPC-over-RDMA handles its own flow control. In addition,
|
|
|
|
|
* make all RNR NAKs visible so we know that RPC-over-RDMA
|
|
|
|
|
* flow control is working correctly (no NAKs should be seen).
|
|
|
|
|
*/
|
2007-09-11 01:51:18 +08:00
|
|
|
ep->rep_remote_cma.flow_control = 0;
|
|
|
|
|
ep->rep_remote_cma.rnr_retry_count = 0;
|
|
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
|
|
out2:
|
2016-03-05 00:28:53 +08:00
|
|
|
ib_free_cq(sendcq);
|
2007-09-11 01:51:18 +08:00
|
|
|
out1:
|
|
|
|
|
return rc;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* rpcrdma_ep_destroy
|
|
|
|
|
*
|
|
|
|
|
* Disconnect and destroy endpoint. After this, the only
|
|
|
|
|
* valid operations on the ep are to free it (if dynamically
|
|
|
|
|
* allocated) or re-create it.
|
|
|
|
|
*/
|
2014-05-28 22:33:16 +08:00
|
|
|
void
|
2007-09-11 01:51:18 +08:00
|
|
|
rpcrdma_ep_destroy(struct rpcrdma_ep *ep, struct rpcrdma_ia *ia)
|
|
|
|
|
{
|
2018-03-20 02:23:16 +08:00
|
|
|
if (ia->ri_id && ia->ri_id->qp) {
|
2016-05-03 02:41:47 +08:00
|
|
|
rpcrdma_ep_disconnect(ep, ia);
|
2008-10-10 03:01:00 +08:00
|
|
|
rdma_destroy_qp(ia->ri_id);
|
|
|
|
|
ia->ri_id->qp = NULL;
|
2007-09-11 01:51:18 +08:00
|
|
|
}
|
|
|
|
|
|
2018-03-20 02:23:16 +08:00
|
|
|
if (ep->rep_attr.recv_cq)
|
|
|
|
|
ib_free_cq(ep->rep_attr.recv_cq);
|
|
|
|
|
if (ep->rep_attr.send_cq)
|
|
|
|
|
ib_free_cq(ep->rep_attr.send_cq);
|
2007-09-11 01:51:18 +08:00
|
|
|
}
|
|
|
|
|
|
2017-04-12 01:23:26 +08:00
|
|
|
/* Re-establish a connection after a device removal event.
|
|
|
|
|
* Unlike a normal reconnection, a fresh PD and a new set
|
|
|
|
|
* of MRs and buffers is needed.
|
|
|
|
|
*/
|
|
|
|
|
static int
|
|
|
|
|
rpcrdma_ep_recreate_xprt(struct rpcrdma_xprt *r_xprt,
|
|
|
|
|
struct rpcrdma_ep *ep, struct rpcrdma_ia *ia)
|
|
|
|
|
{
|
|
|
|
|
int rc, err;
|
|
|
|
|
|
2017-12-21 05:31:29 +08:00
|
|
|
trace_xprtrdma_reinsert(r_xprt);
|
2017-04-12 01:23:26 +08:00
|
|
|
|
|
|
|
|
rc = -EHOSTUNREACH;
|
2017-12-15 09:56:58 +08:00
|
|
|
if (rpcrdma_ia_open(r_xprt))
|
2017-04-12 01:23:26 +08:00
|
|
|
goto out1;
|
|
|
|
|
|
|
|
|
|
rc = -ENOMEM;
|
|
|
|
|
err = rpcrdma_ep_create(ep, ia, &r_xprt->rx_data);
|
|
|
|
|
if (err) {
|
|
|
|
|
pr_err("rpcrdma: rpcrdma_ep_create returned %d\n", err);
|
|
|
|
|
goto out2;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
rc = -ENETUNREACH;
|
|
|
|
|
err = rdma_create_qp(ia->ri_id, ia->ri_pd, &ep->rep_attr);
|
|
|
|
|
if (err) {
|
|
|
|
|
pr_err("rpcrdma: rdma_create_qp returned %d\n", err);
|
|
|
|
|
goto out3;
|
|
|
|
|
}
|
|
|
|
|
|
2017-12-15 09:57:55 +08:00
|
|
|
rpcrdma_mrs_create(r_xprt);
|
2017-04-12 01:23:26 +08:00
|
|
|
return 0;
|
|
|
|
|
|
|
|
|
|
out3:
|
|
|
|
|
rpcrdma_ep_destroy(ep, ia);
|
|
|
|
|
out2:
|
|
|
|
|
rpcrdma_ia_close(ia);
|
|
|
|
|
out1:
|
|
|
|
|
return rc;
|
|
|
|
|
}
|
|
|
|
|
|
2017-04-12 01:23:18 +08:00
|
|
|
static int
|
|
|
|
|
rpcrdma_ep_reconnect(struct rpcrdma_xprt *r_xprt, struct rpcrdma_ep *ep,
|
|
|
|
|
struct rpcrdma_ia *ia)
|
|
|
|
|
{
|
|
|
|
|
struct rdma_cm_id *id, *old;
|
|
|
|
|
int err, rc;
|
|
|
|
|
|
2017-12-21 05:31:29 +08:00
|
|
|
trace_xprtrdma_reconnect(r_xprt);
|
2017-04-12 01:23:18 +08:00
|
|
|
|
|
|
|
|
rpcrdma_ep_disconnect(ep, ia);
|
|
|
|
|
|
|
|
|
|
rc = -EHOSTUNREACH;
|
2017-12-15 09:56:58 +08:00
|
|
|
id = rpcrdma_create_id(r_xprt, ia);
|
2017-04-12 01:23:18 +08:00
|
|
|
if (IS_ERR(id))
|
|
|
|
|
goto out;
|
|
|
|
|
|
|
|
|
|
/* As long as the new ID points to the same device as the
|
|
|
|
|
* old ID, we can reuse the transport's existing PD and all
|
|
|
|
|
* previously allocated MRs. Also, the same device means
|
|
|
|
|
* the transport's previous DMA mappings are still valid.
|
|
|
|
|
*
|
|
|
|
|
* This is a sanity check only. There should be no way these
|
|
|
|
|
* point to two different devices here.
|
|
|
|
|
*/
|
|
|
|
|
old = id;
|
|
|
|
|
rc = -ENETUNREACH;
|
|
|
|
|
if (ia->ri_device != id->device) {
|
|
|
|
|
pr_err("rpcrdma: can't reconnect on different device!\n");
|
|
|
|
|
goto out_destroy;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
err = rdma_create_qp(id, ia->ri_pd, &ep->rep_attr);
|
2018-12-19 23:59:39 +08:00
|
|
|
if (err)
|
2017-04-12 01:23:18 +08:00
|
|
|
goto out_destroy;
|
|
|
|
|
|
|
|
|
|
/* Atomically replace the transport's ID and QP. */
|
|
|
|
|
rc = 0;
|
|
|
|
|
old = ia->ri_id;
|
|
|
|
|
ia->ri_id = id;
|
|
|
|
|
rdma_destroy_qp(old);
|
|
|
|
|
|
|
|
|
|
out_destroy:
|
2017-04-12 01:23:34 +08:00
|
|
|
rdma_destroy_id(old);
|
2017-04-12 01:23:18 +08:00
|
|
|
out:
|
|
|
|
|
return rc;
|
|
|
|
|
}
|
|
|
|
|
|
2007-09-11 01:51:18 +08:00
|
|
|
/*
|
|
|
|
|
* Connect unconnected endpoint.
|
|
|
|
|
*/
|
|
|
|
|
int
|
|
|
|
|
rpcrdma_ep_connect(struct rpcrdma_ep *ep, struct rpcrdma_ia *ia)
|
|
|
|
|
{
|
2017-02-09 06:00:35 +08:00
|
|
|
struct rpcrdma_xprt *r_xprt = container_of(ia, struct rpcrdma_xprt,
|
|
|
|
|
rx_ia);
|
2018-10-02 02:26:08 +08:00
|
|
|
struct rpc_xprt *xprt = &r_xprt->rx_xprt;
|
2017-04-12 01:23:18 +08:00
|
|
|
int rc;
|
2007-09-11 01:51:18 +08:00
|
|
|
|
|
|
|
|
retry:
|
2017-04-12 01:23:18 +08:00
|
|
|
switch (ep->rep_connected) {
|
|
|
|
|
case 0:
|
2014-05-28 22:34:07 +08:00
|
|
|
dprintk("RPC: %s: connecting...\n", __func__);
|
|
|
|
|
rc = rdma_create_qp(ia->ri_id, ia->ri_pd, &ep->rep_attr);
|
|
|
|
|
if (rc) {
|
2017-04-12 01:23:18 +08:00
|
|
|
rc = -ENETUNREACH;
|
|
|
|
|
goto out_noupdate;
|
2014-05-28 22:34:07 +08:00
|
|
|
}
|
2017-04-12 01:23:18 +08:00
|
|
|
break;
|
2017-04-12 01:23:26 +08:00
|
|
|
case -ENODEV:
|
|
|
|
|
rc = rpcrdma_ep_recreate_xprt(r_xprt, ep, ia);
|
|
|
|
|
if (rc)
|
|
|
|
|
goto out_noupdate;
|
|
|
|
|
break;
|
2017-04-12 01:23:18 +08:00
|
|
|
default:
|
|
|
|
|
rc = rpcrdma_ep_reconnect(r_xprt, ep, ia);
|
|
|
|
|
if (rc)
|
|
|
|
|
goto out;
|
2007-09-11 01:51:18 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
ep->rep_connected = 0;
|
2018-10-02 02:26:08 +08:00
|
|
|
xprt_clear_connected(xprt);
|
|
|
|
|
|
2018-07-28 22:46:47 +08:00
|
|
|
rpcrdma_post_recvs(r_xprt, true);
|
2007-09-11 01:51:18 +08:00
|
|
|
|
|
|
|
|
rc = rdma_connect(ia->ri_id, &ep->rep_remote_cma);
|
2018-12-19 23:59:39 +08:00
|
|
|
if (rc)
|
2007-09-11 01:51:18 +08:00
|
|
|
goto out;
|
|
|
|
|
|
|
|
|
|
wait_event_interruptible(ep->rep_connect_wait, ep->rep_connected != 0);
|
|
|
|
|
if (ep->rep_connected <= 0) {
|
2017-02-09 06:00:35 +08:00
|
|
|
if (ep->rep_connected == -EAGAIN)
|
2007-09-11 01:51:18 +08:00
|
|
|
goto retry;
|
|
|
|
|
rc = ep->rep_connected;
|
2017-02-09 06:00:35 +08:00
|
|
|
goto out;
|
2007-09-11 01:51:18 +08:00
|
|
|
}
|
|
|
|
|
|
2017-02-09 06:00:35 +08:00
|
|
|
dprintk("RPC: %s: connected\n", __func__);
|
2018-05-05 03:35:20 +08:00
|
|
|
|
2007-09-11 01:51:18 +08:00
|
|
|
out:
|
|
|
|
|
if (rc)
|
|
|
|
|
ep->rep_connected = rc;
|
2017-04-12 01:23:18 +08:00
|
|
|
|
|
|
|
|
out_noupdate:
|
2007-09-11 01:51:18 +08:00
|
|
|
return rc;
|
|
|
|
|
}
|
|
|
|
|
|
2018-12-19 23:58:29 +08:00
|
|
|
/**
|
|
|
|
|
* rpcrdma_ep_disconnect - Disconnect underlying transport
|
|
|
|
|
* @ep: endpoint to disconnect
|
|
|
|
|
* @ia: associated interface adapter
|
2007-09-11 01:51:18 +08:00
|
|
|
*
|
|
|
|
|
* This is separate from destroy to facilitate the ability
|
|
|
|
|
* to reconnect without recreating the endpoint.
|
|
|
|
|
*
|
|
|
|
|
* This call is not reentrant, and must not be made in parallel
|
|
|
|
|
* on the same endpoint.
|
|
|
|
|
*/
|
2014-07-30 05:25:55 +08:00
|
|
|
void
|
2007-09-11 01:51:18 +08:00
|
|
|
rpcrdma_ep_disconnect(struct rpcrdma_ep *ep, struct rpcrdma_ia *ia)
|
|
|
|
|
{
|
2018-12-19 23:58:29 +08:00
|
|
|
struct rpcrdma_xprt *r_xprt = container_of(ep, struct rpcrdma_xprt,
|
|
|
|
|
rx_ep);
|
2007-09-11 01:51:18 +08:00
|
|
|
int rc;
|
|
|
|
|
|
2018-12-19 23:58:29 +08:00
|
|
|
/* returns without wait if ID is not connected */
|
2007-09-11 01:51:18 +08:00
|
|
|
rc = rdma_disconnect(ia->ri_id);
|
2017-12-21 05:31:29 +08:00
|
|
|
if (!rc)
|
2007-09-11 01:51:18 +08:00
|
|
|
wait_event_interruptible(ep->rep_connect_wait,
|
|
|
|
|
ep->rep_connected != 1);
|
2017-12-21 05:31:29 +08:00
|
|
|
else
|
2007-09-11 01:51:18 +08:00
|
|
|
ep->rep_connected = rc;
|
2018-12-19 23:58:29 +08:00
|
|
|
trace_xprtrdma_disconnect(r_xprt, rc);
|
2016-05-03 02:41:47 +08:00
|
|
|
|
2018-12-19 23:58:29 +08:00
|
|
|
rpcrdma_xprt_drain(r_xprt);
|
2007-09-11 01:51:18 +08:00
|
|
|
}
|
|
|
|
|
|
xprtrdma: Add data structure to manage RDMA Send arguments
Problem statement:
Recently Sagi Grimberg <sagi@grimberg.me> observed that kernel RDMA-
enabled storage initiators don't handle delayed Send completion
correctly. If Send completion is delayed beyond the end of a ULP
transaction, the ULP may release resources that are still being used
by the HCA to complete a long-running Send operation.
This is a common design trait amongst our initiators. Most Send
operations are faster than the ULP transaction they are part of.
Waiting for a completion for these is typically unnecessary.
Infrequently, a network partition or some other problem crops up
where an ordering problem can occur. In NFS parlance, the RPC Reply
arrives and completes the RPC, but the HCA is still retrying the
Send WR that conveyed the RPC Call. In this case, the HCA can try
to use memory that has been invalidated or DMA unmapped, and the
connection is lost. If that memory has been re-used for something
else (possibly not related to NFS), and the Send retransmission
exposes that data on the wire.
Thus we cannot assume that it is safe to release Send-related
resources just because a ULP reply has arrived.
After some analysis, we have determined that the completion
housekeeping will not be difficult for xprtrdma:
- Inline Send buffers are registered via the local DMA key, and
are already left DMA mapped for the lifetime of a transport
connection, thus no additional handling is necessary for those
- Gathered Sends involving page cache pages _will_ need to
DMA unmap those pages after the Send completes. But like
inline send buffers, they are registered via the local DMA key,
and thus will not need to be invalidated
In addition, RPC completion will need to wait for Send completion
in the latter case. However, nearly always, the Send that conveys
the RPC Call will have completed long before the RPC Reply
arrives, and thus no additional latency will be accrued.
Design notes:
In this patch, the rpcrdma_sendctx object is introduced, and a
lock-free circular queue is added to manage a set of them per
transport.
The RPC client's send path already prevents sending more than one
RPC Call at the same time. This allows us to treat the consumer
side of the queue (rpcrdma_sendctx_get_locked) as if there is a
single consumer thread.
The producer side of the queue (rpcrdma_sendctx_put_locked) is
invoked only from the Send completion handler, which is a single
thread of execution (soft IRQ).
The only care that needs to be taken is with the tail index, which
is shared between the producer and consumer. Only the producer
updates the tail index. The consumer compares the head with the
tail to ensure that the a sendctx that is in use is never handed
out again (or, expressed more conventionally, the queue is empty).
When the sendctx queue empties completely, there are enough Sends
outstanding that posting more Send operations can result in a Send
Queue overflow. In this case, the ULP is told to wait and try again.
This introduces strong Send Queue accounting to xprtrdma.
As a final touch, Jason Gunthorpe <jgunthorpe@obsidianresearch.com>
suggested a mechanism that does not require signaling every Send.
We signal once every N Sends, and perform SGE unmapping of N Send
operations during that one completion.
Reported-by: Sagi Grimberg <sagi@grimberg.me>
Suggested-by: Jason Gunthorpe <jgunthorpe@obsidianresearch.com>
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
2017-10-20 22:48:12 +08:00
|
|
|
/* Fixed-size circular FIFO queue. This implementation is wait-free and
|
|
|
|
|
* lock-free.
|
|
|
|
|
*
|
|
|
|
|
* Consumer is the code path that posts Sends. This path dequeues a
|
|
|
|
|
* sendctx for use by a Send operation. Multiple consumer threads
|
|
|
|
|
* are serialized by the RPC transport lock, which allows only one
|
|
|
|
|
* ->send_request call at a time.
|
|
|
|
|
*
|
|
|
|
|
* Producer is the code path that handles Send completions. This path
|
|
|
|
|
* enqueues a sendctx that has been completed. Multiple producer
|
|
|
|
|
* threads are serialized by the ib_poll_cq() function.
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
/* rpcrdma_sendctxs_destroy() assumes caller has already quiesced
|
|
|
|
|
* queue activity, and ib_drain_qp has flushed all remaining Send
|
|
|
|
|
* requests.
|
|
|
|
|
*/
|
|
|
|
|
static void rpcrdma_sendctxs_destroy(struct rpcrdma_buffer *buf)
|
|
|
|
|
{
|
|
|
|
|
unsigned long i;
|
|
|
|
|
|
|
|
|
|
for (i = 0; i <= buf->rb_sc_last; i++)
|
|
|
|
|
kfree(buf->rb_sc_ctxs[i]);
|
|
|
|
|
kfree(buf->rb_sc_ctxs);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static struct rpcrdma_sendctx *rpcrdma_sendctx_create(struct rpcrdma_ia *ia)
|
|
|
|
|
{
|
|
|
|
|
struct rpcrdma_sendctx *sc;
|
|
|
|
|
|
|
|
|
|
sc = kzalloc(sizeof(*sc) +
|
|
|
|
|
ia->ri_max_send_sges * sizeof(struct ib_sge),
|
|
|
|
|
GFP_KERNEL);
|
|
|
|
|
if (!sc)
|
|
|
|
|
return NULL;
|
|
|
|
|
|
|
|
|
|
sc->sc_wr.wr_cqe = &sc->sc_cqe;
|
|
|
|
|
sc->sc_wr.sg_list = sc->sc_sges;
|
|
|
|
|
sc->sc_wr.opcode = IB_WR_SEND;
|
|
|
|
|
sc->sc_cqe.done = rpcrdma_wc_send;
|
|
|
|
|
return sc;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static int rpcrdma_sendctxs_create(struct rpcrdma_xprt *r_xprt)
|
|
|
|
|
{
|
|
|
|
|
struct rpcrdma_buffer *buf = &r_xprt->rx_buf;
|
|
|
|
|
struct rpcrdma_sendctx *sc;
|
|
|
|
|
unsigned long i;
|
|
|
|
|
|
|
|
|
|
/* Maximum number of concurrent outstanding Send WRs. Capping
|
|
|
|
|
* the circular queue size stops Send Queue overflow by causing
|
|
|
|
|
* the ->send_request call to fail temporarily before too many
|
|
|
|
|
* Sends are posted.
|
|
|
|
|
*/
|
|
|
|
|
i = buf->rb_max_requests + RPCRDMA_MAX_BC_REQUESTS;
|
|
|
|
|
dprintk("RPC: %s: allocating %lu send_ctxs\n", __func__, i);
|
|
|
|
|
buf->rb_sc_ctxs = kcalloc(i, sizeof(sc), GFP_KERNEL);
|
|
|
|
|
if (!buf->rb_sc_ctxs)
|
|
|
|
|
return -ENOMEM;
|
|
|
|
|
|
|
|
|
|
buf->rb_sc_last = i - 1;
|
|
|
|
|
for (i = 0; i <= buf->rb_sc_last; i++) {
|
|
|
|
|
sc = rpcrdma_sendctx_create(&r_xprt->rx_ia);
|
|
|
|
|
if (!sc)
|
|
|
|
|
goto out_destroy;
|
|
|
|
|
|
|
|
|
|
sc->sc_xprt = r_xprt;
|
|
|
|
|
buf->rb_sc_ctxs[i] = sc;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
|
|
out_destroy:
|
|
|
|
|
rpcrdma_sendctxs_destroy(buf);
|
|
|
|
|
return -ENOMEM;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/* The sendctx queue is not guaranteed to have a size that is a
|
|
|
|
|
* power of two, thus the helpers in circ_buf.h cannot be used.
|
|
|
|
|
* The other option is to use modulus (%), which can be expensive.
|
|
|
|
|
*/
|
|
|
|
|
static unsigned long rpcrdma_sendctx_next(struct rpcrdma_buffer *buf,
|
|
|
|
|
unsigned long item)
|
|
|
|
|
{
|
|
|
|
|
return likely(item < buf->rb_sc_last) ? item + 1 : 0;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/**
|
|
|
|
|
* rpcrdma_sendctx_get_locked - Acquire a send context
|
|
|
|
|
* @buf: transport buffers from which to acquire an unused context
|
|
|
|
|
*
|
|
|
|
|
* Returns pointer to a free send completion context; or NULL if
|
|
|
|
|
* the queue is empty.
|
|
|
|
|
*
|
|
|
|
|
* Usage: Called to acquire an SGE array before preparing a Send WR.
|
|
|
|
|
*
|
|
|
|
|
* The caller serializes calls to this function (per rpcrdma_buffer),
|
|
|
|
|
* and provides an effective memory barrier that flushes the new value
|
|
|
|
|
* of rb_sc_head.
|
|
|
|
|
*/
|
|
|
|
|
struct rpcrdma_sendctx *rpcrdma_sendctx_get_locked(struct rpcrdma_buffer *buf)
|
|
|
|
|
{
|
|
|
|
|
struct rpcrdma_xprt *r_xprt;
|
|
|
|
|
struct rpcrdma_sendctx *sc;
|
|
|
|
|
unsigned long next_head;
|
|
|
|
|
|
|
|
|
|
next_head = rpcrdma_sendctx_next(buf, buf->rb_sc_head);
|
|
|
|
|
|
|
|
|
|
if (next_head == READ_ONCE(buf->rb_sc_tail))
|
|
|
|
|
goto out_emptyq;
|
|
|
|
|
|
|
|
|
|
/* ORDER: item must be accessed _before_ head is updated */
|
|
|
|
|
sc = buf->rb_sc_ctxs[next_head];
|
|
|
|
|
|
|
|
|
|
/* Releasing the lock in the caller acts as a memory
|
|
|
|
|
* barrier that flushes rb_sc_head.
|
|
|
|
|
*/
|
|
|
|
|
buf->rb_sc_head = next_head;
|
|
|
|
|
|
|
|
|
|
return sc;
|
|
|
|
|
|
|
|
|
|
out_emptyq:
|
|
|
|
|
/* The queue is "empty" if there have not been enough Send
|
|
|
|
|
* completions recently. This is a sign the Send Queue is
|
|
|
|
|
* backing up. Cause the caller to pause and try again.
|
|
|
|
|
*/
|
2018-05-05 03:35:57 +08:00
|
|
|
set_bit(RPCRDMA_BUF_F_EMPTY_SCQ, &buf->rb_flags);
|
xprtrdma: Add data structure to manage RDMA Send arguments
Problem statement:
Recently Sagi Grimberg <sagi@grimberg.me> observed that kernel RDMA-
enabled storage initiators don't handle delayed Send completion
correctly. If Send completion is delayed beyond the end of a ULP
transaction, the ULP may release resources that are still being used
by the HCA to complete a long-running Send operation.
This is a common design trait amongst our initiators. Most Send
operations are faster than the ULP transaction they are part of.
Waiting for a completion for these is typically unnecessary.
Infrequently, a network partition or some other problem crops up
where an ordering problem can occur. In NFS parlance, the RPC Reply
arrives and completes the RPC, but the HCA is still retrying the
Send WR that conveyed the RPC Call. In this case, the HCA can try
to use memory that has been invalidated or DMA unmapped, and the
connection is lost. If that memory has been re-used for something
else (possibly not related to NFS), and the Send retransmission
exposes that data on the wire.
Thus we cannot assume that it is safe to release Send-related
resources just because a ULP reply has arrived.
After some analysis, we have determined that the completion
housekeeping will not be difficult for xprtrdma:
- Inline Send buffers are registered via the local DMA key, and
are already left DMA mapped for the lifetime of a transport
connection, thus no additional handling is necessary for those
- Gathered Sends involving page cache pages _will_ need to
DMA unmap those pages after the Send completes. But like
inline send buffers, they are registered via the local DMA key,
and thus will not need to be invalidated
In addition, RPC completion will need to wait for Send completion
in the latter case. However, nearly always, the Send that conveys
the RPC Call will have completed long before the RPC Reply
arrives, and thus no additional latency will be accrued.
Design notes:
In this patch, the rpcrdma_sendctx object is introduced, and a
lock-free circular queue is added to manage a set of them per
transport.
The RPC client's send path already prevents sending more than one
RPC Call at the same time. This allows us to treat the consumer
side of the queue (rpcrdma_sendctx_get_locked) as if there is a
single consumer thread.
The producer side of the queue (rpcrdma_sendctx_put_locked) is
invoked only from the Send completion handler, which is a single
thread of execution (soft IRQ).
The only care that needs to be taken is with the tail index, which
is shared between the producer and consumer. Only the producer
updates the tail index. The consumer compares the head with the
tail to ensure that the a sendctx that is in use is never handed
out again (or, expressed more conventionally, the queue is empty).
When the sendctx queue empties completely, there are enough Sends
outstanding that posting more Send operations can result in a Send
Queue overflow. In this case, the ULP is told to wait and try again.
This introduces strong Send Queue accounting to xprtrdma.
As a final touch, Jason Gunthorpe <jgunthorpe@obsidianresearch.com>
suggested a mechanism that does not require signaling every Send.
We signal once every N Sends, and perform SGE unmapping of N Send
operations during that one completion.
Reported-by: Sagi Grimberg <sagi@grimberg.me>
Suggested-by: Jason Gunthorpe <jgunthorpe@obsidianresearch.com>
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
2017-10-20 22:48:12 +08:00
|
|
|
r_xprt = container_of(buf, struct rpcrdma_xprt, rx_buf);
|
|
|
|
|
r_xprt->rx_stats.empty_sendctx_q++;
|
|
|
|
|
return NULL;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/**
|
|
|
|
|
* rpcrdma_sendctx_put_locked - Release a send context
|
|
|
|
|
* @sc: send context to release
|
|
|
|
|
*
|
|
|
|
|
* Usage: Called from Send completion to return a sendctxt
|
|
|
|
|
* to the queue.
|
|
|
|
|
*
|
|
|
|
|
* The caller serializes calls to this function (per rpcrdma_buffer).
|
|
|
|
|
*/
|
2018-05-05 03:35:41 +08:00
|
|
|
static void
|
|
|
|
|
rpcrdma_sendctx_put_locked(struct rpcrdma_sendctx *sc)
|
xprtrdma: Add data structure to manage RDMA Send arguments
Problem statement:
Recently Sagi Grimberg <sagi@grimberg.me> observed that kernel RDMA-
enabled storage initiators don't handle delayed Send completion
correctly. If Send completion is delayed beyond the end of a ULP
transaction, the ULP may release resources that are still being used
by the HCA to complete a long-running Send operation.
This is a common design trait amongst our initiators. Most Send
operations are faster than the ULP transaction they are part of.
Waiting for a completion for these is typically unnecessary.
Infrequently, a network partition or some other problem crops up
where an ordering problem can occur. In NFS parlance, the RPC Reply
arrives and completes the RPC, but the HCA is still retrying the
Send WR that conveyed the RPC Call. In this case, the HCA can try
to use memory that has been invalidated or DMA unmapped, and the
connection is lost. If that memory has been re-used for something
else (possibly not related to NFS), and the Send retransmission
exposes that data on the wire.
Thus we cannot assume that it is safe to release Send-related
resources just because a ULP reply has arrived.
After some analysis, we have determined that the completion
housekeeping will not be difficult for xprtrdma:
- Inline Send buffers are registered via the local DMA key, and
are already left DMA mapped for the lifetime of a transport
connection, thus no additional handling is necessary for those
- Gathered Sends involving page cache pages _will_ need to
DMA unmap those pages after the Send completes. But like
inline send buffers, they are registered via the local DMA key,
and thus will not need to be invalidated
In addition, RPC completion will need to wait for Send completion
in the latter case. However, nearly always, the Send that conveys
the RPC Call will have completed long before the RPC Reply
arrives, and thus no additional latency will be accrued.
Design notes:
In this patch, the rpcrdma_sendctx object is introduced, and a
lock-free circular queue is added to manage a set of them per
transport.
The RPC client's send path already prevents sending more than one
RPC Call at the same time. This allows us to treat the consumer
side of the queue (rpcrdma_sendctx_get_locked) as if there is a
single consumer thread.
The producer side of the queue (rpcrdma_sendctx_put_locked) is
invoked only from the Send completion handler, which is a single
thread of execution (soft IRQ).
The only care that needs to be taken is with the tail index, which
is shared between the producer and consumer. Only the producer
updates the tail index. The consumer compares the head with the
tail to ensure that the a sendctx that is in use is never handed
out again (or, expressed more conventionally, the queue is empty).
When the sendctx queue empties completely, there are enough Sends
outstanding that posting more Send operations can result in a Send
Queue overflow. In this case, the ULP is told to wait and try again.
This introduces strong Send Queue accounting to xprtrdma.
As a final touch, Jason Gunthorpe <jgunthorpe@obsidianresearch.com>
suggested a mechanism that does not require signaling every Send.
We signal once every N Sends, and perform SGE unmapping of N Send
operations during that one completion.
Reported-by: Sagi Grimberg <sagi@grimberg.me>
Suggested-by: Jason Gunthorpe <jgunthorpe@obsidianresearch.com>
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
2017-10-20 22:48:12 +08:00
|
|
|
{
|
|
|
|
|
struct rpcrdma_buffer *buf = &sc->sc_xprt->rx_buf;
|
|
|
|
|
unsigned long next_tail;
|
|
|
|
|
|
|
|
|
|
/* Unmap SGEs of previously completed by unsignaled
|
|
|
|
|
* Sends by walking up the queue until @sc is found.
|
|
|
|
|
*/
|
|
|
|
|
next_tail = buf->rb_sc_tail;
|
|
|
|
|
do {
|
|
|
|
|
next_tail = rpcrdma_sendctx_next(buf, next_tail);
|
|
|
|
|
|
|
|
|
|
/* ORDER: item must be accessed _before_ tail is updated */
|
|
|
|
|
rpcrdma_unmap_sendctx(buf->rb_sc_ctxs[next_tail]);
|
|
|
|
|
|
|
|
|
|
} while (buf->rb_sc_ctxs[next_tail] != sc);
|
|
|
|
|
|
|
|
|
|
/* Paired with READ_ONCE */
|
|
|
|
|
smp_store_release(&buf->rb_sc_tail, next_tail);
|
2018-05-05 03:35:57 +08:00
|
|
|
|
|
|
|
|
if (test_and_clear_bit(RPCRDMA_BUF_F_EMPTY_SCQ, &buf->rb_flags)) {
|
|
|
|
|
smp_mb__after_atomic();
|
|
|
|
|
xprt_write_space(&sc->sc_xprt->rx_xprt);
|
|
|
|
|
}
|
xprtrdma: Add data structure to manage RDMA Send arguments
Problem statement:
Recently Sagi Grimberg <sagi@grimberg.me> observed that kernel RDMA-
enabled storage initiators don't handle delayed Send completion
correctly. If Send completion is delayed beyond the end of a ULP
transaction, the ULP may release resources that are still being used
by the HCA to complete a long-running Send operation.
This is a common design trait amongst our initiators. Most Send
operations are faster than the ULP transaction they are part of.
Waiting for a completion for these is typically unnecessary.
Infrequently, a network partition or some other problem crops up
where an ordering problem can occur. In NFS parlance, the RPC Reply
arrives and completes the RPC, but the HCA is still retrying the
Send WR that conveyed the RPC Call. In this case, the HCA can try
to use memory that has been invalidated or DMA unmapped, and the
connection is lost. If that memory has been re-used for something
else (possibly not related to NFS), and the Send retransmission
exposes that data on the wire.
Thus we cannot assume that it is safe to release Send-related
resources just because a ULP reply has arrived.
After some analysis, we have determined that the completion
housekeeping will not be difficult for xprtrdma:
- Inline Send buffers are registered via the local DMA key, and
are already left DMA mapped for the lifetime of a transport
connection, thus no additional handling is necessary for those
- Gathered Sends involving page cache pages _will_ need to
DMA unmap those pages after the Send completes. But like
inline send buffers, they are registered via the local DMA key,
and thus will not need to be invalidated
In addition, RPC completion will need to wait for Send completion
in the latter case. However, nearly always, the Send that conveys
the RPC Call will have completed long before the RPC Reply
arrives, and thus no additional latency will be accrued.
Design notes:
In this patch, the rpcrdma_sendctx object is introduced, and a
lock-free circular queue is added to manage a set of them per
transport.
The RPC client's send path already prevents sending more than one
RPC Call at the same time. This allows us to treat the consumer
side of the queue (rpcrdma_sendctx_get_locked) as if there is a
single consumer thread.
The producer side of the queue (rpcrdma_sendctx_put_locked) is
invoked only from the Send completion handler, which is a single
thread of execution (soft IRQ).
The only care that needs to be taken is with the tail index, which
is shared between the producer and consumer. Only the producer
updates the tail index. The consumer compares the head with the
tail to ensure that the a sendctx that is in use is never handed
out again (or, expressed more conventionally, the queue is empty).
When the sendctx queue empties completely, there are enough Sends
outstanding that posting more Send operations can result in a Send
Queue overflow. In this case, the ULP is told to wait and try again.
This introduces strong Send Queue accounting to xprtrdma.
As a final touch, Jason Gunthorpe <jgunthorpe@obsidianresearch.com>
suggested a mechanism that does not require signaling every Send.
We signal once every N Sends, and perform SGE unmapping of N Send
operations during that one completion.
Reported-by: Sagi Grimberg <sagi@grimberg.me>
Suggested-by: Jason Gunthorpe <jgunthorpe@obsidianresearch.com>
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
2017-10-20 22:48:12 +08:00
|
|
|
}
|
|
|
|
|
|
2016-06-30 01:54:00 +08:00
|
|
|
static void
|
2017-12-15 09:57:55 +08:00
|
|
|
rpcrdma_mrs_create(struct rpcrdma_xprt *r_xprt)
|
2016-06-30 01:54:00 +08:00
|
|
|
{
|
|
|
|
|
struct rpcrdma_buffer *buf = &r_xprt->rx_buf;
|
|
|
|
|
struct rpcrdma_ia *ia = &r_xprt->rx_ia;
|
|
|
|
|
unsigned int count;
|
|
|
|
|
LIST_HEAD(free);
|
|
|
|
|
LIST_HEAD(all);
|
|
|
|
|
|
2018-10-02 02:25:20 +08:00
|
|
|
for (count = 0; count < ia->ri_max_segs; count++) {
|
2017-12-15 09:57:55 +08:00
|
|
|
struct rpcrdma_mr *mr;
|
2016-06-30 01:54:00 +08:00
|
|
|
int rc;
|
|
|
|
|
|
2017-12-15 09:57:55 +08:00
|
|
|
mr = kzalloc(sizeof(*mr), GFP_KERNEL);
|
|
|
|
|
if (!mr)
|
2016-06-30 01:54:00 +08:00
|
|
|
break;
|
|
|
|
|
|
2018-12-19 23:59:01 +08:00
|
|
|
rc = frwr_init_mr(ia, mr);
|
2016-06-30 01:54:00 +08:00
|
|
|
if (rc) {
|
2017-12-15 09:57:55 +08:00
|
|
|
kfree(mr);
|
2016-06-30 01:54:00 +08:00
|
|
|
break;
|
|
|
|
|
}
|
|
|
|
|
|
2017-12-15 09:57:55 +08:00
|
|
|
mr->mr_xprt = r_xprt;
|
2016-06-30 01:54:00 +08:00
|
|
|
|
2017-12-15 09:57:55 +08:00
|
|
|
list_add(&mr->mr_list, &free);
|
|
|
|
|
list_add(&mr->mr_all, &all);
|
2016-06-30 01:54:00 +08:00
|
|
|
}
|
|
|
|
|
|
2017-12-15 09:57:55 +08:00
|
|
|
spin_lock(&buf->rb_mrlock);
|
|
|
|
|
list_splice(&free, &buf->rb_mrs);
|
2016-06-30 01:54:00 +08:00
|
|
|
list_splice(&all, &buf->rb_all);
|
|
|
|
|
r_xprt->rx_stats.mrs_allocated += count;
|
2017-12-15 09:57:55 +08:00
|
|
|
spin_unlock(&buf->rb_mrlock);
|
2017-12-21 05:31:21 +08:00
|
|
|
trace_xprtrdma_createmrs(r_xprt, count);
|
2018-03-01 04:30:44 +08:00
|
|
|
|
|
|
|
|
xprt_write_space(&r_xprt->rx_xprt);
|
2016-06-30 01:54:00 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static void
|
|
|
|
|
rpcrdma_mr_refresh_worker(struct work_struct *work)
|
|
|
|
|
{
|
|
|
|
|
struct rpcrdma_buffer *buf = container_of(work, struct rpcrdma_buffer,
|
|
|
|
|
rb_refresh_worker.work);
|
|
|
|
|
struct rpcrdma_xprt *r_xprt = container_of(buf, struct rpcrdma_xprt,
|
|
|
|
|
rx_buf);
|
|
|
|
|
|
2017-12-15 09:57:55 +08:00
|
|
|
rpcrdma_mrs_create(r_xprt);
|
2016-06-30 01:54:00 +08:00
|
|
|
}
|
|
|
|
|
|
2015-10-25 05:27:43 +08:00
|
|
|
struct rpcrdma_req *
|
2015-01-22 00:03:52 +08:00
|
|
|
rpcrdma_create_req(struct rpcrdma_xprt *r_xprt)
|
|
|
|
|
{
|
2015-10-25 05:27:43 +08:00
|
|
|
struct rpcrdma_buffer *buffer = &r_xprt->rx_buf;
|
2018-03-01 04:31:05 +08:00
|
|
|
struct rpcrdma_regbuf *rb;
|
2015-01-22 00:03:52 +08:00
|
|
|
struct rpcrdma_req *req;
|
|
|
|
|
|
2015-01-22 00:04:16 +08:00
|
|
|
req = kzalloc(sizeof(*req), GFP_KERNEL);
|
2015-01-22 00:03:52 +08:00
|
|
|
if (req == NULL)
|
2015-01-22 00:04:16 +08:00
|
|
|
return ERR_PTR(-ENOMEM);
|
2015-01-22 00:03:52 +08:00
|
|
|
|
2018-03-01 04:31:05 +08:00
|
|
|
rb = rpcrdma_alloc_regbuf(RPCRDMA_HDRBUF_SIZE,
|
|
|
|
|
DMA_TO_DEVICE, GFP_KERNEL);
|
|
|
|
|
if (IS_ERR(rb)) {
|
|
|
|
|
kfree(req);
|
|
|
|
|
return ERR_PTR(-ENOMEM);
|
|
|
|
|
}
|
|
|
|
|
req->rl_rdmabuf = rb;
|
|
|
|
|
xdr_buf_init(&req->rl_hdrbuf, rb->rg_base, rdmab_length(rb));
|
|
|
|
|
req->rl_buffer = buffer;
|
|
|
|
|
INIT_LIST_HEAD(&req->rl_registered);
|
|
|
|
|
|
2018-12-19 23:59:33 +08:00
|
|
|
spin_lock(&buffer->rb_lock);
|
2015-10-25 05:27:43 +08:00
|
|
|
list_add(&req->rl_all, &buffer->rb_allreqs);
|
2018-12-19 23:59:33 +08:00
|
|
|
spin_unlock(&buffer->rb_lock);
|
2015-01-22 00:03:52 +08:00
|
|
|
return req;
|
|
|
|
|
}
|
|
|
|
|
|
2018-05-05 03:35:20 +08:00
|
|
|
static int
|
|
|
|
|
rpcrdma_create_rep(struct rpcrdma_xprt *r_xprt, bool temp)
|
2015-01-22 00:03:52 +08:00
|
|
|
{
|
|
|
|
|
struct rpcrdma_create_data_internal *cdata = &r_xprt->rx_data;
|
2017-12-15 09:56:09 +08:00
|
|
|
struct rpcrdma_buffer *buf = &r_xprt->rx_buf;
|
2015-01-22 00:03:52 +08:00
|
|
|
struct rpcrdma_rep *rep;
|
|
|
|
|
int rc;
|
|
|
|
|
|
|
|
|
|
rc = -ENOMEM;
|
2015-01-22 00:04:25 +08:00
|
|
|
rep = kzalloc(sizeof(*rep), GFP_KERNEL);
|
2015-01-22 00:03:52 +08:00
|
|
|
if (rep == NULL)
|
|
|
|
|
goto out;
|
|
|
|
|
|
2016-09-15 22:56:26 +08:00
|
|
|
rep->rr_rdmabuf = rpcrdma_alloc_regbuf(cdata->inline_rsize,
|
2016-09-15 22:56:10 +08:00
|
|
|
DMA_FROM_DEVICE, GFP_KERNEL);
|
2015-01-22 00:04:25 +08:00
|
|
|
if (IS_ERR(rep->rr_rdmabuf)) {
|
|
|
|
|
rc = PTR_ERR(rep->rr_rdmabuf);
|
2015-01-22 00:03:52 +08:00
|
|
|
goto out_free;
|
2015-01-22 00:04:25 +08:00
|
|
|
}
|
2017-08-04 02:30:03 +08:00
|
|
|
xdr_buf_init(&rep->rr_hdrbuf, rep->rr_rdmabuf->rg_base,
|
|
|
|
|
rdmab_length(rep->rr_rdmabuf));
|
2015-01-22 00:03:52 +08:00
|
|
|
|
2016-09-15 22:57:49 +08:00
|
|
|
rep->rr_cqe.done = rpcrdma_wc_receive;
|
2015-05-26 23:51:37 +08:00
|
|
|
rep->rr_rxprt = r_xprt;
|
2017-10-17 03:01:30 +08:00
|
|
|
INIT_WORK(&rep->rr_work, rpcrdma_deferred_completion);
|
2016-09-15 22:56:51 +08:00
|
|
|
rep->rr_recv_wr.next = NULL;
|
|
|
|
|
rep->rr_recv_wr.wr_cqe = &rep->rr_cqe;
|
|
|
|
|
rep->rr_recv_wr.sg_list = &rep->rr_rdmabuf->rg_iov;
|
|
|
|
|
rep->rr_recv_wr.num_sge = 1;
|
2018-05-05 03:35:20 +08:00
|
|
|
rep->rr_temp = temp;
|
2017-12-15 09:56:09 +08:00
|
|
|
|
|
|
|
|
spin_lock(&buf->rb_lock);
|
|
|
|
|
list_add(&rep->rr_list, &buf->rb_recv_bufs);
|
|
|
|
|
spin_unlock(&buf->rb_lock);
|
|
|
|
|
return 0;
|
2015-01-22 00:03:52 +08:00
|
|
|
|
|
|
|
|
out_free:
|
|
|
|
|
kfree(rep);
|
|
|
|
|
out:
|
2017-12-15 09:56:09 +08:00
|
|
|
return rc;
|
2015-01-22 00:03:52 +08:00
|
|
|
}
|
|
|
|
|
|
2007-09-11 01:51:18 +08:00
|
|
|
int
|
2015-01-22 00:03:44 +08:00
|
|
|
rpcrdma_buffer_create(struct rpcrdma_xprt *r_xprt)
|
2007-09-11 01:51:18 +08:00
|
|
|
{
|
2015-01-22 00:03:44 +08:00
|
|
|
struct rpcrdma_buffer *buf = &r_xprt->rx_buf;
|
2007-09-11 01:51:18 +08:00
|
|
|
int i, rc;
|
|
|
|
|
|
2018-10-02 02:26:29 +08:00
|
|
|
buf->rb_flags = 0;
|
2015-10-25 05:27:02 +08:00
|
|
|
buf->rb_max_requests = r_xprt->rx_data.max_requests;
|
2015-10-25 05:27:43 +08:00
|
|
|
buf->rb_bc_srv_max_requests = 0;
|
2017-12-15 09:57:55 +08:00
|
|
|
spin_lock_init(&buf->rb_mrlock);
|
2016-06-30 01:52:54 +08:00
|
|
|
spin_lock_init(&buf->rb_lock);
|
2017-12-15 09:57:55 +08:00
|
|
|
INIT_LIST_HEAD(&buf->rb_mrs);
|
2016-06-30 01:54:00 +08:00
|
|
|
INIT_LIST_HEAD(&buf->rb_all);
|
|
|
|
|
INIT_DELAYED_WORK(&buf->rb_refresh_worker,
|
|
|
|
|
rpcrdma_mr_refresh_worker);
|
2007-09-11 01:51:18 +08:00
|
|
|
|
2017-12-15 09:57:55 +08:00
|
|
|
rpcrdma_mrs_create(r_xprt);
|
2007-09-11 01:51:18 +08:00
|
|
|
|
2015-10-25 05:27:02 +08:00
|
|
|
INIT_LIST_HEAD(&buf->rb_send_bufs);
|
2015-10-25 05:27:43 +08:00
|
|
|
INIT_LIST_HEAD(&buf->rb_allreqs);
|
2007-09-11 01:51:18 +08:00
|
|
|
for (i = 0; i < buf->rb_max_requests; i++) {
|
|
|
|
|
struct rpcrdma_req *req;
|
|
|
|
|
|
2015-01-22 00:03:52 +08:00
|
|
|
req = rpcrdma_create_req(r_xprt);
|
|
|
|
|
if (IS_ERR(req)) {
|
2007-09-11 01:51:18 +08:00
|
|
|
dprintk("RPC: %s: request buffer %d alloc"
|
|
|
|
|
" failed\n", __func__, i);
|
2015-01-22 00:03:52 +08:00
|
|
|
rc = PTR_ERR(req);
|
2007-09-11 01:51:18 +08:00
|
|
|
goto out;
|
|
|
|
|
}
|
2017-06-08 23:52:12 +08:00
|
|
|
list_add(&req->rl_list, &buf->rb_send_bufs);
|
2015-10-25 05:27:02 +08:00
|
|
|
}
|
|
|
|
|
|
2018-07-28 22:46:47 +08:00
|
|
|
buf->rb_credits = 1;
|
2015-10-25 05:27:02 +08:00
|
|
|
INIT_LIST_HEAD(&buf->rb_recv_bufs);
|
2015-01-22 00:03:52 +08:00
|
|
|
|
xprtrdma: Add data structure to manage RDMA Send arguments
Problem statement:
Recently Sagi Grimberg <sagi@grimberg.me> observed that kernel RDMA-
enabled storage initiators don't handle delayed Send completion
correctly. If Send completion is delayed beyond the end of a ULP
transaction, the ULP may release resources that are still being used
by the HCA to complete a long-running Send operation.
This is a common design trait amongst our initiators. Most Send
operations are faster than the ULP transaction they are part of.
Waiting for a completion for these is typically unnecessary.
Infrequently, a network partition or some other problem crops up
where an ordering problem can occur. In NFS parlance, the RPC Reply
arrives and completes the RPC, but the HCA is still retrying the
Send WR that conveyed the RPC Call. In this case, the HCA can try
to use memory that has been invalidated or DMA unmapped, and the
connection is lost. If that memory has been re-used for something
else (possibly not related to NFS), and the Send retransmission
exposes that data on the wire.
Thus we cannot assume that it is safe to release Send-related
resources just because a ULP reply has arrived.
After some analysis, we have determined that the completion
housekeeping will not be difficult for xprtrdma:
- Inline Send buffers are registered via the local DMA key, and
are already left DMA mapped for the lifetime of a transport
connection, thus no additional handling is necessary for those
- Gathered Sends involving page cache pages _will_ need to
DMA unmap those pages after the Send completes. But like
inline send buffers, they are registered via the local DMA key,
and thus will not need to be invalidated
In addition, RPC completion will need to wait for Send completion
in the latter case. However, nearly always, the Send that conveys
the RPC Call will have completed long before the RPC Reply
arrives, and thus no additional latency will be accrued.
Design notes:
In this patch, the rpcrdma_sendctx object is introduced, and a
lock-free circular queue is added to manage a set of them per
transport.
The RPC client's send path already prevents sending more than one
RPC Call at the same time. This allows us to treat the consumer
side of the queue (rpcrdma_sendctx_get_locked) as if there is a
single consumer thread.
The producer side of the queue (rpcrdma_sendctx_put_locked) is
invoked only from the Send completion handler, which is a single
thread of execution (soft IRQ).
The only care that needs to be taken is with the tail index, which
is shared between the producer and consumer. Only the producer
updates the tail index. The consumer compares the head with the
tail to ensure that the a sendctx that is in use is never handed
out again (or, expressed more conventionally, the queue is empty).
When the sendctx queue empties completely, there are enough Sends
outstanding that posting more Send operations can result in a Send
Queue overflow. In this case, the ULP is told to wait and try again.
This introduces strong Send Queue accounting to xprtrdma.
As a final touch, Jason Gunthorpe <jgunthorpe@obsidianresearch.com>
suggested a mechanism that does not require signaling every Send.
We signal once every N Sends, and perform SGE unmapping of N Send
operations during that one completion.
Reported-by: Sagi Grimberg <sagi@grimberg.me>
Suggested-by: Jason Gunthorpe <jgunthorpe@obsidianresearch.com>
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
2017-10-20 22:48:12 +08:00
|
|
|
rc = rpcrdma_sendctxs_create(r_xprt);
|
|
|
|
|
if (rc)
|
|
|
|
|
goto out;
|
|
|
|
|
|
2018-12-19 23:58:29 +08:00
|
|
|
buf->rb_completion_wq = alloc_workqueue("rpcrdma-%s",
|
|
|
|
|
WQ_MEM_RECLAIM | WQ_HIGHPRI,
|
|
|
|
|
0,
|
|
|
|
|
r_xprt->rx_xprt.address_strings[RPC_DISPLAY_ADDR]);
|
2019-01-05 21:06:07 +08:00
|
|
|
if (!buf->rb_completion_wq) {
|
|
|
|
|
rc = -ENOMEM;
|
2018-12-19 23:58:29 +08:00
|
|
|
goto out;
|
2019-01-05 21:06:07 +08:00
|
|
|
}
|
2018-12-19 23:58:29 +08:00
|
|
|
|
2007-09-11 01:51:18 +08:00
|
|
|
return 0;
|
|
|
|
|
out:
|
|
|
|
|
rpcrdma_buffer_destroy(buf);
|
|
|
|
|
return rc;
|
|
|
|
|
}
|
|
|
|
|
|
2015-01-22 00:03:52 +08:00
|
|
|
static void
|
2016-09-15 22:56:26 +08:00
|
|
|
rpcrdma_destroy_rep(struct rpcrdma_rep *rep)
|
2015-01-22 00:03:52 +08:00
|
|
|
{
|
2016-09-15 22:56:26 +08:00
|
|
|
rpcrdma_free_regbuf(rep->rr_rdmabuf);
|
2015-01-22 00:03:52 +08:00
|
|
|
kfree(rep);
|
|
|
|
|
}
|
|
|
|
|
|
2018-12-19 23:59:33 +08:00
|
|
|
/**
|
|
|
|
|
* rpcrdma_req_destroy - Destroy an rpcrdma_req object
|
|
|
|
|
* @req: unused object to be destroyed
|
|
|
|
|
*
|
|
|
|
|
* This function assumes that the caller prevents concurrent device
|
|
|
|
|
* unload and transport tear-down.
|
|
|
|
|
*/
|
2015-10-25 05:27:43 +08:00
|
|
|
void
|
2018-12-19 23:59:33 +08:00
|
|
|
rpcrdma_req_destroy(struct rpcrdma_req *req)
|
2015-01-22 00:03:52 +08:00
|
|
|
{
|
2018-12-19 23:59:33 +08:00
|
|
|
list_del(&req->rl_all);
|
|
|
|
|
|
2016-09-15 22:56:26 +08:00
|
|
|
rpcrdma_free_regbuf(req->rl_recvbuf);
|
|
|
|
|
rpcrdma_free_regbuf(req->rl_sendbuf);
|
|
|
|
|
rpcrdma_free_regbuf(req->rl_rdmabuf);
|
2015-01-22 00:03:52 +08:00
|
|
|
kfree(req);
|
|
|
|
|
}
|
|
|
|
|
|
2016-06-30 01:54:00 +08:00
|
|
|
static void
|
2017-12-15 09:57:55 +08:00
|
|
|
rpcrdma_mrs_destroy(struct rpcrdma_buffer *buf)
|
2016-06-30 01:54:00 +08:00
|
|
|
{
|
|
|
|
|
struct rpcrdma_xprt *r_xprt = container_of(buf, struct rpcrdma_xprt,
|
|
|
|
|
rx_buf);
|
2017-12-15 09:57:55 +08:00
|
|
|
struct rpcrdma_mr *mr;
|
2016-06-30 01:54:00 +08:00
|
|
|
unsigned int count;
|
|
|
|
|
|
|
|
|
|
count = 0;
|
2017-12-15 09:57:55 +08:00
|
|
|
spin_lock(&buf->rb_mrlock);
|
2016-06-30 01:54:00 +08:00
|
|
|
while (!list_empty(&buf->rb_all)) {
|
2017-12-15 09:57:55 +08:00
|
|
|
mr = list_entry(buf->rb_all.next, struct rpcrdma_mr, mr_all);
|
|
|
|
|
list_del(&mr->mr_all);
|
2016-06-30 01:54:00 +08:00
|
|
|
|
2017-12-15 09:57:55 +08:00
|
|
|
spin_unlock(&buf->rb_mrlock);
|
xprtrdma: Fix list corruption / DMAR errors during MR recovery
The ro_release_mr methods check whether mr->mr_list is empty.
Therefore, be sure to always use list_del_init when removing an MR
linked into a list using that field. Otherwise, when recovering from
transport failures or device removal, list corruption can result, or
MRs can get mapped or unmapped an odd number of times, resulting in
IOMMU-related failures.
In general this fix is appropriate back to v4.8. However, code
changes since then make it impossible to apply this patch directly
to stable kernels. The fix would have to be applied by hand or
reworked for kernels earlier than v4.16.
Backport guidance -- there are several cases:
- When creating an MR, initialize mr_list so that using list_empty
on an as-yet-unused MR is safe.
- When an MR is being handled by the remote invalidation path,
ensure that mr_list is reinitialized when it is removed from
rl_registered.
- When an MR is being handled by rpcrdma_destroy_mrs, it is removed
from mr_all, but it may still be on an rl_registered list. In
that case, the MR needs to be removed from that list before being
released.
- Other cases are covered by using list_del_init in rpcrdma_mr_pop.
Fixes: 9d6b04097882 ('xprtrdma: Place registered MWs on a ... ')
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
2018-05-01 23:37:14 +08:00
|
|
|
|
|
|
|
|
/* Ensure MW is not on any rl_registered list */
|
|
|
|
|
if (!list_empty(&mr->mr_list))
|
|
|
|
|
list_del(&mr->mr_list);
|
|
|
|
|
|
2018-12-19 23:59:01 +08:00
|
|
|
frwr_release_mr(mr);
|
2016-06-30 01:54:00 +08:00
|
|
|
count++;
|
2017-12-15 09:57:55 +08:00
|
|
|
spin_lock(&buf->rb_mrlock);
|
2016-06-30 01:54:00 +08:00
|
|
|
}
|
2017-12-15 09:57:55 +08:00
|
|
|
spin_unlock(&buf->rb_mrlock);
|
2016-06-30 01:54:00 +08:00
|
|
|
r_xprt->rx_stats.mrs_allocated = 0;
|
|
|
|
|
|
|
|
|
|
dprintk("RPC: %s: released %u MRs\n", __func__, count);
|
|
|
|
|
}
|
|
|
|
|
|
2018-12-20 00:00:37 +08:00
|
|
|
/**
|
|
|
|
|
* rpcrdma_buffer_destroy - Release all hw resources
|
|
|
|
|
* @buf: root control block for resources
|
|
|
|
|
*
|
|
|
|
|
* ORDERING: relies on a prior ib_drain_qp :
|
|
|
|
|
* - No more Send or Receive completions can occur
|
|
|
|
|
* - All MRs, reps, and reqs are returned to their free lists
|
|
|
|
|
*/
|
2007-09-11 01:51:18 +08:00
|
|
|
void
|
|
|
|
|
rpcrdma_buffer_destroy(struct rpcrdma_buffer *buf)
|
|
|
|
|
{
|
2017-04-12 01:22:29 +08:00
|
|
|
cancel_delayed_work_sync(&buf->rb_refresh_worker);
|
2016-06-30 01:52:54 +08:00
|
|
|
|
2018-12-19 23:58:29 +08:00
|
|
|
if (buf->rb_completion_wq) {
|
|
|
|
|
destroy_workqueue(buf->rb_completion_wq);
|
|
|
|
|
buf->rb_completion_wq = NULL;
|
|
|
|
|
}
|
|
|
|
|
|
xprtrdma: Add data structure to manage RDMA Send arguments
Problem statement:
Recently Sagi Grimberg <sagi@grimberg.me> observed that kernel RDMA-
enabled storage initiators don't handle delayed Send completion
correctly. If Send completion is delayed beyond the end of a ULP
transaction, the ULP may release resources that are still being used
by the HCA to complete a long-running Send operation.
This is a common design trait amongst our initiators. Most Send
operations are faster than the ULP transaction they are part of.
Waiting for a completion for these is typically unnecessary.
Infrequently, a network partition or some other problem crops up
where an ordering problem can occur. In NFS parlance, the RPC Reply
arrives and completes the RPC, but the HCA is still retrying the
Send WR that conveyed the RPC Call. In this case, the HCA can try
to use memory that has been invalidated or DMA unmapped, and the
connection is lost. If that memory has been re-used for something
else (possibly not related to NFS), and the Send retransmission
exposes that data on the wire.
Thus we cannot assume that it is safe to release Send-related
resources just because a ULP reply has arrived.
After some analysis, we have determined that the completion
housekeeping will not be difficult for xprtrdma:
- Inline Send buffers are registered via the local DMA key, and
are already left DMA mapped for the lifetime of a transport
connection, thus no additional handling is necessary for those
- Gathered Sends involving page cache pages _will_ need to
DMA unmap those pages after the Send completes. But like
inline send buffers, they are registered via the local DMA key,
and thus will not need to be invalidated
In addition, RPC completion will need to wait for Send completion
in the latter case. However, nearly always, the Send that conveys
the RPC Call will have completed long before the RPC Reply
arrives, and thus no additional latency will be accrued.
Design notes:
In this patch, the rpcrdma_sendctx object is introduced, and a
lock-free circular queue is added to manage a set of them per
transport.
The RPC client's send path already prevents sending more than one
RPC Call at the same time. This allows us to treat the consumer
side of the queue (rpcrdma_sendctx_get_locked) as if there is a
single consumer thread.
The producer side of the queue (rpcrdma_sendctx_put_locked) is
invoked only from the Send completion handler, which is a single
thread of execution (soft IRQ).
The only care that needs to be taken is with the tail index, which
is shared between the producer and consumer. Only the producer
updates the tail index. The consumer compares the head with the
tail to ensure that the a sendctx that is in use is never handed
out again (or, expressed more conventionally, the queue is empty).
When the sendctx queue empties completely, there are enough Sends
outstanding that posting more Send operations can result in a Send
Queue overflow. In this case, the ULP is told to wait and try again.
This introduces strong Send Queue accounting to xprtrdma.
As a final touch, Jason Gunthorpe <jgunthorpe@obsidianresearch.com>
suggested a mechanism that does not require signaling every Send.
We signal once every N Sends, and perform SGE unmapping of N Send
operations during that one completion.
Reported-by: Sagi Grimberg <sagi@grimberg.me>
Suggested-by: Jason Gunthorpe <jgunthorpe@obsidianresearch.com>
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
2017-10-20 22:48:12 +08:00
|
|
|
rpcrdma_sendctxs_destroy(buf);
|
|
|
|
|
|
2015-10-25 05:27:02 +08:00
|
|
|
while (!list_empty(&buf->rb_recv_bufs)) {
|
|
|
|
|
struct rpcrdma_rep *rep;
|
2007-09-11 01:51:18 +08:00
|
|
|
|
2018-05-05 03:35:36 +08:00
|
|
|
rep = list_first_entry(&buf->rb_recv_bufs,
|
|
|
|
|
struct rpcrdma_rep, rr_list);
|
|
|
|
|
list_del(&rep->rr_list);
|
2016-09-15 22:56:26 +08:00
|
|
|
rpcrdma_destroy_rep(rep);
|
2007-09-11 01:51:18 +08:00
|
|
|
}
|
|
|
|
|
|
2018-12-19 23:59:33 +08:00
|
|
|
while (!list_empty(&buf->rb_send_bufs)) {
|
2015-10-25 05:27:02 +08:00
|
|
|
struct rpcrdma_req *req;
|
2014-05-28 22:32:09 +08:00
|
|
|
|
2018-12-19 23:59:33 +08:00
|
|
|
req = list_first_entry(&buf->rb_send_bufs,
|
|
|
|
|
struct rpcrdma_req, rl_list);
|
|
|
|
|
list_del(&req->rl_list);
|
|
|
|
|
rpcrdma_req_destroy(req);
|
2015-10-25 05:27:02 +08:00
|
|
|
}
|
2014-05-28 22:32:09 +08:00
|
|
|
|
2017-12-15 09:57:55 +08:00
|
|
|
rpcrdma_mrs_destroy(buf);
|
2007-09-11 01:51:18 +08:00
|
|
|
}
|
|
|
|
|
|
2017-12-15 09:57:55 +08:00
|
|
|
/**
|
|
|
|
|
* rpcrdma_mr_get - Allocate an rpcrdma_mr object
|
|
|
|
|
* @r_xprt: controlling transport
|
|
|
|
|
*
|
|
|
|
|
* Returns an initialized rpcrdma_mr or NULL if no free
|
|
|
|
|
* rpcrdma_mr objects are available.
|
|
|
|
|
*/
|
|
|
|
|
struct rpcrdma_mr *
|
|
|
|
|
rpcrdma_mr_get(struct rpcrdma_xprt *r_xprt)
|
2014-07-30 05:24:36 +08:00
|
|
|
{
|
2015-05-26 23:52:06 +08:00
|
|
|
struct rpcrdma_buffer *buf = &r_xprt->rx_buf;
|
2017-12-15 09:57:55 +08:00
|
|
|
struct rpcrdma_mr *mr = NULL;
|
2015-05-26 23:52:06 +08:00
|
|
|
|
2017-12-15 09:57:55 +08:00
|
|
|
spin_lock(&buf->rb_mrlock);
|
|
|
|
|
if (!list_empty(&buf->rb_mrs))
|
|
|
|
|
mr = rpcrdma_mr_pop(&buf->rb_mrs);
|
|
|
|
|
spin_unlock(&buf->rb_mrlock);
|
2015-05-26 23:52:06 +08:00
|
|
|
|
2017-12-15 09:57:55 +08:00
|
|
|
if (!mr)
|
|
|
|
|
goto out_nomrs;
|
|
|
|
|
return mr;
|
2016-06-30 01:54:00 +08:00
|
|
|
|
2017-12-15 09:57:55 +08:00
|
|
|
out_nomrs:
|
2017-12-21 05:31:21 +08:00
|
|
|
trace_xprtrdma_nomrs(r_xprt);
|
2017-04-12 01:23:10 +08:00
|
|
|
if (r_xprt->rx_ep.rep_connected != -ENODEV)
|
|
|
|
|
schedule_delayed_work(&buf->rb_refresh_worker, 0);
|
2016-06-30 01:54:00 +08:00
|
|
|
|
|
|
|
|
/* Allow the reply handler and refresh worker to run */
|
|
|
|
|
cond_resched();
|
|
|
|
|
|
|
|
|
|
return NULL;
|
2014-07-30 05:24:36 +08:00
|
|
|
}
|
|
|
|
|
|
2017-12-15 09:58:04 +08:00
|
|
|
static void
|
|
|
|
|
__rpcrdma_mr_put(struct rpcrdma_buffer *buf, struct rpcrdma_mr *mr)
|
|
|
|
|
{
|
|
|
|
|
spin_lock(&buf->rb_mrlock);
|
|
|
|
|
rpcrdma_mr_push(mr, &buf->rb_mrs);
|
|
|
|
|
spin_unlock(&buf->rb_mrlock);
|
|
|
|
|
}
|
|
|
|
|
|
2017-12-15 09:57:55 +08:00
|
|
|
/**
|
|
|
|
|
* rpcrdma_mr_put - Release an rpcrdma_mr object
|
|
|
|
|
* @mr: object to release
|
|
|
|
|
*
|
|
|
|
|
*/
|
2015-05-26 23:52:06 +08:00
|
|
|
void
|
2017-12-15 09:57:55 +08:00
|
|
|
rpcrdma_mr_put(struct rpcrdma_mr *mr)
|
2017-12-15 09:58:04 +08:00
|
|
|
{
|
|
|
|
|
__rpcrdma_mr_put(&mr->mr_xprt->rx_buf, mr);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/**
|
|
|
|
|
* rpcrdma_mr_unmap_and_put - DMA unmap an MR and release it
|
|
|
|
|
* @mr: object to release
|
|
|
|
|
*
|
|
|
|
|
*/
|
|
|
|
|
void
|
|
|
|
|
rpcrdma_mr_unmap_and_put(struct rpcrdma_mr *mr)
|
2014-07-30 05:24:36 +08:00
|
|
|
{
|
2017-12-15 09:57:55 +08:00
|
|
|
struct rpcrdma_xprt *r_xprt = mr->mr_xprt;
|
2014-07-30 05:24:36 +08:00
|
|
|
|
2018-12-19 23:58:13 +08:00
|
|
|
if (mr->mr_dir != DMA_NONE) {
|
|
|
|
|
trace_xprtrdma_mr_unmap(mr);
|
|
|
|
|
ib_dma_unmap_sg(r_xprt->rx_ia.ri_device,
|
|
|
|
|
mr->mr_sg, mr->mr_nents, mr->mr_dir);
|
|
|
|
|
mr->mr_dir = DMA_NONE;
|
|
|
|
|
}
|
2017-12-15 09:58:04 +08:00
|
|
|
__rpcrdma_mr_put(&r_xprt->rx_buf, mr);
|
2014-07-30 05:24:36 +08:00
|
|
|
}
|
|
|
|
|
|
2018-05-05 03:35:20 +08:00
|
|
|
/**
|
|
|
|
|
* rpcrdma_buffer_get - Get a request buffer
|
|
|
|
|
* @buffers: Buffer pool from which to obtain a buffer
|
2016-09-06 23:22:49 +08:00
|
|
|
*
|
2018-05-05 03:35:20 +08:00
|
|
|
* Returns a fresh rpcrdma_req, or NULL if none are available.
|
2007-09-11 01:51:18 +08:00
|
|
|
*/
|
|
|
|
|
struct rpcrdma_req *
|
|
|
|
|
rpcrdma_buffer_get(struct rpcrdma_buffer *buffers)
|
|
|
|
|
{
|
|
|
|
|
struct rpcrdma_req *req;
|
2015-05-26 23:52:35 +08:00
|
|
|
|
2015-10-25 05:27:27 +08:00
|
|
|
spin_lock(&buffers->rb_lock);
|
2018-05-05 03:35:31 +08:00
|
|
|
req = list_first_entry_or_null(&buffers->rb_send_bufs,
|
|
|
|
|
struct rpcrdma_req, rl_list);
|
|
|
|
|
if (req)
|
|
|
|
|
list_del_init(&req->rl_list);
|
2015-10-25 05:27:27 +08:00
|
|
|
spin_unlock(&buffers->rb_lock);
|
2015-10-25 05:27:02 +08:00
|
|
|
return req;
|
2007-09-11 01:51:18 +08:00
|
|
|
}
|
|
|
|
|
|
2018-05-05 03:35:20 +08:00
|
|
|
/**
|
|
|
|
|
* rpcrdma_buffer_put - Put request/reply buffers back into pool
|
|
|
|
|
* @req: object to return
|
|
|
|
|
*
|
2007-09-11 01:51:18 +08:00
|
|
|
*/
|
|
|
|
|
void
|
|
|
|
|
rpcrdma_buffer_put(struct rpcrdma_req *req)
|
|
|
|
|
{
|
|
|
|
|
struct rpcrdma_buffer *buffers = req->rl_buffer;
|
2015-10-25 05:27:02 +08:00
|
|
|
struct rpcrdma_rep *rep = req->rl_reply;
|
2007-09-11 01:51:18 +08:00
|
|
|
|
2015-10-25 05:27:02 +08:00
|
|
|
req->rl_reply = NULL;
|
|
|
|
|
|
2015-10-25 05:27:27 +08:00
|
|
|
spin_lock(&buffers->rb_lock);
|
2018-05-05 03:35:20 +08:00
|
|
|
list_add(&req->rl_list, &buffers->rb_send_bufs);
|
xprtrdma: Fix receive buffer accounting
An RPC can terminate before its reply arrives, if a credential
problem or a soft timeout occurs. After this happens, xprtrdma
reports it is out of Receive buffers.
A Receive buffer is posted before each RPC is sent, and returned to
the buffer pool when a reply is received. If no reply is received
for an RPC, that Receive buffer remains posted. But xprtrdma tries
to post another when the next RPC is sent.
If this happens a few dozen times, there are no receive buffers left
to be posted at send time. I don't see a way for a transport
connection to recover at that point, and it will spit warnings and
unnecessarily delay RPCs on occasion for its remaining lifetime.
Commit 1e465fd4ff47 ("xprtrdma: Replace send and receive arrays")
removed a little bit of logic to detect this case and not provide
a Receive buffer so no more buffers are posted, and then transport
operation continues correctly. We didn't understand what that logic
did, and it wasn't commented, so it was removed as part of the
overhaul to support backchannel requests.
Restore it, but be wary of the need to keep extra Receives posted
to deal with backchannel requests.
Fixes: 1e465fd4ff47 ("xprtrdma: Replace send and receive arrays")
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
Reviewed-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
Signed-off-by: Trond Myklebust <trond.myklebust@primarydata.com>
2016-09-06 23:22:58 +08:00
|
|
|
if (rep) {
|
2018-05-05 03:35:20 +08:00
|
|
|
if (!rep->rr_temp) {
|
|
|
|
|
list_add(&rep->rr_list, &buffers->rb_recv_bufs);
|
|
|
|
|
rep = NULL;
|
|
|
|
|
}
|
xprtrdma: Fix receive buffer accounting
An RPC can terminate before its reply arrives, if a credential
problem or a soft timeout occurs. After this happens, xprtrdma
reports it is out of Receive buffers.
A Receive buffer is posted before each RPC is sent, and returned to
the buffer pool when a reply is received. If no reply is received
for an RPC, that Receive buffer remains posted. But xprtrdma tries
to post another when the next RPC is sent.
If this happens a few dozen times, there are no receive buffers left
to be posted at send time. I don't see a way for a transport
connection to recover at that point, and it will spit warnings and
unnecessarily delay RPCs on occasion for its remaining lifetime.
Commit 1e465fd4ff47 ("xprtrdma: Replace send and receive arrays")
removed a little bit of logic to detect this case and not provide
a Receive buffer so no more buffers are posted, and then transport
operation continues correctly. We didn't understand what that logic
did, and it wasn't commented, so it was removed as part of the
overhaul to support backchannel requests.
Restore it, but be wary of the need to keep extra Receives posted
to deal with backchannel requests.
Fixes: 1e465fd4ff47 ("xprtrdma: Replace send and receive arrays")
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
Reviewed-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
Signed-off-by: Trond Myklebust <trond.myklebust@primarydata.com>
2016-09-06 23:22:58 +08:00
|
|
|
}
|
2015-10-25 05:27:27 +08:00
|
|
|
spin_unlock(&buffers->rb_lock);
|
2018-05-05 03:35:20 +08:00
|
|
|
if (rep)
|
|
|
|
|
rpcrdma_destroy_rep(rep);
|
2007-09-11 01:51:18 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Put reply buffers back into pool when not attached to
|
2014-05-28 22:32:34 +08:00
|
|
|
* request. This happens in error conditions.
|
2007-09-11 01:51:18 +08:00
|
|
|
*/
|
|
|
|
|
void
|
|
|
|
|
rpcrdma_recv_buffer_put(struct rpcrdma_rep *rep)
|
|
|
|
|
{
|
2015-05-26 23:51:37 +08:00
|
|
|
struct rpcrdma_buffer *buffers = &rep->rr_rxprt->rx_buf;
|
2007-09-11 01:51:18 +08:00
|
|
|
|
2018-05-05 03:35:20 +08:00
|
|
|
if (!rep->rr_temp) {
|
|
|
|
|
spin_lock(&buffers->rb_lock);
|
|
|
|
|
list_add(&rep->rr_list, &buffers->rb_recv_bufs);
|
|
|
|
|
spin_unlock(&buffers->rb_lock);
|
|
|
|
|
} else {
|
|
|
|
|
rpcrdma_destroy_rep(rep);
|
|
|
|
|
}
|
2007-09-11 01:51:18 +08:00
|
|
|
}
|
|
|
|
|
|
2015-01-22 00:04:00 +08:00
|
|
|
/**
|
2016-09-15 22:56:10 +08:00
|
|
|
* rpcrdma_alloc_regbuf - allocate and DMA-map memory for SEND/RECV buffers
|
2015-01-22 00:04:00 +08:00
|
|
|
* @size: size of buffer to be allocated, in bytes
|
2016-09-15 22:56:10 +08:00
|
|
|
* @direction: direction of data movement
|
2015-01-22 00:04:00 +08:00
|
|
|
* @flags: GFP flags
|
|
|
|
|
*
|
xprtrdma: Delay DMA mapping Send and Receive buffers
Currently, each regbuf is allocated and DMA mapped at the same time.
This is done during transport creation.
When a device driver is unloaded, every DMA-mapped buffer in use by
a transport has to be unmapped, and then remapped to the new
device if the driver is loaded again. Remapping will have to be done
_after_ the connect worker has set up the new device.
But there's an ordering problem:
call_allocate, which invokes xprt_rdma_allocate which calls
rpcrdma_alloc_regbuf to allocate Send buffers, happens _before_
the connect worker can run to set up the new device.
Instead, at transport creation, allocate each buffer, but leave it
unmapped. Once the RPC carries these buffers into ->send_request, by
which time a transport connection should have been established,
check to see that the RPC's buffers have been DMA mapped. If not,
map them there.
When device driver unplug support is added, it will simply unmap all
the transport's regbufs, but it doesn't have to deallocate the
underlying memory.
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
2016-09-15 22:56:18 +08:00
|
|
|
* Returns an ERR_PTR, or a pointer to a regbuf, a buffer that
|
|
|
|
|
* can be persistently DMA-mapped for I/O.
|
2015-01-22 00:04:00 +08:00
|
|
|
*
|
|
|
|
|
* xprtrdma uses a regbuf for posting an outgoing RDMA SEND, or for
|
2016-09-15 22:56:10 +08:00
|
|
|
* receiving the payload of RDMA RECV operations. During Long Calls
|
2018-12-19 23:59:01 +08:00
|
|
|
* or Replies they may be registered externally via frwr_map.
|
2015-01-22 00:04:00 +08:00
|
|
|
*/
|
|
|
|
|
struct rpcrdma_regbuf *
|
2016-09-15 22:56:26 +08:00
|
|
|
rpcrdma_alloc_regbuf(size_t size, enum dma_data_direction direction,
|
|
|
|
|
gfp_t flags)
|
2015-01-22 00:04:00 +08:00
|
|
|
{
|
|
|
|
|
struct rpcrdma_regbuf *rb;
|
|
|
|
|
|
|
|
|
|
rb = kmalloc(sizeof(*rb) + size, flags);
|
|
|
|
|
if (rb == NULL)
|
xprtrdma: Delay DMA mapping Send and Receive buffers
Currently, each regbuf is allocated and DMA mapped at the same time.
This is done during transport creation.
When a device driver is unloaded, every DMA-mapped buffer in use by
a transport has to be unmapped, and then remapped to the new
device if the driver is loaded again. Remapping will have to be done
_after_ the connect worker has set up the new device.
But there's an ordering problem:
call_allocate, which invokes xprt_rdma_allocate which calls
rpcrdma_alloc_regbuf to allocate Send buffers, happens _before_
the connect worker can run to set up the new device.
Instead, at transport creation, allocate each buffer, but leave it
unmapped. Once the RPC carries these buffers into ->send_request, by
which time a transport connection should have been established,
check to see that the RPC's buffers have been DMA mapped. If not,
map them there.
When device driver unplug support is added, it will simply unmap all
the transport's regbufs, but it doesn't have to deallocate the
underlying memory.
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
2016-09-15 22:56:18 +08:00
|
|
|
return ERR_PTR(-ENOMEM);
|
2015-01-22 00:04:00 +08:00
|
|
|
|
xprtrdma: Delay DMA mapping Send and Receive buffers
Currently, each regbuf is allocated and DMA mapped at the same time.
This is done during transport creation.
When a device driver is unloaded, every DMA-mapped buffer in use by
a transport has to be unmapped, and then remapped to the new
device if the driver is loaded again. Remapping will have to be done
_after_ the connect worker has set up the new device.
But there's an ordering problem:
call_allocate, which invokes xprt_rdma_allocate which calls
rpcrdma_alloc_regbuf to allocate Send buffers, happens _before_
the connect worker can run to set up the new device.
Instead, at transport creation, allocate each buffer, but leave it
unmapped. Once the RPC carries these buffers into ->send_request, by
which time a transport connection should have been established,
check to see that the RPC's buffers have been DMA mapped. If not,
map them there.
When device driver unplug support is added, it will simply unmap all
the transport's regbufs, but it doesn't have to deallocate the
underlying memory.
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
2016-09-15 22:56:18 +08:00
|
|
|
rb->rg_device = NULL;
|
2016-09-15 22:56:10 +08:00
|
|
|
rb->rg_direction = direction;
|
xprtrdma: Delay DMA mapping Send and Receive buffers
Currently, each regbuf is allocated and DMA mapped at the same time.
This is done during transport creation.
When a device driver is unloaded, every DMA-mapped buffer in use by
a transport has to be unmapped, and then remapped to the new
device if the driver is loaded again. Remapping will have to be done
_after_ the connect worker has set up the new device.
But there's an ordering problem:
call_allocate, which invokes xprt_rdma_allocate which calls
rpcrdma_alloc_regbuf to allocate Send buffers, happens _before_
the connect worker can run to set up the new device.
Instead, at transport creation, allocate each buffer, but leave it
unmapped. Once the RPC carries these buffers into ->send_request, by
which time a transport connection should have been established,
check to see that the RPC's buffers have been DMA mapped. If not,
map them there.
When device driver unplug support is added, it will simply unmap all
the transport's regbufs, but it doesn't have to deallocate the
underlying memory.
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
2016-09-15 22:56:18 +08:00
|
|
|
rb->rg_iov.length = size;
|
2015-01-22 00:04:00 +08:00
|
|
|
|
|
|
|
|
return rb;
|
xprtrdma: Delay DMA mapping Send and Receive buffers
Currently, each regbuf is allocated and DMA mapped at the same time.
This is done during transport creation.
When a device driver is unloaded, every DMA-mapped buffer in use by
a transport has to be unmapped, and then remapped to the new
device if the driver is loaded again. Remapping will have to be done
_after_ the connect worker has set up the new device.
But there's an ordering problem:
call_allocate, which invokes xprt_rdma_allocate which calls
rpcrdma_alloc_regbuf to allocate Send buffers, happens _before_
the connect worker can run to set up the new device.
Instead, at transport creation, allocate each buffer, but leave it
unmapped. Once the RPC carries these buffers into ->send_request, by
which time a transport connection should have been established,
check to see that the RPC's buffers have been DMA mapped. If not,
map them there.
When device driver unplug support is added, it will simply unmap all
the transport's regbufs, but it doesn't have to deallocate the
underlying memory.
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
2016-09-15 22:56:18 +08:00
|
|
|
}
|
2015-01-22 00:04:00 +08:00
|
|
|
|
xprtrdma: Delay DMA mapping Send and Receive buffers
Currently, each regbuf is allocated and DMA mapped at the same time.
This is done during transport creation.
When a device driver is unloaded, every DMA-mapped buffer in use by
a transport has to be unmapped, and then remapped to the new
device if the driver is loaded again. Remapping will have to be done
_after_ the connect worker has set up the new device.
But there's an ordering problem:
call_allocate, which invokes xprt_rdma_allocate which calls
rpcrdma_alloc_regbuf to allocate Send buffers, happens _before_
the connect worker can run to set up the new device.
Instead, at transport creation, allocate each buffer, but leave it
unmapped. Once the RPC carries these buffers into ->send_request, by
which time a transport connection should have been established,
check to see that the RPC's buffers have been DMA mapped. If not,
map them there.
When device driver unplug support is added, it will simply unmap all
the transport's regbufs, but it doesn't have to deallocate the
underlying memory.
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
2016-09-15 22:56:18 +08:00
|
|
|
/**
|
|
|
|
|
* __rpcrdma_map_regbuf - DMA-map a regbuf
|
|
|
|
|
* @ia: controlling rpcrdma_ia
|
|
|
|
|
* @rb: regbuf to be mapped
|
|
|
|
|
*/
|
|
|
|
|
bool
|
|
|
|
|
__rpcrdma_dma_map_regbuf(struct rpcrdma_ia *ia, struct rpcrdma_regbuf *rb)
|
|
|
|
|
{
|
2017-04-12 01:23:02 +08:00
|
|
|
struct ib_device *device = ia->ri_device;
|
|
|
|
|
|
xprtrdma: Delay DMA mapping Send and Receive buffers
Currently, each regbuf is allocated and DMA mapped at the same time.
This is done during transport creation.
When a device driver is unloaded, every DMA-mapped buffer in use by
a transport has to be unmapped, and then remapped to the new
device if the driver is loaded again. Remapping will have to be done
_after_ the connect worker has set up the new device.
But there's an ordering problem:
call_allocate, which invokes xprt_rdma_allocate which calls
rpcrdma_alloc_regbuf to allocate Send buffers, happens _before_
the connect worker can run to set up the new device.
Instead, at transport creation, allocate each buffer, but leave it
unmapped. Once the RPC carries these buffers into ->send_request, by
which time a transport connection should have been established,
check to see that the RPC's buffers have been DMA mapped. If not,
map them there.
When device driver unplug support is added, it will simply unmap all
the transport's regbufs, but it doesn't have to deallocate the
underlying memory.
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
2016-09-15 22:56:18 +08:00
|
|
|
if (rb->rg_direction == DMA_NONE)
|
|
|
|
|
return false;
|
|
|
|
|
|
2017-04-12 01:23:02 +08:00
|
|
|
rb->rg_iov.addr = ib_dma_map_single(device,
|
xprtrdma: Delay DMA mapping Send and Receive buffers
Currently, each regbuf is allocated and DMA mapped at the same time.
This is done during transport creation.
When a device driver is unloaded, every DMA-mapped buffer in use by
a transport has to be unmapped, and then remapped to the new
device if the driver is loaded again. Remapping will have to be done
_after_ the connect worker has set up the new device.
But there's an ordering problem:
call_allocate, which invokes xprt_rdma_allocate which calls
rpcrdma_alloc_regbuf to allocate Send buffers, happens _before_
the connect worker can run to set up the new device.
Instead, at transport creation, allocate each buffer, but leave it
unmapped. Once the RPC carries these buffers into ->send_request, by
which time a transport connection should have been established,
check to see that the RPC's buffers have been DMA mapped. If not,
map them there.
When device driver unplug support is added, it will simply unmap all
the transport's regbufs, but it doesn't have to deallocate the
underlying memory.
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
2016-09-15 22:56:18 +08:00
|
|
|
(void *)rb->rg_base,
|
|
|
|
|
rdmab_length(rb),
|
|
|
|
|
rb->rg_direction);
|
2018-12-20 00:00:06 +08:00
|
|
|
if (ib_dma_mapping_error(device, rdmab_addr(rb))) {
|
|
|
|
|
trace_xprtrdma_dma_maperr(rdmab_addr(rb));
|
xprtrdma: Delay DMA mapping Send and Receive buffers
Currently, each regbuf is allocated and DMA mapped at the same time.
This is done during transport creation.
When a device driver is unloaded, every DMA-mapped buffer in use by
a transport has to be unmapped, and then remapped to the new
device if the driver is loaded again. Remapping will have to be done
_after_ the connect worker has set up the new device.
But there's an ordering problem:
call_allocate, which invokes xprt_rdma_allocate which calls
rpcrdma_alloc_regbuf to allocate Send buffers, happens _before_
the connect worker can run to set up the new device.
Instead, at transport creation, allocate each buffer, but leave it
unmapped. Once the RPC carries these buffers into ->send_request, by
which time a transport connection should have been established,
check to see that the RPC's buffers have been DMA mapped. If not,
map them there.
When device driver unplug support is added, it will simply unmap all
the transport's regbufs, but it doesn't have to deallocate the
underlying memory.
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
2016-09-15 22:56:18 +08:00
|
|
|
return false;
|
2018-12-20 00:00:06 +08:00
|
|
|
}
|
xprtrdma: Delay DMA mapping Send and Receive buffers
Currently, each regbuf is allocated and DMA mapped at the same time.
This is done during transport creation.
When a device driver is unloaded, every DMA-mapped buffer in use by
a transport has to be unmapped, and then remapped to the new
device if the driver is loaded again. Remapping will have to be done
_after_ the connect worker has set up the new device.
But there's an ordering problem:
call_allocate, which invokes xprt_rdma_allocate which calls
rpcrdma_alloc_regbuf to allocate Send buffers, happens _before_
the connect worker can run to set up the new device.
Instead, at transport creation, allocate each buffer, but leave it
unmapped. Once the RPC carries these buffers into ->send_request, by
which time a transport connection should have been established,
check to see that the RPC's buffers have been DMA mapped. If not,
map them there.
When device driver unplug support is added, it will simply unmap all
the transport's regbufs, but it doesn't have to deallocate the
underlying memory.
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
2016-09-15 22:56:18 +08:00
|
|
|
|
2017-04-12 01:23:02 +08:00
|
|
|
rb->rg_device = device;
|
xprtrdma: Delay DMA mapping Send and Receive buffers
Currently, each regbuf is allocated and DMA mapped at the same time.
This is done during transport creation.
When a device driver is unloaded, every DMA-mapped buffer in use by
a transport has to be unmapped, and then remapped to the new
device if the driver is loaded again. Remapping will have to be done
_after_ the connect worker has set up the new device.
But there's an ordering problem:
call_allocate, which invokes xprt_rdma_allocate which calls
rpcrdma_alloc_regbuf to allocate Send buffers, happens _before_
the connect worker can run to set up the new device.
Instead, at transport creation, allocate each buffer, but leave it
unmapped. Once the RPC carries these buffers into ->send_request, by
which time a transport connection should have been established,
check to see that the RPC's buffers have been DMA mapped. If not,
map them there.
When device driver unplug support is added, it will simply unmap all
the transport's regbufs, but it doesn't have to deallocate the
underlying memory.
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
2016-09-15 22:56:18 +08:00
|
|
|
rb->rg_iov.lkey = ia->ri_pd->local_dma_lkey;
|
|
|
|
|
return true;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static void
|
|
|
|
|
rpcrdma_dma_unmap_regbuf(struct rpcrdma_regbuf *rb)
|
|
|
|
|
{
|
2018-02-01 01:34:13 +08:00
|
|
|
if (!rb)
|
|
|
|
|
return;
|
|
|
|
|
|
xprtrdma: Delay DMA mapping Send and Receive buffers
Currently, each regbuf is allocated and DMA mapped at the same time.
This is done during transport creation.
When a device driver is unloaded, every DMA-mapped buffer in use by
a transport has to be unmapped, and then remapped to the new
device if the driver is loaded again. Remapping will have to be done
_after_ the connect worker has set up the new device.
But there's an ordering problem:
call_allocate, which invokes xprt_rdma_allocate which calls
rpcrdma_alloc_regbuf to allocate Send buffers, happens _before_
the connect worker can run to set up the new device.
Instead, at transport creation, allocate each buffer, but leave it
unmapped. Once the RPC carries these buffers into ->send_request, by
which time a transport connection should have been established,
check to see that the RPC's buffers have been DMA mapped. If not,
map them there.
When device driver unplug support is added, it will simply unmap all
the transport's regbufs, but it doesn't have to deallocate the
underlying memory.
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
2016-09-15 22:56:18 +08:00
|
|
|
if (!rpcrdma_regbuf_is_mapped(rb))
|
|
|
|
|
return;
|
|
|
|
|
|
|
|
|
|
ib_dma_unmap_single(rb->rg_device, rdmab_addr(rb),
|
|
|
|
|
rdmab_length(rb), rb->rg_direction);
|
|
|
|
|
rb->rg_device = NULL;
|
2015-01-22 00:04:00 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/**
|
|
|
|
|
* rpcrdma_free_regbuf - deregister and free registered buffer
|
|
|
|
|
* @rb: regbuf to be deregistered and freed
|
|
|
|
|
*/
|
|
|
|
|
void
|
2016-09-15 22:56:26 +08:00
|
|
|
rpcrdma_free_regbuf(struct rpcrdma_regbuf *rb)
|
2015-01-22 00:04:00 +08:00
|
|
|
{
|
xprtrdma: Delay DMA mapping Send and Receive buffers
Currently, each regbuf is allocated and DMA mapped at the same time.
This is done during transport creation.
When a device driver is unloaded, every DMA-mapped buffer in use by
a transport has to be unmapped, and then remapped to the new
device if the driver is loaded again. Remapping will have to be done
_after_ the connect worker has set up the new device.
But there's an ordering problem:
call_allocate, which invokes xprt_rdma_allocate which calls
rpcrdma_alloc_regbuf to allocate Send buffers, happens _before_
the connect worker can run to set up the new device.
Instead, at transport creation, allocate each buffer, but leave it
unmapped. Once the RPC carries these buffers into ->send_request, by
which time a transport connection should have been established,
check to see that the RPC's buffers have been DMA mapped. If not,
map them there.
When device driver unplug support is added, it will simply unmap all
the transport's regbufs, but it doesn't have to deallocate the
underlying memory.
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
2016-09-15 22:56:18 +08:00
|
|
|
rpcrdma_dma_unmap_regbuf(rb);
|
2015-08-04 01:03:20 +08:00
|
|
|
kfree(rb);
|
2015-01-22 00:04:00 +08:00
|
|
|
}
|
|
|
|
|
|
2018-12-20 00:00:32 +08:00
|
|
|
/**
|
|
|
|
|
* rpcrdma_ep_post - Post WRs to a transport's Send Queue
|
|
|
|
|
* @ia: transport's device information
|
|
|
|
|
* @ep: transport's RDMA endpoint information
|
|
|
|
|
* @req: rpcrdma_req containing the Send WR to post
|
2007-09-11 01:51:18 +08:00
|
|
|
*
|
2018-12-20 00:00:32 +08:00
|
|
|
* Returns 0 if the post was successful, otherwise -ENOTCONN
|
|
|
|
|
* is returned.
|
2007-09-11 01:51:18 +08:00
|
|
|
*/
|
|
|
|
|
int
|
|
|
|
|
rpcrdma_ep_post(struct rpcrdma_ia *ia,
|
|
|
|
|
struct rpcrdma_ep *ep,
|
|
|
|
|
struct rpcrdma_req *req)
|
|
|
|
|
{
|
xprtrdma: Add data structure to manage RDMA Send arguments
Problem statement:
Recently Sagi Grimberg <sagi@grimberg.me> observed that kernel RDMA-
enabled storage initiators don't handle delayed Send completion
correctly. If Send completion is delayed beyond the end of a ULP
transaction, the ULP may release resources that are still being used
by the HCA to complete a long-running Send operation.
This is a common design trait amongst our initiators. Most Send
operations are faster than the ULP transaction they are part of.
Waiting for a completion for these is typically unnecessary.
Infrequently, a network partition or some other problem crops up
where an ordering problem can occur. In NFS parlance, the RPC Reply
arrives and completes the RPC, but the HCA is still retrying the
Send WR that conveyed the RPC Call. In this case, the HCA can try
to use memory that has been invalidated or DMA unmapped, and the
connection is lost. If that memory has been re-used for something
else (possibly not related to NFS), and the Send retransmission
exposes that data on the wire.
Thus we cannot assume that it is safe to release Send-related
resources just because a ULP reply has arrived.
After some analysis, we have determined that the completion
housekeeping will not be difficult for xprtrdma:
- Inline Send buffers are registered via the local DMA key, and
are already left DMA mapped for the lifetime of a transport
connection, thus no additional handling is necessary for those
- Gathered Sends involving page cache pages _will_ need to
DMA unmap those pages after the Send completes. But like
inline send buffers, they are registered via the local DMA key,
and thus will not need to be invalidated
In addition, RPC completion will need to wait for Send completion
in the latter case. However, nearly always, the Send that conveys
the RPC Call will have completed long before the RPC Reply
arrives, and thus no additional latency will be accrued.
Design notes:
In this patch, the rpcrdma_sendctx object is introduced, and a
lock-free circular queue is added to manage a set of them per
transport.
The RPC client's send path already prevents sending more than one
RPC Call at the same time. This allows us to treat the consumer
side of the queue (rpcrdma_sendctx_get_locked) as if there is a
single consumer thread.
The producer side of the queue (rpcrdma_sendctx_put_locked) is
invoked only from the Send completion handler, which is a single
thread of execution (soft IRQ).
The only care that needs to be taken is with the tail index, which
is shared between the producer and consumer. Only the producer
updates the tail index. The consumer compares the head with the
tail to ensure that the a sendctx that is in use is never handed
out again (or, expressed more conventionally, the queue is empty).
When the sendctx queue empties completely, there are enough Sends
outstanding that posting more Send operations can result in a Send
Queue overflow. In this case, the ULP is told to wait and try again.
This introduces strong Send Queue accounting to xprtrdma.
As a final touch, Jason Gunthorpe <jgunthorpe@obsidianresearch.com>
suggested a mechanism that does not require signaling every Send.
We signal once every N Sends, and perform SGE unmapping of N Send
operations during that one completion.
Reported-by: Sagi Grimberg <sagi@grimberg.me>
Suggested-by: Jason Gunthorpe <jgunthorpe@obsidianresearch.com>
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
2017-10-20 22:48:12 +08:00
|
|
|
struct ib_send_wr *send_wr = &req->rl_sendctx->sc_wr;
|
xprtrdma: Use gathered Send for large inline messages
An RPC Call message that is sent inline but that has a data payload
(ie, one or more items in rq_snd_buf's page list) must be "pulled
up:"
- call_allocate has to reserve enough RPC Call buffer space to
accommodate the data payload
- call_transmit has to memcopy the rq_snd_buf's page list and tail
into its head iovec before it is sent
As the inline threshold is increased beyond its current 1KB default,
however, this means data payloads of more than a few KB are copied
by the host CPU. For example, if the inline threshold is increased
just to 4KB, then NFS WRITE requests up to 4KB would involve a
memcpy of the NFS WRITE's payload data into the RPC Call buffer.
This is an undesirable amount of participation by the host CPU.
The inline threshold may be much larger than 4KB in the future,
after negotiation with a peer server.
Instead of copying the components of rq_snd_buf into its head iovec,
construct a gather list of these components, and send them all in
place. The same approach is already used in the Linux server's
RPC-over-RDMA reply path.
This mechanism also eliminates the need for rpcrdma_tail_pullup,
which is used to manage the XDR pad and trailing inline content when
a Read list is present.
This requires that the pages in rq_snd_buf's page list be DMA-mapped
during marshaling, and unmapped when a data-bearing RPC is
completed. This is slightly less efficient for very small I/O
payloads, but significantly more efficient as data payload size and
inline threshold increase past a kilobyte.
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
2016-09-15 22:57:24 +08:00
|
|
|
int rc;
|
2007-09-11 01:51:18 +08:00
|
|
|
|
2017-10-20 22:48:36 +08:00
|
|
|
if (!ep->rep_send_count ||
|
|
|
|
|
test_bit(RPCRDMA_REQ_F_TX_RESOURCES, &req->rl_flags)) {
|
xprtrdma: Add data structure to manage RDMA Send arguments
Problem statement:
Recently Sagi Grimberg <sagi@grimberg.me> observed that kernel RDMA-
enabled storage initiators don't handle delayed Send completion
correctly. If Send completion is delayed beyond the end of a ULP
transaction, the ULP may release resources that are still being used
by the HCA to complete a long-running Send operation.
This is a common design trait amongst our initiators. Most Send
operations are faster than the ULP transaction they are part of.
Waiting for a completion for these is typically unnecessary.
Infrequently, a network partition or some other problem crops up
where an ordering problem can occur. In NFS parlance, the RPC Reply
arrives and completes the RPC, but the HCA is still retrying the
Send WR that conveyed the RPC Call. In this case, the HCA can try
to use memory that has been invalidated or DMA unmapped, and the
connection is lost. If that memory has been re-used for something
else (possibly not related to NFS), and the Send retransmission
exposes that data on the wire.
Thus we cannot assume that it is safe to release Send-related
resources just because a ULP reply has arrived.
After some analysis, we have determined that the completion
housekeeping will not be difficult for xprtrdma:
- Inline Send buffers are registered via the local DMA key, and
are already left DMA mapped for the lifetime of a transport
connection, thus no additional handling is necessary for those
- Gathered Sends involving page cache pages _will_ need to
DMA unmap those pages after the Send completes. But like
inline send buffers, they are registered via the local DMA key,
and thus will not need to be invalidated
In addition, RPC completion will need to wait for Send completion
in the latter case. However, nearly always, the Send that conveys
the RPC Call will have completed long before the RPC Reply
arrives, and thus no additional latency will be accrued.
Design notes:
In this patch, the rpcrdma_sendctx object is introduced, and a
lock-free circular queue is added to manage a set of them per
transport.
The RPC client's send path already prevents sending more than one
RPC Call at the same time. This allows us to treat the consumer
side of the queue (rpcrdma_sendctx_get_locked) as if there is a
single consumer thread.
The producer side of the queue (rpcrdma_sendctx_put_locked) is
invoked only from the Send completion handler, which is a single
thread of execution (soft IRQ).
The only care that needs to be taken is with the tail index, which
is shared between the producer and consumer. Only the producer
updates the tail index. The consumer compares the head with the
tail to ensure that the a sendctx that is in use is never handed
out again (or, expressed more conventionally, the queue is empty).
When the sendctx queue empties completely, there are enough Sends
outstanding that posting more Send operations can result in a Send
Queue overflow. In this case, the ULP is told to wait and try again.
This introduces strong Send Queue accounting to xprtrdma.
As a final touch, Jason Gunthorpe <jgunthorpe@obsidianresearch.com>
suggested a mechanism that does not require signaling every Send.
We signal once every N Sends, and perform SGE unmapping of N Send
operations during that one completion.
Reported-by: Sagi Grimberg <sagi@grimberg.me>
Suggested-by: Jason Gunthorpe <jgunthorpe@obsidianresearch.com>
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
2017-10-20 22:48:12 +08:00
|
|
|
send_wr->send_flags |= IB_SEND_SIGNALED;
|
|
|
|
|
ep->rep_send_count = ep->rep_send_batch;
|
|
|
|
|
} else {
|
|
|
|
|
send_wr->send_flags &= ~IB_SEND_SIGNALED;
|
|
|
|
|
--ep->rep_send_count;
|
|
|
|
|
}
|
2017-12-21 05:30:40 +08:00
|
|
|
|
2018-12-19 23:59:01 +08:00
|
|
|
rc = frwr_send(ia, req);
|
2017-12-21 05:30:40 +08:00
|
|
|
trace_xprtrdma_post_send(req, rc);
|
2007-09-11 01:51:18 +08:00
|
|
|
if (rc)
|
2017-12-21 05:30:40 +08:00
|
|
|
return -ENOTCONN;
|
2016-06-30 01:53:43 +08:00
|
|
|
return 0;
|
2007-09-11 01:51:18 +08:00
|
|
|
}
|
|
|
|
|
|
2018-12-19 23:58:24 +08:00
|
|
|
static void
|
2018-05-05 03:35:20 +08:00
|
|
|
rpcrdma_post_recvs(struct rpcrdma_xprt *r_xprt, bool temp)
|
2015-10-25 05:27:43 +08:00
|
|
|
{
|
2018-05-05 03:35:20 +08:00
|
|
|
struct rpcrdma_buffer *buf = &r_xprt->rx_buf;
|
2018-12-19 23:58:24 +08:00
|
|
|
struct rpcrdma_ep *ep = &r_xprt->rx_ep;
|
2018-05-05 03:35:20 +08:00
|
|
|
struct ib_recv_wr *wr, *bad_wr;
|
|
|
|
|
int needed, count, rc;
|
2015-10-25 05:27:43 +08:00
|
|
|
|
2018-10-02 02:26:35 +08:00
|
|
|
rc = 0;
|
|
|
|
|
count = 0;
|
2018-05-05 03:35:20 +08:00
|
|
|
needed = buf->rb_credits + (buf->rb_bc_srv_max_requests << 1);
|
2018-12-19 23:58:24 +08:00
|
|
|
if (ep->rep_receive_count > needed)
|
2018-10-02 02:26:35 +08:00
|
|
|
goto out;
|
2018-12-19 23:58:24 +08:00
|
|
|
needed -= ep->rep_receive_count;
|
2015-10-25 05:27:43 +08:00
|
|
|
|
2018-05-05 03:35:20 +08:00
|
|
|
count = 0;
|
|
|
|
|
wr = NULL;
|
|
|
|
|
while (needed) {
|
|
|
|
|
struct rpcrdma_regbuf *rb;
|
|
|
|
|
struct rpcrdma_rep *rep;
|
2015-10-25 05:27:43 +08:00
|
|
|
|
2018-05-05 03:35:20 +08:00
|
|
|
spin_lock(&buf->rb_lock);
|
|
|
|
|
rep = list_first_entry_or_null(&buf->rb_recv_bufs,
|
|
|
|
|
struct rpcrdma_rep, rr_list);
|
|
|
|
|
if (likely(rep))
|
|
|
|
|
list_del(&rep->rr_list);
|
|
|
|
|
spin_unlock(&buf->rb_lock);
|
|
|
|
|
if (!rep) {
|
|
|
|
|
if (rpcrdma_create_rep(r_xprt, temp))
|
|
|
|
|
break;
|
|
|
|
|
continue;
|
|
|
|
|
}
|
2015-10-25 05:27:43 +08:00
|
|
|
|
2018-05-05 03:35:20 +08:00
|
|
|
rb = rep->rr_rdmabuf;
|
|
|
|
|
if (!rpcrdma_regbuf_is_mapped(rb)) {
|
|
|
|
|
if (!__rpcrdma_dma_map_regbuf(&r_xprt->rx_ia, rb)) {
|
|
|
|
|
rpcrdma_recv_buffer_put(rep);
|
|
|
|
|
break;
|
|
|
|
|
}
|
|
|
|
|
}
|
2015-10-25 05:27:43 +08:00
|
|
|
|
2018-05-05 03:35:20 +08:00
|
|
|
trace_xprtrdma_post_recv(rep->rr_recv_wr.wr_cqe);
|
|
|
|
|
rep->rr_recv_wr.next = wr;
|
|
|
|
|
wr = &rep->rr_recv_wr;
|
|
|
|
|
++count;
|
|
|
|
|
--needed;
|
|
|
|
|
}
|
|
|
|
|
if (!count)
|
2018-10-02 02:26:35 +08:00
|
|
|
goto out;
|
2018-05-05 03:35:20 +08:00
|
|
|
|
2018-07-19 00:25:32 +08:00
|
|
|
rc = ib_post_recv(r_xprt->rx_ia.ri_id->qp, wr,
|
|
|
|
|
(const struct ib_recv_wr **)&bad_wr);
|
2018-05-05 03:35:20 +08:00
|
|
|
if (rc) {
|
|
|
|
|
for (wr = bad_wr; wr; wr = wr->next) {
|
|
|
|
|
struct rpcrdma_rep *rep;
|
|
|
|
|
|
|
|
|
|
rep = container_of(wr, struct rpcrdma_rep, rr_recv_wr);
|
|
|
|
|
rpcrdma_recv_buffer_put(rep);
|
|
|
|
|
--count;
|
|
|
|
|
}
|
|
|
|
|
}
|
2018-12-19 23:58:24 +08:00
|
|
|
ep->rep_receive_count += count;
|
2018-10-02 02:26:35 +08:00
|
|
|
out:
|
2018-05-05 03:35:20 +08:00
|
|
|
trace_xprtrdma_post_recvs(r_xprt, count, rc);
|
2015-10-25 05:27:43 +08:00
|
|
|
}
|
![](/avatar/511417582bfa18e9a88da2ada8339d64?size=36 "\\"Talpey, Thomas\"){kind=small}
