mirror of
https://github.com/edk2-porting/linux-next.git
synced 2024-12-29 23:53:55 +08:00
e122d845a0
The changes introduced to allow rxrpc calls to be retried creates an issue
when it comes to refcounting afs_call structs. The problem is that when
rxrpc_send_data() queues the last packet for an asynchronous call, the
following sequence can occur:
(1) The notify_end_tx callback is invoked which causes the state in the
afs_call to be changed from AFS_CALL_CL_REQUESTING or
AFS_CALL_SV_REPLYING.
(2) afs_deliver_to_call() can then process event notifications from rxrpc
on the async_work queue.
(3) Delivery of events, such as an abort from the server, can cause the
afs_call state to be changed to AFS_CALL_COMPLETE on async_work.
(4) For an asynchronous call, afs_process_async_call() notes that the call
is complete and tried to clean up all the refs on async_work.
(5) rxrpc_send_data() might return the amount of data transferred
(success) or an error - which could in turn reflect a local error or a
received error.
Synchronising the clean up after rxrpc_kernel_send_data() returns an error
with the asynchronous cleanup is then tricky to get right.
Mostly revert commit c038a58ccf. The two API
functions the original commit added aren't currently used. This makes
rxrpc_kernel_send_data() always return successfully if it queued the data
it was given.
Note that this doesn't affect synchronous calls since their Rx notification
function merely pokes a wait queue and does not refcounting. The
asynchronous call notification function *has* to do refcounting and pass a
ref over the work item to avoid the need to sync the workqueue in call
cleanup.
Signed-off-by: David Howells <dhowells@redhat.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
1135 lines
44 KiB
Plaintext
1135 lines
44 KiB
Plaintext
======================
|
|
RxRPC NETWORK PROTOCOL
|
|
======================
|
|
|
|
The RxRPC protocol driver provides a reliable two-phase transport on top of UDP
|
|
that can be used to perform RxRPC remote operations. This is done over sockets
|
|
of AF_RXRPC family, using sendmsg() and recvmsg() with control data to send and
|
|
receive data, aborts and errors.
|
|
|
|
Contents of this document:
|
|
|
|
(*) Overview.
|
|
|
|
(*) RxRPC protocol summary.
|
|
|
|
(*) AF_RXRPC driver model.
|
|
|
|
(*) Control messages.
|
|
|
|
(*) Socket options.
|
|
|
|
(*) Security.
|
|
|
|
(*) Example client usage.
|
|
|
|
(*) Example server usage.
|
|
|
|
(*) AF_RXRPC kernel interface.
|
|
|
|
(*) Configurable parameters.
|
|
|
|
|
|
========
|
|
OVERVIEW
|
|
========
|
|
|
|
RxRPC is a two-layer protocol. There is a session layer which provides
|
|
reliable virtual connections using UDP over IPv4 (or IPv6) as the transport
|
|
layer, but implements a real network protocol; and there's the presentation
|
|
layer which renders structured data to binary blobs and back again using XDR
|
|
(as does SunRPC):
|
|
|
|
+-------------+
|
|
| Application |
|
|
+-------------+
|
|
| XDR | Presentation
|
|
+-------------+
|
|
| RxRPC | Session
|
|
+-------------+
|
|
| UDP | Transport
|
|
+-------------+
|
|
|
|
|
|
AF_RXRPC provides:
|
|
|
|
(1) Part of an RxRPC facility for both kernel and userspace applications by
|
|
making the session part of it a Linux network protocol (AF_RXRPC).
|
|
|
|
(2) A two-phase protocol. The client transmits a blob (the request) and then
|
|
receives a blob (the reply), and the server receives the request and then
|
|
transmits the reply.
|
|
|
|
(3) Retention of the reusable bits of the transport system set up for one call
|
|
to speed up subsequent calls.
|
|
|
|
(4) A secure protocol, using the Linux kernel's key retention facility to
|
|
manage security on the client end. The server end must of necessity be
|
|
more active in security negotiations.
|
|
|
|
AF_RXRPC does not provide XDR marshalling/presentation facilities. That is
|
|
left to the application. AF_RXRPC only deals in blobs. Even the operation ID
|
|
is just the first four bytes of the request blob, and as such is beyond the
|
|
kernel's interest.
|
|
|
|
|
|
Sockets of AF_RXRPC family are:
|
|
|
|
(1) created as type SOCK_DGRAM;
|
|
|
|
(2) provided with a protocol of the type of underlying transport they're going
|
|
to use - currently only PF_INET is supported.
|
|
|
|
|
|
The Andrew File System (AFS) is an example of an application that uses this and
|
|
that has both kernel (filesystem) and userspace (utility) components.
|
|
|
|
|
|
======================
|
|
RXRPC PROTOCOL SUMMARY
|
|
======================
|
|
|
|
An overview of the RxRPC protocol:
|
|
|
|
(*) RxRPC sits on top of another networking protocol (UDP is the only option
|
|
currently), and uses this to provide network transport. UDP ports, for
|
|
example, provide transport endpoints.
|
|
|
|
(*) RxRPC supports multiple virtual "connections" from any given transport
|
|
endpoint, thus allowing the endpoints to be shared, even to the same
|
|
remote endpoint.
|
|
|
|
(*) Each connection goes to a particular "service". A connection may not go
|
|
to multiple services. A service may be considered the RxRPC equivalent of
|
|
a port number. AF_RXRPC permits multiple services to share an endpoint.
|
|
|
|
(*) Client-originating packets are marked, thus a transport endpoint can be
|
|
shared between client and server connections (connections have a
|
|
direction).
|
|
|
|
(*) Up to a billion connections may be supported concurrently between one
|
|
local transport endpoint and one service on one remote endpoint. An RxRPC
|
|
connection is described by seven numbers:
|
|
|
|
Local address }
|
|
Local port } Transport (UDP) address
|
|
Remote address }
|
|
Remote port }
|
|
Direction
|
|
Connection ID
|
|
Service ID
|
|
|
|
(*) Each RxRPC operation is a "call". A connection may make up to four
|
|
billion calls, but only up to four calls may be in progress on a
|
|
connection at any one time.
|
|
|
|
(*) Calls are two-phase and asymmetric: the client sends its request data,
|
|
which the service receives; then the service transmits the reply data
|
|
which the client receives.
|
|
|
|
(*) The data blobs are of indefinite size, the end of a phase is marked with a
|
|
flag in the packet. The number of packets of data making up one blob may
|
|
not exceed 4 billion, however, as this would cause the sequence number to
|
|
wrap.
|
|
|
|
(*) The first four bytes of the request data are the service operation ID.
|
|
|
|
(*) Security is negotiated on a per-connection basis. The connection is
|
|
initiated by the first data packet on it arriving. If security is
|
|
requested, the server then issues a "challenge" and then the client
|
|
replies with a "response". If the response is successful, the security is
|
|
set for the lifetime of that connection, and all subsequent calls made
|
|
upon it use that same security. In the event that the server lets a
|
|
connection lapse before the client, the security will be renegotiated if
|
|
the client uses the connection again.
|
|
|
|
(*) Calls use ACK packets to handle reliability. Data packets are also
|
|
explicitly sequenced per call.
|
|
|
|
(*) There are two types of positive acknowledgment: hard-ACKs and soft-ACKs.
|
|
A hard-ACK indicates to the far side that all the data received to a point
|
|
has been received and processed; a soft-ACK indicates that the data has
|
|
been received but may yet be discarded and re-requested. The sender may
|
|
not discard any transmittable packets until they've been hard-ACK'd.
|
|
|
|
(*) Reception of a reply data packet implicitly hard-ACK's all the data
|
|
packets that make up the request.
|
|
|
|
(*) An call is complete when the request has been sent, the reply has been
|
|
received and the final hard-ACK on the last packet of the reply has
|
|
reached the server.
|
|
|
|
(*) An call may be aborted by either end at any time up to its completion.
|
|
|
|
|
|
=====================
|
|
AF_RXRPC DRIVER MODEL
|
|
=====================
|
|
|
|
About the AF_RXRPC driver:
|
|
|
|
(*) The AF_RXRPC protocol transparently uses internal sockets of the transport
|
|
protocol to represent transport endpoints.
|
|
|
|
(*) AF_RXRPC sockets map onto RxRPC connection bundles. Actual RxRPC
|
|
connections are handled transparently. One client socket may be used to
|
|
make multiple simultaneous calls to the same service. One server socket
|
|
may handle calls from many clients.
|
|
|
|
(*) Additional parallel client connections will be initiated to support extra
|
|
concurrent calls, up to a tunable limit.
|
|
|
|
(*) Each connection is retained for a certain amount of time [tunable] after
|
|
the last call currently using it has completed in case a new call is made
|
|
that could reuse it.
|
|
|
|
(*) Each internal UDP socket is retained [tunable] for a certain amount of
|
|
time [tunable] after the last connection using it discarded, in case a new
|
|
connection is made that could use it.
|
|
|
|
(*) A client-side connection is only shared between calls if they have have
|
|
the same key struct describing their security (and assuming the calls
|
|
would otherwise share the connection). Non-secured calls would also be
|
|
able to share connections with each other.
|
|
|
|
(*) A server-side connection is shared if the client says it is.
|
|
|
|
(*) ACK'ing is handled by the protocol driver automatically, including ping
|
|
replying.
|
|
|
|
(*) SO_KEEPALIVE automatically pings the other side to keep the connection
|
|
alive [TODO].
|
|
|
|
(*) If an ICMP error is received, all calls affected by that error will be
|
|
aborted with an appropriate network error passed through recvmsg().
|
|
|
|
|
|
Interaction with the user of the RxRPC socket:
|
|
|
|
(*) A socket is made into a server socket by binding an address with a
|
|
non-zero service ID.
|
|
|
|
(*) In the client, sending a request is achieved with one or more sendmsgs,
|
|
followed by the reply being received with one or more recvmsgs.
|
|
|
|
(*) The first sendmsg for a request to be sent from a client contains a tag to
|
|
be used in all other sendmsgs or recvmsgs associated with that call. The
|
|
tag is carried in the control data.
|
|
|
|
(*) connect() is used to supply a default destination address for a client
|
|
socket. This may be overridden by supplying an alternate address to the
|
|
first sendmsg() of a call (struct msghdr::msg_name).
|
|
|
|
(*) If connect() is called on an unbound client, a random local port will
|
|
bound before the operation takes place.
|
|
|
|
(*) A server socket may also be used to make client calls. To do this, the
|
|
first sendmsg() of the call must specify the target address. The server's
|
|
transport endpoint is used to send the packets.
|
|
|
|
(*) Once the application has received the last message associated with a call,
|
|
the tag is guaranteed not to be seen again, and so it can be used to pin
|
|
client resources. A new call can then be initiated with the same tag
|
|
without fear of interference.
|
|
|
|
(*) In the server, a request is received with one or more recvmsgs, then the
|
|
the reply is transmitted with one or more sendmsgs, and then the final ACK
|
|
is received with a last recvmsg.
|
|
|
|
(*) When sending data for a call, sendmsg is given MSG_MORE if there's more
|
|
data to come on that call.
|
|
|
|
(*) When receiving data for a call, recvmsg flags MSG_MORE if there's more
|
|
data to come for that call.
|
|
|
|
(*) When receiving data or messages for a call, MSG_EOR is flagged by recvmsg
|
|
to indicate the terminal message for that call.
|
|
|
|
(*) A call may be aborted by adding an abort control message to the control
|
|
data. Issuing an abort terminates the kernel's use of that call's tag.
|
|
Any messages waiting in the receive queue for that call will be discarded.
|
|
|
|
(*) Aborts, busy notifications and challenge packets are delivered by recvmsg,
|
|
and control data messages will be set to indicate the context. Receiving
|
|
an abort or a busy message terminates the kernel's use of that call's tag.
|
|
|
|
(*) The control data part of the msghdr struct is used for a number of things:
|
|
|
|
(*) The tag of the intended or affected call.
|
|
|
|
(*) Sending or receiving errors, aborts and busy notifications.
|
|
|
|
(*) Notifications of incoming calls.
|
|
|
|
(*) Sending debug requests and receiving debug replies [TODO].
|
|
|
|
(*) When the kernel has received and set up an incoming call, it sends a
|
|
message to server application to let it know there's a new call awaiting
|
|
its acceptance [recvmsg reports a special control message]. The server
|
|
application then uses sendmsg to assign a tag to the new call. Once that
|
|
is done, the first part of the request data will be delivered by recvmsg.
|
|
|
|
(*) The server application has to provide the server socket with a keyring of
|
|
secret keys corresponding to the security types it permits. When a secure
|
|
connection is being set up, the kernel looks up the appropriate secret key
|
|
in the keyring and then sends a challenge packet to the client and
|
|
receives a response packet. The kernel then checks the authorisation of
|
|
the packet and either aborts the connection or sets up the security.
|
|
|
|
(*) The name of the key a client will use to secure its communications is
|
|
nominated by a socket option.
|
|
|
|
|
|
Notes on sendmsg:
|
|
|
|
(*) MSG_WAITALL can be set to tell sendmsg to ignore signals if the peer is
|
|
making progress at accepting packets within a reasonable time such that we
|
|
manage to queue up all the data for transmission. This requires the
|
|
client to accept at least one packet per 2*RTT time period.
|
|
|
|
If this isn't set, sendmsg() will return immediately, either returning
|
|
EINTR/ERESTARTSYS if nothing was consumed or returning the amount of data
|
|
consumed.
|
|
|
|
|
|
Notes on recvmsg:
|
|
|
|
(*) If there's a sequence of data messages belonging to a particular call on
|
|
the receive queue, then recvmsg will keep working through them until:
|
|
|
|
(a) it meets the end of that call's received data,
|
|
|
|
(b) it meets a non-data message,
|
|
|
|
(c) it meets a message belonging to a different call, or
|
|
|
|
(d) it fills the user buffer.
|
|
|
|
If recvmsg is called in blocking mode, it will keep sleeping, awaiting the
|
|
reception of further data, until one of the above four conditions is met.
|
|
|
|
(2) MSG_PEEK operates similarly, but will return immediately if it has put any
|
|
data in the buffer rather than sleeping until it can fill the buffer.
|
|
|
|
(3) If a data message is only partially consumed in filling a user buffer,
|
|
then the remainder of that message will be left on the front of the queue
|
|
for the next taker. MSG_TRUNC will never be flagged.
|
|
|
|
(4) If there is more data to be had on a call (it hasn't copied the last byte
|
|
of the last data message in that phase yet), then MSG_MORE will be
|
|
flagged.
|
|
|
|
|
|
================
|
|
CONTROL MESSAGES
|
|
================
|
|
|
|
AF_RXRPC makes use of control messages in sendmsg() and recvmsg() to multiplex
|
|
calls, to invoke certain actions and to report certain conditions. These are:
|
|
|
|
MESSAGE ID SRT DATA MEANING
|
|
======================= === =========== ===============================
|
|
RXRPC_USER_CALL_ID sr- User ID App's call specifier
|
|
RXRPC_ABORT srt Abort code Abort code to issue/received
|
|
RXRPC_ACK -rt n/a Final ACK received
|
|
RXRPC_NET_ERROR -rt error num Network error on call
|
|
RXRPC_BUSY -rt n/a Call rejected (server busy)
|
|
RXRPC_LOCAL_ERROR -rt error num Local error encountered
|
|
RXRPC_NEW_CALL -r- n/a New call received
|
|
RXRPC_ACCEPT s-- n/a Accept new call
|
|
RXRPC_EXCLUSIVE_CALL s-- n/a Make an exclusive client call
|
|
RXRPC_UPGRADE_SERVICE s-- n/a Client call can be upgraded
|
|
RXRPC_TX_LENGTH s-- data len Total length of Tx data
|
|
|
|
(SRT = usable in Sendmsg / delivered by Recvmsg / Terminal message)
|
|
|
|
(*) RXRPC_USER_CALL_ID
|
|
|
|
This is used to indicate the application's call ID. It's an unsigned long
|
|
that the app specifies in the client by attaching it to the first data
|
|
message or in the server by passing it in association with an RXRPC_ACCEPT
|
|
message. recvmsg() passes it in conjunction with all messages except
|
|
those of the RXRPC_NEW_CALL message.
|
|
|
|
(*) RXRPC_ABORT
|
|
|
|
This is can be used by an application to abort a call by passing it to
|
|
sendmsg, or it can be delivered by recvmsg to indicate a remote abort was
|
|
received. Either way, it must be associated with an RXRPC_USER_CALL_ID to
|
|
specify the call affected. If an abort is being sent, then error EBADSLT
|
|
will be returned if there is no call with that user ID.
|
|
|
|
(*) RXRPC_ACK
|
|
|
|
This is delivered to a server application to indicate that the final ACK
|
|
of a call was received from the client. It will be associated with an
|
|
RXRPC_USER_CALL_ID to indicate the call that's now complete.
|
|
|
|
(*) RXRPC_NET_ERROR
|
|
|
|
This is delivered to an application to indicate that an ICMP error message
|
|
was encountered in the process of trying to talk to the peer. An
|
|
errno-class integer value will be included in the control message data
|
|
indicating the problem, and an RXRPC_USER_CALL_ID will indicate the call
|
|
affected.
|
|
|
|
(*) RXRPC_BUSY
|
|
|
|
This is delivered to a client application to indicate that a call was
|
|
rejected by the server due to the server being busy. It will be
|
|
associated with an RXRPC_USER_CALL_ID to indicate the rejected call.
|
|
|
|
(*) RXRPC_LOCAL_ERROR
|
|
|
|
This is delivered to an application to indicate that a local error was
|
|
encountered and that a call has been aborted because of it. An
|
|
errno-class integer value will be included in the control message data
|
|
indicating the problem, and an RXRPC_USER_CALL_ID will indicate the call
|
|
affected.
|
|
|
|
(*) RXRPC_NEW_CALL
|
|
|
|
This is delivered to indicate to a server application that a new call has
|
|
arrived and is awaiting acceptance. No user ID is associated with this,
|
|
as a user ID must subsequently be assigned by doing an RXRPC_ACCEPT.
|
|
|
|
(*) RXRPC_ACCEPT
|
|
|
|
This is used by a server application to attempt to accept a call and
|
|
assign it a user ID. It should be associated with an RXRPC_USER_CALL_ID
|
|
to indicate the user ID to be assigned. If there is no call to be
|
|
accepted (it may have timed out, been aborted, etc.), then sendmsg will
|
|
return error ENODATA. If the user ID is already in use by another call,
|
|
then error EBADSLT will be returned.
|
|
|
|
(*) RXRPC_EXCLUSIVE_CALL
|
|
|
|
This is used to indicate that a client call should be made on a one-off
|
|
connection. The connection is discarded once the call has terminated.
|
|
|
|
(*) RXRPC_UPGRADE_SERVICE
|
|
|
|
This is used to make a client call to probe if the specified service ID
|
|
may be upgraded by the server. The caller must check msg_name returned to
|
|
recvmsg() for the service ID actually in use. The operation probed must
|
|
be one that takes the same arguments in both services.
|
|
|
|
Once this has been used to establish the upgrade capability (or lack
|
|
thereof) of the server, the service ID returned should be used for all
|
|
future communication to that server and RXRPC_UPGRADE_SERVICE should no
|
|
longer be set.
|
|
|
|
(*) RXRPC_TX_LENGTH
|
|
|
|
This is used to inform the kernel of the total amount of data that is
|
|
going to be transmitted by a call (whether in a client request or a
|
|
service response). If given, it allows the kernel to encrypt from the
|
|
userspace buffer directly to the packet buffers, rather than copying into
|
|
the buffer and then encrypting in place. This may only be given with the
|
|
first sendmsg() providing data for a call. EMSGSIZE will be generated if
|
|
the amount of data actually given is different.
|
|
|
|
This takes a parameter of __s64 type that indicates how much will be
|
|
transmitted. This may not be less than zero.
|
|
|
|
The symbol RXRPC__SUPPORTED is defined as one more than the highest control
|
|
message type supported. At run time this can be queried by means of the
|
|
RXRPC_SUPPORTED_CMSG socket option (see below).
|
|
|
|
|
|
==============
|
|
SOCKET OPTIONS
|
|
==============
|
|
|
|
AF_RXRPC sockets support a few socket options at the SOL_RXRPC level:
|
|
|
|
(*) RXRPC_SECURITY_KEY
|
|
|
|
This is used to specify the description of the key to be used. The key is
|
|
extracted from the calling process's keyrings with request_key() and
|
|
should be of "rxrpc" type.
|
|
|
|
The optval pointer points to the description string, and optlen indicates
|
|
how long the string is, without the NUL terminator.
|
|
|
|
(*) RXRPC_SECURITY_KEYRING
|
|
|
|
Similar to above but specifies a keyring of server secret keys to use (key
|
|
type "keyring"). See the "Security" section.
|
|
|
|
(*) RXRPC_EXCLUSIVE_CONNECTION
|
|
|
|
This is used to request that new connections should be used for each call
|
|
made subsequently on this socket. optval should be NULL and optlen 0.
|
|
|
|
(*) RXRPC_MIN_SECURITY_LEVEL
|
|
|
|
This is used to specify the minimum security level required for calls on
|
|
this socket. optval must point to an int containing one of the following
|
|
values:
|
|
|
|
(a) RXRPC_SECURITY_PLAIN
|
|
|
|
Encrypted checksum only.
|
|
|
|
(b) RXRPC_SECURITY_AUTH
|
|
|
|
Encrypted checksum plus packet padded and first eight bytes of packet
|
|
encrypted - which includes the actual packet length.
|
|
|
|
(c) RXRPC_SECURITY_ENCRYPTED
|
|
|
|
Encrypted checksum plus entire packet padded and encrypted, including
|
|
actual packet length.
|
|
|
|
(*) RXRPC_UPGRADEABLE_SERVICE
|
|
|
|
This is used to indicate that a service socket with two bindings may
|
|
upgrade one bound service to the other if requested by the client. optval
|
|
must point to an array of two unsigned short ints. The first is the
|
|
service ID to upgrade from and the second the service ID to upgrade to.
|
|
|
|
(*) RXRPC_SUPPORTED_CMSG
|
|
|
|
This is a read-only option that writes an int into the buffer indicating
|
|
the highest control message type supported.
|
|
|
|
|
|
========
|
|
SECURITY
|
|
========
|
|
|
|
Currently, only the kerberos 4 equivalent protocol has been implemented
|
|
(security index 2 - rxkad). This requires the rxkad module to be loaded and,
|
|
on the client, tickets of the appropriate type to be obtained from the AFS
|
|
kaserver or the kerberos server and installed as "rxrpc" type keys. This is
|
|
normally done using the klog program. An example simple klog program can be
|
|
found at:
|
|
|
|
http://people.redhat.com/~dhowells/rxrpc/klog.c
|
|
|
|
The payload provided to add_key() on the client should be of the following
|
|
form:
|
|
|
|
struct rxrpc_key_sec2_v1 {
|
|
uint16_t security_index; /* 2 */
|
|
uint16_t ticket_length; /* length of ticket[] */
|
|
uint32_t expiry; /* time at which expires */
|
|
uint8_t kvno; /* key version number */
|
|
uint8_t __pad[3];
|
|
uint8_t session_key[8]; /* DES session key */
|
|
uint8_t ticket[0]; /* the encrypted ticket */
|
|
};
|
|
|
|
Where the ticket blob is just appended to the above structure.
|
|
|
|
|
|
For the server, keys of type "rxrpc_s" must be made available to the server.
|
|
They have a description of "<serviceID>:<securityIndex>" (eg: "52:2" for an
|
|
rxkad key for the AFS VL service). When such a key is created, it should be
|
|
given the server's secret key as the instantiation data (see the example
|
|
below).
|
|
|
|
add_key("rxrpc_s", "52:2", secret_key, 8, keyring);
|
|
|
|
A keyring is passed to the server socket by naming it in a sockopt. The server
|
|
socket then looks the server secret keys up in this keyring when secure
|
|
incoming connections are made. This can be seen in an example program that can
|
|
be found at:
|
|
|
|
http://people.redhat.com/~dhowells/rxrpc/listen.c
|
|
|
|
|
|
====================
|
|
EXAMPLE CLIENT USAGE
|
|
====================
|
|
|
|
A client would issue an operation by:
|
|
|
|
(1) An RxRPC socket is set up by:
|
|
|
|
client = socket(AF_RXRPC, SOCK_DGRAM, PF_INET);
|
|
|
|
Where the third parameter indicates the protocol family of the transport
|
|
socket used - usually IPv4 but it can also be IPv6 [TODO].
|
|
|
|
(2) A local address can optionally be bound:
|
|
|
|
struct sockaddr_rxrpc srx = {
|
|
.srx_family = AF_RXRPC,
|
|
.srx_service = 0, /* we're a client */
|
|
.transport_type = SOCK_DGRAM, /* type of transport socket */
|
|
.transport.sin_family = AF_INET,
|
|
.transport.sin_port = htons(7000), /* AFS callback */
|
|
.transport.sin_address = 0, /* all local interfaces */
|
|
};
|
|
bind(client, &srx, sizeof(srx));
|
|
|
|
This specifies the local UDP port to be used. If not given, a random
|
|
non-privileged port will be used. A UDP port may be shared between
|
|
several unrelated RxRPC sockets. Security is handled on a basis of
|
|
per-RxRPC virtual connection.
|
|
|
|
(3) The security is set:
|
|
|
|
const char *key = "AFS:cambridge.redhat.com";
|
|
setsockopt(client, SOL_RXRPC, RXRPC_SECURITY_KEY, key, strlen(key));
|
|
|
|
This issues a request_key() to get the key representing the security
|
|
context. The minimum security level can be set:
|
|
|
|
unsigned int sec = RXRPC_SECURITY_ENCRYPTED;
|
|
setsockopt(client, SOL_RXRPC, RXRPC_MIN_SECURITY_LEVEL,
|
|
&sec, sizeof(sec));
|
|
|
|
(4) The server to be contacted can then be specified (alternatively this can
|
|
be done through sendmsg):
|
|
|
|
struct sockaddr_rxrpc srx = {
|
|
.srx_family = AF_RXRPC,
|
|
.srx_service = VL_SERVICE_ID,
|
|
.transport_type = SOCK_DGRAM, /* type of transport socket */
|
|
.transport.sin_family = AF_INET,
|
|
.transport.sin_port = htons(7005), /* AFS volume manager */
|
|
.transport.sin_address = ...,
|
|
};
|
|
connect(client, &srx, sizeof(srx));
|
|
|
|
(5) The request data should then be posted to the server socket using a series
|
|
of sendmsg() calls, each with the following control message attached:
|
|
|
|
RXRPC_USER_CALL_ID - specifies the user ID for this call
|
|
|
|
MSG_MORE should be set in msghdr::msg_flags on all but the last part of
|
|
the request. Multiple requests may be made simultaneously.
|
|
|
|
An RXRPC_TX_LENGTH control message can also be specified on the first
|
|
sendmsg() call.
|
|
|
|
If a call is intended to go to a destination other than the default
|
|
specified through connect(), then msghdr::msg_name should be set on the
|
|
first request message of that call.
|
|
|
|
(6) The reply data will then be posted to the server socket for recvmsg() to
|
|
pick up. MSG_MORE will be flagged by recvmsg() if there's more reply data
|
|
for a particular call to be read. MSG_EOR will be set on the terminal
|
|
read for a call.
|
|
|
|
All data will be delivered with the following control message attached:
|
|
|
|
RXRPC_USER_CALL_ID - specifies the user ID for this call
|
|
|
|
If an abort or error occurred, this will be returned in the control data
|
|
buffer instead, and MSG_EOR will be flagged to indicate the end of that
|
|
call.
|
|
|
|
A client may ask for a service ID it knows and ask that this be upgraded to a
|
|
better service if one is available by supplying RXRPC_UPGRADE_SERVICE on the
|
|
first sendmsg() of a call. The client should then check srx_service in the
|
|
msg_name filled in by recvmsg() when collecting the result. srx_service will
|
|
hold the same value as given to sendmsg() if the upgrade request was ignored by
|
|
the service - otherwise it will be altered to indicate the service ID the
|
|
server upgraded to. Note that the upgraded service ID is chosen by the server.
|
|
The caller has to wait until it sees the service ID in the reply before sending
|
|
any more calls (further calls to the same destination will be blocked until the
|
|
probe is concluded).
|
|
|
|
|
|
====================
|
|
EXAMPLE SERVER USAGE
|
|
====================
|
|
|
|
A server would be set up to accept operations in the following manner:
|
|
|
|
(1) An RxRPC socket is created by:
|
|
|
|
server = socket(AF_RXRPC, SOCK_DGRAM, PF_INET);
|
|
|
|
Where the third parameter indicates the address type of the transport
|
|
socket used - usually IPv4.
|
|
|
|
(2) Security is set up if desired by giving the socket a keyring with server
|
|
secret keys in it:
|
|
|
|
keyring = add_key("keyring", "AFSkeys", NULL, 0,
|
|
KEY_SPEC_PROCESS_KEYRING);
|
|
|
|
const char secret_key[8] = {
|
|
0xa7, 0x83, 0x8a, 0xcb, 0xc7, 0x83, 0xec, 0x94 };
|
|
add_key("rxrpc_s", "52:2", secret_key, 8, keyring);
|
|
|
|
setsockopt(server, SOL_RXRPC, RXRPC_SECURITY_KEYRING, "AFSkeys", 7);
|
|
|
|
The keyring can be manipulated after it has been given to the socket. This
|
|
permits the server to add more keys, replace keys, etc. while it is live.
|
|
|
|
(3) A local address must then be bound:
|
|
|
|
struct sockaddr_rxrpc srx = {
|
|
.srx_family = AF_RXRPC,
|
|
.srx_service = VL_SERVICE_ID, /* RxRPC service ID */
|
|
.transport_type = SOCK_DGRAM, /* type of transport socket */
|
|
.transport.sin_family = AF_INET,
|
|
.transport.sin_port = htons(7000), /* AFS callback */
|
|
.transport.sin_address = 0, /* all local interfaces */
|
|
};
|
|
bind(server, &srx, sizeof(srx));
|
|
|
|
More than one service ID may be bound to a socket, provided the transport
|
|
parameters are the same. The limit is currently two. To do this, bind()
|
|
should be called twice.
|
|
|
|
(4) If service upgrading is required, first two service IDs must have been
|
|
bound and then the following option must be set:
|
|
|
|
unsigned short service_ids[2] = { from_ID, to_ID };
|
|
setsockopt(server, SOL_RXRPC, RXRPC_UPGRADEABLE_SERVICE,
|
|
service_ids, sizeof(service_ids));
|
|
|
|
This will automatically upgrade connections on service from_ID to service
|
|
to_ID if they request it. This will be reflected in msg_name obtained
|
|
through recvmsg() when the request data is delivered to userspace.
|
|
|
|
(5) The server is then set to listen out for incoming calls:
|
|
|
|
listen(server, 100);
|
|
|
|
(6) The kernel notifies the server of pending incoming connections by sending
|
|
it a message for each. This is received with recvmsg() on the server
|
|
socket. It has no data, and has a single dataless control message
|
|
attached:
|
|
|
|
RXRPC_NEW_CALL
|
|
|
|
The address that can be passed back by recvmsg() at this point should be
|
|
ignored since the call for which the message was posted may have gone by
|
|
the time it is accepted - in which case the first call still on the queue
|
|
will be accepted.
|
|
|
|
(7) The server then accepts the new call by issuing a sendmsg() with two
|
|
pieces of control data and no actual data:
|
|
|
|
RXRPC_ACCEPT - indicate connection acceptance
|
|
RXRPC_USER_CALL_ID - specify user ID for this call
|
|
|
|
(8) The first request data packet will then be posted to the server socket for
|
|
recvmsg() to pick up. At that point, the RxRPC address for the call can
|
|
be read from the address fields in the msghdr struct.
|
|
|
|
Subsequent request data will be posted to the server socket for recvmsg()
|
|
to collect as it arrives. All but the last piece of the request data will
|
|
be delivered with MSG_MORE flagged.
|
|
|
|
All data will be delivered with the following control message attached:
|
|
|
|
RXRPC_USER_CALL_ID - specifies the user ID for this call
|
|
|
|
(9) The reply data should then be posted to the server socket using a series
|
|
of sendmsg() calls, each with the following control messages attached:
|
|
|
|
RXRPC_USER_CALL_ID - specifies the user ID for this call
|
|
|
|
MSG_MORE should be set in msghdr::msg_flags on all but the last message
|
|
for a particular call.
|
|
|
|
(10) The final ACK from the client will be posted for retrieval by recvmsg()
|
|
when it is received. It will take the form of a dataless message with two
|
|
control messages attached:
|
|
|
|
RXRPC_USER_CALL_ID - specifies the user ID for this call
|
|
RXRPC_ACK - indicates final ACK (no data)
|
|
|
|
MSG_EOR will be flagged to indicate that this is the final message for
|
|
this call.
|
|
|
|
(11) Up to the point the final packet of reply data is sent, the call can be
|
|
aborted by calling sendmsg() with a dataless message with the following
|
|
control messages attached:
|
|
|
|
RXRPC_USER_CALL_ID - specifies the user ID for this call
|
|
RXRPC_ABORT - indicates abort code (4 byte data)
|
|
|
|
Any packets waiting in the socket's receive queue will be discarded if
|
|
this is issued.
|
|
|
|
Note that all the communications for a particular service take place through
|
|
the one server socket, using control messages on sendmsg() and recvmsg() to
|
|
determine the call affected.
|
|
|
|
|
|
=========================
|
|
AF_RXRPC KERNEL INTERFACE
|
|
=========================
|
|
|
|
The AF_RXRPC module also provides an interface for use by in-kernel utilities
|
|
such as the AFS filesystem. This permits such a utility to:
|
|
|
|
(1) Use different keys directly on individual client calls on one socket
|
|
rather than having to open a whole slew of sockets, one for each key it
|
|
might want to use.
|
|
|
|
(2) Avoid having RxRPC call request_key() at the point of issue of a call or
|
|
opening of a socket. Instead the utility is responsible for requesting a
|
|
key at the appropriate point. AFS, for instance, would do this during VFS
|
|
operations such as open() or unlink(). The key is then handed through
|
|
when the call is initiated.
|
|
|
|
(3) Request the use of something other than GFP_KERNEL to allocate memory.
|
|
|
|
(4) Avoid the overhead of using the recvmsg() call. RxRPC messages can be
|
|
intercepted before they get put into the socket Rx queue and the socket
|
|
buffers manipulated directly.
|
|
|
|
To use the RxRPC facility, a kernel utility must still open an AF_RXRPC socket,
|
|
bind an address as appropriate and listen if it's to be a server socket, but
|
|
then it passes this to the kernel interface functions.
|
|
|
|
The kernel interface functions are as follows:
|
|
|
|
(*) Begin a new client call.
|
|
|
|
struct rxrpc_call *
|
|
rxrpc_kernel_begin_call(struct socket *sock,
|
|
struct sockaddr_rxrpc *srx,
|
|
struct key *key,
|
|
unsigned long user_call_ID,
|
|
s64 tx_total_len,
|
|
gfp_t gfp,
|
|
rxrpc_notify_rx_t notify_rx,
|
|
bool upgrade);
|
|
|
|
This allocates the infrastructure to make a new RxRPC call and assigns
|
|
call and connection numbers. The call will be made on the UDP port that
|
|
the socket is bound to. The call will go to the destination address of a
|
|
connected client socket unless an alternative is supplied (srx is
|
|
non-NULL).
|
|
|
|
If a key is supplied then this will be used to secure the call instead of
|
|
the key bound to the socket with the RXRPC_SECURITY_KEY sockopt. Calls
|
|
secured in this way will still share connections if at all possible.
|
|
|
|
The user_call_ID is equivalent to that supplied to sendmsg() in the
|
|
control data buffer. It is entirely feasible to use this to point to a
|
|
kernel data structure.
|
|
|
|
tx_total_len is the amount of data the caller is intending to transmit
|
|
with this call (or -1 if unknown at this point). Setting the data size
|
|
allows the kernel to encrypt directly to the packet buffers, thereby
|
|
saving a copy. The value may not be less than -1.
|
|
|
|
notify_rx is a pointer to a function to be called when events such as
|
|
incoming data packets or remote aborts happen.
|
|
|
|
upgrade should be set to true if a client operation should request that
|
|
the server upgrade the service to a better one. The resultant service ID
|
|
is returned by rxrpc_kernel_recv_data().
|
|
|
|
If this function is successful, an opaque reference to the RxRPC call is
|
|
returned. The caller now holds a reference on this and it must be
|
|
properly ended.
|
|
|
|
(*) End a client call.
|
|
|
|
void rxrpc_kernel_end_call(struct socket *sock,
|
|
struct rxrpc_call *call);
|
|
|
|
This is used to end a previously begun call. The user_call_ID is expunged
|
|
from AF_RXRPC's knowledge and will not be seen again in association with
|
|
the specified call.
|
|
|
|
(*) Send data through a call.
|
|
|
|
typedef void (*rxrpc_notify_end_tx_t)(struct sock *sk,
|
|
unsigned long user_call_ID,
|
|
struct sk_buff *skb);
|
|
|
|
int rxrpc_kernel_send_data(struct socket *sock,
|
|
struct rxrpc_call *call,
|
|
struct msghdr *msg,
|
|
size_t len,
|
|
rxrpc_notify_end_tx_t notify_end_rx);
|
|
|
|
This is used to supply either the request part of a client call or the
|
|
reply part of a server call. msg.msg_iovlen and msg.msg_iov specify the
|
|
data buffers to be used. msg_iov may not be NULL and must point
|
|
exclusively to in-kernel virtual addresses. msg.msg_flags may be given
|
|
MSG_MORE if there will be subsequent data sends for this call.
|
|
|
|
The msg must not specify a destination address, control data or any flags
|
|
other than MSG_MORE. len is the total amount of data to transmit.
|
|
|
|
notify_end_rx can be NULL or it can be used to specify a function to be
|
|
called when the call changes state to end the Tx phase. This function is
|
|
called with the call-state spinlock held to prevent any reply or final ACK
|
|
from being delivered first.
|
|
|
|
(*) Receive data from a call.
|
|
|
|
int rxrpc_kernel_recv_data(struct socket *sock,
|
|
struct rxrpc_call *call,
|
|
void *buf,
|
|
size_t size,
|
|
size_t *_offset,
|
|
bool want_more,
|
|
u32 *_abort,
|
|
u16 *_service)
|
|
|
|
This is used to receive data from either the reply part of a client call
|
|
or the request part of a service call. buf and size specify how much
|
|
data is desired and where to store it. *_offset is added on to buf and
|
|
subtracted from size internally; the amount copied into the buffer is
|
|
added to *_offset before returning.
|
|
|
|
want_more should be true if further data will be required after this is
|
|
satisfied and false if this is the last item of the receive phase.
|
|
|
|
There are three normal returns: 0 if the buffer was filled and want_more
|
|
was true; 1 if the buffer was filled, the last DATA packet has been
|
|
emptied and want_more was false; and -EAGAIN if the function needs to be
|
|
called again.
|
|
|
|
If the last DATA packet is processed but the buffer contains less than
|
|
the amount requested, EBADMSG is returned. If want_more wasn't set, but
|
|
more data was available, EMSGSIZE is returned.
|
|
|
|
If a remote ABORT is detected, the abort code received will be stored in
|
|
*_abort and ECONNABORTED will be returned.
|
|
|
|
The service ID that the call ended up with is returned into *_service.
|
|
This can be used to see if a call got a service upgrade.
|
|
|
|
(*) Abort a call.
|
|
|
|
void rxrpc_kernel_abort_call(struct socket *sock,
|
|
struct rxrpc_call *call,
|
|
u32 abort_code);
|
|
|
|
This is used to abort a call if it's still in an abortable state. The
|
|
abort code specified will be placed in the ABORT message sent.
|
|
|
|
(*) Intercept received RxRPC messages.
|
|
|
|
typedef void (*rxrpc_interceptor_t)(struct sock *sk,
|
|
unsigned long user_call_ID,
|
|
struct sk_buff *skb);
|
|
|
|
void
|
|
rxrpc_kernel_intercept_rx_messages(struct socket *sock,
|
|
rxrpc_interceptor_t interceptor);
|
|
|
|
This installs an interceptor function on the specified AF_RXRPC socket.
|
|
All messages that would otherwise wind up in the socket's Rx queue are
|
|
then diverted to this function. Note that care must be taken to process
|
|
the messages in the right order to maintain DATA message sequentiality.
|
|
|
|
The interceptor function itself is provided with the address of the socket
|
|
and handling the incoming message, the ID assigned by the kernel utility
|
|
to the call and the socket buffer containing the message.
|
|
|
|
The skb->mark field indicates the type of message:
|
|
|
|
MARK MEANING
|
|
=============================== =======================================
|
|
RXRPC_SKB_MARK_DATA Data message
|
|
RXRPC_SKB_MARK_FINAL_ACK Final ACK received for an incoming call
|
|
RXRPC_SKB_MARK_BUSY Client call rejected as server busy
|
|
RXRPC_SKB_MARK_REMOTE_ABORT Call aborted by peer
|
|
RXRPC_SKB_MARK_NET_ERROR Network error detected
|
|
RXRPC_SKB_MARK_LOCAL_ERROR Local error encountered
|
|
RXRPC_SKB_MARK_NEW_CALL New incoming call awaiting acceptance
|
|
|
|
The remote abort message can be probed with rxrpc_kernel_get_abort_code().
|
|
The two error messages can be probed with rxrpc_kernel_get_error_number().
|
|
A new call can be accepted with rxrpc_kernel_accept_call().
|
|
|
|
Data messages can have their contents extracted with the usual bunch of
|
|
socket buffer manipulation functions. A data message can be determined to
|
|
be the last one in a sequence with rxrpc_kernel_is_data_last(). When a
|
|
data message has been used up, rxrpc_kernel_data_consumed() should be
|
|
called on it.
|
|
|
|
Messages should be handled to rxrpc_kernel_free_skb() to dispose of. It
|
|
is possible to get extra refs on all types of message for later freeing,
|
|
but this may pin the state of a call until the message is finally freed.
|
|
|
|
(*) Accept an incoming call.
|
|
|
|
struct rxrpc_call *
|
|
rxrpc_kernel_accept_call(struct socket *sock,
|
|
unsigned long user_call_ID);
|
|
|
|
This is used to accept an incoming call and to assign it a call ID. This
|
|
function is similar to rxrpc_kernel_begin_call() and calls accepted must
|
|
be ended in the same way.
|
|
|
|
If this function is successful, an opaque reference to the RxRPC call is
|
|
returned. The caller now holds a reference on this and it must be
|
|
properly ended.
|
|
|
|
(*) Reject an incoming call.
|
|
|
|
int rxrpc_kernel_reject_call(struct socket *sock);
|
|
|
|
This is used to reject the first incoming call on the socket's queue with
|
|
a BUSY message. -ENODATA is returned if there were no incoming calls.
|
|
Other errors may be returned if the call had been aborted (-ECONNABORTED)
|
|
or had timed out (-ETIME).
|
|
|
|
(*) Allocate a null key for doing anonymous security.
|
|
|
|
struct key *rxrpc_get_null_key(const char *keyname);
|
|
|
|
This is used to allocate a null RxRPC key that can be used to indicate
|
|
anonymous security for a particular domain.
|
|
|
|
(*) Get the peer address of a call.
|
|
|
|
void rxrpc_kernel_get_peer(struct socket *sock, struct rxrpc_call *call,
|
|
struct sockaddr_rxrpc *_srx);
|
|
|
|
This is used to find the remote peer address of a call.
|
|
|
|
(*) Set the total transmit data size on a call.
|
|
|
|
void rxrpc_kernel_set_tx_length(struct socket *sock,
|
|
struct rxrpc_call *call,
|
|
s64 tx_total_len);
|
|
|
|
This sets the amount of data that the caller is intending to transmit on a
|
|
call. It's intended to be used for setting the reply size as the request
|
|
size should be set when the call is begun. tx_total_len may not be less
|
|
than zero.
|
|
|
|
(*) Get call RTT.
|
|
|
|
u64 rxrpc_kernel_get_rtt(struct socket *sock, struct rxrpc_call *call);
|
|
|
|
Get the RTT time to the peer in use by a call. The value returned is in
|
|
nanoseconds.
|
|
|
|
(*) Check call still alive.
|
|
|
|
u32 rxrpc_kernel_check_life(struct socket *sock,
|
|
struct rxrpc_call *call);
|
|
void rxrpc_kernel_probe_life(struct socket *sock,
|
|
struct rxrpc_call *call);
|
|
|
|
The first function returns a number that is updated when ACKs are received
|
|
from the peer (notably including PING RESPONSE ACKs which we can elicit by
|
|
sending PING ACKs to see if the call still exists on the server). The
|
|
caller should compare the numbers of two calls to see if the call is still
|
|
alive after waiting for a suitable interval.
|
|
|
|
This allows the caller to work out if the server is still contactable and
|
|
if the call is still alive on the server while waiting for the server to
|
|
process a client operation.
|
|
|
|
The second function causes a ping ACK to be transmitted to try to provoke
|
|
the peer into responding, which would then cause the value returned by the
|
|
first function to change. Note that this must be called in TASK_RUNNING
|
|
state.
|
|
|
|
(*) Get reply timestamp.
|
|
|
|
bool rxrpc_kernel_get_reply_time(struct socket *sock,
|
|
struct rxrpc_call *call,
|
|
ktime_t *_ts)
|
|
|
|
This allows the timestamp on the first DATA packet of the reply of a
|
|
client call to be queried, provided that it is still in the Rx ring. If
|
|
successful, the timestamp will be stored into *_ts and true will be
|
|
returned; false will be returned otherwise.
|
|
|
|
(*) Get remote client epoch.
|
|
|
|
u32 rxrpc_kernel_get_epoch(struct socket *sock,
|
|
struct rxrpc_call *call)
|
|
|
|
This allows the epoch that's contained in packets of an incoming client
|
|
call to be queried. This value is returned. The function always
|
|
successful if the call is still in progress. It shouldn't be called once
|
|
the call has expired. Note that calling this on a local client call only
|
|
returns the local epoch.
|
|
|
|
This value can be used to determine if the remote client has been
|
|
restarted as it shouldn't change otherwise.
|
|
|
|
|
|
=======================
|
|
CONFIGURABLE PARAMETERS
|
|
=======================
|
|
|
|
The RxRPC protocol driver has a number of configurable parameters that can be
|
|
adjusted through sysctls in /proc/net/rxrpc/:
|
|
|
|
(*) req_ack_delay
|
|
|
|
The amount of time in milliseconds after receiving a packet with the
|
|
request-ack flag set before we honour the flag and actually send the
|
|
requested ack.
|
|
|
|
Usually the other side won't stop sending packets until the advertised
|
|
reception window is full (to a maximum of 255 packets), so delaying the
|
|
ACK permits several packets to be ACK'd in one go.
|
|
|
|
(*) soft_ack_delay
|
|
|
|
The amount of time in milliseconds after receiving a new packet before we
|
|
generate a soft-ACK to tell the sender that it doesn't need to resend.
|
|
|
|
(*) idle_ack_delay
|
|
|
|
The amount of time in milliseconds after all the packets currently in the
|
|
received queue have been consumed before we generate a hard-ACK to tell
|
|
the sender it can free its buffers, assuming no other reason occurs that
|
|
we would send an ACK.
|
|
|
|
(*) resend_timeout
|
|
|
|
The amount of time in milliseconds after transmitting a packet before we
|
|
transmit it again, assuming no ACK is received from the receiver telling
|
|
us they got it.
|
|
|
|
(*) max_call_lifetime
|
|
|
|
The maximum amount of time in seconds that a call may be in progress
|
|
before we preemptively kill it.
|
|
|
|
(*) dead_call_expiry
|
|
|
|
The amount of time in seconds before we remove a dead call from the call
|
|
list. Dead calls are kept around for a little while for the purpose of
|
|
repeating ACK and ABORT packets.
|
|
|
|
(*) connection_expiry
|
|
|
|
The amount of time in seconds after a connection was last used before we
|
|
remove it from the connection list. While a connection is in existence,
|
|
it serves as a placeholder for negotiated security; when it is deleted,
|
|
the security must be renegotiated.
|
|
|
|
(*) transport_expiry
|
|
|
|
The amount of time in seconds after a transport was last used before we
|
|
remove it from the transport list. While a transport is in existence, it
|
|
serves to anchor the peer data and keeps the connection ID counter.
|
|
|
|
(*) rxrpc_rx_window_size
|
|
|
|
The size of the receive window in packets. This is the maximum number of
|
|
unconsumed received packets we're willing to hold in memory for any
|
|
particular call.
|
|
|
|
(*) rxrpc_rx_mtu
|
|
|
|
The maximum packet MTU size that we're willing to receive in bytes. This
|
|
indicates to the peer whether we're willing to accept jumbo packets.
|
|
|
|
(*) rxrpc_rx_jumbo_max
|
|
|
|
The maximum number of packets that we're willing to accept in a jumbo
|
|
packet. Non-terminal packets in a jumbo packet must contain a four byte
|
|
header plus exactly 1412 bytes of data. The terminal packet must contain
|
|
a four byte header plus any amount of data. In any event, a jumbo packet
|
|
may not exceed rxrpc_rx_mtu in size.
|