From 4c2274c5ddc593a3ef31f377277a756e52dc04b2 Mon Sep 17 00:00:00 2001 From: Bertrand Marquis Date: Thu, 9 Oct 2025 09:48:14 +0200 Subject: [PATCH 01/11] virtio-msg: 8 byte header with unique identifier Move the header to be 8 bytes and include a unique identifier that can be use to correlate request to responses. Rename the msg_id field to msg_op as having a msg_id and msg_uid field and using identifiers in 2 contexts would have been unclear. Rework ID to OP and change wording message identifer to message operation to be coherent in the rest of the specification. Signed-off-by: Bertrand Marquis --- transport-msg.tex | 87 ++++++++++++++++++++++++++--------------------- 1 file changed, 49 insertions(+), 38 deletions(-) diff --git a/transport-msg.tex b/transport-msg.tex index 79a0a5b..db4e20c 100644 --- a/transport-msg.tex +++ b/transport-msg.tex @@ -286,41 +286,52 @@ \subsubsection{Endianness} \subsubsection{Common Message Format} \label{sec:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Common Message Format} -All virtio-msg exchanges, whether \emph{bus messages} or \emph{transport messages}, -begin with a shared header that indicates how the recipient should parse the -rest of the payload. This header has the following format: +All virtio-msg exchanges, whether \emph{bus messages} or +\emph{transport messages}, begin with an 8 byte header followed by an optional +payload. + +The header layout is: \begin{lstlisting} -struct virtio_msg_message { - uint8_t type; - uint8_t msg_id; - uint16_t dev_num; - uint16_t msg_size; - u8 payload[]; +struct virtio_msg_header { + uint8_t type; /* request/response + bus/transport */ + uint8_t msg_op; /* message operation code */ + uint16_t dev_num; /* device number (0 for bus messages) */ + uint16_t msg_uid; /* correlation identifier (0 for events) */ + uint16_t msg_size; /* total size: header (8) + payload */ + uint8_t payload[]; }; \end{lstlisting} -The fields in this header have the following usage: +Field semantics: \begin{itemize} \item \field{type}: \begin{itemize} - \item Bit[0]: Identifies if a message is a request (0) or a response - to a request (1). - \item Bit[1]: Identifies if a message is a Transport Message (0) or a - Bus Message (1). - \item Bit[2-7] Are reserved for future use and must be zero. + \item Bit[0]: 0=request, 1=response. + \item Bit[1]: 0=Transport Message, 1=Bus Message. + \item Bits[2..7]: \textbf{MUST} be zero; receivers \textbf{MUST} ignore. \end{itemize} - \item \field{msg_id}: - Uniquely identifies which message definition applies (e.g., GET_DEVICES, - GET_DEVICE_FEATURES, SET_CONFIG). The specific range or enumeration of types is - defined in sections \ref{sec:Virtio Transport Options / Virtio Over Messages / Transport Messages} - and \ref{sec:Virtio Transport Options / Virtio Over Messages / Bus Messages}. - \item \field{dev_num}: - Identifies the Device Number the message is targeting or is coming from for - Transport Message and must be zero of Bus messages. - \item \field{msg_size}; - Indicates the total length of the message payload including the header. + \item \field{msg_op}: Operation code identifying the message definition. Ranges + are defined in + \ref{sec:Virtio Transport Options / Virtio Over Messages / Transport Messages} + and + \ref{sec:Virtio Transport Options / Virtio Over Messages / Bus Messages}. + \item \field{dev_num}: For Transport Messages, the target device number; for + Bus Messages \textbf{MUST} be zero. + \item \field{msg_uid}: Non-zero for requests that expect a response; zero + \textbf{MUST} be used only for event (one-way) messages. Responses + \textbf{MUST} echo the request's \field{msg_uid}. + \item \field{msg_size}: Total size in bytes of the complete message (header + + payload). \textbf{MUST} be \(\ge 8\) and \textbf{MUST NOT} exceed the + bus's maximum message size. + \item \field{payload}: Operation-specific data. Unused trailing bytes (if any + introduced by a bus framing) \textbf{MUST} be zero and \textbf{MUST} be + ignored by receivers. \end{itemize} +All reserved header bits and any unspecified header values \textbf{MUST} be +sent as zero and \textbf{MUST} be ignored on receive to preserve forward +compatibility. + \subsection{Bus Operation} \label{sec:Virtio Transport Options / Virtio Over Messages / Bus Operation} @@ -391,8 +402,8 @@ \subsubsection{Monitoring the Bus} \subsubsection{Bus Specific Messages} \label{sec:Virtio Transport Options / Virtio Over Messages / Bus Operation / Bus Specific Messages} -A range of message IDs are reserved for use by the specific bus -implementation. These messages can be used for any implementation specific +A range of message operation identifiers are reserved for use by the specific +bus implementation. These messages can be used for any implementation specific usage. Example usage could include: \begin{itemize} @@ -715,11 +726,11 @@ \subsubsection{Overview} Most transport messages adopt a \emph{request/response} pattern, but some are unidirectional (e.g., asynchronous notifications). -\paragraph{Messages IDs and issuers} +\paragraph{Messages Operation (OP) identifiers and issuers} \begin{tabular}{|l|l|l|} \hline -Name & ID & Sender \\ +Name & OP identifier & Sender \\ \hline \hline Reserved & 0x0 & \\ @@ -756,8 +767,8 @@ \subsubsection{Overview} \hline \end{tabular} -Transport message IDs 0x00 to 0x3F are used for messages that require a response -and IDs 0x40 to 0x7F are used for event messages. Transport message IDs 0x80 +Transport message OPs 0x00 to 0x3F are used for messages that require a response +and OPs 0x40 to 0x7F are used for event messages. Transport message OPs 0x80 and above are reserved by this specification. \paragraph{Mandatory Transport Messages} @@ -1080,11 +1091,11 @@ \subsection{Bus Messages}\label{sec:Virtio Transport Options / Virtio Over Messa firmware tables, device trees). Each bus instance \emph{may} use a subset of or all these messages according to its design. -\paragraph{Messages IDs and issuers} +\paragraph{OPs identifiers and issuers} \begin{tabular}{|l|l|l|} \hline -Name & ID & Sender \\ +Name & OP identifier & Sender \\ \hline \hline Reserved & 0x0 & \\ @@ -1099,14 +1110,14 @@ \subsection{Bus Messages}\label{sec:Virtio Transport Options / Virtio Over Messa \hline \end{tabular} -Bus message IDs below 0x80 are reserved for standardizes (but optional) bus +Bus message OPs below 0x80 are reserved for standardizes (but optional) bus messages. A few are used here and more are expected in the future. Bus message -IDs below 0x40 are used for request/response messages and 0x40 and above for +OPs below 0x40 are used for request/response messages and 0x40 and above for event messages. -Bus message IDs 0x80 and above are bus implementation specific. Bus -implementations \emph{MAY} specify the policy that IDs below 0xC0 be used -for request/response messages and IDs 0xC0 and above are used for event messages. +Bus message OPs 0x80 and above are bus implementation specific. Bus +implementations \emph{MAY} specify the policy that OPs below 0xC0 be used +for request/response messages and OPs 0xC0 and above are used for event messages. \paragraph{Note:} A bus implementation \textbf{is not required} to use these messages if it already provides equivalent functionality through some From e36035ea39dad2bc15c409e8fed1c1d4fc511c23 Mon Sep 17 00:00:00 2001 From: Bill Mills Date: Fri, 17 Oct 2025 09:52:56 -0400 Subject: [PATCH 02/11] virtio-msg: revert chucks of msg_id -> msg_op This reverts commit 4c2274c5ddc593a3ef31f377277a756e52dc04b2. Signed-off-by: Bill Mills --- transport-msg.tex | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/transport-msg.tex b/transport-msg.tex index db4e20c..3d58184 100644 --- a/transport-msg.tex +++ b/transport-msg.tex @@ -402,8 +402,8 @@ \subsubsection{Monitoring the Bus} \subsubsection{Bus Specific Messages} \label{sec:Virtio Transport Options / Virtio Over Messages / Bus Operation / Bus Specific Messages} -A range of message operation identifiers are reserved for use by the specific -bus implementation. These messages can be used for any implementation specific +A range of message IDs are reserved for use by the specific bus +implementation. These messages can be used for any implementation specific usage. Example usage could include: \begin{itemize} @@ -726,11 +726,11 @@ \subsubsection{Overview} Most transport messages adopt a \emph{request/response} pattern, but some are unidirectional (e.g., asynchronous notifications). -\paragraph{Messages Operation (OP) identifiers and issuers} +\paragraph{Messages IDs and issuers} \begin{tabular}{|l|l|l|} \hline -Name & OP identifier & Sender \\ +Name & ID & Sender \\ \hline \hline Reserved & 0x0 & \\ @@ -767,8 +767,8 @@ \subsubsection{Overview} \hline \end{tabular} -Transport message OPs 0x00 to 0x3F are used for messages that require a response -and OPs 0x40 to 0x7F are used for event messages. Transport message OPs 0x80 +Transport message IDs 0x00 to 0x3F are used for messages that require a response +and IDs 0x40 to 0x7F are used for event messages. Transport message IDs 0x80 and above are reserved by this specification. \paragraph{Mandatory Transport Messages} @@ -1091,11 +1091,11 @@ \subsection{Bus Messages}\label{sec:Virtio Transport Options / Virtio Over Messa firmware tables, device trees). Each bus instance \emph{may} use a subset of or all these messages according to its design. -\paragraph{OPs identifiers and issuers} +\paragraph{Messages IDs and issuers} \begin{tabular}{|l|l|l|} \hline -Name & OP identifier & Sender \\ +Name & ID & Sender \\ \hline \hline Reserved & 0x0 & \\ @@ -1110,14 +1110,14 @@ \subsection{Bus Messages}\label{sec:Virtio Transport Options / Virtio Over Messa \hline \end{tabular} -Bus message OPs below 0x80 are reserved for standardizes (but optional) bus +Bus message IDs below 0x80 are reserved for standardizes (but optional) bus messages. A few are used here and more are expected in the future. Bus message -OPs below 0x40 are used for request/response messages and 0x40 and above for +IDs below 0x40 are used for request/response messages and 0x40 and above for event messages. -Bus message OPs 0x80 and above are bus implementation specific. Bus -implementations \emph{MAY} specify the policy that OPs below 0xC0 be used -for request/response messages and OPs 0xC0 and above are used for event messages. +Bus message IDs 0x80 and above are bus implementation specific. Bus +implementations \emph{MAY} specify the policy that IDs below 0xC0 be used +for request/response messages and IDs 0xC0 and above are used for event messages. \paragraph{Note:} A bus implementation \textbf{is not required} to use these messages if it already provides equivalent functionality through some From 8f3ea12bbf0f46b94caf0a578deab4e1ad1c4487 Mon Sep 17 00:00:00 2001 From: Bill Mills Date: Fri, 17 Oct 2025 15:13:51 -0400 Subject: [PATCH 03/11] virtio-msg: rename msg_uid to token and msg_op back to msg_id Rename these as agreed in the meeting on Oct 17. Signed-off-by: Bill Mills --- transport-msg.tex | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/transport-msg.tex b/transport-msg.tex index 3d58184..b273887 100644 --- a/transport-msg.tex +++ b/transport-msg.tex @@ -294,9 +294,9 @@ \subsubsection{Common Message Format} \begin{lstlisting} struct virtio_msg_header { uint8_t type; /* request/response + bus/transport */ - uint8_t msg_op; /* message operation code */ + uint8_t msg_id; /* message id */ uint16_t dev_num; /* device number (0 for bus messages) */ - uint16_t msg_uid; /* correlation identifier (0 for events) */ + uint16_t token; /* correlation identifier (0 for events) */ uint16_t msg_size; /* total size: header (8) + payload */ uint8_t payload[]; }; @@ -310,16 +310,16 @@ \subsubsection{Common Message Format} \item Bit[1]: 0=Transport Message, 1=Bus Message. \item Bits[2..7]: \textbf{MUST} be zero; receivers \textbf{MUST} ignore. \end{itemize} - \item \field{msg_op}: Operation code identifying the message definition. Ranges + \item \field{msg_id}: Message ID identifying the message definition. Ranges are defined in \ref{sec:Virtio Transport Options / Virtio Over Messages / Transport Messages} and \ref{sec:Virtio Transport Options / Virtio Over Messages / Bus Messages}. \item \field{dev_num}: For Transport Messages, the target device number; for Bus Messages \textbf{MUST} be zero. - \item \field{msg_uid}: Non-zero for requests that expect a response; zero + \item \field{token}: Non-zero for requests (which expect a response); zero \textbf{MUST} be used only for event (one-way) messages. Responses - \textbf{MUST} echo the request's \field{msg_uid}. + \textbf{MUST} echo the request's \field{token}. \item \field{msg_size}: Total size in bytes of the complete message (header + payload). \textbf{MUST} be \(\ge 8\) and \textbf{MUST NOT} exceed the bus's maximum message size. From b7d663e4a4b52db371a59ace39c66b475f5d15e2 Mon Sep 17 00:00:00 2001 From: Bertrand Marquis Date: Thu, 9 Oct 2025 09:50:45 +0200 Subject: [PATCH 04/11] virtio-msg: introduce a message correlation protocol Add a sub-section to do a first explanation draft of how the message correlation is to be done using a combination of a message device number and the message unique identifier value. Signed-off-by: Bertrand Marquis --- transport-msg.tex | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/transport-msg.tex b/transport-msg.tex index b273887..35a9f33 100644 --- a/transport-msg.tex +++ b/transport-msg.tex @@ -332,6 +332,29 @@ \subsubsection{Common Message Format} sent as zero and \textbf{MUST} be ignored on receive to preserve forward compatibility. +\subsubsection{Message Correlation} +\label{sec:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Correlation} + +Correlation associates a response with its request: +\begin{itemize} + \item Bus Message key: \field{msg_uid}. + \item Transport Message key: (\field{dev_num}, \field{msg_uid}). +\end{itemize} + +Rules: +\begin{itemize} + \item The driver allocates a non-zero \field{msg_uid} for every request that + expects a response and \textbf{MUST NOT} reuse it while in flight. + \item Event (one-way) messages \textbf{MUST} set \field{msg_uid}=0 and MUST + NOT yield a response. + \item A response \textbf{MUST} echo \field{msg_uid}, \field{dev_num} (for + transport), and \field{msg_op}. + \item Unknown or already completed correlation tuples \textbf{SHOULD} result + in discarding the response without further protocol action. + \item Implementations \textbf{MAY} deliver responses out of order unless a + specific message definition mandates ordering. +\end{itemize} + \subsection{Bus Operation} \label{sec:Virtio Transport Options / Virtio Over Messages / Bus Operation} From b14b390f09cac0f121553d819a5474f9a837ac1e Mon Sep 17 00:00:00 2001 From: Bill Mills Date: Fri, 17 Oct 2025 09:33:54 -0400 Subject: [PATCH 05/11] virtio-msg: fix up message correlation protocol * change msg_uid to token and msg_op back to msg_id * More explicitly define tuple * Use "request originator" not "driver" * Restate that Requests have responses and Events don't in intro then drop those qualifications from the rules. Otherwise it sounds like some requests might not have responses and some events might have them * Specify that an implementation does not need to support sending multiple requests in parallel * Use the word "assign" not "allocate" as it is less suggestive of implementation choice Signed-off-by: Bill Mills --- transport-msg.tex | 32 +++++++++++++++++++++----------- 1 file changed, 21 insertions(+), 11 deletions(-) diff --git a/transport-msg.tex b/transport-msg.tex index 35a9f33..8df3fa1 100644 --- a/transport-msg.tex +++ b/transport-msg.tex @@ -335,22 +335,32 @@ \subsubsection{Common Message Format} \subsubsection{Message Correlation} \label{sec:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Correlation} -Correlation associates a response with its request: +Messages are either requests or events. Requests require a response. Events are +one way and do not have a response. Most requests defined today originate at the +driver side but bus message requests such as \busref{PING} may originate at +either driver side or device side. + +This section defines how responses are correlated to requests. An implementation +does not need to support sending multiple requests in parallel but these rules +allow for that possibility. + +The token field in the message header is part of a tuple that is unique during +a request to response interval. \begin{itemize} - \item Bus Message key: \field{msg_uid}. - \item Transport Message key: (\field{dev_num}, \field{msg_uid}). + \item Bus Message tuple: ( 1, 0, \field{token} ). + \item Transport Message tuple: ( 0, \field{dev_num}, \field{token} ). \end{itemize} Rules: \begin{itemize} - \item The driver allocates a non-zero \field{msg_uid} for every request that - expects a response and \textbf{MUST NOT} reuse it while in flight. - \item Event (one-way) messages \textbf{MUST} set \field{msg_uid}=0 and MUST - NOT yield a response. - \item A response \textbf{MUST} echo \field{msg_uid}, \field{dev_num} (for - transport), and \field{msg_op}. - \item Unknown or already completed correlation tuples \textbf{SHOULD} result - in discarding the response without further protocol action. + \item The request originator assigns a non-zero \field{token} for every + request such that the tuple is unique for all inflight requests. + \item Event (one-way) messages \textbf{MUST} set \field{token}=0 + \item A response \textbf{MUST} echo \field{token}, \field{dev_num} (for + transport), and \field{msg_id}. + \item Reception of unknown or already completed correlation tuples + \textbf{SHOULD} result in discarding the response without further protocol + action. \item Implementations \textbf{MAY} deliver responses out of order unless a specific message definition mandates ordering. \end{itemize} From dadb51ad70754ff214012a1cdaec035b4ecb2b02 Mon Sep 17 00:00:00 2001 From: Bertrand Marquis Date: Thu, 9 Oct 2025 09:51:47 +0200 Subject: [PATCH 06/11] virtio-msg: Relax error handling in the bus Relax error handling in the virtio-msg bus and introduce wording to allow the virtio-msg bus to report error to the virtio-msg transport if a request could not be transmited or a malformed response was received. This prevents the need to generate dummy responses in the bus and make the specification coherent with the current Linux PoC implementation. Signed-off-by: Bertrand Marquis --- transport-msg.tex | 35 +++++++++++++++++++++++++---------- 1 file changed, 25 insertions(+), 10 deletions(-) diff --git a/transport-msg.tex b/transport-msg.tex index 8df3fa1..8e8c3df 100644 --- a/transport-msg.tex +++ b/transport-msg.tex @@ -232,16 +232,31 @@ \subsubsection{Feature Negotiation Blocks} \subsubsection{Error Signaling} \label{sec:Virtio Transport Options / Virtio Over Messages / Basic Concepts / ErrorSignaling} -All legal transactions are defined at the transport level and responses defined. -If the transport level does something invalid or the bus has error conditions, -this \emph{SHOULD} be handled at the bus implementation level. - -How the bus recovers from an error (e.g., by retrying, resetting -devices, or escalating to a bus-wide reset) is environment-specific, but -\emph{MUST} adhere to any mandatory behaviors (see -\ref{sec:Virtio Transport Options / Virtio Over Messages / Bus Operation} -and -\ref{sec:Virtio Transport Options / Virtio Over Messages / Device Operation}). +Errors may arise from: (a) malformed or unsupported transport messages, (b) +transmission or routing issues within a bus implementation, or (c) device-side +failures while processing a valid request. Local detection and recovery are +preferred, but a virtio-msg bus \textbf{MAY} report transmission errors to the +virtio-msg transport when it cannot deliver a request or obtain a response +within a bounded policy. + +The following rules apply: +\begin{itemize} + \item A bus implementation \textbf{MAY} surface a transport-visible failure + (implementation-defined) after exhausting any bounded retry policy for + a transmission error. + \item Malformed headers or unsupported \field{msg_op} values \textbf{SHOULD} + be discarded; the receiver \textbf{MAY} log them and \textbf{SHOULD NOT} + generate further protocol traffic in response. + \item Event (one-way) messages \textbf{MUST NOT} elicit an error response. + \item Recovery actions (retry, selective reset, device removal) are + environment-specific but \textbf{MUST} comply with any normative reset + or status handling semantics described in + \ref{sec:Virtio Transport Options / Virtio Over Messages / Device Operation}. +\end{itemize} + +This specification does not mandate a specific error reporting message for +transmission failures; it only permits a virtio-msg bus to surface such +failures to the virtio-msg transport when silent recovery is not feasible. \subsubsection{Bus vs. Transport Messages} \label{sec:Virtio Transport Options / Virtio Over Messages / Basic Concepts / BusVsTransport} From fca607d8d753137ba095fd7e8cf3b879b32c8f1a Mon Sep 17 00:00:00 2001 From: Bill Mills Date: Fri, 17 Oct 2025 09:38:20 -0400 Subject: [PATCH 07/11] virtio-msg: Fix-up error handling in the bus * change msg_op back to msg_id Signed-off-by: Bill Mills --- transport-msg.tex | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/transport-msg.tex b/transport-msg.tex index 8e8c3df..b8ff6c5 100644 --- a/transport-msg.tex +++ b/transport-msg.tex @@ -244,7 +244,7 @@ \subsubsection{Error Signaling} \item A bus implementation \textbf{MAY} surface a transport-visible failure (implementation-defined) after exhausting any bounded retry policy for a transmission error. - \item Malformed headers or unsupported \field{msg_op} values \textbf{SHOULD} + \item Malformed headers or unsupported \field{msg_id} values \textbf{SHOULD} be discarded; the receiver \textbf{MAY} log them and \textbf{SHOULD NOT} generate further protocol traffic in response. \item Event (one-way) messages \textbf{MUST NOT} elicit an error response. From d8e0eba49b6dec8ba21fff35685de17d303225f3 Mon Sep 17 00:00:00 2001 From: Bill Mills Date: Thu, 23 Oct 2025 22:06:11 -0400 Subject: [PATCH 08/11] virtio-msg: editorial changes from Matias * drop 'underlying' * add MUST to Optional Bus Messages * s/tables/table/ (Matias suggested s/defines/define/ but there is only one table ATM) (Also change table defines to table lists to avoid double "define") * If the bus adds s/it/its/ * Fix feature {itemize} list * drop verb from {Block Index} * Also fix s/32 bits values/32 bit values/ * drop \emph{chooses} in two places * s/defines/defined/ in Endianness section * add "and the buffer has been added to the used ring of the virtqueue" * s/in band/in-band and s/out of band event/out-of-band event * (Did not add "on", the sentience reads ok without it) * Replace sentence with "it should prevent this feature ..." * s/standardizes/standardized/ * Fix sentence about PING being optional Signed-off-by: Bill Mills --- transport-msg.tex | 38 +++++++++++++++++++------------------- 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/transport-msg.tex b/transport-msg.tex index b8ff6c5..6efab5a 100644 --- a/transport-msg.tex +++ b/transport-msg.tex @@ -22,8 +22,7 @@ \subsubsection{Purpose} \item \textbf{Support multiple bus implementations:} Systems may rely on various communication methods such as hypercalls, local IPC, network channels, or device trees for enumerating devices. virtio-msg - defines a common transport interface suitable for any of these underlying - mechanisms. + defines a common transport interface suitable for any of these mechanisms. \item \textbf{Reduce per-bus complexity:} Buses can implement a fully message-based workflow (including optional @@ -95,8 +94,9 @@ \subsubsection{Optional Bus Messages} completely message-based approach to enumeration, hotplug, and bus-wide health. However, these are \emph{not} mandatory if a bus instance already handles those functions via firmware, device tree, or other platform features. The only strict -requirement is that the bus be able to forward device-specific \emph{transport -messages} once a device is recognized, so the virtio-msg driver can manage it. +requirement is that the bus \emph{MUST} be able to forward device-specific +\emph{transport messages} once a device is recognized, so the virtio-msg driver +can manage it. \subsection{Basic Concepts} \label{sec:Virtio Transport Options / Virtio Over Messages / Basic Concepts} @@ -139,7 +139,7 @@ \subsubsection{Transport Revisions and Maximum Message Size} \paragraph{virtio-msg revisions} -The following tables defines the currently defined virtio-msg revisions: +The following table lists the currently defined virtio-msg revisions: \begin{tabular}{ |l|l|l|l| } \hline @@ -153,7 +153,7 @@ \subsubsection{Transport Revisions and Maximum Message Size} correspond to a change in the virtio-msg revision. The maximum message size is from the common transport level point of -view and includes the headers and payload described here. If the bus adds it +view and includes the headers and payload described here. If the bus adds its own overhead (e.x. its own header) this is not included in the maximum message size. The maximum useful message size is currently expected to be 274. This value is large enough to support a GET_CONFIG or SET_CONFIG message with a @@ -218,7 +218,7 @@ \subsubsection{Feature Negotiation Blocks} \msgref{SET_DRIVER_FEATURES}. Each block corresponds to up to 32 features: \begin{itemize} - \item \textbf{Block Index}: Identifies the starting block (e.g., block 0 for + \item \textbf{Block Index}: The starting block (e.g., block 0 for features 0--31, block 1 for features 32--63, etc.). \item \textbf{Number of Blocks}: How many blocks the driver wishes to retrieve or modify in a single message. @@ -270,7 +270,7 @@ \subsubsection{Bus vs. Transport Messages} or assessing bus-wide health (\busref{PING}). These messages are \emph{optional} in environments where device discovery or state changes occur through other means (e.g., device - tree). However, if a bus \emph{chooses} to handle those tasks via messages, + tree). However, if a bus chooses to handle those tasks via messages, it \emph{should} implement the appropriate bus message definitions. \item[\textbf{Transport Messages}:] @@ -295,7 +295,7 @@ \subsubsection{Bus vs. Transport Messages} \subsubsection{Endianness} \label{sec:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Endianness} -All encoding of values and fields defines in the virtio-msg messages \emph{MUST} +All encoding of values and fields defined in the virtio-msg messages \emph{MUST} be encoded in little-endian. \subsubsection{Common Message Format} @@ -686,7 +686,7 @@ \subsubsection{Device Notifications} These notifications may be the result of: \begin{itemize} - \item The same messages received in band on the message channel from the + \item The same messages received in-band on the message channel from the device side bus. \item Manufactured by the driver side bus based on reception of an out-of-band (OoB) notification from the device side. Example OoB @@ -709,7 +709,8 @@ \subsubsection{Device Notifications} \paragraph{EVENT\_USED} \begin{itemize} \item Signifies that one or more buffers in a specific virtqueue have been - processed or consumed by the device. + processed or consumed by the device and the buffer has been added to the + used ring of the virtqueue. \item The driver uses normal virtio methods (e.g., reading the "used" ring) to identify which buffers are complete. \item If a device does not support sending \msgref{EVENT_USED}, the driver @@ -831,7 +832,7 @@ \subsubsection{Overview} \end{itemize} The functionality of the following messages \textbf{MUST} be provided by -in band messages, out of band event notification, or bus implementation based +in-band messages, out-of-band event notification, or bus implementation based polling: \begin{itemize} \item \msgref{EVENT_AVAIL} @@ -1115,8 +1116,8 @@ \subsubsection{Overview} The \textbf{Next wrap} field is the MSB of the 32 bit value. The \textbf{Next offset} field is the other 31 bits. These fields should be 0 if the VIRTIO_F_NOTIFICATION_DATA feature has not been negotiated. If the bus -implementation is using out-of-band notifications, it should refuse to allow -this feature to be negotiated. +implementation is using out-of-band notifications, it should prevent this +feature from being negotiated. \msgdef{EVENT_USED} @@ -1158,7 +1159,7 @@ \subsection{Bus Messages}\label{sec:Virtio Transport Options / Virtio Over Messa \hline \end{tabular} -Bus message IDs below 0x80 are reserved for standardizes (but optional) bus +Bus message IDs below 0x80 are reserved for standardized (but optional) bus messages. A few are used here and more are expected in the future. Bus message IDs below 0x40 are used for request/response messages and 0x40 and above for event messages. @@ -1324,16 +1325,15 @@ \subsubsection{Optional Requirements} and \busref{EVENT_DEVICE} for discovering and managing devices in a message-driven manner. However, this is not mandatory if other enumeration methods (e.g., device tree, ACPI, hypervisor firmware) are used. - \item If a bus \emph{chooses} to implement these messages, it \textbf{MUST} do + \item If a bus chooses to implement these messages, it \textbf{MUST} do so in compliance with their defined formats and semantics (see \ref{sec:Virtio Transport Options / Virtio Over Messages / Bus Messages}). \end{itemize} \paragraph{Optional Bus-Level Messages} \begin{itemize} - \item \busref{PING} for keepalive or health checks is also \emph{MAY} - implement. If used, both sides \textbf{MUST} echo the 32-bit data field - precisely. + \item \busref{PING} \emph{MAY} be implemented for keepalive or health checks. + If used, both sides \textbf{MUST} echo the 32-bit data field precisely. \end{itemize} \paragraph{Runtime Notifications} From 4198de8ae2ee6866b580ecdbb8828ff3bde5e527 Mon Sep 17 00:00:00 2001 From: Bill Mills Date: Sat, 18 Oct 2025 08:51:29 -0400 Subject: [PATCH 09/11] virtio-msg: editorial changes from Peter * add missing ')' * clarify that bus properties are advertised from the bus to the transport layer and change "driver" to bus or transport layer to more clearly show whom is targeted by the requirements * Avoid passive voice and clearly specify which element must conform * s/All encoding of values and fields/All values and fields/ * Use u8 and le16 types for virtio_msg_message * s/must be zero of Bus messages/must be zero for Bus messages/ * use ':' at end of item in list, not ';' * Add missing verb and s/posted/available in Runtime Notifications Signed-off-by: Bill Mills --- transport-msg.tex | 39 ++++++++++++++++++++------------------- 1 file changed, 20 insertions(+), 19 deletions(-) diff --git a/transport-msg.tex b/transport-msg.tex index 6efab5a..5af212f 100644 --- a/transport-msg.tex +++ b/transport-msg.tex @@ -26,7 +26,7 @@ \subsubsection{Purpose} \item \textbf{Reduce per-bus complexity:} Buses can implement a fully message-based workflow (including optional - enumeration via \busref{GET_DEVICES} and hotplug via \busref{EVENT_DEVICE} + enumeration via \busref{GET_DEVICES} and hotplug via \busref{EVENT_DEVICE}) or they can discover and manage devices through alternative means such as platform firmware data. In either case, they forward transport messages to and from each device. @@ -110,7 +110,8 @@ \subsection{Basic Concepts} \subsubsection{Transport Revisions and Maximum Message Size} \label{sec:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Revisions} -Each \textbf{virtio-msg bus instance} advertises: +Each \textbf{virtio-msg bus instance} advertises the following to the transport +layer: \begin{itemize} \item A \textbf{transport revision} indicating the protocol version it supports. This revision is separate from the overall Virtio @@ -122,8 +123,8 @@ \subsubsection{Transport Revisions and Maximum Message Size} \end{itemize} These parameters \emph{MAY} vary between bus instances within the same system. -The driver obtains a bus's revision, maximum message size and list of features -through an \emph{implementation-defined} mechanism, which could be: +The bus implementation obtains a bus's revision, maximum message size and list +of features through an \emph{implementation-defined} mechanism, which could be: \begin{itemize} \item A device tree or firmware method providing bus configuration, \item A message exchange during bus setup, @@ -131,10 +132,10 @@ \subsubsection{Transport Revisions and Maximum Message Size} \item A static definition built into the driver for a known environment. \end{itemize} -After learning these parameters, the driver \emph{MUST} respect them for all -messages involving that bus instance. For example, it \emph{MUST NOT} send a -message exceeding the \textbf{maximum message size}, and it \emph{MUST} avoid -using advanced features or messages unavailable in the bus's advertised +After learning these parameters, the transport layer \emph{MUST} respect them +for all messages involving that bus instance. For example, it \emph{MUST NOT} +send a message exceeding the \textbf{maximum message size}, and it \emph{MUST} +avoid using advanced features or messages unavailable in the bus's advertised \textbf{transport revision}. \paragraph{virtio-msg revisions} @@ -188,8 +189,8 @@ \subsubsection{Configuration Generation Count} or the response to \msgref{GET_CONFIG} which also both provide the device's current configuration count. The device may change any amount of data for one generation count increment. If the change cannot fit in one \msgref{EVENT_CONFIG} -message, it \emph{SHOULD} be signaled to the driver via a \msgref{EVENT_CONFIG} -message with a zero data length and the new generation count. +message, the device \emph{SHOULD} send an \msgref{EVENT_CONFIG} message +with a zero data length and the new generation count to the driver. The device \emph{MUST NOT} provide the same generation count in multiple \msgref{EVENT_CONFIG} messages that contain non-zero length config data. The driver includes its view of the current generation count in @@ -295,7 +296,7 @@ \subsubsection{Bus vs. Transport Messages} \subsubsection{Endianness} \label{sec:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Endianness} -All encoding of values and fields defined in the virtio-msg messages \emph{MUST} +All values and fields defined in the virtio-msg messages \emph{MUST} be encoded in little-endian. \subsubsection{Common Message Format} @@ -308,12 +309,12 @@ \subsubsection{Common Message Format} The header layout is: \begin{lstlisting} struct virtio_msg_header { - uint8_t type; /* request/response + bus/transport */ - uint8_t msg_id; /* message id */ - uint16_t dev_num; /* device number (0 for bus messages) */ - uint16_t token; /* correlation identifier (0 for events) */ - uint16_t msg_size; /* total size: header (8) + payload */ - uint8_t payload[]; + u8 type; /* request/response + bus/transport */ + u8 msg_id; /* message id */ + le16 dev_num; /* device number (0 for bus messages) */ + le16 token; /* correlation identifier (0 for events) */ + le16 msg_size; /* total size: header (8) + payload */ + u8 payload[]; }; \end{lstlisting} @@ -1340,10 +1341,10 @@ \subsubsection{Optional Requirements} \begin{itemize} \item A device or the driver side bus \emph{MUST} send \msgref{EVENT_CONFIG} to inform the driver of configuration of device status changes. - \item A device or the driver side bus \emph{MUST} \msgref{EVENT_USED} + \item A device or the driver side bus \emph{MUST} send \msgref{EVENT_USED} to inform the driver of (likely) buffer completions. \item A driver \emph{MUST} send \msgref{EVENT_AVAIL} to notify the device that - new buffers are posted. + new buffers are available. \end{itemize} \subsubsection{Compliance for Different Environments} From b3bec81c949ec8da004881a31ff6401e6468d846 Mon Sep 17 00:00:00 2001 From: Bill Mills Date: Sat, 18 Oct 2025 08:51:29 -0400 Subject: [PATCH 10/11] virtio-msg: align 8 bytes fields Align 8 byte fields to offsets that are multiples of 8. Messages are byte streams not directly addressable C structures. Custom messages may use any field alignment they wish. However, for the current standard messages there is no harm in aligning them. This only effects GET_VQUEUE and SET_VQUEUE. Signed-off-by: Bill Mills --- transport-msg.tex | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/transport-msg.tex b/transport-msg.tex index 5af212f..7455a73 100644 --- a/transport-msg.tex +++ b/transport-msg.tex @@ -1006,9 +1006,10 @@ \subsubsection{Overview} Answer & 0 & 4 & Virtqueue index \\ & 4 & 4 & Maximum virtqueue size \\ & 8 & 4 & Current virtqueue size \\ -& 12 & 8 & Descriptor address \\ -& 20 & 8 & Driver address \\ -& 28 & 8 & Device address \\ +& 12 & 4 & Reserved (Must Be Zero - MBZ) \\ +& 16 & 8 & Descriptor address \\ +& 24 & 8 & Driver address \\ +& 32 & 8 & Device address \\ \hline \end{tabular} @@ -1027,9 +1028,10 @@ \subsubsection{Overview} Request & 0 & 4 & Virtqueue index \\ & 4 & 4 & Reserved (Must Be Zero - MBZ) \\ & 8 & 4 & Current virtqueue size \\ -& 12 & 8 & Descriptor address \\ -& 20 & 8 & Driver address \\ -& 28 & 8 & Device address \\ +& 12 & 4 & Reserved (Must Be Zero - MBZ) \\ +& 16 & 8 & Descriptor address \\ +& 24 & 8 & Driver address \\ +& 32 & 8 & Device address \\ \hline Answer & 0 & 0 & no extra data \\ \hline From 6bad61741b1b93069c5fbb43a728c4cde63a3e36 Mon Sep 17 00:00:00 2001 From: Bill Mills Date: Thu, 23 Oct 2025 21:49:08 -0400 Subject: [PATCH 11/11] virtio-msg: fix-up response matching as discussed Oct 23 Simplify the response matching as discussed in the PR and in the Oct 23 meeting. * Bus messages have a dev_num and it is normally 0 so we don't need to say that matching dev_num if only for transport messages. * Drop msg_id from matching. Altough the msg_id would normally match there may be some cases where it would not. The token field should be unique for a given dev_num. * This simplifys the description of the matching tuple as well Signed-off-by: Bill Mills --- transport-msg.tex | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/transport-msg.tex b/transport-msg.tex index 7455a73..eaab0eb 100644 --- a/transport-msg.tex +++ b/transport-msg.tex @@ -363,8 +363,7 @@ \subsubsection{Message Correlation} The token field in the message header is part of a tuple that is unique during a request to response interval. \begin{itemize} - \item Bus Message tuple: ( 1, 0, \field{token} ). - \item Transport Message tuple: ( 0, \field{dev_num}, \field{token} ). + \item Message tuple: ( \field{dev_num}, \field{token} ). \end{itemize} Rules: @@ -372,13 +371,10 @@ \subsubsection{Message Correlation} \item The request originator assigns a non-zero \field{token} for every request such that the tuple is unique for all inflight requests. \item Event (one-way) messages \textbf{MUST} set \field{token}=0 - \item A response \textbf{MUST} echo \field{token}, \field{dev_num} (for - transport), and \field{msg_id}. + \item A response \textbf{MUST} echo \field{token} and \field{dev_num}. \item Reception of unknown or already completed correlation tuples \textbf{SHOULD} result in discarding the response without further protocol action. - \item Implementations \textbf{MAY} deliver responses out of order unless a - specific message definition mandates ordering. \end{itemize} \subsection{Bus Operation}