From 8650ec88e6780e31f059addf20d2a572201c3e63 Mon Sep 17 00:00:00 2001 From: Bill Mills Date: Sun, 15 Jun 2025 13:57:56 -0400 Subject: [PATCH 01/12] virtio-msg: Remove EVENT_CONFIG_EMPTY As decided on June 5th, the EVENT_CONFIG_EMPTY message is deleted. Instead of documenting "invalid" settings of device status and configuration generation count, I have removed that use case of EVENT_CONFIG. EVENT_CONFIG can still be used with an without changed configuration data in the payload. More explicit documentation on these two cases has been added. Signed-off-by: Bill Mills --- transport-msg.tex | 59 +++++++++++++++-------------------------------- 1 file changed, 18 insertions(+), 41 deletions(-) diff --git a/transport-msg.tex b/transport-msg.tex index 0d24653..03090c1 100644 --- a/transport-msg.tex +++ b/transport-msg.tex @@ -189,8 +189,9 @@ \subsubsection{Configuration Generation Count} or the response to \msgref{GET_CONFIG} which also both provide the device's current configuration count. The device may change any amount of data for one generation count increment. If the change cannot fit in one \msgref{EVENT_CONFIG} -message, it \emph{SHOULD} be signaled to the driver via a \msgref{EVENT_CONFIG_EMPTY} -message. The device \emph{MUST NOT} provide the same generation count in +message, it \emph{SHOULD} be signaled to the driver via a \msgref{EVENT_CONFIG} +message with a zero data length and the new generation count. +The device \emph{MUST NOT} provide the same generation count in multiple \msgref{EVENT_CONFIG} messages that contain non-zero length config data. The driver includes its view of the current generation count in \msgref{SET_CONFIG} requests. The configuration generation count does not @@ -634,9 +635,6 @@ \subsubsection{Device Notifications} status, and optionally the updated portion of the configuration data. \item If the data is not included, the driver \emph{SHOULD} issue \msgref{GET_CONFIG} to discover the updated configuration. - \item If a device does not support to send \msgref{EVENT_CONFIG}, the driver - \emph{MAY} rely on \msgref{GET_CONFIG}, \msgref{SET_CONFIG} and - \msgref{GET_DEVICE_STATUS} to handle configuration and status changes. \end{itemize} \paragraph{EVENT\_USED} @@ -658,8 +656,8 @@ \subsubsection{Configuration Changes During Operation} The driver \emph{MAY} issue new \msgref{SET_CONFIG} messages after initialization if device configuration needs to be changed at runtime. The device \emph{MAY} also make configuration changes at -runtime and \emph{MUST} signal those changes with \msgref{EVENT_CONFIG} or -\msgref{EVENT_CONFIG_EMPTY} messages. +runtime and \emph{MUST} signal those changes with \msgref{EVENT_CONFIG} +messages. \subsubsection{Virtqueue Changes During Operation} \label{sec:Virtio Transport Options / Virtio Over Messages / Device Operation / VQueueUpdates} @@ -738,13 +736,11 @@ \subsubsection{Overview} \hline \msgref{RESET_VQUEUE} & 0xB & Driver \\ \hline -\msgref{EVENT_CONFIG_EMPTY} & 0x40 & Device \\ +\msgref{EVENT_CONFIG} & 0x40 & Device \\ \hline -\msgref{EVENT_CONFIG} & 0x41 & Device \\ +\msgref{EVENT_AVAIL} & 0x41 & Driver \\ \hline -\msgref{EVENT_AVAIL} & 0x42 & Driver \\ -\hline -\msgref{EVENT_USED} & 0x43 & Device \\ +\msgref{EVENT_USED} & 0x42 & Device \\ \hline \end{tabular} @@ -769,8 +765,7 @@ \subsubsection{Overview} \begin{itemize} \item \msgref{EVENT_AVAIL} \item \msgref{EVENT_USED} - \item \msgref{EVENT_CONFIG} or \msgref{EVENT_CONFIG_EMPTY} - \item \msgref{EVENT_USED} + \item \msgref{EVENT_CONFIG} \end{itemize} \newcommand{\msgdef}[1]{\subsubsection{VIRTIO_MSG_#1}\label{sec:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_#1}} @@ -983,31 +978,6 @@ \subsubsection{Overview} \hline \end{tabular} -\msgdef{EVENT_CONFIG_EMPTY} - -This message is sent by the virtio-msg device and there is no response. - -\begin{tabular}{|l|l|l|l|} -\hline -Type & Offset & Size (bytes) & Content \\ -\hline \hline -Request & 0 & 0 & no extra data \\ -\hline -\end{tabular} - -The \msgref{EVENT_CONFIG_EMPTY} message is used when the extra information in -\msgref{EVENT_CONFIG} is not available. In general \msgref{EVENT_CONFIG} -\emph{SHOULD} be preferred to \msgref{EVENT_CONFIG_EMPTY} but this is not always -possible. If the driver side bus implementation is manufacturing notification -based on polling intervals, it should use this message to notify the driver -side transport layer that it should use \msgref{GET_DEVICE_STATUS} and -\msgref{GET_CONFIG} to query for device side changes. - -This message may also be used when a single configuration change on the device -side cannot be communicated in one \msgref{EVENT_CONFIG} message due to maximum -message size and the device does not desire to represent it as multiple -configuration changes. - \msgdef{EVENT_CONFIG} This message is sent by the virtio-msg device and there is no response. @@ -1019,11 +989,19 @@ \subsubsection{Overview} Request & 0 & 4 & Device status \\ & 4 & 4 & Configuration generation count \\ & 8 & 4 & Configuration offset \\ -& 12 & 4 & Number of bytes \\ +& 12 & 4 & Number of bytes (may be zero) \\ & 16 & ... & Configuration data \\ \hline \end{tabular} +The number of bytes field may be zero. If so the configuration data field +will also be of zero length. This is the normal case when the configuration +generation count has not been changed. + +If the number of bytes field is zero and the configuration generation count is +changed, then the driver is responsible for discovering any changed +configuration data via \msgref{GET_CONFIG} messages. + \msgdef{EVENT_AVAIL} This message is sent by the virtio-msg driver and there is no response. @@ -1268,7 +1246,6 @@ \subsubsection{Optional Requirements} \paragraph{Runtime Notifications} \begin{itemize} \item A device or the driver side bus \emph{MUST} send \msgref{EVENT_CONFIG} - or \msgref{EVENT_CONFIG_EMPTY} to inform the driver of configuration of device status changes. \item A device or the driver side bus \emph{MUST} \msgref{EVENT_USED} to inform the driver of (likely) buffer completions. From 60538a2d9cce222c72896f6c78365c8ba8523d64 Mon Sep 17 00:00:00 2001 From: Bill Mills Date: Sun, 15 Jun 2025 14:02:41 -0400 Subject: [PATCH 02/12] virtio-msg: Fix virtqueue index value for "all virtqueues" The hex value for this case was always intended to be "all 1s" but was mis-specified before. This fixes that. This also makes it clear that for the EVENT_AVAIL use case, this would typically be used on the device side. (The driver side always knows which virtqueue was updated as it originates this event.) Signed-off-by: Bill Mills --- transport-msg.tex | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/transport-msg.tex b/transport-msg.tex index 03090c1..773bd9b 100644 --- a/transport-msg.tex +++ b/transport-msg.tex @@ -1015,10 +1015,10 @@ \subsubsection{Overview} \hline \end{tabular} -If virtqueue index is 0xFFFF then it indicates that all virtqueus should be -polled for available buffers. This is useful when the bus implementation is -using out-of-band notifications, using polling, or when emulating or bridging -to another virtio transport. +If virtqueue index is 0xFFFFFFFF then it indicates that all virtqueues should be +polled for available buffers. This may be useful on the device side when the +bus implementation is using out-of-band notifications, using polling, or when +emulating or bridging to another virtio transport. The \textbf{Next wrap} field is the MSB of the 32 bit value. The \textbf{Next offset} field is the other 31 bits. These fields should be 0 if @@ -1038,8 +1038,8 @@ \subsubsection{Overview} \hline \end{tabular} -If virtqueue index is 0xFFFF then it indicates that all virtqueus should be -polled for used buffers. This is useful when the bus implementation is +If virtqueue index is 0xFFFFFFFF then it indicates that all virtqueues should be +polled for used buffers. This may be useful when the bus implementation is using out-of-band notifications, using polling, or when emulating or bridging to another virtio transport. From 68092972836a945956dd17058fa43fdf8dce5499 Mon Sep 17 00:00:00 2001 From: Bill Mills Date: Sun, 15 Jun 2025 14:05:53 -0400 Subject: [PATCH 03/12] virtio-msg: fix spelling and English usage * writeing -> writing * support to send -> support sending * /msgref -> \msgref Signed-off-by: Bill Mills --- transport-msg.tex | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/transport-msg.tex b/transport-msg.tex index 773bd9b..0659885 100644 --- a/transport-msg.tex +++ b/transport-msg.tex @@ -643,7 +643,7 @@ \subsubsection{Device Notifications} processed or consumed by the device. \item The driver uses normal virtio methods (e.g., reading the “used” ring) to identify which buffers are complete. - \item If a device does not support to send \msgref{EVENT_USED}, the driver + \item If a device does not support sending \msgref{EVENT_USED}, the driver \emph{MAY} rely on standard virtqueue polling mechanisms to detect completion. \end{itemize} @@ -680,7 +680,7 @@ \subsubsection{Device Reset and Shutdown} In some circumstances, the device \emph{MAY} also trigger a reset if it encounters an unrecoverable error. This can be signaled to the driver by sending an \msgref{EVENT_CONFIG} with a \emph{device status} with the DEVICE_NEEDS_RESET -bit set. The driver \emph{SHOULD} reset the device by writeing a status of 0 +bit set. The driver \emph{SHOULD} reset the device by writing a status of 0 via the \msgref{SET_DEVICE_STATUS} message. The driver \emph{MAY} reinitialize the device if it wishes to use the device again. @@ -1214,7 +1214,7 @@ \subsubsection{Mandatory Requirements} the \textbf{maximum message size}, including correct offsets and block counts in \msgref{GET_DEVICE_FEATURES}, \msgref{SET_DRIVER_FEATURES}, \msgref{GET_CONFIG}, and \msgref{SET_CONFIG}. - \item A driver \emph{MUST} handle a /msgref{SET_CONFIG} being rejected for a + \item A driver \emph{MUST} handle a \msgref{SET_CONFIG} being rejected for a mismatched configuration generation count. \item A driver \textbf{MUST} initialize each device’s virtqueues (where needed) via \msgref{SET_VQUEUE} before attempting normal I/O and \textbf{SHOULD} From 9ccb02b33dd4fef97f186a13ec65d959c32d8f5f Mon Sep 17 00:00:00 2001 From: Bill Mills Date: Sun, 15 Jun 2025 15:09:53 -0400 Subject: [PATCH 04/12] virtio-msg: Use EVENT_DEVICE to replace DEVICE_ADDED and DEVICE_REMOVED This uses the EVENT_ name for event messages and replaces two messages with one that includes a state value. This allows future expansion of state to handle things like suspend/resume and request removal. Signed-off-by: Bill Mills --- transport-msg.tex | 55 ++++++++++++++++++++++++++--------------------- 1 file changed, 30 insertions(+), 25 deletions(-) diff --git a/transport-msg.tex b/transport-msg.tex index 0659885..51a62ee 100644 --- a/transport-msg.tex +++ b/transport-msg.tex @@ -27,8 +27,8 @@ \subsubsection{Purpose} \item \textbf{Reduce per-bus complexity:} Buses can implement a fully message-based workflow (including optional - enumeration via \busref{GET_DEVICES} and hotplug via \busref{DEVICE_ADDED} - / \busref{DEVICE_REMOVED}), or they can discover and manage devices through + enumeration via \busref{GET_DEVICES} and hotplug via \busref{EVENT_DEVICE} + or they can discover and manage devices through alternative means such as platform firmware data. In either case, they forward transport messages to and from each device. @@ -49,8 +49,7 @@ \subsubsection{High-Level Architecture} \item Optional message-based queries (\busref{GET_DEVICES}), \item External data sources (e.g., device tree, ACPI tables, hypervisor firmware calls), - \item Dynamic hotplug notifications (optionally via \busref{DEVICE_ADDED} - and \busref{DEVICE_REMOVED}). + \item Dynamic hotplug notifications (optionally via \busref{EVENT_DEVICE}). \end{itemize} Once a device is identified—regardless of discovery method—the bus typically uses \msgref{GET_DEVICE_INFO} to read its device ID and vendor ID, then @@ -92,7 +91,7 @@ \subsubsection{System Topology} \subsubsection{Optional Bus Messages} This specification \emph{defines} messages such as \busref{GET_DEVICES}, -\busref{DEVICE_ADDED}, \busref{DEVICE_REMOVED}, and \busref{PING} for a +\busref{EVENT_DEVICE}, and \busref{PING} for a completely message-based approach to enumeration, hotplug, and bus-wide health. However, these are \emph{not} mandatory if a bus instance already handles those functions via firmware, device tree, or other platform features. The only strict @@ -252,8 +251,8 @@ \subsubsection{Bus vs. Transport Messages} \begin{description} \item[\textbf{Bus Messages}:] Intended for global bus operations such as enumerating device numbers - (\busref{GET_DEVICES}), managing device hotplug events (\busref{DEVICE_ADDED}, - \busref{DEVICE_REMOVED}), or assessing bus-wide health (\busref{PING}). + (\busref{GET_DEVICES}), managing device hotplug events (\busref{EVENT_DEVICE}) + or assessing bus-wide health (\busref{PING}). These messages are \emph{optional} in environments where device discovery or state changes occur through other means (e.g., device tree). However, if a bus \emph{chooses} to handle those tasks via messages, @@ -358,14 +357,14 @@ \subsubsection{Device Hotplug} If the bus supports dynamic addition or removal of devices at runtime, it \emph{MAY} announce these events using messages: \begin{itemize} - \item \busref{DEVICE_ADDED} for a newly available device, - \item \busref{DEVICE_REMOVED} for a device that is no longer accessible. + \item \busref{EVENT_DEVICE} with state ready for a newly available device, + \item \busref{EVENT_DEVICE} with state removed for a device that is no longer accessible. \end{itemize} Alternatively, the bus \emph{MAY} rely on external signals (e.g., an event from the platform or hypervisor) and inform the OS of hotplug changes out-of-band. -This specification \emph{defines} \busref{DEVICE_ADDED} and -\busref{DEVICE_REMOVED} for implementations that prefer a fully message-based +This specification \emph{defines} \busref{EVENT_DEVICE} +for implementations that prefer a fully message-based approach but does not require their use. In any scenario, once notified that a device has appeared, the bus \emph{SHOULD} query the device (e.g., via \msgref{GET_DEVICE_INFO}) and register it with the OS. If a device is @@ -1067,9 +1066,7 @@ \subsection{Bus Messages}\label{sec:Virtio Transport Options / Virtio Over Messa \hline \busref{PING} & 0x3 & Any \\ \hline -\busref{DEVICE_ADDED} & 0x41 & Device \\ -\hline -\busref{DEVICE_REMOVED} & 0x42 & Device \\ +\busref{EVENT_DEVICE} & 0x40 & Device \\ \hline \end{tabular} @@ -1115,7 +1112,7 @@ \subsection{Bus Messages}\label{sec:Virtio Transport Options / Virtio Over Messa device in the least significant bit of the first byte, the eighth device in the most significant bit of the first byte, etc. -\busdef{DEVICE_ADDED} +\busdef{EVENT_DEVICE} This message is sent by the virtio-msg device side bus and there is no response. @@ -1124,21 +1121,29 @@ \subsection{Bus Messages}\label{sec:Virtio Transport Options / Virtio Over Messa Type & Offset & Size (bytes) & Content \\ \hline \hline Request & 0 & 2 & Device Number \\ + & 2 & 2 & Device Bus State \\ \hline \end{tabular} -\busdef{DEVICE_REMOVED} - -This message is sent by the virtio-msg device side bus and there is no response. - -\begin{tabular}{|l|l|l|l|} +\begin{tabular}{|l|l|l|} \hline -Type & Offset & Size (bytes) & Content \\ -\hline \hline -Request & 0 & 2 & Device Number \\ +Device Bus State & Value & Meaning \\ +\hline +\hline +DEVICE_BUS_STATE_READY & 0x0 & Device is present and ready \\ +\hline +DEVICE_BUS_STATE_REMOVED & 0x1 & Device is no longer present \\ +\hline +reserved & 0x2 to 0x7FFF & Reserved for standard use \\ +\hline +any & 0x8000 to 0xFFFF & may be used by bus implementations \\ \hline \end{tabular} +This event is sent when the bus level state of the device has changed. +It can indicate when a device is added or removed. It may also be used by +bus implementations to indicate other states. + \busdef{PING} This message is sent by the virtio-msg driver side bus or device side bus and @@ -1227,8 +1232,8 @@ \subsubsection{Optional Requirements} \paragraph{Bus Messages for Enumeration and Hotplug} \begin{itemize} - \item Implementations \emph{MAY} use \busref{GET_DEVICES}, \busref{DEVICE_ADDED}, - and \busref{DEVICE_REMOVED} for discovering and managing devices in a + \item Implementations \emph{MAY} use \busref{GET_DEVICES} + and \busref{EVENT_DEVICE} for discovering and managing devices in a message-driven manner. However, this is not mandatory if other enumeration methods (e.g., device tree, ACPI, hypervisor firmware) are used. \item If a bus \emph{chooses} to implement these messages, it \textbf{MUST} do From 4bdb8fe72a8f16efe2b47be6aa3b39f5883da6f4 Mon Sep 17 00:00:00 2001 From: Bill Mills Date: Sun, 15 Jun 2025 15:32:02 -0400 Subject: [PATCH 05/12] virtio-msg: fix offset of max num virtqueues in GET_DEVICE_INFO Fix the copy and past error here. Signed-off-by: Bill Mills --- transport-msg.tex | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/transport-msg.tex b/transport-msg.tex index 51a62ee..7ab7c4b 100644 --- a/transport-msg.tex +++ b/transport-msg.tex @@ -784,7 +784,7 @@ \subsubsection{Overview} & 4 & 4 & Vendor ID \\ & 8 & 4 & Number of feature bits \\ & 12 & 4 & Configuration size in bytes \\ -& 12 & 4 & Maximum number of virtqueues \\ +& 16 & 4 & Maximum number of virtqueues \\ \hline \end{tabular} From 36cc50bcad87e00a483bad8f99f7b53be15b5fef Mon Sep 17 00:00:00 2001 From: Bill Mills Date: Sun, 15 Jun 2025 15:39:36 -0400 Subject: [PATCH 06/12] virtio-msg: add support for admin virtqueues and SHM regions * Add support for device owned shared memory segments from virtio v1.3 * Add support for Admin virtqueues from virtio v1.4 (draft) Signed-off-by: Bill Mills --- transport-msg.tex | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) diff --git a/transport-msg.tex b/transport-msg.tex index 7ab7c4b..c8df74e 100644 --- a/transport-msg.tex +++ b/transport-msg.tex @@ -468,12 +468,21 @@ \subsubsection{Device Information} configuration space. \item \textbf{Number of virtqueues}: The maximum number of virtqueues the device supports. + \item \textbf{Number of SHM segments}: The number of device owned + shared memory segments this device has. + \item \textbf{Admin virtqueue starting index}: The virtqueue index for the + first admin virtqueue supported by the device. + \item \textbf{Admin virtqueue count}: The number of admin virtqueues + supported by the device. \end{itemize} Armed with these details, the driver knows how many feature blocks to query, what portion of the configuration space is valid, and which OS-level virtio driver might bind to this device based on the Device ID. +It also knows of any device owned shared memory segments and any admin +virtqueues supported by the device. + \subsubsection{Feature Negotiation} \label{sec:Virtio Transport Options / Virtio Over Messages / Device Initialization / Device Features} @@ -735,6 +744,8 @@ \subsubsection{Overview} \hline \msgref{RESET_VQUEUE} & 0xB & Driver \\ \hline +\msgref{GET_SHM} & 0xC & Driver \\ +\hline \msgref{EVENT_CONFIG} & 0x40 & Device \\ \hline \msgref{EVENT_AVAIL} & 0x41 & Driver \\ @@ -785,6 +796,8 @@ \subsubsection{Overview} & 8 & 4 & Number of feature bits \\ & 12 & 4 & Configuration size in bytes \\ & 16 & 4 & Maximum number of virtqueues \\ +& 20 & 2 & Admin virtqueue start index \\ +& 22 & 2 & Admin virtqueue count \\ \hline \end{tabular} @@ -977,6 +990,30 @@ \subsubsection{Overview} \hline \end{tabular} +\msgdef{GET_SHM} + +This message is sent by the virtio-msg transport driver and requires a +response from the device. + +\begin{tabular}{|l|l|l|l|} +\hline +Type & Offset & Size (bytes) & Content \\ +\hline \hline +Request & 0 & 4 & Shared memory region index \\ +\hline +Answer & 0 & 4 & Shared memory region index \\ +& 4 & 4 & Shared memory region length \\ +& 8 & 4 & Shared memory region address \\ +\hline +\end{tabular} + +If the returned shared memory region length is zero, the memory region does not +exist. + +Note: these shared memory regions are device owned and supported on other virtio +transports since virtio v1.3. These shared memory regions are separate and +distinct from any shared memory regions defined by the bus implementation. + \msgdef{EVENT_CONFIG} This message is sent by the virtio-msg device and there is no response. From 64f52cc40cfcffaade4a50809e653eeb9c37f761 Mon Sep 17 00:00:00 2001 From: Bill Mills Date: Sun, 15 Jun 2025 16:03:13 -0400 Subject: [PATCH 07/12] virtio-msg: Make {SET,GET,EVENT}_CONFIG field order more consistent Make the order of fields: * [config gen count, if present] * config offset * config length * config data Signed-off-by: Bill Mills --- transport-msg.tex | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/transport-msg.tex b/transport-msg.tex index c8df74e..5492de2 100644 --- a/transport-msg.tex +++ b/transport-msg.tex @@ -863,9 +863,9 @@ \subsubsection{Overview} Request & 0 & 4 & Configuration offset in bytes \\ & 4 & 4 & Number of bytes \\ \hline -Answer & 0 & 4 & Configuration offset in bytes \\ -& 4 & 4 & Number of bytes \\ -& 8 & 4 & Configuration generation count \\ +Answer & 0 & 4 & Configuration generation count \\ +& 4 & 4 & Configuration offset in bytes \\ \\ +& 8 & 4 & Number of bytes \\ & 12 & ... & Configuration data \\ \hline \end{tabular} @@ -879,14 +879,14 @@ \subsubsection{Overview} \hline Type & Offset & Size (bytes) & Content \\ \hline \hline -Request & 0 & 4 & Configuration offset in bytes\\ -& 4 & 4 & Number of bytes \\ -& 8 & 4 & Configuration generation count \\ +Request & 0 & 4 & Configuration generation count \\ +& 4 & 4 & Configuration offset in bytes \\ +& 8 & 4 & Number of bytes \\ & 12 & ... & Configuration data \\ \hline -Answer & 0 & 4 & Configuration offset in bytes\\ -& 4 & 4 & Number of bytes, or 0 if rejected \\ -& 8 & 4 & New Configuration generation count \\ +Answer & 0 & 4 & New Configuration generation count \\ +& 4 & 4 & Configuration offset in bytes \\ +& 8 & 4 & Number of bytes, or 0 if rejected \\ & 12 & ... & Configuration data \\ \hline \end{tabular} From 2fe42f1e2dd25d7d948b38d84413118b5a7f516a Mon Sep 17 00:00:00 2001 From: Bill Mills Date: Thu, 19 Jun 2025 09:00:25 -0400 Subject: [PATCH 08/12] virtio-msg: review fixup for EVENT_DEVICE * Use full state names in description * Use 1 & 2 as state values Signed-off-by: Bill Mills --- transport-msg.tex | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/transport-msg.tex b/transport-msg.tex index 5492de2..2dc787d 100644 --- a/transport-msg.tex +++ b/transport-msg.tex @@ -357,8 +357,10 @@ \subsubsection{Device Hotplug} If the bus supports dynamic addition or removal of devices at runtime, it \emph{MAY} announce these events using messages: \begin{itemize} - \item \busref{EVENT_DEVICE} with state ready for a newly available device, - \item \busref{EVENT_DEVICE} with state removed for a device that is no longer accessible. + \item \busref{EVENT_DEVICE} with state DEVICE_BUS_STATE_READY for a newly + available device, + \item \busref{EVENT_DEVICE} with state DEVICE_BUS_STATE_REMOVED for a device + that is no longer accessible. \end{itemize} Alternatively, the bus \emph{MAY} rely on external signals (e.g., an event from @@ -1167,11 +1169,11 @@ \subsection{Bus Messages}\label{sec:Virtio Transport Options / Virtio Over Messa Device Bus State & Value & Meaning \\ \hline \hline -DEVICE_BUS_STATE_READY & 0x0 & Device is present and ready \\ +DEVICE_BUS_STATE_READY & 0x1 & Device is present and ready \\ \hline -DEVICE_BUS_STATE_REMOVED & 0x1 & Device is no longer present \\ +DEVICE_BUS_STATE_REMOVED & 0x2 & Device is no longer present \\ \hline -reserved & 0x2 to 0x7FFF & Reserved for standard use \\ +reserved & 0x3 to 0x7FFF & Reserved for standard use \\ \hline any & 0x8000 to 0xFFFF & may be used by bus implementations \\ \hline From 33ee92f3a5488b39168697866a4185578bb46c4b Mon Sep 17 00:00:00 2001 From: Bill Mills Date: Thu, 19 Jun 2025 09:01:48 -0400 Subject: [PATCH 09/12] virtio-msg: review fixup for EVENT_CONFIG * Use "\emph{MAY} be zero" for config data length Signed-off-by: Bill Mills --- transport-msg.tex | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/transport-msg.tex b/transport-msg.tex index 2dc787d..278e0b8 100644 --- a/transport-msg.tex +++ b/transport-msg.tex @@ -1027,12 +1027,12 @@ \subsubsection{Overview} Request & 0 & 4 & Device status \\ & 4 & 4 & Configuration generation count \\ & 8 & 4 & Configuration offset \\ -& 12 & 4 & Number of bytes (may be zero) \\ +& 12 & 4 & Number of bytes (\emph{MAY} be zero) \\ & 16 & ... & Configuration data \\ \hline \end{tabular} -The number of bytes field may be zero. If so the configuration data field +The number of bytes field \emph{MAY} be zero. If so the configuration data field will also be of zero length. This is the normal case when the configuration generation count has not been changed. From 40415036655ced975f6df2a5abc4b78c132ce87a Mon Sep 17 00:00:00 2001 From: Bill Mills Date: Thu, 19 Jun 2025 09:08:30 -0400 Subject: [PATCH 10/12] virtio-msg: EVENT_{AVAIL.USED} delete "all virtqueues" provision This is hard to describe and real implementations may just use a loop anyway. So delete this for now anyway. A loop is required in the bus when: * One OoB notification can trigger multiple virtio devices * One OoB notification can trigger SOME virtqueues but not all So this value only covers one use case and it may cause more confusion than value. Signed-off-by: Bill Mills --- transport-msg.tex | 10 ---------- 1 file changed, 10 deletions(-) diff --git a/transport-msg.tex b/transport-msg.tex index 278e0b8..01466b6 100644 --- a/transport-msg.tex +++ b/transport-msg.tex @@ -1053,11 +1053,6 @@ \subsubsection{Overview} \hline \end{tabular} -If virtqueue index is 0xFFFFFFFF then it indicates that all virtqueues should be -polled for available buffers. This may be useful on the device side when the -bus implementation is using out-of-band notifications, using polling, or when -emulating or bridging to another virtio transport. - The \textbf{Next wrap} field is the MSB of the 32 bit value. The \textbf{Next offset} field is the other 31 bits. These fields should be 0 if the VIRTIO_F_NOTIFICATION_DATA feature has not been negotiated. If the bus @@ -1076,11 +1071,6 @@ \subsubsection{Overview} \hline \end{tabular} -If virtqueue index is 0xFFFFFFFF then it indicates that all virtqueues should be -polled for used buffers. This may be useful when the bus implementation is -using out-of-band notifications, using polling, or when emulating or bridging -to another virtio transport. - \subsection{Bus Messages}\label{sec:Virtio Transport Options / Virtio Over Messages / Bus Messages} The bus messages defined in this specification facilitate device discovery, From d7c1dfdc54f8fc802826e52897f2499c106a1810 Mon Sep 17 00:00:00 2001 From: Bill Mills Date: Thu, 19 Jun 2025 09:20:28 -0400 Subject: [PATCH 11/12] virtio-msg: adjust cover letter for RFC2 Document what we have done since RFC1. Signed-off-by: Bill Mills --- .prjinfo/sendmail/cover.txt | 55 +++++++++++++++++++++++++++++-------- 1 file changed, 44 insertions(+), 11 deletions(-) diff --git a/.prjinfo/sendmail/cover.txt b/.prjinfo/sendmail/cover.txt index 26911db..ec7ffc7 100644 --- a/.prjinfo/sendmail/cover.txt +++ b/.prjinfo/sendmail/cover.txt @@ -51,25 +51,58 @@ We are working on two reusable bus implementations: * Minimal requirements: bi-directional interrupts and at least one shared memory area -We also anticipate a third: +We also anticipate a few more: * virtio-msg-xen specific to Xen * Usable on any Xen system (including x86 where FF-A does not exist) * Using Xen events and page grants +* virtio-msg-loopback for userspace implemented devices + * Allows user space to provide devices to its own kernel + * This is similar to fuse, cuse or loopback block devices but for virtio + * Once developed this can provide a single kernel demo of virtio-msg + * [Work has begun on this] + +* virtio-msg over admin virtqueues + * This allows any virtio-pci device that supports admin virtqueues to also + support a virtio-msg bus that supports sub devices + * [We are looking for collaborators for this work] + +Changes since RFC1: +* reformated document to better conform to the virtio spec style + - created an introduction chapter + - created a basic concept chapter + - created bus operation and device initialization and operation chapters + - reworked description of transport and bus messages + - attempted a "compliance chapter" + - reused spec macros + - switched to MAY/MUST/SHALL/SHOULD wording + - eliminate the use of terms front-end and back-end and use driver and device +* made the maximum message size variable per bus instance +* use "device number" for virtio-msg device instances on the bus instead of + adding yet another meaning for "device ID" +* added the configuration generation count and its use +* described types of things that can be done with bus specific messages + such as setup of bus level shared memory and out of band notifications +* removed wording of notification being optional at transport level and + described bus level notifications of in-band, out-of-band, and polled + from driver side bus +* removed the ERROR message from transport level. Errors should be handled at + the bus level to better match virtio-pci and virtio-mmio +* removed bus level reset and status from standard bus messages +* replaced bus messages DEVICE_{ADDED,REMOVED} with EVENT_DEVICE +* changed names to GET_DEVICE_FEATURES and SET_DRIVER_FEATURES for clarity +* made SET_DEVICE_STATE return new state as it may not match +* Don't echo back the data in SET_VQUEUE (it cannot change) +* defined request/response vs event message id ranges +* match field size of next offset and wrap to virtio-{pci,mmio} +* added maximum number of virtqueues to DEVICE_INFO +* added admin virtqueue and SHM support + This series is a work in progress and we acknowledge at least the following issues we need to work on: -* Conform to virtio spec nouns (device/driver vs frontend/backend) - and verbs (must/may) -* Perhaps move error definition elsewhere it the spec and align on its symbols - and numeric values -* Allow message size to be greater than 40 bytes and allow bus implementations - to define their max message size -* Add a way to discover the protocol version -* Add a better description of the types of things a bus can do, specifically - including out-of-band notification and memory area sharing/discovery -* Maybe redo configuration generation handling +* Better conformance documentation Background info and work in progress implementations: * HVAC project page with intro slides [1] From 5874b403e0797361361bc22545c9fcf9eca70aa0 Mon Sep 17 00:00:00 2001 From: Bill Mills Date: Thu, 19 Jun 2025 17:10:48 -0400 Subject: [PATCH 12/12] virtio-msg: cover letter, mention more todo items * As discussed Jun 19, mention work on FF-A spec is underway but don't promise a date. * Do likewise for the AMP data structures and first bus message set * Do likewise for aligned implementations and PATCH v1 Signed-off-by: Bill Mills --- .prjinfo/sendmail/cover.txt | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.prjinfo/sendmail/cover.txt b/.prjinfo/sendmail/cover.txt index ec7ffc7..e8557de 100644 --- a/.prjinfo/sendmail/cover.txt +++ b/.prjinfo/sendmail/cover.txt @@ -103,6 +103,9 @@ This series is a work in progress and we acknowledge at least the following issues we need to work on: * Better conformance documentation +* Publish an update to Arm FF-A spec that shows virtio-msg binding (work underway) +* Publish virtio-msg-amp data structures and messages somewhere +* Align implementations to this version and send PATCH v1 (non-rfc) Background info and work in progress implementations: * HVAC project page with intro slides [1]