diff --git a/.prjinfo/sendmail/cover.txt b/.prjinfo/sendmail/cover.txt index 26911dba..e8557de5 100644 --- a/.prjinfo/sendmail/cover.txt +++ b/.prjinfo/sendmail/cover.txt @@ -51,25 +51,61 @@ We are working on two reusable bus implementations: * Minimal requirements: bi-directional interrupts and at least one shared memory area -We also anticipate a third: +We also anticipate a few more: * virtio-msg-xen specific to Xen * Usable on any Xen system (including x86 where FF-A does not exist) * Using Xen events and page grants +* virtio-msg-loopback for userspace implemented devices + * Allows user space to provide devices to its own kernel + * This is similar to fuse, cuse or loopback block devices but for virtio + * Once developed this can provide a single kernel demo of virtio-msg + * [Work has begun on this] + +* virtio-msg over admin virtqueues + * This allows any virtio-pci device that supports admin virtqueues to also + support a virtio-msg bus that supports sub devices + * [We are looking for collaborators for this work] + +Changes since RFC1: +* reformated document to better conform to the virtio spec style + - created an introduction chapter + - created a basic concept chapter + - created bus operation and device initialization and operation chapters + - reworked description of transport and bus messages + - attempted a "compliance chapter" + - reused spec macros + - switched to MAY/MUST/SHALL/SHOULD wording + - eliminate the use of terms front-end and back-end and use driver and device +* made the maximum message size variable per bus instance +* use "device number" for virtio-msg device instances on the bus instead of + adding yet another meaning for "device ID" +* added the configuration generation count and its use +* described types of things that can be done with bus specific messages + such as setup of bus level shared memory and out of band notifications +* removed wording of notification being optional at transport level and + described bus level notifications of in-band, out-of-band, and polled + from driver side bus +* removed the ERROR message from transport level. Errors should be handled at + the bus level to better match virtio-pci and virtio-mmio +* removed bus level reset and status from standard bus messages +* replaced bus messages DEVICE_{ADDED,REMOVED} with EVENT_DEVICE +* changed names to GET_DEVICE_FEATURES and SET_DRIVER_FEATURES for clarity +* made SET_DEVICE_STATE return new state as it may not match +* Don't echo back the data in SET_VQUEUE (it cannot change) +* defined request/response vs event message id ranges +* match field size of next offset and wrap to virtio-{pci,mmio} +* added maximum number of virtqueues to DEVICE_INFO +* added admin virtqueue and SHM support + This series is a work in progress and we acknowledge at least the following issues we need to work on: -* Conform to virtio spec nouns (device/driver vs frontend/backend) - and verbs (must/may) -* Perhaps move error definition elsewhere it the spec and align on its symbols - and numeric values -* Allow message size to be greater than 40 bytes and allow bus implementations - to define their max message size -* Add a way to discover the protocol version -* Add a better description of the types of things a bus can do, specifically - including out-of-band notification and memory area sharing/discovery -* Maybe redo configuration generation handling +* Better conformance documentation +* Publish an update to Arm FF-A spec that shows virtio-msg binding (work underway) +* Publish virtio-msg-amp data structures and messages somewhere +* Align implementations to this version and send PATCH v1 (non-rfc) Background info and work in progress implementations: * HVAC project page with intro slides [1] diff --git a/transport-msg.tex b/transport-msg.tex index 0d24653b..01466b66 100644 --- a/transport-msg.tex +++ b/transport-msg.tex @@ -27,8 +27,8 @@ \subsubsection{Purpose} \item \textbf{Reduce per-bus complexity:} Buses can implement a fully message-based workflow (including optional - enumeration via \busref{GET_DEVICES} and hotplug via \busref{DEVICE_ADDED} - / \busref{DEVICE_REMOVED}), or they can discover and manage devices through + enumeration via \busref{GET_DEVICES} and hotplug via \busref{EVENT_DEVICE} + or they can discover and manage devices through alternative means such as platform firmware data. In either case, they forward transport messages to and from each device. @@ -49,8 +49,7 @@ \subsubsection{High-Level Architecture} \item Optional message-based queries (\busref{GET_DEVICES}), \item External data sources (e.g., device tree, ACPI tables, hypervisor firmware calls), - \item Dynamic hotplug notifications (optionally via \busref{DEVICE_ADDED} - and \busref{DEVICE_REMOVED}). + \item Dynamic hotplug notifications (optionally via \busref{EVENT_DEVICE}). \end{itemize} Once a device is identified—regardless of discovery method—the bus typically uses \msgref{GET_DEVICE_INFO} to read its device ID and vendor ID, then @@ -92,7 +91,7 @@ \subsubsection{System Topology} \subsubsection{Optional Bus Messages} This specification \emph{defines} messages such as \busref{GET_DEVICES}, -\busref{DEVICE_ADDED}, \busref{DEVICE_REMOVED}, and \busref{PING} for a +\busref{EVENT_DEVICE}, and \busref{PING} for a completely message-based approach to enumeration, hotplug, and bus-wide health. However, these are \emph{not} mandatory if a bus instance already handles those functions via firmware, device tree, or other platform features. The only strict @@ -189,8 +188,9 @@ \subsubsection{Configuration Generation Count} or the response to \msgref{GET_CONFIG} which also both provide the device's current configuration count. The device may change any amount of data for one generation count increment. If the change cannot fit in one \msgref{EVENT_CONFIG} -message, it \emph{SHOULD} be signaled to the driver via a \msgref{EVENT_CONFIG_EMPTY} -message. The device \emph{MUST NOT} provide the same generation count in +message, it \emph{SHOULD} be signaled to the driver via a \msgref{EVENT_CONFIG} +message with a zero data length and the new generation count. +The device \emph{MUST NOT} provide the same generation count in multiple \msgref{EVENT_CONFIG} messages that contain non-zero length config data. The driver includes its view of the current generation count in \msgref{SET_CONFIG} requests. The configuration generation count does not @@ -251,8 +251,8 @@ \subsubsection{Bus vs. Transport Messages} \begin{description} \item[\textbf{Bus Messages}:] Intended for global bus operations such as enumerating device numbers - (\busref{GET_DEVICES}), managing device hotplug events (\busref{DEVICE_ADDED}, - \busref{DEVICE_REMOVED}), or assessing bus-wide health (\busref{PING}). + (\busref{GET_DEVICES}), managing device hotplug events (\busref{EVENT_DEVICE}) + or assessing bus-wide health (\busref{PING}). These messages are \emph{optional} in environments where device discovery or state changes occur through other means (e.g., device tree). However, if a bus \emph{chooses} to handle those tasks via messages, @@ -357,14 +357,16 @@ \subsubsection{Device Hotplug} If the bus supports dynamic addition or removal of devices at runtime, it \emph{MAY} announce these events using messages: \begin{itemize} - \item \busref{DEVICE_ADDED} for a newly available device, - \item \busref{DEVICE_REMOVED} for a device that is no longer accessible. + \item \busref{EVENT_DEVICE} with state DEVICE_BUS_STATE_READY for a newly + available device, + \item \busref{EVENT_DEVICE} with state DEVICE_BUS_STATE_REMOVED for a device + that is no longer accessible. \end{itemize} Alternatively, the bus \emph{MAY} rely on external signals (e.g., an event from the platform or hypervisor) and inform the OS of hotplug changes out-of-band. -This specification \emph{defines} \busref{DEVICE_ADDED} and -\busref{DEVICE_REMOVED} for implementations that prefer a fully message-based +This specification \emph{defines} \busref{EVENT_DEVICE} +for implementations that prefer a fully message-based approach but does not require their use. In any scenario, once notified that a device has appeared, the bus \emph{SHOULD} query the device (e.g., via \msgref{GET_DEVICE_INFO}) and register it with the OS. If a device is @@ -468,12 +470,21 @@ \subsubsection{Device Information} configuration space. \item \textbf{Number of virtqueues}: The maximum number of virtqueues the device supports. + \item \textbf{Number of SHM segments}: The number of device owned + shared memory segments this device has. + \item \textbf{Admin virtqueue starting index}: The virtqueue index for the + first admin virtqueue supported by the device. + \item \textbf{Admin virtqueue count}: The number of admin virtqueues + supported by the device. \end{itemize} Armed with these details, the driver knows how many feature blocks to query, what portion of the configuration space is valid, and which OS-level virtio driver might bind to this device based on the Device ID. +It also knows of any device owned shared memory segments and any admin +virtqueues supported by the device. + \subsubsection{Feature Negotiation} \label{sec:Virtio Transport Options / Virtio Over Messages / Device Initialization / Device Features} @@ -634,9 +645,6 @@ \subsubsection{Device Notifications} status, and optionally the updated portion of the configuration data. \item If the data is not included, the driver \emph{SHOULD} issue \msgref{GET_CONFIG} to discover the updated configuration. - \item If a device does not support to send \msgref{EVENT_CONFIG}, the driver - \emph{MAY} rely on \msgref{GET_CONFIG}, \msgref{SET_CONFIG} and - \msgref{GET_DEVICE_STATUS} to handle configuration and status changes. \end{itemize} \paragraph{EVENT\_USED} @@ -645,7 +653,7 @@ \subsubsection{Device Notifications} processed or consumed by the device. \item The driver uses normal virtio methods (e.g., reading the “used” ring) to identify which buffers are complete. - \item If a device does not support to send \msgref{EVENT_USED}, the driver + \item If a device does not support sending \msgref{EVENT_USED}, the driver \emph{MAY} rely on standard virtqueue polling mechanisms to detect completion. \end{itemize} @@ -658,8 +666,8 @@ \subsubsection{Configuration Changes During Operation} The driver \emph{MAY} issue new \msgref{SET_CONFIG} messages after initialization if device configuration needs to be changed at runtime. The device \emph{MAY} also make configuration changes at -runtime and \emph{MUST} signal those changes with \msgref{EVENT_CONFIG} or -\msgref{EVENT_CONFIG_EMPTY} messages. +runtime and \emph{MUST} signal those changes with \msgref{EVENT_CONFIG} +messages. \subsubsection{Virtqueue Changes During Operation} \label{sec:Virtio Transport Options / Virtio Over Messages / Device Operation / VQueueUpdates} @@ -682,7 +690,7 @@ \subsubsection{Device Reset and Shutdown} In some circumstances, the device \emph{MAY} also trigger a reset if it encounters an unrecoverable error. This can be signaled to the driver by sending an \msgref{EVENT_CONFIG} with a \emph{device status} with the DEVICE_NEEDS_RESET -bit set. The driver \emph{SHOULD} reset the device by writeing a status of 0 +bit set. The driver \emph{SHOULD} reset the device by writing a status of 0 via the \msgref{SET_DEVICE_STATUS} message. The driver \emph{MAY} reinitialize the device if it wishes to use the device again. @@ -738,13 +746,13 @@ \subsubsection{Overview} \hline \msgref{RESET_VQUEUE} & 0xB & Driver \\ \hline -\msgref{EVENT_CONFIG_EMPTY} & 0x40 & Device \\ +\msgref{GET_SHM} & 0xC & Driver \\ \hline -\msgref{EVENT_CONFIG} & 0x41 & Device \\ +\msgref{EVENT_CONFIG} & 0x40 & Device \\ \hline -\msgref{EVENT_AVAIL} & 0x42 & Driver \\ +\msgref{EVENT_AVAIL} & 0x41 & Driver \\ \hline -\msgref{EVENT_USED} & 0x43 & Device \\ +\msgref{EVENT_USED} & 0x42 & Device \\ \hline \end{tabular} @@ -769,8 +777,7 @@ \subsubsection{Overview} \begin{itemize} \item \msgref{EVENT_AVAIL} \item \msgref{EVENT_USED} - \item \msgref{EVENT_CONFIG} or \msgref{EVENT_CONFIG_EMPTY} - \item \msgref{EVENT_USED} + \item \msgref{EVENT_CONFIG} \end{itemize} \newcommand{\msgdef}[1]{\subsubsection{VIRTIO_MSG_#1}\label{sec:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_#1}} @@ -790,7 +797,9 @@ \subsubsection{Overview} & 4 & 4 & Vendor ID \\ & 8 & 4 & Number of feature bits \\ & 12 & 4 & Configuration size in bytes \\ -& 12 & 4 & Maximum number of virtqueues \\ +& 16 & 4 & Maximum number of virtqueues \\ +& 20 & 2 & Admin virtqueue start index \\ +& 22 & 2 & Admin virtqueue count \\ \hline \end{tabular} @@ -856,9 +865,9 @@ \subsubsection{Overview} Request & 0 & 4 & Configuration offset in bytes \\ & 4 & 4 & Number of bytes \\ \hline -Answer & 0 & 4 & Configuration offset in bytes \\ -& 4 & 4 & Number of bytes \\ -& 8 & 4 & Configuration generation count \\ +Answer & 0 & 4 & Configuration generation count \\ +& 4 & 4 & Configuration offset in bytes \\ \\ +& 8 & 4 & Number of bytes \\ & 12 & ... & Configuration data \\ \hline \end{tabular} @@ -872,14 +881,14 @@ \subsubsection{Overview} \hline Type & Offset & Size (bytes) & Content \\ \hline \hline -Request & 0 & 4 & Configuration offset in bytes\\ -& 4 & 4 & Number of bytes \\ -& 8 & 4 & Configuration generation count \\ +Request & 0 & 4 & Configuration generation count \\ +& 4 & 4 & Configuration offset in bytes \\ +& 8 & 4 & Number of bytes \\ & 12 & ... & Configuration data \\ \hline -Answer & 0 & 4 & Configuration offset in bytes\\ -& 4 & 4 & Number of bytes, or 0 if rejected \\ -& 8 & 4 & New Configuration generation count \\ +Answer & 0 & 4 & New Configuration generation count \\ +& 4 & 4 & Configuration offset in bytes \\ +& 8 & 4 & Number of bytes, or 0 if rejected \\ & 12 & ... & Configuration data \\ \hline \end{tabular} @@ -983,30 +992,29 @@ \subsubsection{Overview} \hline \end{tabular} -\msgdef{EVENT_CONFIG_EMPTY} +\msgdef{GET_SHM} -This message is sent by the virtio-msg device and there is no response. +This message is sent by the virtio-msg transport driver and requires a +response from the device. \begin{tabular}{|l|l|l|l|} \hline Type & Offset & Size (bytes) & Content \\ \hline \hline -Request & 0 & 0 & no extra data \\ +Request & 0 & 4 & Shared memory region index \\ +\hline +Answer & 0 & 4 & Shared memory region index \\ +& 4 & 4 & Shared memory region length \\ +& 8 & 4 & Shared memory region address \\ \hline \end{tabular} -The \msgref{EVENT_CONFIG_EMPTY} message is used when the extra information in -\msgref{EVENT_CONFIG} is not available. In general \msgref{EVENT_CONFIG} -\emph{SHOULD} be preferred to \msgref{EVENT_CONFIG_EMPTY} but this is not always -possible. If the driver side bus implementation is manufacturing notification -based on polling intervals, it should use this message to notify the driver -side transport layer that it should use \msgref{GET_DEVICE_STATUS} and -\msgref{GET_CONFIG} to query for device side changes. +If the returned shared memory region length is zero, the memory region does not +exist. -This message may also be used when a single configuration change on the device -side cannot be communicated in one \msgref{EVENT_CONFIG} message due to maximum -message size and the device does not desire to represent it as multiple -configuration changes. +Note: these shared memory regions are device owned and supported on other virtio +transports since virtio v1.3. These shared memory regions are separate and +distinct from any shared memory regions defined by the bus implementation. \msgdef{EVENT_CONFIG} @@ -1019,11 +1027,19 @@ \subsubsection{Overview} Request & 0 & 4 & Device status \\ & 4 & 4 & Configuration generation count \\ & 8 & 4 & Configuration offset \\ -& 12 & 4 & Number of bytes \\ +& 12 & 4 & Number of bytes (\emph{MAY} be zero) \\ & 16 & ... & Configuration data \\ \hline \end{tabular} +The number of bytes field \emph{MAY} be zero. If so the configuration data field +will also be of zero length. This is the normal case when the configuration +generation count has not been changed. + +If the number of bytes field is zero and the configuration generation count is +changed, then the driver is responsible for discovering any changed +configuration data via \msgref{GET_CONFIG} messages. + \msgdef{EVENT_AVAIL} This message is sent by the virtio-msg driver and there is no response. @@ -1037,11 +1053,6 @@ \subsubsection{Overview} \hline \end{tabular} -If virtqueue index is 0xFFFF then it indicates that all virtqueus should be -polled for available buffers. This is useful when the bus implementation is -using out-of-band notifications, using polling, or when emulating or bridging -to another virtio transport. - The \textbf{Next wrap} field is the MSB of the 32 bit value. The \textbf{Next offset} field is the other 31 bits. These fields should be 0 if the VIRTIO_F_NOTIFICATION_DATA feature has not been negotiated. If the bus @@ -1060,11 +1071,6 @@ \subsubsection{Overview} \hline \end{tabular} -If virtqueue index is 0xFFFF then it indicates that all virtqueus should be -polled for used buffers. This is useful when the bus implementation is -using out-of-band notifications, using polling, or when emulating or bridging -to another virtio transport. - \subsection{Bus Messages}\label{sec:Virtio Transport Options / Virtio Over Messages / Bus Messages} The bus messages defined in this specification facilitate device discovery, @@ -1089,9 +1095,7 @@ \subsection{Bus Messages}\label{sec:Virtio Transport Options / Virtio Over Messa \hline \busref{PING} & 0x3 & Any \\ \hline -\busref{DEVICE_ADDED} & 0x41 & Device \\ -\hline -\busref{DEVICE_REMOVED} & 0x42 & Device \\ +\busref{EVENT_DEVICE} & 0x40 & Device \\ \hline \end{tabular} @@ -1137,7 +1141,7 @@ \subsection{Bus Messages}\label{sec:Virtio Transport Options / Virtio Over Messa device in the least significant bit of the first byte, the eighth device in the most significant bit of the first byte, etc. -\busdef{DEVICE_ADDED} +\busdef{EVENT_DEVICE} This message is sent by the virtio-msg device side bus and there is no response. @@ -1146,21 +1150,29 @@ \subsection{Bus Messages}\label{sec:Virtio Transport Options / Virtio Over Messa Type & Offset & Size (bytes) & Content \\ \hline \hline Request & 0 & 2 & Device Number \\ + & 2 & 2 & Device Bus State \\ \hline \end{tabular} -\busdef{DEVICE_REMOVED} - -This message is sent by the virtio-msg device side bus and there is no response. - -\begin{tabular}{|l|l|l|l|} +\begin{tabular}{|l|l|l|} \hline -Type & Offset & Size (bytes) & Content \\ -\hline \hline -Request & 0 & 2 & Device Number \\ +Device Bus State & Value & Meaning \\ +\hline +\hline +DEVICE_BUS_STATE_READY & 0x1 & Device is present and ready \\ +\hline +DEVICE_BUS_STATE_REMOVED & 0x2 & Device is no longer present \\ +\hline +reserved & 0x3 to 0x7FFF & Reserved for standard use \\ +\hline +any & 0x8000 to 0xFFFF & may be used by bus implementations \\ \hline \end{tabular} +This event is sent when the bus level state of the device has changed. +It can indicate when a device is added or removed. It may also be used by +bus implementations to indicate other states. + \busdef{PING} This message is sent by the virtio-msg driver side bus or device side bus and @@ -1236,7 +1248,7 @@ \subsubsection{Mandatory Requirements} the \textbf{maximum message size}, including correct offsets and block counts in \msgref{GET_DEVICE_FEATURES}, \msgref{SET_DRIVER_FEATURES}, \msgref{GET_CONFIG}, and \msgref{SET_CONFIG}. - \item A driver \emph{MUST} handle a /msgref{SET_CONFIG} being rejected for a + \item A driver \emph{MUST} handle a \msgref{SET_CONFIG} being rejected for a mismatched configuration generation count. \item A driver \textbf{MUST} initialize each device’s virtqueues (where needed) via \msgref{SET_VQUEUE} before attempting normal I/O and \textbf{SHOULD} @@ -1249,8 +1261,8 @@ \subsubsection{Optional Requirements} \paragraph{Bus Messages for Enumeration and Hotplug} \begin{itemize} - \item Implementations \emph{MAY} use \busref{GET_DEVICES}, \busref{DEVICE_ADDED}, - and \busref{DEVICE_REMOVED} for discovering and managing devices in a + \item Implementations \emph{MAY} use \busref{GET_DEVICES} + and \busref{EVENT_DEVICE} for discovering and managing devices in a message-driven manner. However, this is not mandatory if other enumeration methods (e.g., device tree, ACPI, hypervisor firmware) are used. \item If a bus \emph{chooses} to implement these messages, it \textbf{MUST} do @@ -1268,7 +1280,6 @@ \subsubsection{Optional Requirements} \paragraph{Runtime Notifications} \begin{itemize} \item A device or the driver side bus \emph{MUST} send \msgref{EVENT_CONFIG} - or \msgref{EVENT_CONFIG_EMPTY} to inform the driver of configuration of device status changes. \item A device or the driver side bus \emph{MUST} \msgref{EVENT_USED} to inform the driver of (likely) buffer completions.