Stream API

[chmousset]

I'm a bit confused to how the StreamIO interface is supposed to be used.

The doc says

The protocol
implemented in the handshake is identical to the Ready/Valid
protocol from the AXI spec for channels.

So I assume:

• data.is_some() is the equivalent of AXI's valid signal
• ready is AXI's counterpart
• a 'transfer' occurs when data.is_some() && ready
• data.is_some() and ready can be independently set
• data.is_some() and ready can be driven however you like (including in async), so long as they don't depend on each other

Is that correct?

The doc also mentions

The stream cores
all have registered inputs and outputs, so as to avoid
combinatorial pathways between the input and output.

AFAIU, the current IPs (widgets?) respect that convention but the choice of a sync or async stream is up to the designer.

Is there a plan to enforce StreamIO interfaces to be always sync, is is that just a convention?

Also in the doc:

A generic Stream IO type that holds a data and ready
signal. Note that in a Stream interface, these
signals are generally in opposite directions.

In AXI stream they are always in opposite direction.
Is that a typo or is there intent to support other type of streams?

[samitbasu]

Ooo... Good questions.

It may take a bit of time to write thoughtful answers to all of these. But roughly speaking:

1. I had assumed that the interfaces were synchronous. As far as I understand it, the AXI bus protocol handshakes are synchronous, and referenced to the bus clock.
2. I know that people use AXI streams to cross clock domains, but I always assumed that they put clock crossing FIFOs upstream of the bus interface, and ran the bus side at the AXI clock frequency.
3. I should update the documentation to make it clear how the AXI signals map to RHDL streams.
4. The register/not-registered thing is more complicated than I originally thought. In general, I prefer to have designs that can be composed without introducing either long path timing issues or potential logic loops. But I'm not sure if that should be "imposed" on people using the cores. Of course, they can always write their own, but I'm not sure where it will land. The cores that I have been working on are registered on one side. Which means that if you stack them, they have only a single register between stages. But that might not be the right thing to do.
5. As for enforcement, I can currently enforce that there are no combinatorial paths between the input and output of any Stream core I write, and I think that is important (and required by the AXI spec as I understand it anyway). But it's an "opt-in" enforcement, in the sense that you have to run the analysis pass from drc on your widget.

I hope that provides some light. I know the docs need work. It's top of my priority list once I get past the current development hurdle.

[chmousset]

OK, thanks for the info.

I think I might have brought a little confusion.
The AXI spec isn't very long, and really only the section 2.2 is the most important to understand to implement reliable streams.

1. To rephrase what I meant: the stream interface's transfers is indeed synchronous to the clock, although the control signals don't technically have to be registered. As per the doc: "A transfer takes place only when both TVALID and TREADY are asserted [on a rising edge of ACLK]"
2. AFAIK, it's generally the case. It's usually implemented with cascaded DFF or dualport RAM (for data) + grey counter to pass FIFO level across clock domains.
3. yes! ❤️
4. From experience working with migen/LiteX, making stream interface comb isn't a problem when primitives like buffers, FIFO and CDC can be easily instantiated. If ever you don't meet timing, it's easy to add a buffer where needed. If you have bursts-induced congestion, add a FIFO. The first advantage is obviously a lower usage of LC but the second, maybe even more important one, is that it makes streams "cheap" to use and de-facto interface for everything. It drastically eases testing and integrating cores.
5. The most important thing you need to be AXI-stream valid is to enforce there is no comb path between the valid and ready of the same stream. It's perfectly valid to make a core with a sink and source stream with only comb logic between them (something you try to do with .map() right ?)

If you are open to suggestions I can make a proposal for architecture with some "ground rules" and a couple of practical examples. Would that be OK?

[samitbasu]

That would amazing. Thank you!

[chmousset]

Sure thing!
First, a bit of context

At the beginning of (professionally) working with FPGA, I used raw Verilog and our system was getting difficult to debug, let alone improve. The toplevel was a complex, fragile 2000 LOC Verilog monster that was very hard to test, with hard-to-track edge cases. Building would take 15 minutes, only to spend another 15 minutes testing the hours simulating, digging through each signal (there were >1000) to find out I made a typo in one module's inputs.
Quality was bad, and so was quality of life.

So, 6 years ago we switched to LiteX. Since then, I have probably built close to 100 modules using LiteX's Steam interface.
The first class of problems that nearly vanished entirely thanks to LiteX was missing signals/typos in modules' interfaces, thanks to LiteX's Record (rhdl's equivalent of using struct). Instead of connecting signals individually, a single statement connects all of them at once. This reduced the toplevel size by over 5 folds, and errors by over 10 folds.

The second class of problems that was eliminated was bugs in the "glue logic" that caused stalls, repeated patterns or missing data.
Streams were largely responsible for that, especially since LiteX comes with a library of Stream-compatible building blocks like FIFO, arbiter, MUX, gearboxes or CDC.

The third thing that happened was a huge improvement in quality. Because we now use Streams for nearly everything,

• we don't spend time figuring out how interfaces should behave: we focus on functional, not piping
• documentation is easier for you to write and faster for others to read
• we have built good tooling to simulate and test each module, with ~70% of our modules having unit tests that run in seconds
• we also have good tooling to test modules individually on real hardware (LiteX's wishbone and scope ecosystem is great for that)
• it's trivial to add more tests if we discover an edge case
• we don't have to change tests if we add some internal buffer or change an FSM, as the test are more functional
• it's trivial to add a buffer when we don't meet timing, or a FIFO when we can't catch whole bursts of data
• it's trivial to move pipelines from one clock domain to another
• it's trivial to re-use a module instead of adapting them per-project

So, yes, I'm a big fan of Streams.

Feedback from existing API

I think the existing API is a good start, but has a few quirks to polish

• Option vs valid: I think Option<> is clearer and simpler to understand when coming from rust, but if the goal is cross-compatibility with other streams, valid might be simpler to understand. I don't have strong preference, it sort of depends where you want to take rhdl to.
• registering all sources and/or sink as a mean to guarantee the stream will 'work' and timings are met is counterproductive.
  It increases the LC usage and increase latency even when it's not required. As a result, the designer might be tempted to make bigger, more complex and specific modules instead of smaller, reusable ones to limit LC usage. This hurts testing and re-usability.
  If instead you make it easy to insert a buffer of FIFO if required, where required, stream become 0-cost abstraction and enable more modularization.
• streams are easy if you understand a few simple rules (see below). In my experience, explaining them and give means to verify it's done right is better than trying to hide complexity. Especially if stream are capable and 'cheap enough' to replace many other module-specific interfaces.

Proposed API

The API I propose is largely copied from LiteX's streams. It's itself inspired from AXI's streams, but simplified.
To be frank, I am not proficient enough in Rust to pretend coming out with a great, practical rust API. I can however give the big picture.

Naming convention

• payload: the data being transferred.
• param: optional data, like parameters.
• endpoint: the collection of valid, ready, payload etc. signals
• sink: an endpoint that consumes data
• source: an endpoint that produces data
• transaction: the transfer of a single payload value (ie. last 1 clock cycle)
• packet: the collection of one or more transactions

Mandatory signals

Streams don't have to be complicated to be useful: you just need 2 signals to validate a transaction.

• valid (generated by the source): when set it to 1, indicates that the payload, param first and last are valid
• ready (generated by the sink): when set to 1, indicates that the payload can be consumed
• a clock signal (can be implicit)

A valid transaction happens on a rising clock edge if (valid & ready).

Optional signals

• payload (generated by the source): struct that holds the data to be transferred. Value to be ignored if ~valid. Can change between each transaction.
• param (generated by the source): struct. Valid if valid. Can change between each packet.
• first (generated by the source): set to 1 to indicate the first transaction of a packet. Can change between each transaction.
• last (generated by the source): set to 1 to indicate the last transaction of a packet. Can change between each transaction.

Notes:

• first & last is valid: it indicates a 1 transaction packet.
• a stream with an empty payload do have use, like for arbitration

Rules

We've already seen

• if valid is 1, the source's payload, param, first and last are valid
• if ready is 1, the sink is ready to receive a transaction
• ready and valid can change state at any time, independently
• packets are delimited by first and last

There is a single additional rule you need to apply to have a working endpoint:

• no combinatory path between the ready and valid of the same endpoint. NEVER!

Beyond the streams

Stream-compatible library

The most useful stream-compatible components are

• sync FIFO (both endpoint in the same clock domain)
• async FIFO (endpoints are in different clock domains)
• Clock Domain Crossing buffer (a 1-deep variation of the async FIFO, to avoid using block RAM)
• sync buffer (a 1-deep variation of the sync FIFO, to avoid using block RAM)

LiteX does have a collection of other generic stream-compatible modules like gearboxes (~(de)serializers), arbiter, ... they can be very useful, but can be built at a later stage.

LiteX cross-compatibility

LiteX has its caveats, IMO mainly due to lack of strong-typing, but nevertheless comes with a ton of stream-compatible modules from the simplest UART to an Ethernet UDP streamer.
It also enables to easily create a complete SoC, so I think there is an opportunity of using LiteX modules from rhdl, or vice-versa.

Simulation

From simulation, 3 generic methods on endpoint can be useful for unit testing:

• push: drive valid and payload, watch ready to push a transaction
• pop: drive ready and watch valid to read a transaction
• `watch: same as pop, but don't drive ready
  With that, you can make functional test without thinking of jiggling control signals.
  And if there is a mean of detecting combinatorial path between ready and valid, that's all you really need.

Next steps

As I wrote, I don't think I'm proficient enough in rust to build the base API.
However, I can work on writing documentation and test if you wish.