Stream conformance tool

[defagos]

As a stream encoding/packaging engineer it would be helpful to have a tool that can perform a series of conformance tests so that many obvious errors or incompatibilities can be immediately identified, without involving the player team directly.

Another advantage is that such a tool can be improved as new issues arise, extending its coverage.

Hints

• We could write a tool (script) relying on tools we already use (mediastreamvalidator, mediainfo, ffmpeg, ffprobe) or parsing to quickly generate a report, e.g. with:
  • Issues are reported by the tools (e.g. HLS playlist issues).
  • Type of the stream based on playlist length.
  • Check of segment size vs GOP size.
  • Check of obvious implementation errors (e.g. zero-length WebVTT cues).
  • Check of ts length vs declared length in a media playlist.
  • Segment beginning with I-frame.
  • PTS/discontinuities.
  • Alignment between variants.
  • Codec consistency between variants.
  • ...
• The tool could be available online for quick copy/paste when a stream URL needs to be validated.
• This tool could then be used by us to quickly generate a report when a stream must be investigated. It could also be provided to other teams to do the same.
• We could also think about implementing a tool around encoder diagnostic APIs.
• Some inspiration:
  • https://players.akamai.com/stream-validator
  • https://ott.dolby.com/OnDel_tools/validator/Start_Here.html
  • https://hlsanalyzer.com/
• Some references:
  • Behavioral matrix for DEFAULT and AUTOSELECT #45
• The tool could also be helpful to build a regression testing solution so that changes to the encoding or packaging can be validated automatically.
• We could also discuss if/how we could check result end-to-end on a wide range of physical devices, e.g. using BrowserStack. Maybe the Play Next Q&A team would be interested to collaborate as well.

Acceptance criteria

• ...

Tasks

• ...