Currently, we are only properly assessing semantic segmentation performance.
We can move on to assessing object detection performance. In the intermediate steps, we can also assess over-segmentation and under-segmentation using metrics such as ROM-RUM.
This can help assess the performance of current object detection algorithm, which is: Semantic Segmentation + Contour.
We can move on to methods such as Semantic Segmentation + Clustering (e.g. DBSCAN).
While we may not move to panoptic segmentation at the moment, this will be on the radar.