Not sure yet why and when this happens (I can't create a source that always leads to this behavior).
I suspect the problem is the way we block the pads using the flag GST_PAD_PROBE_TYPE_BLOCK_DOWNSTREAM. I've tried to use the flag GST_PAD_PROBE_TYPE_BLOCKING (which seems to be more suitable), but the whole presentation stalls.