This reverts commit 04aa09c4bc
and reintroduces 0ff5567a30 that
was temporarily reverted due to minor regressions.
It also reverts e5bce8b4ce that fixed FATE refs.
The fate-ffm change is caused by field_order now being set
on the output format because the first frame arrives earlier.
The fate-mxf change is assumed to be the same.