avcodec/decode: reset codec on receiving packet after EOF in compat_decode

In commit 061a0c14 ("decode: restructure the core decoding code"), the
deprecated avcodec_decode_* APIs were reworked so that they called into the
new avcodec_send_packet / avcodec_receive_frame API. This had the side effect
of prohibiting sending new packets containing data after a drain
packet, but in previous versions of FFmpeg this "worked" and some
applications relied on it.

To restore some compatibility, reset the codec if we receive a new non-drain
packet using the old API after draining has completed. While this does
not give the same behaviour as the old API did, in the majority of cases
it works and it does not require changes to any other part of the decoding
code.

Fixes ticket #6775
Signed-off-by: James Cowgill <jcowgill@debian.org>
Signed-off-by: Marton Balint <cus@passwd.hu>
(cherry picked from commit 02ba4b91)

Changelog

@@ -2,6 +2,7 @@ Entries are sorted chronologically from oldest to youngest within each release,
 releases are sorted from youngest to oldest.
 
 version 3.4.1:
+- avcodec/decode: reset codec on receiving packet after EOF in compat_decode
 - avcodec/diracdsp: Fix integer overflow in PUT_SIGNED_RECT_CLAMPED()
 - avcodec/dirac_dwt: Fix integer overflows in COMPOSE_DAUB97*
 - avcodec/extract_extradata_bsf: Fix leak discovered via fuzzing

libavcodec/decode.c

@@ -753,6 +753,11 @@ static int compat_decode(AVCodecContext *avctx, AVFrame *frame,
 
     av_assert0(avci->compat_decode_consumed == 0);
 
+    if (avci->draining_done && pkt && pkt->size != 0) {
+        av_log(avctx, AV_LOG_WARNING, "Got unexpected packet after EOF\n");
+        avcodec_flush_buffers(avctx);
+    }
+
     *got_frame = 0;
     avci->compat_decode = 1;