Fixes assertion failure
Fixes out of memory access

Fixes: test_casex.ivf
Found-by: Tyson Smith <twsmith@mozilla.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 7c97946d)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>