Fixes: integer overflows
Reported-by: "Xiaohan Wang (王消寒)" <xhwang@chromium.org>

Based on limits in "8.5 Transform coefficient decoding process and picture
construction process prior to deblocking  filter process"
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit f26a63c4)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>