• Ronald S. Bultje's avatar
    swscale: fix overflows in output of RGB48 pixels. · be1bafc3
    Ronald S. Bultje authored
    For certain types of filters where the intermediate sum of coefficients
    can go above the fixed-point equivalent of 1.0 in the middle of a filter,
    the sum of a 31-bit calculation can overflow in both directions and can
    thus not be represented in a 32-bit signed or unsigned integer. To work
    around this, we subtract 0x40000000 from a signed integer base, so that
    we're halfway signed/unsigned, which makes it fit even if it overflows.
    After the filter finishes, we add the scaled bias back after a shift.
    
    We use the same trick for 16-bit bpc YUV output routines.
    Signed-off-by: 's avatarMans Rullgard <mans@mansr.com>
    be1bafc3
swscale.c 106 KB