• Frank Barchard's avatar
    ubsan overflow fix for multiply by 0x01010101 · 9e39c1f2
    Frank Barchard authored
    This is an UBSan error reported by libjingle
    
    [ RUN      ] WebRtcVideoFrameTest.ConvertToYUY2BufferStride
    [000:000] (videoframe.cc:375): Validate frame passed. format: I420 bpp: 12 size: 1280x720 bytes: 1382400 expected: 1382400 sample[0..3]: 73, 73, 73, 73
    ../../chromium/src/third_party/libyuv/source/row_gcc.cc:2903:25: runtime error: signed integer overflow: 128 * 16843009 cannot be represented in type 'int'
    [8/614] WebRtcVideoFrameTest.ConvertToYUY2BufferStride returned/aborted with exit code 1 (32 ms)
    [9/614] WebRtcVideoFrameTest.ConvertToYUY2BufferInverted (29 ms)
    Note: Google Test filter = WebRtcVideoFrameTest.ConvertToYUY2BufferInverted
    
    The source is uint8 and the multiply is by 0x01010101 to replicate the byte to 4 bytes.
    Changing the constant to 0x01010101u should avoid overflow.
    
    R=harryjin@google.com
    TBR=harryjin@google.com
    BUG=libyuv:563
    
    Review URL: https://codereview.chromium.org/1657533005 .
    9e39c1f2
Name
Last commit
Last update
chromium Loading commit data...
include Loading commit data...
source Loading commit data...
third_party/gflags Loading commit data...
tools Loading commit data...
unit_test Loading commit data...
util Loading commit data...
.gitignore Loading commit data...
.gn Loading commit data...
AUTHORS Loading commit data...
Android.mk Loading commit data...
BUILD.gn Loading commit data...
CMakeLists.txt Loading commit data...
DEPS Loading commit data...
LICENSE Loading commit data...
LICENSE_THIRD_PARTY Loading commit data...
OWNERS Loading commit data...
PATENTS Loading commit data...
PRESUBMIT.py Loading commit data...
README.chromium Loading commit data...
all.gyp Loading commit data...
codereview.settings Loading commit data...
download_vs_toolchain.py Loading commit data...
drover.properties Loading commit data...
gyp_libyuv Loading commit data...
gyp_libyuv.py Loading commit data...
libyuv.gyp Loading commit data...
libyuv.gypi Loading commit data...
libyuv_nacl.gyp Loading commit data...
libyuv_test.gyp Loading commit data...
linux.mk Loading commit data...
public.mk Loading commit data...
setup_links.py Loading commit data...
sync_chromium.py Loading commit data...
winarm.mk Loading commit data...