Commit graph

41429 commits

Author SHA1 Message Date
Michael Niedermayer
27db9c8288 avcodec/dpcm: Fix integer overflow in AV_CODEC_ID_GREMLIN_DPCM
Fixes: signed integer overflow: -2147479324 + -32568 cannot be represented in type 'int'
Fixes: 20103/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_GREMLIN_DPCM_fuzzer-5667667579240448

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit b1aecad9ea)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2020-07-03 12:10:23 +02:00
Michael Niedermayer
52b9e7f530 avcodec/wmalosslessdec: Fix integer overflow with sliding in padding bits
Fixes: signed integer overflow: -53716100 * 256 cannot be represented in type 'int'
Fixes: 20143/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_WMALOSSLESS_fuzzer-5716604000403456

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit b8a0be9352)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2020-07-03 12:10:23 +02:00
Michael Niedermayer
3653108a5c avcodec/wmalosslessdec: Fix loop in revert_acfilter()
Fixes: out of array read
Fixes: 20059/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_WMALOSSLESS_fuzzer-5691776237305856

No testcase except the fuzzed one.

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 5584c0bb94)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2020-07-03 12:10:23 +02:00
Michael Niedermayer
c4dccb509d avcodec/lagarith: Sanity check scale
A value of 24 and above can collaps the range to 0 which would not work.

Fixes: Timeout (75sec -> 21sec)
Fixes: 18707/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_LAGARITH_fuzzer-5708950892969984

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit fb3855342b)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2020-07-03 12:10:23 +02:00
Michael Niedermayer
d75d754b84 avcodec/apedec: Fix integer overflows in predictor_decode_mono_3950()
Fixes: signed integer overflow: -2147407150 + -1871606 cannot be represented in type 'int'
Fixes: 18702/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_APE_fuzzer-5679095417667584

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit eb64a5c6f9)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2020-07-03 12:10:23 +02:00
Michael Niedermayer
9f30a1694c avcodec/ralf: Fix integer overflow in apply_lpc()
Fixes: signed integer overflow: 2147482897 + 2048 cannot be represented in type 'int'
Fixes: 19240/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_RALF_fuzzer-5743240326414336
Fixes: 19869/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_RALF_fuzzer-5150136636538880

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit fd313d8cf8)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2020-07-03 12:10:23 +02:00
Michael Niedermayer
af1bfe5bc3 avcodec/dca_lbr: Fix some error codes and error passing
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit bfea054a75)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2020-07-03 12:10:23 +02:00
Michael Niedermayer
9f30473846 avcodec/wmavoice: Fix rounding and integer anomalies in calc_input_response()
Fixes: out of array access
Fixes: inf is outside the range of representable values of type 'int'
Fixes: signed integer overflow: -9223372036854775808 - 1 cannot be represented in type 'long'
Fixes: 19316/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_WMAVOICE_fuzzer-5677369365102592

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 38d3758444)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2020-07-03 12:10:23 +02:00
Michael Niedermayer
20c0ea160e avcodec/wmavoice: sanity check block_align
This limit is roughly based on the bitreader limit, its likely a much tighter limit
could be used

Fixes: left shift of 1965039647 by 1 places cannot be represented in type 'int'
Fixes: 19545/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_WMAVOICE_fuzzer-5695391899320320

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 6847e22c8c)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2020-07-03 12:10:23 +02:00
Michael Niedermayer
958db2c438 avcodec/pcm: Fix invalid shift in pcm_decode_frame for LXF
Fixes: left shift of 32 by 28 places cannot be represented in type 'int'
Fixes: 19472/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_PCM_LXF_fuzzer-5704364320096256

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 985d3666f6)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2020-07-03 12:10:23 +02:00
Michael Niedermayer
bc77d29d9e avcodec/snappy: Sanity check bytestream2_get_levarint()
Fixes: left shift of 79 by 28 places cannot be represented in type 'int'
Fixes: 20202/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_HAP_fuzzer-5719004081815552
Fixes: 20219/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_HAP_fuzzer-5641738677125120
Fixes: 20389/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_HAP_fuzzer-5680721517871104

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit be54da2117)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2020-07-03 12:10:23 +02:00
Michael Niedermayer
5e83098085 avcodec/mlpdsp: Fix a invalid shift in ff_mlp_rematrix_channel()
Fixes: left shift of negative value -2
Fixes: 20305/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_TRUEHD_fuzzer-5677196618498048

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Reviewed-by: Jai Luthra <me@jailuthra.in>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit fcc9f13717)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2020-07-03 12:10:23 +02:00
Michael Niedermayer
51c084bf72 avcodec/avdct: Clear IDCTDSPContext context
Fixes use of uninitialized variable and segfault

Reviewed-by: Paul B Mahol <onemda@gmail.com>
Reviewed-by: James Almer <jamrial@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit b82825eba8)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2020-07-03 12:10:23 +02:00
Michael Niedermayer
9ed9c8cc51 avcodec/x86/diracdsp: Fix high bits on Windows x86_64
Found-by: james
(cherry picked from commit 24af459d1e)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2020-07-03 12:10:23 +02:00
Michael Niedermayer
862c0048ec avcodec/wmalosslessdec: Fix multiple integer overflows
Fixes: left shift of 3329 by 20 places cannot be represented in type 'int'
Fixes: signed integer overflow: -199378355 + -1948950833 cannot be represented in type 'int'
Fixes: 19837/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_WMALOSSLESS_fuzzer-5752565837070336
Fixes: 19839/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_WMALOSSLESS_fuzzer-5767483265122304

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 422202516c)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2020-07-03 12:10:23 +02:00
Michael Niedermayer
109cab1487 avcodec/apedec: Fix undefined integer overflow in decode_array_0000()
Fixes: signed integer overflow: -2143289344 - 6246400 cannot be represented in type 'int'
Fixes: 19239/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_APE_fuzzer-5173755680915456

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit a3655bb02c)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2020-07-03 12:10:23 +02:00
Michael Niedermayer
b7c2f4d298 avcodec/smacker: Check space before decoding type
Fixes: Timeout (232sec -> 280ms)
Fixes: 19682/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_SMACKER_fuzzer-5654129649385472

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 6f5c18da59)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2020-07-03 12:10:23 +02:00
Michael Niedermayer
3f42af0342 avcodec/rawdec: Use linesize in b64a
Fixes: out of array access
Fixes: 19750/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_RAWVIDEO_fuzzer-5074834119983104

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 2b5b9d5dac)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2020-07-03 12:10:23 +02:00
Michael Niedermayer
57278dac15 avcodec/iff: Over-allocate ham_palbuf for HAM6 IFF-PBM
IFF-PBM-HAM6 can read out of array without this overallocation
Fixes: Out of array read
Fixes: 19752/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_IFF_ILBM_fuzzer-5675331403120640

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 8652f4e7a1)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2020-07-03 12:10:23 +02:00
Michael Niedermayer
e1bea6d412 avcodec/x86/diracdsp: Fix incorrect src addressing in dequant_subband_32()
Fixes: Segfault (not reproducable with asm, which made this hard to debug)
Fixes: decoding errors
Fixes: 19854/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_DIRAC_fuzzer-5729372837511168

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Reviewed-by: Paul B Mahol <onemda@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 0694b60b7b)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2020-07-03 12:10:23 +02:00
Michael Niedermayer
88326e29dc avcodec/hevc_mp4toannexb_bsf: check that nalu size doesnt overflow
Fixes: Out of array access
Fixes: 19299/clusterfuzz-testcase-minimized-ffmpeg_BSF_HEVC_MP4TOANNEXB_fuzzer-5169193398042624

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit a8ceb2a72f)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2020-07-03 12:10:23 +02:00
Michael Niedermayer
a0d0e9b245 avcodec/hevc_mp4toannexb_bsf: Avoid NULL memcpy()
Fixes: invalid memcpy use
Fixes: 19299/clusterfuzz-testcase-minimized-ffmpeg_BSF_HEVC_MP4TOANNEXB_fuzzer-5169193398042624

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 1e23b5a706)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2020-07-03 12:10:23 +02:00
Michael Niedermayer
831ac9ad46 avcodec/cbs_av1: Check leb128 values read
"It is a requirement of bitstream conformance that the value returned from the leb128 parsing process is less than or equal
to (1 << 32) - 1."

Fixes: assertion failure
Fixes: 19293/clusterfuzz-testcase-minimized-ffmpeg_BSF_TRACE_HEADERS_fuzzer-5749508361420800

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit a70d836364)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2020-07-03 12:10:23 +02:00
Michael Niedermayer
623ecb2ec8 avcodec/wmalosslessdec: move channel check up
Fixes: out of array access
Fixes: 2nd part of 18429/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_WMALOSSLESS_fuzzer-6210814364614656

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Reviewed-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 891bcc4acc)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2020-07-03 12:10:23 +02:00
Michael Niedermayer
94a1280fb0 avcodec/cbs_h2645: Skip all 0 NAL units
Fixes: assertion failure
Fixes: 19286/clusterfuzz-testcase-minimized-ffmpeg_BSF_H264_REDUNDANT_PPS_fuzzer-5707990724509696

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 285138ef14)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2020-07-03 12:10:23 +02:00
Michael Niedermayer
7be077fcf8 avcodec/adpcm: Fix overflow in FFABS() IMA_EA_EACS
Fixes: negation of -2147483648 cannot be represented in type 'int'; cast to an unsigned type to negate this value to itself
Fixes: 19235/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_ADPCM_IMA_EA_EACS_fuzzer-5680878952382464

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 794352ae9d)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2020-07-03 12:10:23 +02:00
Michael Niedermayer
b36592ce07 avcodec/alac: Fix integer overflow in LPC coefficient adaption
Fixes: signed integer overflow: 267693597 * 10 cannot be represented in type 'int'
Fixes: 19237/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_ALAC_fuzzer-5755407700328448

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 6a865cec5e)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2020-07-03 12:10:23 +02:00
Michael Niedermayer
84048cf37f avcodec/g729postfilter: Optimize out overflowing multiplication from apply_tilt_comp()
Fixes: signed integer overflow: -1114392282 * 2 cannot be represented in type 'int'
Fixes: 19236/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_G729_fuzzer-5741678938030080

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit c0bd5fa43d)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2020-07-03 12:10:23 +02:00
Michael Niedermayer
fabd42043a avcodec/vc1dec: Check field_mode for sprites
Fixes: Out of array read
Fixes: 19263/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_VC1IMAGE_fuzzer-5389219325542400

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 32fb919836)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2020-07-03 12:10:23 +02:00
Michael Niedermayer
f2a0bde8f3 avcodec/vc1dec: Limit bits by the actual bitstream size
Fixes: Timeout (350 ->19sec)
Fixes: 19249/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_VC1IMAGE_fuzzer-6566896438870016

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit c56a52a82c)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2020-07-03 12:10:23 +02:00
Michael Niedermayer
f061fe9aa1 avcodec/vmdaudio: Check block_align more
Fixes: signed integer overflow: 2147483647 + 1 cannot be represented in type 'int'
Fixes: 19788/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_VMDAUDIO_fuzzer-5743379690553344

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Reviewed-by: Paul B Mahol <onemda@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 06f6857b54)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2020-07-03 12:10:23 +02:00
Andreas Rheinhardt
96f18b6427 avcodec/cinepakenc: Fix invalid shifts
Fixes: left shift of 1 by 31 places cannot be represented in type 'int'.
Affected the FATE-tests vsynth1-cinepak, vsynth2-cinepak and
vsynth_lena-cinepak. Also fixes ticket #8220.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
Reviewed-by: Paul B Mahol <onemda@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit e3650dcfc9)
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
2020-07-02 00:27:48 +02:00
Andreas Rheinhardt
946d2893a9 avcodec/cbs_h2645: Fix potential out-of-bounds array access
The maximum allowed index for an array access is FF_ARRAY_ELEMS - 1; yet
the current code allowed FF_ARRAY_ELEMS. This wasn't dangerous in practice,
as parameter sets with invalid ids were already filtered out during
reading.

Found via PVS-Studio (see ticket #8156).

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
(cherry picked from commit f3333c3c67)
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
2020-07-02 00:25:46 +02:00
Andreas Rheinhardt
5db4c91ef5 h264_redundant_pps: Fix memleak in case of errors
Now the fragment is uninitialized and the input packet freed in case of
errors.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@googlemail.com>
Signed-off-by: Mark Thompson <sw@jkqxz.net>
(cherry picked from commit 40b74abfca)
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
2020-07-02 00:20:34 +02:00
Andreas Rheinhardt
2b35185006 avcodec/cavsdsp: Fix undefined left shifts of negative numbers
Affected the ffmpeg-filter_colorkey FATE-test (but only if the C version
of idct8_add is used and not e.g. the x86 SSE2 version).

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>

Fixes: left shift of negative value -107
Fixes: 20398/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_CAVS_fuzzer-5725389278412800

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 0f0f2ab0c3)
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
2020-07-01 23:18:20 +02:00
Andreas Rheinhardt
d9e4783f06 avcodec/ra144enc: Fix invalid left shift of negative number
by replacing it with a multiplication. Said multiplication can't
overflow an int32_t because lpc_coefs is limited to 16 bit precision.

Fixes the FACE-test acodec-ra144 as well as part of #8217.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit e3fb9af6f1)
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
2020-07-01 23:04:12 +02:00
Andreas Rheinhardt
68e67bb187 avcodec/adxenc: Avoid undefined left shift of negative numbers
Replace "((a << shift) + b) >> shift" by "a + (b >> shift)". This avoids
a left shift which also happens to trigger undefined behaviour in case "a"
is negative. This affected the FATE-tests acodec-adpcm-adx and
acodec-adpcm-adx-trellis; it also fixes ticket #8008.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 59a9d65e0d)
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
2020-07-01 23:03:43 +02:00
Andreas Rheinhardt
d620f2eeb4 avcodec/adpcm: Fix undefined left shifts of negative numbers
Affected the adpcm-afc, adpcm-ea-1, adpcm-ea-2, adpcm-ea-maxis-xa,
adpcm-thp and ea-cdata FATE-tests. Also fixes ticket #8487.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 3ad8af51b7)
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
2020-07-01 23:03:23 +02:00
Andreas Rheinhardt
02368962f5 avcodec/proresenc_anatoliy: Fix invalid left shift of negative number
This fixes ticket #7997 as well as the vsynth*-prores_# FATE-tests
(where * ranges over { 1, 2, 3, _lena } and # over { , _int, _444,
_444_int }).

(Given that prev_dc is in the range -0xC000..0x3FFF, no overflow can
happen upon multiplication with 2.)

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 198081efb7)
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
2020-07-01 23:01:29 +02:00
Andreas Rheinhardt
02d9561eab avcodec/jpeg2000dwt: Fix undefined shifts of negative numbers
Affected the vsynth*-jpeg2000 and the vsynth*-jpeg2000-97 FATE tests
(where * ranges over { 1, 2, 3, _lena }) as well as ticket #7983.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 5cf593adcd)
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
2020-07-01 22:30:42 +02:00
Andreas Rheinhardt
ceed3603e1 avcodec/ituh263dec: Fix undefined left shift of negative number
Fixes ticket #8160.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 324487b596)
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
2020-07-01 22:30:16 +02:00
Andreas Rheinhardt
ab3b008dad avcodec/dnxhdenc: Fix undefined left shifts of negative numbers
Affected 61 FATE-tests: 60 vsynth tests and lavf-mxf_opatom.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit b7f156e8cb)
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
2020-07-01 22:29:36 +02:00
Andreas Rheinhardt
438773db51 avcodec/exr: Fix undefined left shifts of negative numbers
Affected the FATE-tests exr-rgb-scanline-pxr24-half-uint32-13x9 and
exr-rgb-scanline-pxr24-uint32.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 8b0f949906)
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
2020-07-01 22:28:15 +02:00
Andreas Rheinhardt
2b9ddde1d0 avcodec/pcm: Fix undefined shifts
Fixes the acodec-pcm-u16[lb]e FATE-tests.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
(cherry picked from commit 69473bec6f)
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
2020-07-01 22:26:42 +02:00
Andreas Rheinhardt
3497fd2dd7 avcodec/wavpackenc: Fix undefined shifts
Fixes ticket #8161 and the acodec-wavpack FATE-test.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
(cherry picked from commit 84974c6fb5)
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
2020-07-01 22:26:04 +02:00
Andreas Rheinhardt
9083b1054a avcodec/ac3enc: Fix invalid shift
Fixes the FATE-tests unknown_layout-ac3, ac3-fixed-encode, ac3-encode
and eac3-encode. It furthermore fixes the ac3-encoder bugs mentioned in
tickets #7994, #8144 and #8159.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 2f289ec914)
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
2020-07-01 22:25:21 +02:00
Andreas Rheinhardt
0284b72c8a avcodec/tdsc: Fix undefined shifts
Fixes the tdsc FATE-test.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 86bef10e7a)
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
2020-07-01 22:20:51 +02:00
Andreas Rheinhardt
9a3edef421 avcodec/ttaenc: Fix undefined shift
ttaenc contained (1 << unary) - 1 as an argument for a function
expecting an unsigned int. unary can be as big as 31 in this case.
The type of the shift and the whole expression is int, because 1 fits
into an integer, so that the behaviour is undefined if unary == 31
as the result of the shift can't be represented in an int §. Subtraction
by 1 (which makes the result of the whole expression representable in
an int) doesn't change that this is undefined (it usually leads to
signed integer overflow which is undefined, too).

The solution is simple: Make 1 unsigned to change the type of the
whole expression to unsigned int (as the function expects anyway).

Fixes ticket #8153.

§: This of course presupposes the common int range of -2^31..2^31-1

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
Reviewed-by: Paul B Mahol <onemda@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 3ab488a540)
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
2020-07-01 22:18:52 +02:00
Andreas Rheinhardt
8d90a8cb37 avcodec/cbs_av1: Fix writing uvlc numbers >= INT_MAX
Fixes: assertion failure
Fixes: left shift of 1 by 31 places cannot be represented in type 'int'
Fixes: 23264/clusterfuzz-testcase-minimized-ffmpeg_BSF_AV1_METADATA_fuzzer-6308429248593920

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
(cherry picked from commit 6f06c17a55)
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
2020-07-01 17:16:06 +02:00
Andreas Rheinhardt
0df1210631 avcodec/bitstream: Don't check for undefined behaviour after it happened
Reviewed-by: Michael Niedermayer <michael@niedermayer.cc>
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
(cherry picked from commit 5e196dac22)
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
2020-07-01 16:59:01 +02:00