Commit graph

23276 commits

Author SHA1 Message Date
Michael Niedermayer
17788711ac avcodec/h264_mp4toannexb_bsf: Reorder operations in nal_size check
Fixes Ticket4778

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 2bb54b82b5)
2015-10-10 01:22:43 +02:00
wm4
928473a187 avcodec/mp3: fix skipping zeros
Commits 43bc5cf9 and c5371f77 add code for skipping initial zeros in mp3
packets. This code forgot to report to the user that data was skipped at
all.

Since audio codecs allow partial packet decoding, the user application
has to rely on the return value. It will remove the data reported as
consumed by the decoder, and feed it to the decoder again. This resulted
in the mp3 frame after the zero region to be decoded over and over
again, until the zero region was finally skipped by the application.

Fix this by including the amount of skipped bytes to the number of
consumed bytes returned by the decode call.

Fixes trac ticket #4890.
(cherry picked from commit cb1da9fb8d)
2015-10-10 01:21:20 +02:00
Michael Niedermayer
581bfd9c6f avcodec/x86/h264_weight: handle weight1=128
Fix ticket4596

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit e100966575)
2015-06-19 11:57:16 +02:00
Michael Niedermayer
7e7772c13a avcodec/mjpegdec: Skip blocks which are outside the visible area
Fixes out of array accesses
Fixes: ffmpeg_mjpeg_crash.avi

Found-by: Thomas Lindroth <thomas.lindroth@gmail.com>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 08509c8f86)

Conflicts:

	libavcodec/mjpegdec.c
(cherry picked from commit b881a97b9977b79dfe3ce02d61542c630fe78c14)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-02-11 11:59:23 +01:00
Michael Niedermayer
ac8ef33f9a avcodec/h264_slice: ignore SAR changes in slices after the first
Fixes race condition and null pointer dereference
Fixes: signal_sigsegv_1472ac3_468_cov_2915641226_CABACI3_Sony_B.jsv

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 38d5241b7f)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Conflicts:

	libavcodec/h264_slice.c
2015-02-07 14:14:28 +01:00
Michael Niedermayer
4de40be507 avcodec/h264_slice: Check picture structure before setting the related fields
This might fix a hypothetical race condition

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit f111831ed6)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Conflicts:

	libavcodec/h264_slice.c

Conflicts:

	libavcodec/h264.c
2015-02-07 14:14:24 +01:00
Michael Niedermayer
64e50b2f2a avcodec/h264_slice: Do not change frame_num after the first slice
Fixes potential race condition
Fixes: signal_sigsegv_1472ac3_468_cov_2915641226_CABACI3_Sony_B.jsv

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit f906982c94)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Conflicts:

	libavcodec/h264_slice.c
2015-02-07 14:11:25 +01:00
Michael Niedermayer
763c753348 avcodec/h264: Be more strict on rejecting pps/sps changes
Fixes race condition
Fixes: signal_sigsegv_1472ac3_468_cov_2915641226_CABACI3_Sony_B.jsv

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 6fafc62b0b)

Conflicts:

	libavcodec/h264.c
2015-02-06 15:53:48 +01:00
Michael Niedermayer
4ef5605fc9 avcodec/h264_ps: More completely check the bit depths
Fixes out of array read
Fixes: asan_static-oob_30328b6_719_cov_3325483287_H264_artifacts_motion.h264

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 69aa79365c)

Conflicts:

	libavcodec/h264_ps.c

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-02-06 04:51:35 +01:00
Michael Niedermayer
fd6a9fcd42 avcodec/mjpegdec: Check number of components for JPEG-LS
Fixes out of array accesses
Fixes: asan_heap-oob_1c1a4ea_1242_cov_2274415971_TESTcmyk.jpg

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit fabbfaa095)

Conflicts:

	libavcodec/mjpegdec.c
2015-02-04 21:27:46 +01:00
Michael Niedermayer
20be3ea442 avcodec/mjpegdec: Check escape sequence validity
Fixes assertion failure
Fixes: asan_heap-oob_1c1a4ea_1242_cov_2274415971_TESTcmyk.jpg

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-02-04 21:27:44 +01:00
Michael Niedermayer
7881d3b2a5 avcodec/mpegvideo_enc: Fix number suffixes in rc_buffer_size calculation
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 4531e2c489)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-02-01 20:33:43 +01:00
Michael Niedermayer
5f2902bdc7 avcodec/h264_cabac: use int instead of long for mbb_xy
The mb address fits in int

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 592ba6ec10)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-02-01 20:33:00 +01:00
Michael Niedermayer
a86cc0cc1b avcodec/flac_parser: fix handling EOF if no headers are found
Fixes assertion failure
Fixes Ticket4269

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit c4d85fc23c)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-01-17 02:13:11 +01:00
wm4
8a3a3e3154 avcodec/dvdsubdec: error on bitmaps with size 0
Attemtping to decode them could lead to invalid writes with some fuzzed
samples.

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit bcaa9099b3)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-01-08 02:11:29 +01:00
wm4
931f4313b2 avcodec/dvdsubdec: fix out of bounds accesses
The code blindly trusted buffer offsets read from the file in the RLE
decoder. Explicitly check the offset. Also error out on other RLE
decoding errors.

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit c9151de7c4)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-01-05 05:13:27 +01:00
Michael Niedermayer
c60caa5769 avcodec/indeo3: ensure offsets are non negative
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 368642361f)

Conflicts:

	libavcodec/indeo3.c
2014-12-18 18:58:35 +01:00
Michael Niedermayer
96e2a4ba74 avcodec/h264: Check *log2_weight_denom
Fixes undefined behavior
Fixes: signal_sigsegv_14768d2_2248_cov_3629497219_h264_h264___pi_20070614T182942.h264
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 61296d41e2)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-12-18 03:48:46 +01:00
Michael Niedermayer
a9c0f905aa avcodec/h264: Clear delayed_pic on deallocation
Fixes use of freed memory

Fixes: case5_av_frame_copy_props.mp4
Found-by: Michal Zalewski <lcamtuf@coredump.cx>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit e8714f6f93)

Conflicts:

	libavcodec/h264.c
2014-12-17 22:17:32 +01:00
Michael Niedermayer
42b7d224bc avcodec/indeo3: use signed variables to avoid underflow
Fixes out of array read
Fixes: signal_sigsegv_1b0a4da_1865_cov_2167818389_computer_anger.avi
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 3305acdc92)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-12-17 05:24:50 +01:00
Michael Niedermayer
45bb57e009 avcodec/h264: make the first field of H264Context an AVClass
Fixes use of freed memory
Fixes: asan_heap-uaf_3660f67_757_cov_1257014655_Hi422FR1_SONY_A.jsv
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit f3b5b139ad)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-12-17 02:14:37 +01:00
Michael Niedermayer
ece0c9c4b0 avcodec/utvideodec: Fix handling of slice_height=0
Fixes out of array accesses
Fixes: asan_heap-oob_25bcd7e_3783_cov_3553517262_utvideo_rgba_median.avi
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 3881606240)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-12-16 23:06:36 +01:00
Michael Niedermayer
0da0d7754e avcodec/vmdvideo: Check len before using it in method 3
Fixes out of array access
Fixes: asan_heap-oob_4d23ba_91_cov_3853393937_128.vmd

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 3030fb7e0d)

Conflicts:

	libavcodec/vmdvideo.c
2014-12-16 19:43:41 +01:00
Michael Niedermayer
69a92ae397 avcodec/motion_est: use 2x8x8 for interlaced qpel
Fixes out of array read
Fixes Ticket4121

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit b50e003e1c)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-12-02 00:26:19 +01:00
Michael Niedermayer
d723f92dbf avcodec/pngdec: Check IHDR/IDAT order
Fixes out of array access
Fixes: asan_heap-oob_20a6c26_2690_cov_3434532168_mail.png
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 79ceaf827b)

Conflicts:

	libavcodec/pngdec.c
2014-12-02 00:26:19 +01:00
Michael Niedermayer
11e5106c56 avcodec/flacdec: Call ff_flacdsp_init() unconditionally
Fixes out of array access
Fixes: signal_sigsegv_324b135_3398_cov_246853371_short.flac
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit e5c01ccdf5)

Conflicts:

	libavcodec/flacdec.c
2014-12-02 00:26:19 +01:00
Michael Niedermayer
a65573c45d avcodec/utils: Check that the data is complete in avpriv_bprint_to_extradata()
Fixes out of array read
Fixes: asan_heap-oob_4d2250_814_cov_2745172097_JACOsub_capability_tester.jss
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 3d5d95db3f)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-12-02 00:26:19 +01:00
Michael Niedermayer
6c28673cb3 avcodec/mjpegdec: Fix context fields becoming inconsistent
Fixes out of array access
Fixes: asan_heap-oob_1ca4f85_2760_cov_144449187_miss_congeniality_pegasus_ljpg.avi
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 0eecf40935)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-12-02 00:26:19 +01:00
Michael Niedermayer
01384e3dde avcodec/wmaprodec: Fix integer overflow in sfb_offsets initialization
Fixes out of array read
Fixes: asan_heap-oob_2aec5b0_1828_classical_22_16_2_16000_v3c_0_exclusive_0_29.wma
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 5dcb99033d)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-11-10 23:49:08 +01:00
Michael Niedermayer
2d39d8ffc1 avcodec/utvideodec: fix assumtation that slice_height >= 1
Fixes out of array read
Fixes: asan_heap-oob_2573085_3783_utvideo_rgba_median.avi
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 7656c4c6e6)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-11-10 23:38:50 +01:00
Michael Niedermayer
5d33142fd6 avcodec/options_table fix min of audio channels and sample rate
Found-by: Lukasz Marek <lukasz.m.luki2@gmail.com>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 206c98f303)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-11-03 13:30:09 +01:00
Michael Niedermayer
96dac432f7 avcodec/svq1dec: zero terminate embedded message before printing
Fixes out of array access
Fixes: asan_stack-oob_49b1e5_10_009.mov
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit e91ba2efa9)

Conflicts:

	libavcodec/svq1dec.c
2014-11-02 11:15:12 +01:00
Michael Niedermayer
b92ccfefc3 avcodec/utils: Align dimensions by at least their chroma sub-sampling factors.
Fixes: out of array accesses
Fixes: asan_heap-oob_112c6b3_13_012.avi
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit df74811cd5)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-11-02 11:14:25 +01:00
Michael Niedermayer
6e45a99a7c avcodec/dnxhddec: treat pix_fmt like width/height
Fixes out of array accesses
Fixes: asan_heap-oob_22c9a39_16_015.mxf
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit f3c0e0bf6f)

Conflicts:

	libavcodec/dnxhddec.c

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-11-02 11:14:25 +01:00
Michael Niedermayer
d59804bafb avcodec/dxa: check dimensions
Fixes out of array access
Fixes: asan_heap-oob_11222fb_21_020.dxa
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit e70312dfc2)

Conflicts:

	libavcodec/dxa.c
2014-11-02 11:14:25 +01:00
Michael Niedermayer
35cf24eee9 avcodec/dirac_arith: fix integer overflow
Fixes: asan_heap-oob_1078676_9_008.drc
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 39680cacee)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-11-02 11:14:25 +01:00
Michael Niedermayer
341cf9ed81 avcodec/diracdec: Tighter checks on CODEBLOCKS_X/Y
Fixes very long but finite loop
Fixes: asan_heap-oob_107866c_42_041.drc
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 5145d22b88)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-11-02 11:14:25 +01:00
Michael Niedermayer
c36f5df34a avcodec/diracdec: Use 64bit in calculation of codeblock coordinates
Fixes integer overflow
Fixes out of array read
Fixes: asan_heap-oob_107866c_42_041.drc
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 526886e606)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-11-02 11:14:25 +01:00
Michael Niedermayer
25a9823eb4 avcodec/h264_slice: Clear table pointers to avoid stale pointers
Might fix Ticket3889

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 547fce9585)

Conflicts:

	libavcodec/h264_slice.c

Conflicts:

	libavcodec/h264.c
2014-11-02 02:22:14 +01:00
Carl Eugen Hoyos
a8fd50bb4e Fix compilation after aa9d7058. 2014-11-02 00:43:50 +01:00
Michael Niedermayer
9282c96071 avcodec/h264_parser: rewrite the parse_nal_units() loop logic based on h264.c
Fixes Ticket4011

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 69a9a90d2e)

Conflicts:

	libavcodec/h264_parser.c
2014-11-01 20:27:01 +01:00
Michael Niedermayer
aa9d705871 avcodec/h264: simplify find_start_code()
this also uses avpriv_find_start_code(), though no speed change is expected as
the area searched is generally small

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 3b678da5e3)

Conflicts:

	libavcodec/h264.h
2014-11-01 20:12:25 +01:00
Michael Niedermayer
7ddf252c7e Move get_avc_nalsize() and find_start_code() to h264.h
This allows sharing them with the h264 parser

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 4898440f6b)

Conflicts:

	libavcodec/h264.c
	libavcodec/h264.h
2014-11-01 20:12:16 +01:00
Michael Niedermayer
7fc97160c2 avcodec/h264: Check mode before considering mixed mode intra prediction
Fixes out of array read
Fixes: asan_heap-oob_e476fc_2_asan_heap-oob_1333ec6_61_CAMACI3_Sony_C.jsv

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 9734a7a1de)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-10-04 15:04:43 +02:00
Michael Niedermayer
4865948d2e avcodec/smc: fix off by 1 error
Fixes out of array access
Fixes: asan_heap-oob_1685bf0_5_asan_heap-oob_1f35116_430_smc.mov

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit c727401aa9)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-10-03 23:24:27 +02:00
Michael Niedermayer
891376d261 avcodec/qpeg: fix off by 1 error in MV bounds check
Fixes out of array access
Fixes: asan_heap-oob_153760f_4_asan_heap-oob_1d7a4cf_164_VWbig6.avi

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit dd3bfe3cc1)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-10-03 21:32:15 +02:00
Michael Niedermayer
24a5cd720d avcodec/gifdec: factorize interleave end handling out
also change it to a loop
Fixes out of array access
Fixes: asan_heap-oob_ca5410_8_asan_heap-oob_ca5410_97_ID_LSD_Size_Less_Then_Data_Inter_3.gif

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 8f1457864b)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-10-03 20:28:52 +02:00
Michael Niedermayer
75f5fe165c avcodec/cinepak: fix integer underflow
Fixes out of array access
Fixes: asan_heap-oob_4da0ba_6_asan_heap-oob_4da0ba_241_cvid_crash.avi

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit e7e5114c50)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-10-03 19:47:43 +02:00
Michael Niedermayer
665f70209a avcodec/pngdec: Calculate MPNG bytewidth more defensively
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit e830902934)

Conflicts:

	libavcodec/pngdec.c
2014-10-03 18:36:41 +02:00
Michael Niedermayer
12c8e4021c avcodec/pngdec: Check bits per pixel before setting monoblack pixel format
Fixes out of array accesses
Fixes: asan_heap-oob_14dbfcf_4_asan_heap-oob_1ce5767_179_add_method_small.png

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 3e2b745020)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-10-03 18:35:38 +02:00