Commit graph

38546 commits

Author SHA1 Message Date
Michael Niedermayer
e95fcfe8fb avcodec/lpc: signed integer overflow in compute_lpc_coefs() (aacdec_fixed)
Fixes: runtime error: signed integer overflow: -1575818955 + -915383657 cannot be represented in type 'int'
Fixes: 2224/clusterfuzz-testcase-minimized-6208559949807616

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-06-21 02:25:23 +02:00
John Rummell
966a0a814d avcodec/decode: Update decode_simple_internal() to get the side data correctly.
Use avci->last_pkt_props to get the side data. Using |pkt| doesn't work
when FF_API_MERGE_SD is set, as the compressed side data is expanded into
|tmp|, leaving the original |pkt| unchanged.

Signed-off-by: James Almer <jamrial@gmail.com>
2017-06-20 17:14:17 -03:00
James Darnley
8221c71703 avcodec/x86: allow future 8-bit simple idct to use slightly different coefficients 2017-06-20 16:12:25 +02:00
James Darnley
9d11fedd11 avcodec/mdec: override IDCT choice before initing DSP structs 2017-06-20 13:59:51 +02:00
James Darnley
d2597fb0c1 avcodec/x86: modify simple_idct10 macros to add an action paramter 2017-06-20 13:35:01 +02:00
James Darnley
8781330d80 avcodec/x86: cleanup simple_idct10
Use named arguments for the functions so we can remove a define.  The
stride/linesize argument is now ptrdiff_t type so we no longer need to
sign extend the register.
2017-06-20 13:34:38 +02:00
James Darnley
e3db94302c avcodec/x86/mpegenc: support transpose permuation type 2017-06-20 12:12:13 +02:00
James Darnley
fa30a0a548 avcodec/x86/mpegenc: check IDCT permutation type is a valid value 2017-06-20 12:12:13 +02:00
Michael Niedermayer
ae6f6d4e34 avcodec/x86/mpegvideo: Use intra scantable in dct_unquantize_h263_intra_mmx()
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-06-20 00:07:51 +02:00
Anton Mitrofanov
cf231b68da avcodec/h264: Fix mix of lossless and lossy MBs decoding
Signed-off-by: Ronald S. Bultje <rsbultje@gmail.com>
2017-06-19 14:33:40 -04:00
Anton Mitrofanov
06dda70f1e avcodec/h264_mb: Fix 8x8dct in lossless for new versions of x264
Signed-off-by: Ronald S. Bultje <rsbultje@gmail.com>
2017-06-19 14:33:39 -04:00
Anton Mitrofanov
840b41b2a6 avcodec/h264_cabac: Fix CABAC+8x8dct in 4:4:4
Use the correct ctxIdxInc calculation for coded_block_flag.
Keep old behavior for old versions of x264 for backward compatibility.

Signed-off-by: Ronald S. Bultje <rsbultje@gmail.com>
2017-06-19 14:33:39 -04:00
Michael Niedermayer
5f89747086 avcodec/wavpack: Fix undefined integer negation
Fixes: runtime error: negation of -2147483648 cannot be represented in type 'int'; cast to an unsigned type to negate this value to itself
Fixes: 2291/clusterfuzz-testcase-minimized-5538453481586688

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-06-19 17:54:40 +02:00
Michael Niedermayer
cf7edbd6c5 avcodec/aacdec_fixed: Check s for being too small
Fixes: runtime error: shift exponent -8 is negative
Fixes: 2286/clusterfuzz-testcase-minimized-5711764169687040

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-06-19 17:54:40 +02:00
Michael Niedermayer
f670c13f13 avcodec: Rename ff_mpv_decode_mb() to ff_mpv_reconstruct_mb
The new name more accuratly describes what the function does

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-06-19 17:54:40 +02:00
James Almer
8bb59e6742 x86/aacpsdsp: add ff_ps_hybrid_analysis_ileave_sse
About 2x faster than the c version.
2017-06-18 22:34:22 -03:00
James Almer
e229df9478 x86/aacpsdsp: add ff_ps_hybrid_synthesis_deint_{sse,sse4}
About 2x faster than the c version.
2017-06-18 22:33:27 -03:00
Michael Niedermayer
14b834c45a avcodec/htmlsubtitles: Factor open brace handling into its own function
Suggested-by: wm4
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-06-18 14:50:35 +02:00
Michael Niedermayer
4132218b87 avcodec/htmlsubtitles: Replace very slow redundant sscanf() calls by cleaner and faster code
This reduces the worst case from O(n²) to O(n) time

Fixes Timeout
Fixes: 2127/clusterfuzz-testcase-minimized-6595787859427328

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-06-18 14:50:30 +02:00
Michael Niedermayer
27c2006805 avcodec/takdec: Fixes: integer overflow in AV_SAMPLE_FMT_U8P output
Fixes: runtime error: signed integer overflow: 2147483543 + 128 cannot be represented in type 'int'
Fixes: 2234/clusterfuzz-testcase-minimized-6266896041115648

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-06-18 14:38:41 +02:00
Michael Niedermayer
9a6503f496 avcodec/iff: Cleanup on init failure
Fixes: memleak
Fixes: 2272/clusterfuzz-testcase-minimized-5059103858622464

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-06-18 14:05:57 +02:00
Michael Niedermayer
c746f92a8e avcodec/jpeg2000dsp: Reorder operations in ict_int() to avoid 2 integer overflows
Fixes: runtime error: signed integer overflow: 58065 * 51981 cannot be represented in type 'int'
Fixes: 2271/clusterfuzz-testcase-minimized-5778297776504832

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-06-17 22:36:54 +02:00
Michael Niedermayer
c94326c1fc avcodec/hevcpred_template: Fix left shift of negative value
Fixes: runtime error: left shift of negative value -1
Fixes: 2250/clusterfuzz-testcase-minimized-5693382112313344

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-06-17 22:36:54 +02:00
Michael Niedermayer
1edbf5e20c avcodec/hevcdec: Fix signed integer overflow in decode_lt_rps()
Fixes: runtime error: signed integer overflow: 2147483647 + 6 cannot be represented in type 'int'
Fixes: 2263/clusterfuzz-testcase-minimized-4800359627227136

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-06-17 00:34:48 +02:00
Michael Niedermayer
9b65dbf734 avcodec/gdv: Fix undefined shift
Fixes: runtime error: left shift of 255 by 24 places cannot be represented in type 'int'
Fixes: 2249/clusterfuzz-testcase-minimized-5388542379294720

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-06-16 23:32:28 +02:00
Michael Niedermayer
dfb61ea263 avcodec/jpeg2000dec: Check nonzerobits more completely
Fixes: runtime error: shift exponent 36 is too large for 32-bit type 'int'
Fixes: 2239/clusterfuzz-testcase-minimized-5639766592716800

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-06-16 20:32:39 +02:00
Michael Niedermayer
16d6cc2168 avcodec/wavpack: Change wp_log2() to unsigned
Fixes: runtime error: signed integer overflow: 2143315325 + 4186162 cannot be represented in type 'int'
Fixes: 2134/clusterfuzz-testcase-minimized-4619258405322752

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-06-16 20:32:39 +02:00
Michael Niedermayer
e77ddd31a8 avcodec/shorten: Sanity check maxnlpc
Fixes OOM
Fixes: 2131/clusterfuzz-testcase-minimized-4718045157130240

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-06-16 20:32:39 +02:00
James Almer
623d217ed1 avcodec/aacps: move checks for valid length outside the stereo_interpolate dsp function
Signed-off-by: James Almer <jamrial@gmail.com>
2017-06-15 23:49:40 -03:00
James Almer
b3446862bf x86/vorbisdsp: optimize ff_vorbis_inverse_coupling_sse
About 7% faster.
2017-06-15 23:20:05 -03:00
Michael Niedermayer
c0607d88ee avcodec/parser: assert that there is a past buffer if theres a reference into the past
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-06-16 01:35:35 +02:00
Michael Niedermayer
3c716682a8 avcodec/truemotion2: Move skip computation after checks
Fixes: runtime error: signed integer overflow: 630067357 * 4 cannot be represented in type 'int'
Fixes: 2233/clusterfuzz-testcase-minimized-5943031318446080

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-06-16 00:21:30 +02:00
Michael Niedermayer
e3fadc57c5 avcodec/jpeg2000: Fixes integer overflow in ff_jpeg2000_ceildivpow2()
Fixes: runtime error: negation of -2147483648 cannot be represented in type 'int'; cast to an unsigned type to negate this value to itself
Fixes: 2231/clusterfuzz-testcase-minimized-4565181982048256

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-06-16 00:21:30 +02:00
Michael Niedermayer
611b356274 avcodec/dnxhd_parser: Do not return invalid value from dnxhd_find_frame_end() on error
Fixes: Null pointer dereference

Fixes: CVE-2017-9608
Found-by: Yihan Lian
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-06-16 00:21:30 +02:00
Rostislav Pehlivanov
b52b398c30 vc2enc: decrease default strictness level
Given how incredibly limited the official specifications are (limiting all use
to only the most common broadcasting formats), permit all supported inputs
by default. This makes the encoder more useful.
2017-06-15 18:30:08 +01:00
Tyler Jones
752dd1952a vorbisenc: Stop tracking number of samples per frame
Each frame is now padded with 0 values if not enough samples are
present, and all frames are guaranteed to have exactly
1 << (venc->log2_blocksize[1] - 1) samples.

Signed-off-by: Tyler Jones <tdjones879@gmail.com>
Reviewed-by: Rostislav Pehlivanov <atomnuker@gmail.com>
2017-06-15 16:42:49 +01:00
Tyler Jones
f57f665183 vorbisenc: Apply and output correct length window and mdct
Usage of blocksize, window, mode, and mdct indexes are switched from
default 0 to a default of 1 to better align with specs. A flag of 0
should correspond with short windows, a flag of 1 with long.

Signed-off-by: Tyler Jones <tdjones879@gmail.com>
Reviewed-by: Rostislav Pehlivanov <atomnuker@gmail.com>
2017-06-15 16:42:49 +01:00
Tyler Jones
5a2ad7ede3 vorbisenc: Separate copying audio samples from windowing
Audio samples are shifted around when copying from the frame queue so that
analysis can be done without negatively impacting calculation of the MDCT.

Window coefficients are applied to the current two overlapped windows
simultaneously instead of applying overlap for the next frame ahead of time.
This improves readability when applying windows of varying lengths.

Signed-off-by: Tyler Jones <tdjones879@gmail.com>
Reviewed-by: Rostislav Pehlivanov <atomnuker@gmail.com>
2017-06-15 16:42:49 +01:00
Michael Niedermayer
bc40674462 avcodec/hevcdec: Check nb_sps
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-06-15 03:09:40 +02:00
Michael Niedermayer
1cb4ef526d avcodec/hevc_refs: Check nb_refs in add_candidate_ref()
Fixes: runtime error: index 16 out of bounds for type 'int [16]'
Fixes: 2209/clusterfuzz-testcase-minimized-5012343912136704

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-06-15 03:09:40 +02:00
Michael Niedermayer
12245ab1f6 avcodec/mpeg4videodec: Check sprite delta upshift against overflowing.
Fixes: runtime error: signed integer overflow: -268386304 * 16 cannot be represented in type 'int'
Fixes: 2204/clusterfuzz-testcase-minimized-5616756909408256

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-06-15 00:16:20 +02:00
Michael Niedermayer
0a87be404a avcodec/mpeg4videodec: Fix integer overflow in num_sprite_warping_points=2 case
Fixes: runtime error: signed integer overflow: 131072 + 2147352576 cannot be represented in type 'int'
Fixes: 2192/clusterfuzz-testcase-minimized-5370387988742144

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-06-15 00:16:20 +02:00
Michael Niedermayer
900fe8ee5d avcodec/dnxhdenc: Assert that frame size is not assigned an error code
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-06-15 00:16:20 +02:00
Mark Thompson
88a2e4504d hevc: Fix scaling list prediction delta for the 32x32 inter matrix
Fixes ticket #6356.
2017-06-14 23:08:26 +01:00
Mark Thompson
3882063174 vaapi: Add external control of allow-profile-mismatch
Uses the just-added ALLOW_PROFILE_MISMATCH flag.

(cherry picked from commit 7acb90333a)
2017-06-14 22:27:06 +01:00
Mark Thompson
49ae8a5e87 lavc: Add flag to allow profile mismatch with hardware decoding
(cherry picked from commit 64a5260c69)
2017-06-14 22:27:04 +01:00
Mark Thompson
b658b5399e vaapi_encode: Use gop_size consistently in RC parameters
The non-H.26[45] codecs already use this form.  Since we don't
currently generate I frames for codecs which support them separately
to IDR, the p_per_i variable is set to infinity by default so that it
doesn't interfere with any other calculation.  (All the code for I
frames still exists, and it works for H.264 if set manually.)

(cherry picked from commit 6af014f402)
2017-06-14 22:26:32 +01:00
Mark Thompson
28aedeed19 qsvenc: Allow use of hw_device_ctx to make the internal session
(cherry picked from commit 3d197514e6)
2017-06-14 22:26:32 +01:00
Mark Thompson
8aa3c2df1a qsvdec: Allow use of hw_device_ctx to make the internal session
(cherry picked from commit 8848ba0bd6)
2017-06-14 22:26:32 +01:00
Mark Thompson
91c3b50d74 qsv: Add ability to create a session from a device
(cherry picked from commit 4936a48b1e)
2017-06-14 22:26:32 +01:00