Commit graph

38555 commits

Author SHA1 Message Date
Diego Biurrun
fd502f4f5f build: Generalize yasm/nasm-related variable names
None of them are specific to the YASM assembler.

(Cherry-picked from libav commit 39e208f4d4)

Signed-off-by: James Almer <jamrial@gmail.com>
2017-06-21 17:00:29 -03:00
Ronald S. Bultje
e639d09199 mdec: stop preferring the simple IDCT.
This was added in e3e3c82555, probably
as a workaround for the fact that the quant table was not permutated
and the IDCT coefficients are, meaning that you'd only get correct
reconstruction if the IDCT permutation was an identity matrix, which
happens to be the case when you use the simple IDCT. The quant table
permutation bug was fixed in 42dd1434bf,
meaning this workaround is no longer necessary.

In practical terms, before 42dd1434bf,
the PSNR between decodes of the fate-mdec using simple (C) or simplemmx
IDCTs was 35. After 42dd1434bf, it's 90.
2017-06-21 09:01:32 -04:00
Ronald S. Bultje
42dd1434bf mdec: use correctly permutated quant matrix for dequantization. 2017-06-21 08:59:22 -04:00
Memphiz
9e85c5d6a7 aarch64: vp9 16bpp: Fix assembling with Xcode 6.2 and older
Properly use the b.eq form instead of the nonstandard form (which
both gas and newer clang accept though), and expand the register
lists that used a range (which the Xcode 6.2 clang, based on clang
3.5 svn, didn't support).

Signed-off-by: Martin Storsjö <martin@martin.st>
2017-06-21 09:08:14 +03:00
Memphiz
998609ddb8 aarch64: vp9: Fix assembling with Xcode 6.2 and older
Properly use the b.eq/b.ge forms instead of the nonstandard forms
(which both gas and newer clang accept though), and expand the
register list that used a range (which the Xcode 6.2 clang, based
on clang 3.5 svn, didn't support).

This is cherrypicked from libav commit
a970f9de86.

Signed-off-by: Martin Storsjö <martin@martin.st>
2017-06-21 09:08:13 +03:00
Michael Niedermayer
70a7df049c avcodec/mpegvideo: Use intra_scantable in dct_unquantize_h263_intra_c()
Reviewed-by: James Darnley <jdarnley@obe.tv>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-06-21 02:25:23 +02:00
Michael Niedermayer
d7b3d5c3f2 avcodec/hevc_filter: Fix invalid shift
Fixes: runtime error: left shift of negative value -1

Fixes: 2299/clusterfuzz-testcase-minimized-4843509351710720

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-06-21 02:25:23 +02:00
Michael Niedermayer
fdfbcbdd14 avcodec/hevcdec: Check beta and tc offset in hls_slice_header()
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-06-21 02:25:23 +02:00
Michael Niedermayer
5443c4bdf4 avcodec/mpeg4videodec: Fix overflow in virtual_ref computation
Fixes: runtime error: signed integer overflow: 262144 * -16120 cannot be represented in type 'int'
Fixes: 2292/clusterfuzz-testcase-minimized-6156080415506432

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-06-21 02:25:23 +02:00
Michael Niedermayer
e95fcfe8fb avcodec/lpc: signed integer overflow in compute_lpc_coefs() (aacdec_fixed)
Fixes: runtime error: signed integer overflow: -1575818955 + -915383657 cannot be represented in type 'int'
Fixes: 2224/clusterfuzz-testcase-minimized-6208559949807616

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-06-21 02:25:23 +02:00
John Rummell
966a0a814d avcodec/decode: Update decode_simple_internal() to get the side data correctly.
Use avci->last_pkt_props to get the side data. Using |pkt| doesn't work
when FF_API_MERGE_SD is set, as the compressed side data is expanded into
|tmp|, leaving the original |pkt| unchanged.

Signed-off-by: James Almer <jamrial@gmail.com>
2017-06-20 17:14:17 -03:00
James Darnley
8221c71703 avcodec/x86: allow future 8-bit simple idct to use slightly different coefficients 2017-06-20 16:12:25 +02:00
James Darnley
9d11fedd11 avcodec/mdec: override IDCT choice before initing DSP structs 2017-06-20 13:59:51 +02:00
James Darnley
d2597fb0c1 avcodec/x86: modify simple_idct10 macros to add an action paramter 2017-06-20 13:35:01 +02:00
James Darnley
8781330d80 avcodec/x86: cleanup simple_idct10
Use named arguments for the functions so we can remove a define.  The
stride/linesize argument is now ptrdiff_t type so we no longer need to
sign extend the register.
2017-06-20 13:34:38 +02:00
James Darnley
e3db94302c avcodec/x86/mpegenc: support transpose permuation type 2017-06-20 12:12:13 +02:00
James Darnley
fa30a0a548 avcodec/x86/mpegenc: check IDCT permutation type is a valid value 2017-06-20 12:12:13 +02:00
Michael Niedermayer
ae6f6d4e34 avcodec/x86/mpegvideo: Use intra scantable in dct_unquantize_h263_intra_mmx()
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-06-20 00:07:51 +02:00
Anton Mitrofanov
cf231b68da avcodec/h264: Fix mix of lossless and lossy MBs decoding
Signed-off-by: Ronald S. Bultje <rsbultje@gmail.com>
2017-06-19 14:33:40 -04:00
Anton Mitrofanov
06dda70f1e avcodec/h264_mb: Fix 8x8dct in lossless for new versions of x264
Signed-off-by: Ronald S. Bultje <rsbultje@gmail.com>
2017-06-19 14:33:39 -04:00
Anton Mitrofanov
840b41b2a6 avcodec/h264_cabac: Fix CABAC+8x8dct in 4:4:4
Use the correct ctxIdxInc calculation for coded_block_flag.
Keep old behavior for old versions of x264 for backward compatibility.

Signed-off-by: Ronald S. Bultje <rsbultje@gmail.com>
2017-06-19 14:33:39 -04:00
Michael Niedermayer
5f89747086 avcodec/wavpack: Fix undefined integer negation
Fixes: runtime error: negation of -2147483648 cannot be represented in type 'int'; cast to an unsigned type to negate this value to itself
Fixes: 2291/clusterfuzz-testcase-minimized-5538453481586688

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-06-19 17:54:40 +02:00
Michael Niedermayer
cf7edbd6c5 avcodec/aacdec_fixed: Check s for being too small
Fixes: runtime error: shift exponent -8 is negative
Fixes: 2286/clusterfuzz-testcase-minimized-5711764169687040

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-06-19 17:54:40 +02:00
Michael Niedermayer
f670c13f13 avcodec: Rename ff_mpv_decode_mb() to ff_mpv_reconstruct_mb
The new name more accuratly describes what the function does

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-06-19 17:54:40 +02:00
James Almer
8bb59e6742 x86/aacpsdsp: add ff_ps_hybrid_analysis_ileave_sse
About 2x faster than the c version.
2017-06-18 22:34:22 -03:00
James Almer
e229df9478 x86/aacpsdsp: add ff_ps_hybrid_synthesis_deint_{sse,sse4}
About 2x faster than the c version.
2017-06-18 22:33:27 -03:00
Michael Niedermayer
14b834c45a avcodec/htmlsubtitles: Factor open brace handling into its own function
Suggested-by: wm4
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-06-18 14:50:35 +02:00
Michael Niedermayer
4132218b87 avcodec/htmlsubtitles: Replace very slow redundant sscanf() calls by cleaner and faster code
This reduces the worst case from O(n²) to O(n) time

Fixes Timeout
Fixes: 2127/clusterfuzz-testcase-minimized-6595787859427328

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-06-18 14:50:30 +02:00
Michael Niedermayer
27c2006805 avcodec/takdec: Fixes: integer overflow in AV_SAMPLE_FMT_U8P output
Fixes: runtime error: signed integer overflow: 2147483543 + 128 cannot be represented in type 'int'
Fixes: 2234/clusterfuzz-testcase-minimized-6266896041115648

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-06-18 14:38:41 +02:00
Michael Niedermayer
9a6503f496 avcodec/iff: Cleanup on init failure
Fixes: memleak
Fixes: 2272/clusterfuzz-testcase-minimized-5059103858622464

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-06-18 14:05:57 +02:00
Michael Niedermayer
c746f92a8e avcodec/jpeg2000dsp: Reorder operations in ict_int() to avoid 2 integer overflows
Fixes: runtime error: signed integer overflow: 58065 * 51981 cannot be represented in type 'int'
Fixes: 2271/clusterfuzz-testcase-minimized-5778297776504832

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-06-17 22:36:54 +02:00
Michael Niedermayer
c94326c1fc avcodec/hevcpred_template: Fix left shift of negative value
Fixes: runtime error: left shift of negative value -1
Fixes: 2250/clusterfuzz-testcase-minimized-5693382112313344

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-06-17 22:36:54 +02:00
Michael Niedermayer
1edbf5e20c avcodec/hevcdec: Fix signed integer overflow in decode_lt_rps()
Fixes: runtime error: signed integer overflow: 2147483647 + 6 cannot be represented in type 'int'
Fixes: 2263/clusterfuzz-testcase-minimized-4800359627227136

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-06-17 00:34:48 +02:00
Michael Niedermayer
9b65dbf734 avcodec/gdv: Fix undefined shift
Fixes: runtime error: left shift of 255 by 24 places cannot be represented in type 'int'
Fixes: 2249/clusterfuzz-testcase-minimized-5388542379294720

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-06-16 23:32:28 +02:00
Michael Niedermayer
dfb61ea263 avcodec/jpeg2000dec: Check nonzerobits more completely
Fixes: runtime error: shift exponent 36 is too large for 32-bit type 'int'
Fixes: 2239/clusterfuzz-testcase-minimized-5639766592716800

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-06-16 20:32:39 +02:00
Michael Niedermayer
16d6cc2168 avcodec/wavpack: Change wp_log2() to unsigned
Fixes: runtime error: signed integer overflow: 2143315325 + 4186162 cannot be represented in type 'int'
Fixes: 2134/clusterfuzz-testcase-minimized-4619258405322752

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-06-16 20:32:39 +02:00
Michael Niedermayer
e77ddd31a8 avcodec/shorten: Sanity check maxnlpc
Fixes OOM
Fixes: 2131/clusterfuzz-testcase-minimized-4718045157130240

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-06-16 20:32:39 +02:00
James Almer
623d217ed1 avcodec/aacps: move checks for valid length outside the stereo_interpolate dsp function
Signed-off-by: James Almer <jamrial@gmail.com>
2017-06-15 23:49:40 -03:00
James Almer
b3446862bf x86/vorbisdsp: optimize ff_vorbis_inverse_coupling_sse
About 7% faster.
2017-06-15 23:20:05 -03:00
Michael Niedermayer
c0607d88ee avcodec/parser: assert that there is a past buffer if theres a reference into the past
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-06-16 01:35:35 +02:00
Michael Niedermayer
3c716682a8 avcodec/truemotion2: Move skip computation after checks
Fixes: runtime error: signed integer overflow: 630067357 * 4 cannot be represented in type 'int'
Fixes: 2233/clusterfuzz-testcase-minimized-5943031318446080

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-06-16 00:21:30 +02:00
Michael Niedermayer
e3fadc57c5 avcodec/jpeg2000: Fixes integer overflow in ff_jpeg2000_ceildivpow2()
Fixes: runtime error: negation of -2147483648 cannot be represented in type 'int'; cast to an unsigned type to negate this value to itself
Fixes: 2231/clusterfuzz-testcase-minimized-4565181982048256

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-06-16 00:21:30 +02:00
Michael Niedermayer
611b356274 avcodec/dnxhd_parser: Do not return invalid value from dnxhd_find_frame_end() on error
Fixes: Null pointer dereference

Fixes: CVE-2017-9608
Found-by: Yihan Lian
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-06-16 00:21:30 +02:00
Rostislav Pehlivanov
b52b398c30 vc2enc: decrease default strictness level
Given how incredibly limited the official specifications are (limiting all use
to only the most common broadcasting formats), permit all supported inputs
by default. This makes the encoder more useful.
2017-06-15 18:30:08 +01:00
Tyler Jones
752dd1952a vorbisenc: Stop tracking number of samples per frame
Each frame is now padded with 0 values if not enough samples are
present, and all frames are guaranteed to have exactly
1 << (venc->log2_blocksize[1] - 1) samples.

Signed-off-by: Tyler Jones <tdjones879@gmail.com>
Reviewed-by: Rostislav Pehlivanov <atomnuker@gmail.com>
2017-06-15 16:42:49 +01:00
Tyler Jones
f57f665183 vorbisenc: Apply and output correct length window and mdct
Usage of blocksize, window, mode, and mdct indexes are switched from
default 0 to a default of 1 to better align with specs. A flag of 0
should correspond with short windows, a flag of 1 with long.

Signed-off-by: Tyler Jones <tdjones879@gmail.com>
Reviewed-by: Rostislav Pehlivanov <atomnuker@gmail.com>
2017-06-15 16:42:49 +01:00
Tyler Jones
5a2ad7ede3 vorbisenc: Separate copying audio samples from windowing
Audio samples are shifted around when copying from the frame queue so that
analysis can be done without negatively impacting calculation of the MDCT.

Window coefficients are applied to the current two overlapped windows
simultaneously instead of applying overlap for the next frame ahead of time.
This improves readability when applying windows of varying lengths.

Signed-off-by: Tyler Jones <tdjones879@gmail.com>
Reviewed-by: Rostislav Pehlivanov <atomnuker@gmail.com>
2017-06-15 16:42:49 +01:00
Michael Niedermayer
bc40674462 avcodec/hevcdec: Check nb_sps
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-06-15 03:09:40 +02:00
Michael Niedermayer
1cb4ef526d avcodec/hevc_refs: Check nb_refs in add_candidate_ref()
Fixes: runtime error: index 16 out of bounds for type 'int [16]'
Fixes: 2209/clusterfuzz-testcase-minimized-5012343912136704

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-06-15 03:09:40 +02:00
Michael Niedermayer
12245ab1f6 avcodec/mpeg4videodec: Check sprite delta upshift against overflowing.
Fixes: runtime error: signed integer overflow: -268386304 * 16 cannot be represented in type 'int'
Fixes: 2204/clusterfuzz-testcase-minimized-5616756909408256

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-06-15 00:16:20 +02:00