Commit graph

13713 commits

Author SHA1 Message Date
Michael Niedermayer
0eca0da06e svq1dec: call avcodec_set_dimensions() after dimensions changed.
Fixes NGS00148, CVE-2011-4579

Found-by: Phillip Langlois
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

(cherry picked from commit 6e24b9488e)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-12-23 15:55:38 +01:00
Thierry Foucu
ba4b08b789 vp6: Fix illegal read.
Found with Address Sanitizer

Signed-off-by: Alex Converse <alex.converse@gmail.com>
(cherry picked from commit e0966eb140)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-12-04 09:21:09 +01:00
Alex Converse
67a7ed623b vp6: Fix illegal read.
(cherry picked from commit 2a6eb06254)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-12-04 09:20:49 +01:00
Laurent Aimar
c76505e0de vp6: Reset the internal state when aborting key frames header parsing
It prevents leaving the state only half initialized.

Signed-off-by: Janne Grunau <janne-libav@jannau.net>
(cherry picked from commit a72cad0a6c)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-12-04 09:20:28 +01:00
Laurent Aimar
30c08e2261 vp6: Check for huffman tree build errors
Signed-off-by: Janne Grunau <janne-libav@jannau.net>
(cherry picked from commit 066fff755a)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-12-04 09:20:10 +01:00
Dustin Brody
7367cbec1b vp6: partially propagate huffman tree building errors during coeff model parsing and fix misspelling
Signed-off-by: Ronald S. Bultje <rsbultje@gmail.com>
(cherry picked from commit f913eeea43)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-12-04 09:19:29 +01:00
Justin Ruggles
7347205351 qdm2: check output buffer size before decoding
(cherry picked from commit 7d49f79f1c)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-12-04 08:55:55 +01:00
Laurent Aimar
0d93d5c461 Fix out of bound reads in the QDM2 decoder.
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
Signed-off-by: Justin Ruggles <justin.ruggles@gmail.com>
(cherry picked from commit 5a19acb17c)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-12-04 08:55:55 +01:00
Laurent Aimar
a31ccacb1a Check for out of bound writes in the QDM2 decoder.
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
Signed-off-by: Justin Ruggles <justin.ruggles@gmail.com>
(cherry picked from commit 291d74a46d)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-12-04 08:44:09 +01:00
Laurent Aimar
494cfacdb9 vmd: fix segfaults on corruped streams
Signed-off-by: Janne Grunau <janne-libav@jannau.net>
Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-12-03 21:07:07 +01:00
Anton Khirnov
07624cfeaa Add a version bump and APIchanges entry for avcodec_open2 and avformat_find_stream_info. 2011-11-19 10:22:27 +01:00
Baptiste Coudurier
23f0d0f16b lavc: fix parentheses placement in avcodec_open2().
Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit 1d36fb13b0)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2011-11-19 10:22:27 +01:00
Anton Khirnov
47953c33ea lavc: introduce avcodec_open2() as a replacement for avcodec_open().
Adds support for decoder-private options and makes setting other options
simpler.
(cherry picked from commit 0b950fe240)

Conflicts:

	libavcodec/avcodec.h

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2011-11-19 10:22:26 +01:00
Mans Rullgard
ecda54a640 ppc: fix 32-bit PIC build
On 32-bit ppc, the GOT pointer must be loaded manually.
This adds a "get_got" assembler macro to compute the
GOT address.  The "movrel" macro is updated to take an
additional parameter containing the GOT address since
no register is reserved for this purpose on ppc32.
These changes have no effect on ppc64 builds.

Signed-off-by: Mans Rullgard <mans@mansr.com>
(cherry picked from commit 6e4a35ced9)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-09-26 19:31:32 +02:00
Laurent Aimar
2bbb142a14 rv34: Check for invalid slice offsets
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit 4cc7732386)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2011-09-26 19:26:33 +02:00
Laurent Aimar
b4a1bf0bbf rv34: Fix potential overreads
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit b4ed3d78cb)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2011-09-26 19:26:33 +02:00
Laurent Aimar
f0bcba238a rv34: Avoid NULL dereference on corrupted bitstream
rv34_decode_slice() can return without allocating any pictures.

Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit d0f6ab0298)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2011-09-26 19:26:33 +02:00
Laurent Aimar
28d948ac44 rv10: Reject slices that does not have the same type as the first one
This prevents crashes with some corrupted bitstreams.

Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit 4a29b47186)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2011-09-26 19:26:33 +02:00
Laurent Aimar
54a178f28f Fixed size given to init_get_bits().
init_get_bits() takes a number of bits and not a number of bytes as
its size argument.

Signed-off-by: Alex Converse <alex.converse@gmail.com>
(cherry picked from commit b59efc9434)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2011-09-26 19:26:32 +02:00
Michael Niedermayer
78cd2e18a4 smacker: fix a few off by 1 errors
stereo & 16bit is untested due to lack of samples

Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit 5166376f24)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2011-09-26 19:26:32 +02:00
Laurent Aimar
0d93b03e68 Check for invalid VLC value in smacker decoder.
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit 6489455495)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2011-09-26 19:26:32 +02:00
Laurent Aimar
5b1f79b092 Check and propagate errors when VLC trees cannot be built in smacker decoder.
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit 9676ffba83)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2011-09-26 19:26:32 +02:00
Alex Converse
61ddc8271d xan: Add some buffer checks
(cherry picked from commit 0872bb23b4)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2011-09-26 19:26:32 +02:00
Laurent Aimar
e6694dce1c Fixed size given to init_get_bits() in xan decoder.
(cherry picked from commit 393d5031c6)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2011-09-26 19:26:32 +02:00
Laurent Aimar
9b30b7b9bf Fixed segfault with wavpack decoder on corrupted decorrelation terms sub-blocks.
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit 8bfea4ab4e)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2011-09-26 19:26:32 +02:00
Alex Converse
384ed15c2a cljr: init_get_bits size in bits instead of bytes
(cherry picked from commit 0c1f5b93d9)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2011-09-26 19:26:32 +02:00
Alex Converse
6550e2b5c5 indeo2: fail if input buffer too small
(cherry picked from commit b7ce4f1d1c)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2011-09-26 19:26:31 +02:00
Alex Converse
af32fa929a indeo2: init_get_bits size in bits instead of bytes
(cherry picked from commit 68ca330cbd)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2011-09-26 19:26:31 +02:00
Laurent Aimar
07b3c4cde5 ffv1: Fixed size given to init_get_bits() in decoder.
init_get_bits() takes a number of bits and not a number of bytes as
its size argument.

Signed-off-by: Alex Converse <alex.converse@gmail.com>
(cherry picked from commit 46b004959b)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2011-09-26 19:26:31 +02:00
Alex Converse
5d4c065476 wavpack: Check error codes rather than working around error conditions.
(cherry picked from commit dba2b63a98)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2011-09-26 19:26:31 +02:00
Laurent Aimar
4b84e995ad Fixed invalid access in wavpack decoder on corrupted bitstream.
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit 55354b7de2)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2011-09-26 19:26:31 +02:00
Laurent Aimar
685940da4c Fixed invalid writes in wavpack decoder on corrupted bitstreams.
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit 0aedab0340)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2011-09-26 19:26:31 +02:00
Laurent Aimar
aee461277a Fixed invalid access in wavpack decoder on corrupted extra bits sub-blocks.
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit beefafda63)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2011-09-26 19:26:31 +02:00
Anton Khirnov
a4f2973b2d lavc: fix type for thread_type option
It should be flags, not int.
(cherry picked from commit fb47997edb)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2011-09-26 19:26:31 +02:00
Alberto Delmás
db5e27f94e VC1: Fix first/last row checks with slices
In some places 0/mb_height were used in place of start_mb_y/end_mb_y.

Fixes SA00049, SA00058, SA10091, SA10097, SA10131, SA20021, SA30030

Improves PSNR in SA00054, SA00059, SA00060, SA10096, SA10098, SA20022,
SA30031, SA30032, SA40012, SA40013

Signed-off-by: Ronald S. Bultje <rsbultje@gmail.com>
(cherry picked from commit 1cf82cab08)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2011-09-26 19:26:30 +02:00
Ronald S. Bultje
97ce2a29b6 vc1: properly zero coded_block[] edges on new slice entry.
Previously, we would leave the left edge uninitialized, which led to
CBP prediction errors on slice edges, e.g. in SA10098.vc1.
(cherry picked from commit d4b9974465)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2011-09-26 19:26:30 +02:00
Jeff Downs
ce8f40a7b9 h264: fix PCM intra-coded blocks in monochrome case
Signed-off-by: Diego Biurrun <diego@biurrun.de>
(cherry picked from commit 6581e161c5)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2011-09-26 19:26:30 +02:00
Jeff Downs
45b3f7c71e h264: correct implicit weight table computation for long ref pics
Correct computation of implicit weight tables when referencing pictures
that are marked for long reference.

Signed-off-by: Diego Biurrun <diego@biurrun.de>
(cherry picked from commit 87cf70eb23)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2011-09-26 19:26:30 +02:00
Jeff Downs
8ad6555f82 h264: correct the check for invalid long term frame index in MMCO decode
The current check on MMCO parameters prohibits a "max long term frame index
plus 1" of 16 (frame idx of 15) for the "set max long term frame index" MMCO.
Fix this off-by-one error to allow the full range of legal values.

Signed-off-by: Diego Biurrun <diego@biurrun.de>
(cherry picked from commit 29a09eae9a)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2011-09-26 19:26:30 +02:00
Alex Converse
b4099a6dc5 aac: Only output configure if audio was found.
Audio found is not triggered on a CCE because a CCE alone has no output.

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
(cherry picked from commit d8425ed4af)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2011-09-26 19:26:30 +02:00
Kostya Shishkov
dec458b900 rv10/20: tell decoder to use edge emulation
This removes out-of-edge motion compensation artifacts (easily spotted green
blocks in avplay, gray blocks in transcoding), for example here:
http://samples.libav.org/samples/real/tv_watching_t1.rm

Signed-off-by: Diego Biurrun <diego@biurrun.de>
(cherry picked from commit 331971116d)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2011-09-26 19:26:30 +02:00
Reimar Döffinger
28321b777f VC-1: fix reading of custom PAR.
Custom PAR num/denum are in 1-256 range.

Signed-off-by: Reimar Döffinger <Reimar.Doeffinger@gmx.de>
Signed-off-by: Diego Biurrun <diego@biurrun.de>
(cherry picked from commit 0e86965514)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2011-09-26 19:26:30 +02:00
Dustin Brody
59a22afa0b h264: notice memory allocation failure
Signed-off-by: Ronald S. Bultje <rsbultje@gmail.com>
(cherry picked from commit bac3ab13ea)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2011-09-26 19:26:30 +02:00
Justin Ruggles
042934e786 Remove incorrect info in documentation of AVCodecContext.bits_per_raw_sample.
bits_per_raw_sample is used in video as well, where sample_fmt is not used.
(cherry picked from commit d271d5b215)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2011-09-26 19:26:30 +02:00
Baptiste Coudurier
67163d751b libx264: do not set pic quality if no frame is output
Avoids uninitialized reads.

Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit 5caa2de19e)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2011-09-26 19:26:30 +02:00
Alex Converse
96a453eb85 aac: Remove some suspicious illegal memcpy()s from LTP.
(cherry picked from commit a6c49f18ab)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2011-09-26 19:26:29 +02:00
Ronald S. Bultje
b3b97559bb vp3/theora: flush after seek.
(cherry picked from commit 8dcf518430)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2011-09-26 19:26:29 +02:00
Diego Biurrun
44c718cf71 rv30: return AVERROR(EINVAL) instead of EINVAL
On some platforms EINVAL could be positive, ensure we return negative values.
(cherry picked from commit e5985185d2)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2011-09-26 19:26:29 +02:00
Mans Rullgard
99ec59adbd Fix incorrect max_lowres values
Signed-off-by: Mans Rullgard <mans@mansr.com>
(cherry picked from commit e23a05ab06)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2011-09-26 19:26:29 +02:00
Rafaël Carré
3ed12b97be Do not decode RV30 files if the extradata is too small
Signed-off-by: Diego Biurrun <diego@biurrun.de>
(cherry picked from commit 289c60001f)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2011-09-26 19:26:29 +02:00