Commit graph

24158 commits

Author SHA1 Message Date
Michael Niedermayer
57eb787ed3 Merge remote-tracking branch 'qatar/release/0.6' into release/0.6
* qatar/release/0.6: (58 commits)
  Bump version number for 0.6.4 release.
  qdm2: check output buffer size before decoding
  Fix qdm2 decoder packet handling to match the api
  4xm: Add a check in decode_i_frame to prevent buffer overreads
  wma: initialize prev_block_len_bits, next_block_len_bits, and block_len_bits.
  swscale: #include "libavutil/mathematics.h"
  vp3dec: Check coefficient index in vp3_dequant()
  svq1dec: call avcodec_set_dimensions() after dimensions changed.
  vp6: Fix illegal read.
  vp6: Fix illegal read.
  vp6: Reset the internal state when aborting key frames header parsing
  vp6: Check for huffman tree build errors
  vp6: partially propagate huffman tree building errors during coeff model parsing and fix misspelling
  Fix out of bound reads in the QDM2 decoder.
  Check for out of bound writes in the QDM2 decoder.
  vmd: fix segfaults on corruped streams
  rv34: Check for invalid slice offsets
  rv34: Fix potential overreads
  rv34: Avoid NULL dereference on corrupted bitstream
  rv10: Reject slices that does not have the same type as the first one
  ...

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2011-12-25 01:24:40 +01:00
Reinhard Tartler
dbe7e209df Bump version number for 0.6.4 release. 2011-12-24 15:59:10 +01:00
Justin Ruggles
cfb9b47a1e qdm2: check output buffer size before decoding
(cherry picked from commit 7d49f79f1c)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
(cherry picked from commit 7347205351)

Conflicts:

	libavcodec/qdm2.c
2011-12-24 15:57:17 +01:00
Baptiste Coudurier
b26c1a8b7e Fix qdm2 decoder packet handling to match the api
Originally committed as revision 25767 to svn://svn.ffmpeg.org/ffmpeg/trunk
2011-12-24 15:54:51 +01:00
Shitiz Garg
ccd2ca0246 4xm: Add a check in decode_i_frame to prevent buffer overreads
Fixes bugzilla #135

Signed-off-by: Janne Grunau <janne-libav@jannau.net>
(cherry picked from commit 355d917c0b)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
(cherry picked from commit d912a30c7d)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-12-24 15:47:57 +01:00
Justin Ruggles
92b964969b wma: initialize prev_block_len_bits, next_block_len_bits, and block_len_bits.
The initial values are not checked against the number of block sizes.
Initializing them to frame_len_bits will result in a block size index of 0
in these cases instead of something that might be out-of-range.

Fixes Bug 81.
(cherry picked from commit 05d1e45d1f)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
(cherry picked from commit 8dba5608dc)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-12-24 15:47:57 +01:00
Reinhard Tartler
ca87ec53e9 swscale: #include "libavutil/mathematics.h"
this file uses the M_PI macro since
4e74187db2, so include the correct header
directly.

Signed-off-by: Reinhard Tartler <siretart@tauware.de>

(cherry picked from commit 5089ce1b5a)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
(cherry picked from commit 851098c9e0)

Conflicts:

	libswscale/utils.c
2011-12-24 15:47:57 +01:00
Reinhard Tartler
bd071de29a vp3dec: Check coefficient index in vp3_dequant()
Based on a patch by Michael Niedermayer <michaelni@gmx.at>

Fixes NGS00145, CVE-2011-4352

Found-by: Phillip Langlois
Signed-off-by: Reinhard Tartler <siretart@tauware.de>

(cherry picked from commit 8b94df0f20)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
(cherry picked from commit bba709214a)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-12-24 15:47:57 +01:00
Michael Niedermayer
8ddc0b491d svq1dec: call avcodec_set_dimensions() after dimensions changed.
Fixes NGS00148, CVE-2011-4579

Found-by: Phillip Langlois
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

(cherry picked from commit 6e24b9488e)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
(cherry picked from commit 0eca0da06e)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-12-24 15:47:57 +01:00
Thierry Foucu
94aacaf508 vp6: Fix illegal read.
Found with Address Sanitizer

Signed-off-by: Alex Converse <alex.converse@gmail.com>
(cherry picked from commit e0966eb140)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
(cherry picked from commit ba4b08b789)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-12-24 15:47:57 +01:00
Alex Converse
8d68083298 vp6: Fix illegal read.
(cherry picked from commit 2a6eb06254)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
(cherry picked from commit 67a7ed623b)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-12-24 15:47:57 +01:00
Laurent Aimar
e28bb18fdc vp6: Reset the internal state when aborting key frames header parsing
It prevents leaving the state only half initialized.

Signed-off-by: Janne Grunau <janne-libav@jannau.net>
(cherry picked from commit a72cad0a6c)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
(cherry picked from commit c76505e0de)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-12-24 15:47:57 +01:00
Laurent Aimar
a62779d986 vp6: Check for huffman tree build errors
Signed-off-by: Janne Grunau <janne-libav@jannau.net>
(cherry picked from commit 066fff755a)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
(cherry picked from commit 30c08e2261)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-12-24 15:47:57 +01:00
Dustin Brody
201fcfb894 vp6: partially propagate huffman tree building errors during coeff model parsing and fix misspelling
Signed-off-by: Ronald S. Bultje <rsbultje@gmail.com>
(cherry picked from commit f913eeea43)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
(cherry picked from commit 7367cbec1b)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-12-24 15:47:56 +01:00
Laurent Aimar
8856c4c5c9 Fix out of bound reads in the QDM2 decoder.
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
Signed-off-by: Justin Ruggles <justin.ruggles@gmail.com>
(cherry picked from commit 5a19acb17c)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
(cherry picked from commit 0d93d5c461)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-12-24 15:17:00 +01:00
Laurent Aimar
0f7bf1786e Check for out of bound writes in the QDM2 decoder.
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
Signed-off-by: Justin Ruggles <justin.ruggles@gmail.com>
(cherry picked from commit 291d74a46d)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
(cherry picked from commit a31ccacb1a)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-12-24 15:16:51 +01:00
Laurent Aimar
b99366faef vmd: fix segfaults on corruped streams
Signed-off-by: Janne Grunau <janne-libav@jannau.net>
Signed-off-by: Reinhard Tartler <siretart@tauware.de>
(cherry picked from commit 494cfacdb9)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-12-24 15:16:36 +01:00
Laurent Aimar
da0900e8bb rv34: Check for invalid slice offsets
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit 4cc7732386)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit 2bbb142a14)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-12-24 12:20:33 +01:00
Laurent Aimar
d5551d7884 rv34: Fix potential overreads
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit b4ed3d78cb)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit b4a1bf0bbf)

Conflicts:

	libavcodec/rv34.c

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-12-24 12:20:33 +01:00
Laurent Aimar
35f1888585 rv34: Avoid NULL dereference on corrupted bitstream
rv34_decode_slice() can return without allocating any pictures.

Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit d0f6ab0298)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-12-24 12:20:33 +01:00
Laurent Aimar
7cd7461ec8 rv10: Reject slices that does not have the same type as the first one
This prevents crashes with some corrupted bitstreams.

Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit 4a29b47186)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit 28d948ac44)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-12-24 12:20:33 +01:00
Laurent Aimar
23f622de60 oggdec: fix out of bound write in the ogg demuxer
Between ogg_save() and ogg_restore() calls, the number of streams
could have been reduced.

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
(cherry picked from commit 0e7efb9d23)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit a3d471e500)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-12-24 12:20:33 +01:00
Michael Niedermayer
19a99b6e6b smacker: fix a few off by 1 errors
stereo & 16bit is untested due to lack of samples

Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit 5166376f24)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit 78cd2e18a4)

Conflicts:

	libavcodec/smacker.c

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-12-24 12:20:33 +01:00
Laurent Aimar
117e04cdfa Check for invalid VLC value in smacker decoder.
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit 6489455495)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-12-24 12:20:33 +01:00
Laurent Aimar
5d6fe49ac9 Check and propagate errors when VLC trees cannot be built in smacker decoder.
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit 9676ffba83)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-12-24 12:20:33 +01:00
Laurent Aimar
9f28eede5e Fixed off by one packet size allocation in the smacker demuxer.
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit a92d0fa5d2)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-12-24 12:20:33 +01:00
Laurent Aimar
6f70111e81 Check for invalid packet size in the smacker demuxer.
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit e055932f56)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-12-24 12:20:33 +01:00
Laurent Aimar
4492523938 ape demuxer: fix segfault on memory allocation failure.
Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit 273aab99bf)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit 4ee014309c)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-12-24 12:20:33 +01:00
Laurent Aimar
a97e82c487 Fixed size given to init_get_bits() in xan decoder.
(cherry picked from commit 393d5031c6)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-12-24 12:20:33 +01:00
Kostya Shishkov
f79f3a946f smacker demuxer: handle possible av_realloc() failure.
Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit 47a8589f7b)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit 0b9b3570a3)

Conflicts:

	libavformat/smacker.c

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-12-24 12:20:33 +01:00
Laurent Aimar
5394cdf775 Fixed segfault with wavpack decoder on corrupted decorrelation terms sub-blocks.
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit 8bfea4ab4e)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-12-24 12:20:33 +01:00
Alex Converse
7f163e5a87 indeo2: fail if input buffer too small
(cherry picked from commit b7ce4f1d1c)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-12-24 12:20:33 +01:00
Alex Converse
23999c45bc indeo2: init_get_bits size in bits instead of bytes
(cherry picked from commit 68ca330cbd)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-12-24 12:20:33 +01:00
Alex Converse
14fae6eab0 wavpack: Check error codes rather than working around error conditions.
(cherry picked from commit dba2b63a98)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit 5d4c065476)

Conflicts:

	libavcodec/wavpack.c

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-12-24 12:20:33 +01:00
Laurent Aimar
f5a8c4242e Fixed invalid writes in wavpack decoder on corrupted bitstreams.
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit 0aedab0340)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit 685940da4c)

Conflicts:

	libavcodec/wavpack.c

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-12-24 12:20:32 +01:00
Laurent Aimar
1edc513bcf Fixed invalid access in wavpack decoder on corrupted bitstream.
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit 55354b7de2)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit 4b84e995ad)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-12-24 12:20:32 +01:00
Laurent Aimar
9864518544 Fixed invalid access in wavpack decoder on corrupted extra bits sub-blocks.
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit beefafda63)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-12-24 12:20:32 +01:00
Sean McGovern
4ccb8f5b7b cpu detection: avoid a signed overflow
1<<31 overflows because 1 is signed, so force it to unsigned.

Signed-off-by: Ronald S. Bultje <rsbultje@gmail.com>
(cherry picked from commit 5938e02185)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-12-24 12:20:32 +01:00
Jeff Downs
01b9a6e447 h264: correct implicit weight table computation for long ref pics
Correct computation of implicit weight tables when referencing pictures
that are marked for long reference.

Signed-off-by: Diego Biurrun <diego@biurrun.de>
(cherry picked from commit 87cf70eb23)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-12-24 12:20:32 +01:00
Jeff Downs
34d2fe6860 h264: correct the check for invalid long term frame index in MMCO decode
The current check on MMCO parameters prohibits a "max long term frame index
plus 1" of 16 (frame idx of 15) for the "set max long term frame index" MMCO.
Fix this off-by-one error to allow the full range of legal values.

Signed-off-by: Diego Biurrun <diego@biurrun.de>
(cherry picked from commit 29a09eae9a)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-12-24 12:20:32 +01:00
Kostya Shishkov
e7746f834a rv10/20: tell decoder to use edge emulation
This removes out-of-edge motion compensation artifacts (easily spotted green
blocks in avplay, gray blocks in transcoding), for example here:
http://samples.libav.org/samples/real/tv_watching_t1.rm

Signed-off-by: Diego Biurrun <diego@biurrun.de>
(cherry picked from commit 331971116d)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-12-24 12:20:32 +01:00
Luca Barbato
973bdafe05 flvenc: use int64_t to store offsets
Metadata currently is written only at the start of the file in normal
cases, when transcoding from a rtmp source metadata could be
written later and the offset recorded can exceed 32bit.

Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit 7f5bf4fbaf)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit fe3e7297fe)

Conflicts:

	libavformat/flvenc.c

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-12-24 12:20:32 +01:00
Reimar Döffinger
f4a5a730d8 VC-1: fix reading of custom PAR.
Custom PAR num/denum are in 1-256 range.

Signed-off-by: Reimar Döffinger <Reimar.Doeffinger@gmx.de>
Signed-off-by: Diego Biurrun <diego@biurrun.de>
(cherry picked from commit 0e86965514)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-12-24 12:20:32 +01:00
Michael Niedermayer
603a282f8f Merge remote-tracking branch 'qatar/release/0.6' into release/0.6
* qatar/release/0.6:
  update version
  Release notes and changelog for 0.6.3
  Fix memory (re)allocation in matroskadec.c, related to MSVR-11-0080.
  cavs: fix some crashes with invalid bitstreams
  libvo-aacenc: Sync up with 0.7.2
  Fix MMX rgb24 to yuv conversion with gcc 4.6
  oggdec: prevent heap corruption.
  Fix ff_imdct_calc_sse() on gcc-4.6
  backport libvo-aacenc wrapper for aac encoding

Conflicts:
	Changelog
	VERSION

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2011-11-06 01:50:29 +01:00
Dustin Brody
36c196bca4 h264: notice memory allocation failure
Signed-off-by: Ronald S. Bultje <rsbultje@gmail.com>
(cherry picked from commit bac3ab13ea)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit 59a22afa0b)

Conflicts:

	libavcodec/h264.c

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-11-05 13:18:32 +01:00
Baptiste Coudurier
7b733e4b7f libx264: do not set pic quality if no frame is output
Avoids uninitialized reads.

Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit 5caa2de19e)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-11-05 13:18:32 +01:00
Alex Converse
e07086d8ae mxfdec: Include FF_INPUT_BUFFER_PADDING_SIZE when allocating extradata.
This prevents out of bounds reads when extradata is being decoded.
(cherry picked from commit 1f6f58d585)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-11-05 13:18:32 +01:00
Diego Biurrun
d2411412f1 rv30: return AVERROR(EINVAL) instead of EINVAL
On some platforms EINVAL could be positive, ensure we return negative values.
(cherry picked from commit e5985185d2)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-11-05 13:18:32 +01:00
Rafaël Carré
0facc63ff6 Do not decode RV30 files if the extradata is too small
Signed-off-by: Diego Biurrun <diego@biurrun.de>
(cherry picked from commit 289c60001f)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-11-05 13:18:32 +01:00
Mans Rullgard
5ed9457260 aacps: skip some memcpy() if src and dst would be equal
Signed-off-by: Mans Rullgard <mans@mansr.com>
(cherry picked from commit e5902d60ce)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2011-11-05 13:18:32 +01:00