avcodec/vvc/ctu: check coeff before multiply

ff_vvc_palette_escape_val() can return AVERROR in which case the
coeff*scale will overflow.

Fixes: runtime error: signed integer overflow: -1094995529 * 6528 cannot
       be represented in type 'int'
Fixes: 435225406/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_VVC_fuzzer-5118570024730624
Found-by: OSS-Fuzz
Signed-off-by: Kacper Michajłow <kasper93@gmail.com>
This commit is contained in:
Kacper Michajłow 2025-08-06 18:58:10 +02:00
parent df5199c63f
commit 75f30b4d3e

View file

@ -2054,9 +2054,9 @@ static int palette_subblock_data(VVCLocalContext *lc,
const int v = PALETTE_INDEX(xc, yc);
if (v == esc) {
const int coeff = ff_vvc_palette_escape_val(lc, (1 << sps->bit_depth) - 1);
const int pixel = av_clip_intp2(RSHIFT(coeff * scale, 6), sps->bit_depth);
if (coeff < 0)
return AVERROR_INVALIDDATA;
const int pixel = av_clip_intp2(RSHIFT(coeff * scale, 6), sps->bit_depth);
PALETTE_SET_PIXEL(xc, yc, pixel);
} else {
PALETTE_SET_PIXEL(xc, yc, plt->entries[v]);