When transquant bypass is enabled in HBD streams, decoded frames
mismatched the reference output due to two issues:
1. Intra boundary filtering:
In the 8-bit path, boundary filtering for horizontal (mode 10) and
vertical (mode 26) intra predictions is disabled when transquant bypass
is active with implicit RDPCM. However, the HBD reconstruction path was
passing raw u1_luma_pred_mode instead of disable_boundary_filter, and the
HBD filter functions ignored the mode parameter.
2. SAO Scratch Buffer Overflow:
During shifted CTB SAO processing, transquant bypass blocks are temporarily
backed up into pu1_tmp_buf_luma and restored afterwards to keep them lossless.
In HBD, each sample requires 2 bytes (UWORD16), but the static scratch buffer
was allocated for 8-bit samples (sizeof(UWORD8)).
Because the shifted CTB processing window spans beyond standard CTB boundaries
to handle delayed neighbor boundaries, the 16-bit luma backup exceeded the
allocated buffer capacity and overflowed into pu1_tmp_buf_chroma. Consequently,
chroma backup overwrote the luma backup data, resulting in corrupted samples
upon restoration.
This commit resolves the above issues with the following changes:
- common/ihevc_hbd_intra_pred_filters.c: Check 'disable_boundary_filter' to bypass
boundary filtering in ihevc_hbd_intra_pred_luma_horz and ihevc_hbd_intra_pred_luma_ver.
- decoder/ihevcd_iquant_itrans_recon_ctb.c: Pass disable_boundary_filter
to apf_hbd_intra_pred_luma for modes 10 and 26.
- decoder/ihevcd_api.c: Scale SAO temporary scratch buffer allocation
(pu1_tmp_buf_luma and pu1_tmp_buf_chroma) by sizeof(UWORD16) to prevent
buffer collision in HBD mode.