Commit graph

187 commits

Author SHA1 Message Date
Harish Mahendrakar
b308dc1318 Fix stack buffer overflow in ih264d_process_intra_mb am: f69e34419b am: cbaa0dc3f9 am: 943e36e510
am: 6a897f0c3a

Change-Id: I75f379b049ee2336b50d0fffe77a6a842896819c
2017-05-10 16:43:41 +00:00
Harish Mahendrakar
cbaa0dc3f9 Fix stack buffer overflow in ih264d_process_intra_mb
am: f69e34419b

Change-Id: I154d8dd94d11be2d34b8085c30e24f6bec62d9eb
2017-05-10 16:20:37 +00:00
Harish Mahendrakar
f69e34419b Fix stack buffer overflow in ih264d_process_intra_mb
Aligned the sizes of au1_ngbr_pels to ensure SSE42 functions do not
result in stack buffer overflow

Bug: 36490809

Change-Id: I0bfe493f94647046013759b3ec9db3c627ac471e
2017-05-10 15:58:24 +00:00
Marco Nelissen
0d7aed6e94 Merge "Decoder: Fix in reference list initialization." into mnc-dev am: 0015860fdc am: 9208f9461f am: c783f2efbe
am: 68c59343e1

Change-Id: I1c765b8f5a0d82e673ecf5e64df2616ffb4ce444
2017-04-06 23:21:12 +00:00
Marco Nelissen
9208f9461f Merge "Decoder: Fix in reference list initialization." into mnc-dev
am: 0015860fdc

Change-Id: I731793974cd3607fed07cb7895234e5f9484a0ea
2017-04-06 23:10:11 +00:00
Marco Nelissen
0015860fdc Merge "Decoder: Fix in reference list initialization." into mnc-dev 2017-04-06 23:00:48 +00:00
Ray Essick
aff1a88f83 Merge "Decoder: Fixes in accessing mbaff flag in error cases" into mnc-dev am: 7ace2f5ca2 am: ba33452066 am: e8faa24c74
am: 57ba056850

Change-Id: I17a61a91bafc8dc1c888b2d084f5b64e2adb01cf
2017-04-05 23:45:06 +00:00
Ray Essick
ba33452066 Merge "Decoder: Fixes in accessing mbaff flag in error cases" into mnc-dev
am: 7ace2f5ca2

Change-Id: I5dde0d284cb06fa2faddae483900df1102d9bc52
2017-04-05 23:36:35 +00:00
Ray Essick
7ace2f5ca2 Merge "Decoder: Fixes in accessing mbaff flag in error cases" into mnc-dev 2017-04-05 23:29:45 +00:00
Ray Essick
24c16eb2b5 Merge "Fix in the case of MMCO 3 (long term reference idx)." into mnc-dev am: 64b617346c am: bfb5f29471 am: 98a81bb017
am: 14276b25d1

Change-Id: Idc8f7d3cc1c5869a11d0abc2bdf1aedf38fa8224
2017-04-05 21:22:26 +00:00
Ray Essick
bfb5f29471 Merge "Fix in the case of MMCO 3 (long term reference idx)." into mnc-dev
am: 64b617346c

Change-Id: Ia57a698ac23ee4abeefe194c8585301941984771
2017-04-05 21:13:56 +00:00
Ray Essick
64b617346c Merge "Fix in the case of MMCO 3 (long term reference idx)." into mnc-dev 2017-04-05 21:09:18 +00:00
Harish Mahendrakar
0f2f2b5fde Decoder: Fixes in accessing mbaff flag in error cases
ps_dec->ps_cur_slice->u1_mbaff_frame_flag is updated in ih264d_start_of_pic().
So updated value should be used after calling ih264d_start_of_pic()

Bug: 33974623
Test: ran POC from bug
Change-Id: I0f1ff5e01ed39767f493f197791e51b0da74952f
(cherry picked from commit 3f6937a003)
2017-03-23 23:46:33 +00:00
Hamsalekha S
93954f5e9a Decoder: Fix in reference list initialization.
In the case of error, initialize the new reference list1 with the first
picture in default list0 instead of default list1, as first picture in
list1 could still be invalid.

Bug: 36035074

Change-Id: I7ab493ee7a157cbefcd4da8389ff1ff899c16b7f
2017-03-22 16:55:01 +00:00
Ray Essick
3a669baf18 Merge "Decoder: Fixed error handling for dangling fields" into mnc-dev am: 78334b2a24 am: cd5ebec24a am: f48987d2fc
am: 1cbcdd7f16

Change-Id: I2adb58066b67a321c9b7275874a7ebb04e8e7c9d
2017-03-14 22:30:50 +00:00
Ray Essick
cd5ebec24a Merge "Decoder: Fixed error handling for dangling fields" into mnc-dev
am: 78334b2a24

Change-Id: Id5cb9826e72c59890526300ad9e8833b61691c20
2017-03-14 22:15:20 +00:00
Ray Essick
78334b2a24 Merge "Decoder: Fixed error handling for dangling fields" into mnc-dev 2017-03-14 22:08:38 +00:00
Hamsalekha S
46e96d40db Fix in the case of MMCO 3 (long term reference idx).
Increment number of long term reference buffers only when both top field
and bottom field have been set as long term.

[backport for M/N from master]

Bug: 35584425
Test: ran POC - no hang, no segfault.

Change-Id: I94e3857944da675eda38f8e1a9bd887f48bff524
(cherry picked from commit 6fa5df8811)
2017-03-04 20:13:09 +00:00
Marco Nelissen
2e4b53cfb4 resolve merge conflicts of 3654ad0 to mnc-dr-dev am: 37345554fe am: 33d9d00a3c
am: 858542d83e

Change-Id: I8376533d60ce908a6a08e260f57cb6163a7fa2dd
2017-02-14 23:01:52 +00:00
Marco Nelissen
37345554fe resolve merge conflicts of 3654ad0 to mnc-dr-dev
Bug: 33818508
Bug: 34013472
Change-Id: I2e99cbceba1c00555d624e8975522725e362362b
2017-02-14 13:49:16 -08:00
Marco Nelissen
3654ad0da5 Merge changes Iec2941f1,I38be0e4c into mnc-dev
* changes:
  Decoder: Add supported width check for MBaff streams
  Decoder: Added a check for unsupported resolutions
2017-02-14 21:30:39 +00:00
Harish Mahendrakar
c011eeade7 Decoder: Fixed initialization of first_slice_in_pic am: 0b23c81c3d am: 16fa990ce1 am: ab79eb3973
am: a37d120eb2

Change-Id: Ib3eb4da620fc964a15a35d8e862b91e55a611bd4
2017-02-14 17:08:36 +00:00
Harish Mahendrakar
df2fa01c9e Decoder: Moved end of pic processing to end of decode call am: 494561291a am: ba7f9e2aed am: 26bb8b98ee
am: 89fa016b5f

Change-Id: Ib8de0752f0e90dfea59d9e8380d2006dd9643312
2017-02-14 17:07:16 +00:00
Harish Mahendrakar
3007985473 Decoder: Treat first slice in a picture as part of new picture always am: 8b5fd8f24e am: 8b75cdae9b am: fb68ae8960
am: a8a3dc84b0

Change-Id: I172945f3e9d84d77ae21d1f737258851529f94c7
2017-02-14 17:07:07 +00:00
Harish Mahendrakar
16fa990ce1 Decoder: Fixed initialization of first_slice_in_pic
am: 0b23c81c3d

Change-Id: I55e49ca9616a151456342bad7673d0986098b448
2017-02-14 17:03:06 +00:00
Harish Mahendrakar
ba7f9e2aed Decoder: Moved end of pic processing to end of decode call
am: 494561291a

Change-Id: Id3867f5dd0788f12f6827ed311a309d1fa58c535
2017-02-14 17:00:16 +00:00
Harish Mahendrakar
8b75cdae9b Decoder: Treat first slice in a picture as part of new picture always
am: 8b5fd8f24e

Change-Id: Ia14e9fae4a08143ff86439b5440ab9cf5e3052e3
2017-02-14 17:00:07 +00:00
Harish Mahendrakar
0b23c81c3d Decoder: Fixed initialization of first_slice_in_pic
To handle some errors, first_slice_in_pic was being set to 2.
This is now cleaned up and first_slice_in_pic is set to 1 only once per pic.
This will ensure picture level initializations are done only once even in case
of error clips

Bug: 33717589
Bug: 33551775
Bug: 33716442
Bug: 33677995

Change-Id: If341436b3cbaa724017eedddd88c2e6fac36d8ba
2017-02-14 16:59:09 +00:00
Harish Mahendrakar
494561291a Decoder: Moved end of pic processing to end of decode call
ih264d_end_of_pic() was called after parsing slice of a new picture.
This is now being done at the end of decode of the current picture.
decode_gaps_in_frame_num which needs frame_num of new slice is now
done after decoding frame_num in new slice.

This helps in handling errors in picaff streams with gaps in frames

Bug: 33588051
Bug: 33641588
Bug: 34097231
Change-Id: I1a26e611aaa2c19e2043e05a210849bd21b22220
2017-02-14 16:58:07 +00:00
Harish Mahendrakar
8b5fd8f24e Decoder: Treat first slice in a picture as part of new picture always
This is needed to decode streams with consecutive IDRs.

Bug: 34097231
Test: successful run of POC in security bug
Change-Id: Ib737a4ef4b8c5bb7a57c90292102dd28af0615fe
2017-02-14 04:41:02 +00:00
Harish Mahendrakar
69b5191865 Decoder: Add supported width check for MBaff streams
In case of MBAff streams, decoder processes two rows at a time,
this limits maximum supported width to 1920 for MBAff streams.

Bug: 33818508
Bug: 34013472
Change-Id: Iec2941f116cf3c36b63013a930319960023a3b42
2017-02-13 15:30:57 -08:00
Harish Mahendrakar
9d60a0c5da Decoder: Added a check for unsupported resolutions
Decoder now returns an error for resolutions greater than 3840x2176

Bug: 24542936

Change-Id: I38be0e4c5cf2a980bfd4c781f3b49171f73b5ccb
2017-02-13 15:30:46 -08:00
Harish Mahendrakar
1f0ab0b089 DO NOT MERGE: Decoder: Fixes an out of bound write in bitstream buffer
[this is for mnc-dev only, not any other mnc-*-dev flavors;
there is a different patch for mnc-dr-dev, nyc-* and going forward]

After emulation prevention, data is written as an int,
so at least 3 additional bytes should be available.
And since bitstream functions read 8 bytes ahead, 8 extra bytes
should be available in the bitstream buffer.

Bug: 33934721
Test: Ittiam testing, POC in the bug no longer fails
Change-Id: I444ec6f85d01b0bade9f827e15c4b476779d6c69
2017-02-13 14:44:11 -08:00
Ray Essick
933d00bfdc Merge "Decoder: Return correct error code for slice header errors" into mnc-dev am: 51ecbcabfb am: cf05624312 am: f0bcc3dfaa
am: 3780d87265

Change-Id: I16d98f94966bbcdc80205cb838b2c8b6ae37fa4e
2017-02-13 19:29:08 +00:00
Ray Essick
ec93be1ae9 Merge "Decoder: Initialize default reference buffers for all pictures" into mnc-dev am: d61abc5195 am: 16306f7347 am: 0fbfecd7ce
am: 6fe876a014

Change-Id: I43344486e8b3b0bc1f7a905969d9f9bed133822b
2017-02-13 19:28:57 +00:00
Harish Mahendrakar
2785504d50 Fix in returning end of bitstream error for MBAFF am: 7950bf47b6 am: 73f74a84e1 am: f23ebcfab1
am: c8b03b10df

Change-Id: Ife3b0ade038472f264b21298313614f01d3fcd59
2017-02-13 19:28:49 +00:00
Ray Essick
cf05624312 Merge "Decoder: Return correct error code for slice header errors" into mnc-dev
am: 51ecbcabfb

Change-Id: Ib486869b38d66e1fb0fdeffe4316dce6a5ff9fd9
2017-02-13 19:21:06 +00:00
Ray Essick
16306f7347 Merge "Decoder: Initialize default reference buffers for all pictures" into mnc-dev
am: d61abc5195

Change-Id: Ieb21ecb1c3152bde97adc7cfcfc4d49afcafd93b
2017-02-13 19:20:57 +00:00
Harish Mahendrakar
73f74a84e1 Fix in returning end of bitstream error for MBAFF
am: 7950bf47b6

Change-Id: If45518b6d58c393313dbcb425f7aa2e7bec02056
2017-02-13 19:20:47 +00:00
Ray Essick
51ecbcabfb Merge "Decoder: Return correct error code for slice header errors" into mnc-dev 2017-02-13 19:15:36 +00:00
Ray Essick
d61abc5195 Merge "Decoder: Initialize default reference buffers for all pictures" into mnc-dev 2017-02-13 19:15:31 +00:00
Harish Mahendrakar
9a00f562a6 Decoder: Return correct error code for slice header errors
Return ERROR_INV_SLICE_HDR_T instead of ERROR_INV_SPS_PPS_T for slice
header errors.

Bug: 34097915
Change-Id: I45d14a71f2322ff349058baaf65fb0f3c1140fba
2017-02-13 19:14:59 +00:00
Harish Mahendrakar
f634481e94 Decoder: Initialize default reference buffers for all pictures
Reference buffer is now initialized to default value for
each pic before decoding the first slice in the pic

Bug: 34097866
Change-Id: Id64b123af2188217ce833f11db0e6c0681d41dfd
2017-02-13 19:14:50 +00:00
Harish Mahendrakar
7950bf47b6 Fix in returning end of bitstream error for MBAFF
In case of MBAFF streams, slices should terminate on
even MB boundary. If bytes are exhausted with odd number
of MBs decoded for MBAff, then treat that as error.

Bug: 33933140

Change-Id: Ifc26b66ff8ebdb3aec5c0d6c512e4cac3f54c5b7
2017-02-13 19:14:18 +00:00
Harish Mahendrakar
73a3e320d8 Decoder: Fixes an out of bound write in bitstream buffer am: 33ef7de9dd am: 0aab4958fb
am: 9f72b6eca1

Change-Id: I3f25b86b4ad13ad3b7e9a85e51789145d4b1b0ab
2017-02-13 18:49:46 +00:00
Harish Mahendrakar
33ef7de9dd Decoder: Fixes an out of bound write in bitstream buffer
[for mnc-dr-dev and later;  mnc-dev gets a different patch]

After emulation prevention, data is written as an int,
so at least 3 additional bytes should be available.
And since bitstream functions read 8 bytes ahead, 8 extra bytes
should be available in the bitstream buffer.

Bug: 33934721

Change-Id: I444ec6f85d01b0bade9f827e15c4b476779d6c69
2017-01-19 09:17:00 -08:00
Harish Mahendrakar
f8b3e57266 Decoder: Initialize ps_cur_slice->u1_mbaff_frame_flag correctly for error cases am: 1d5640f2f9 am: 5a26569fa3 am: 18a58ec2b6
am: 575fdb281f

Change-Id: I735691b67ab0b41a8560b0bdd9aaaa5a3947b5d7
2017-01-18 21:17:41 +00:00
Harish Mahendrakar
5a26569fa3 Decoder: Initialize ps_cur_slice->u1_mbaff_frame_flag correctly for error cases
am: 1d5640f2f9

Change-Id: I32e960c752ccbc991f8ce04455618a534f1f52c7
2017-01-18 21:11:11 +00:00
Ray Essick
43a184ad48 Merge "Decoder: Initialize ps_cur_slice->u1_mbaff_frame_flag correctly for error cases" into mnc-dev 2017-01-18 21:00:19 +00:00
Marco Nelissen
d5424f66e3 Merge "DO NOT MERGE Decoder: Increase memory allocation for weights & offsets for interlaced clips" into mnc-dev 2017-01-18 18:42:47 +00:00