On Tue, Jul 07, 2020 at 10:57:26PM +0200, Heinrich Schuchardt wrote:
From: Heiko Stuebner heiko.stuebner@theobroma-systems.com
When calculating rrtmp/rr rsa_gen_key_prop() tries to make (((rlen + 31) >> 5) + 1) steps in the rr uint32_t array and (((rlen + 7) >> 3) + 1) / 4 steps in uint32_t rrtmp[] with rlen being num_bits * 2
On a 4096bit key this comes down to to 257 uint32_t elements in rr and 256 elements in rrtmp but with the current allocation rr and rrtmp only have 129 uint32_t elements.
On 2048bit keys this works by chance as the defined max_rsa_size=4096 allocates a suitable number of elements, but with an actual 4096bit key this results in other memory parts getting overwritten.
So as suggested by Heinrich Schuchardt just use the actual bit-size of the key as base for the size calculation, in turn making the code compatible to any future keysizes.
Suggested-by: Heinrich Schuchardt xypron.debian@gmx.de Signed-off-by: Heiko Stuebner heiko.stuebner@theobroma-systems.com Reviewed-by: Simon Glass sjg@chromium.org
rrtmp needs 2 + (((*prop)->num_bits * 2) >> 5) array elements.
Reviewed-by: Heinrich Schuchardt xypron.glpk@gmx.de
Applied to u-boot/master, thanks!