[U-Boot] Unable to compile U-Boot for Tegra K1 with modified SPL_TEXT_BASE

I am trying to build latest U-Boot for my custom board based on Tegra TK1. The downstream U-Boot provided by Nvidia builds and works fine on my board. But when I try to build mainline U-Boot and flash it to my board using tegra-uboot-flasher scripts, I get no output when I boot the board.

Then I noticed that the downstream U-Boot has CONFIG_SPL_TEXT_BASE set to 0x83D88000 while in mainline U-Boot, it is set to 0x80108000. When I tried to modify this to any value other than 0x80108000, I get following error while building:

binman: 'I' format requires 0 <= number <= 4294967295

Can someone guide what can I do to remove this error and get the latest U-Boot working on Tegra K1?