changing CONFIG_BAUDRATE to 9600 but it has no effect, it just stays at 115200. Any ideas?

Note that that CONFIG parameter is ONLY used if there is no baudrate settin on the environment in flash. Change the environment variable, and it should work. (Same thing with CONFIG_CLOCK_IN_MHZ....)