Here is the boot log:
===== U-Boot 2020.10 (Nov 28 2020 - 18:14:31 +0000)
DRAM: 3.9 GiB RPI 4 Model B (0xc03111) MMC: mmcnr@7e300000: 1, emmc2@7e340000: 0 Loading Environment from FAT... *** Warning - bad CRC, using default environment
In: serial Out: serial Err: serial Net: eth0: ethernet@7d580000 PCIe BRCM: link up, 5.0 Gbps x1 (SSC) starting USB... Bus usb@7e980000: USB DWC2 Bus xhci_pci: Register 5000420 NbrPorts 5 Starting the controller USB XHCI 1.00 scanning bus usb@7e980000 for devices... 1 USB Device(s) found scanning bus xhci_pci for devices... WARN halted endpoint, queueing URB anyway. Unexpected XHCI event TRB, skipping... (3db6b570 00000000 13000000 02008401) BUG at drivers/usb/host/xhci-ring.c:498/abort_td()! BUG! resetting ... =====
Unplugging the USB keyboard fixed it. It seems to also happen for other kinds of USB devices: https://lists.denx.de/pipermail/u-boot/2020-November/433620.html