Great idea Ulf.

We (Xerox Corp) load the FPGAs in uBoot because they have the PCIe interface "hardware" and Linux comes up much easier if the PCIe hardware exists and is operational on boot instead of attempting to add it it later when Linux is up and you have to do a hot plug install.

If we have FPGAs that don't have PCIe hardware, then we load them from Linux after the board is booted.

The FPGA images are loaded from a network server by fetching the file using TFTP instead of a SPI device which adds cost and complexity to the hardware design. They are then load into the FPGA using a serial bitblt process.

Yes, it does slow down the boot by a couple of seconds, but life is all about trade offs, and for us this is a good one.

Rob Westfall Xerox Corporation

On Mon, Sep 14, 2020 at 7:49 AM Michal Simek monstr@monstr.eu wrote:

Hi,

On 07. 09. 20 19:03, Ulf Samuelsson wrote:

...

Hi, The FPGA support in u-boot looks a bit old-fashioned to me. I would like to have device-tree support and would like comments on my approach.

Definitely there is a space for improvement. In connection to DT - we should likely create fpga DM class for it.

...

==================================================================== BACKGROUND Need to support loading 2 x Cyclone-10 LP FPGAs using Passive Serial. Raw binary files (RBF) are stored in boot flash, and u-boot will load both FPGAs before booting the OS. The OS should be able to update the FPGAs as well. The FPGAs must be independently configurable.

Hardwarewise, I have an SPI bus with 4 chip selects. SPI MOSI, SCK, CS2 and CS3 are connected to the synchronous serial bus of the FPGA (DATA0, DCLK, the FPGA1.nCS0 and the FPGA2.nCS0)

Intel normally connect FPGA nCS0 to ground but I connect this to the SPI CS. The advantage is that I can have several FPGAs which are programmed independent to each other.

The FPGAs require three other signals (nCONFIG, nSTATUS and CONF_DONE). Each FPGA have their own set of those signals. ====================================================================

I have not found device-tree support for FPGAs in u-boot. Since the FPGAs are connected to the SPI, I thought adding them as MTD devices would result in a nice interface.

==================================================================== Example Device tree connected to a bit-bang SPI driver supporting multiple chip select.

The MTD configuration supplies

• Name of the FPGA (fpga property)
• The size of the configuration data
• Sideband signals for starting and checking configuration.

&soft_mcspi { compatible = "mcspi-gpio"; pinctrl-names = "default"; pinctrl-0 = <&soft_mcspi1_pins>; status = "okay"; gpio-sck = <&gpio0 7 GPIO_ACTIVE_LOW>; gpio-miso = <&gpio1 8 GPIO_ACTIVE_HIGH>; gpio-mosi = <&gpio1 9 GPIO_ACTIVE_HIGH>; cs-gpios = <&gpio0 12 GPIO_ACTIVE_LOW>, <&gpio0 13 GPIO_ACTIVE_LOW>, <&gpio0 17 GPIO_ACTIVE_LOW>, <&gpio0 16 GPIO_ACTIVE_LOW>; num-chipselects = <4>; #address-cells = <1>; #size-cells = <0>; ... sspy-fpga-cfg@3 { /* Intel Cyclone 10, 10CL025 */ #address-cells = <1>; #size-cells = <1>; compatible = "intel,cyclone10"; reg = <3>; /* Chip select 0 */ spi-max-frequency = <1000000>; fpga = "spyf"; config-size = <718569>; gpio-nconfig = <&gpio3 21 GPIO_ACTIVE_HIGH>; gpio-nstatus = <&gpio3 17 GPIO_ACTIVE_HIGH>; gpio-conf_done = <&gpio0 18 GPIO_ACTIVE_HIGH>; gpio-crc_error = <&gpio2 18 GPIO_ACTIVE_HIGH>; }; }; ====================================================================

With a Cyclone 10 MTD driver, is would appear in the "mtd list" command and configuring the fpga would be done using

$ mtd write.raw spyf <ram-addr>

The driver would have to implement dummy read and erase commands.

An alternative would be to add flags which would be tested by the mtd command which would block the read and erase if the mtd device is an FPGA or simply test for that in the mtd command.

What would be best?

The MTD driver would start configuration by toggling gpio-nconfig low and then wait for gpio-nstatus to go high.

If OK, it would call the spi_xfer routine for the parent spi slave (soft-mcspi)

When the spi write returns, the FPGA MTD driver would await the CONF_DONE signal before returning.

====================================================================

What do you guys think of such an approach?

You are writing that fpga should be loaded before OS. You likely have any reason for it but I think that make sense to take a look at it from different angle. You have 4 fpgas and when you load all of them in u-boot you are slowing down boot. Anyway I think that make sense to take a look at Linux first. https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/dri...

I see 4 spi based drivers where I expect none is using MTD layer for programming. You will find out dt binding for these devices and the same binding should be used also in u-boot.

Thanks, Michal

-- Michal Simek, Ing. (M.Eng), OpenPGP -> KeyID: FE3D1F91 w: www.monstr.eu p: +42-0-721842854 Maintainer of Linux kernel - Xilinx Microblaze Maintainer of Linux kernel - Xilinx Zynq ARM and ZynqMP ARM64 SoCs U-Boot custodian - Xilinx Microblaze/Zynq/ZynqMP/Versal SoCs

Den 2020-09-14 kl. 13:49, skrev Michal Simek:

Hi,

On 07. 09. 20 19:03, Ulf Samuelsson wrote:

...

Hi, The FPGA support in u-boot looks a bit old-fashioned to me. I would like to have device-tree support and would like comments on my approach.

Definitely there is a space for improvement. In connection to DT - we should likely create fpga DM class for it.

I do not think that it is neccessary. At least I did not need it to implement passive serial programming.

I have it working right now, and can use the MTD command and just add the FPGA as a device under the the SPI bus.

If a DM class is added, then either the MTD command needs to be updated, or a new command developed.

...

==================================================================== BACKGROUND Need to support loading 2 x Cyclone-10 LP FPGAs using Passive Serial. Raw binary files (RBF) are stored in boot flash, and u-boot will load both FPGAs before booting the OS. The OS should be able to update the FPGAs as well. The FPGAs must be independently configurable.

Hardwarewise, I have an SPI bus with 4 chip selects. SPI MOSI, SCK, CS2 and CS3 are connected to the synchronous serial bus of the FPGA (DATA0, DCLK, the FPGA1.nCS0 and the FPGA2.nCS0)

Intel normally connect FPGA nCS0 to ground but I connect this to the SPI CS. The advantage is that I can have several FPGAs which are programmed independent to each other.

The FPGAs require three other signals (nCONFIG, nSTATUS and CONF_DONE). Each FPGA have their own set of those signals. ====================================================================

I have not found device-tree support for FPGAs in u-boot. Since the FPGAs are connected to the SPI, I thought adding them as MTD devices would result in a nice interface.

==================================================================== Example Device tree connected to a bit-bang SPI driver supporting multiple chip select.

The MTD configuration supplies

• Name of the FPGA (fpga property)
• The size of the configuration data
• Sideband signals for starting and checking configuration.

&soft_mcspi {     compatible = "mcspi-gpio";     pinctrl-names = "default";     pinctrl-0 = <&soft_mcspi1_pins>;     status = "okay";     gpio-sck  = <&gpio0 7 GPIO_ACTIVE_LOW>;     gpio-miso = <&gpio1 8 GPIO_ACTIVE_HIGH>;     gpio-mosi = <&gpio1 9 GPIO_ACTIVE_HIGH>;     cs-gpios =        <&gpio0 12 GPIO_ACTIVE_LOW>,        <&gpio0 13 GPIO_ACTIVE_LOW>,        <&gpio0 17 GPIO_ACTIVE_LOW>,        <&gpio0 16 GPIO_ACTIVE_LOW>;     num-chipselects = <4>;     #address-cells = <1>;     #size-cells = <0>;     ...     sspy-fpga-cfg@3 {    /* Intel Cyclone 10, 10CL025 */         #address-cells = <1>;         #size-cells = <1>;         compatible = "intel,cyclone10";         reg = <3>; /* Chip select 0 */         spi-max-frequency = <1000000>;         fpga = "spyf";         config-size    = <718569>;         gpio-nconfig    = <&gpio3 21 GPIO_ACTIVE_HIGH>;         gpio-nstatus    = <&gpio3 17 GPIO_ACTIVE_HIGH>;         gpio-conf_done    = <&gpio0 18 GPIO_ACTIVE_HIGH>;         gpio-crc_error    = <&gpio2 18 GPIO_ACTIVE_HIGH>;     }; }; ====================================================================

With a Cyclone 10 MTD driver, is would appear in the "mtd list" command and configuring the fpga would be done using

$ mtd write.raw spyf <ram-addr>

The driver would have to implement dummy read and erase commands.

An alternative would be to add flags which would be tested by the mtd command which would block the read and erase if the mtd device is an FPGA or simply test for that in the mtd command.

What would be best?

The MTD driver would start configuration by toggling gpio-nconfig low and then wait for gpio-nstatus to go high.

If OK, it would call the spi_xfer routine for the parent spi slave (soft-mcspi)

When the spi write returns, the FPGA MTD driver would await the CONF_DONE signal before returning.

====================================================================

What do you guys think of such an approach?

You are writing that fpga should be loaded before OS. You likely have any reason for it but I think that make sense to take a look at it from different angle. You have 4 fpgas and when you load all of them in u-boot you are slowing down boot.

The FPGA goes into a system where the FPGAs needs to be running as soon as possible. It cannot wait for linux to boot.

We only have two FPGAs each on a separate chip select (2 & 3). One of the FPGAs contains peripherals accessible over SPI, so chip select 0 goes to a user SPI slave interface inside this FPGA. SPI chipselect 1 goes to a dedicated SPI peripheral.

================= I am currently using a bitbanging driver and get programming times of 3 & 9 seconds. They are not particularily large so they should load in less than a second with a H/W driver. The H/W driver we are using (omap-spi) does not support GPIO chip selects, and only two are supported so it needs to be extended.

I hope to do the load during the bootdelay. =================

Anyway I think that make sense to take a look at Linux first. https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/dri...

I see 4 spi based drivers where I expect none is using MTD layer for programming. You will find out dt binding for these devices and the same binding should be used also in u-boot.

I looked at the linux drivers and the altera-ps-spi.c is the correct starting point. It only supports cyclone 5 and Arria 10 and I am using Cyclone 10. The linux drivers needs to have Cyclone 10 support added.

The main difference is delay timing, not functionality..

I also found a device tree using this: imx6q-evi.dts It is the only one using the driver.

+++++++ &ecspi1 { cs-gpios = <&gpio4 10 GPIO_ACTIVE_LOW>; pinctrl-names = "default"; pinctrl-0 = <&pinctrl_ecspi1 &pinctrl_ecspi1cs>; status = "okay";

fpga: fpga@0 { compatible = "altr,fpga-passive-serial"; spi-max-frequency = <20000000>; reg = <0>; pinctrl-0 = <&pinctrl_fpgaspi>; nconfig-gpios = <&gpio4 9 GPIO_ACTIVE_LOW>; nstat-gpios = <&gpio4 11 GPIO_ACTIVE_LOW>; }; };

This does not use the confd pin, but the driver supports it. -------

compared to my current: &soft_mcspi { compatible = "mcspi-gpio"; pinctrl-names = "default"; pinctrl-0 = <&soft_mcspi1_pins>; status = "okay"; gpio-sck = <&gpio0 7 GPIO_ACTIVE_LOW>; gpio-miso = <&gpio1 8 GPIO_ACTIVE_HIGH>; gpio-mosi = <&gpio1 9 GPIO_ACTIVE_HIGH>; cs-gpios = <&gpio0 12 GPIO_ACTIVE_LOW>, <&gpio0 13 GPIO_ACTIVE_LOW>, <&gpio0 17 GPIO_ACTIVE_LOW>, <&gpio0 16 GPIO_ACTIVE_LOW>; num-chipselects = <4>; #address-cells = <1>; #size-cells = <0>; ... sspy-fpga-cfg@3 { /* Intel Cyclone 10, 10CL025 */ #address-cells = <1>; #size-cells = <1>; compatible = "intel,cyclone10"; reg = <3>; /* Chip select 0 */ spi-max-frequency = <1000000>; fpga = "spyf"; config-size = <718569>; gpio-nconfig = <&gpio3 21 GPIO_ACTIVE_HIGH>; gpio-nstatus = <&gpio3 17 GPIO_ACTIVE_HIGH>; gpio-conf_done = <&gpio0 18 GPIO_ACTIVE_HIGH>; gpio-crc_error = <&gpio2 18 GPIO_ACTIVE_HIGH>; }; };

So I can rename the compatible name and the gpios to get it to work.

The current linux driver does not support * config size. Without this, the config size needs to be specified on the command string, which I do not like. * the crc_error pin * an optional fpga name which is desirable. The driver will generate a generated name using "dev_driver_string". Is something similar available in u-boot?

I guess a linux driver will simply ignore them unless updated.

================= The SPI drivers are not using MTD. They are using FPGA manager, which I feels is a bit unneccessary.

You need dedicated applications to program the FPGA this way. Using an mtd approach, I expect you can do

cat fpga-config.rbf > /dev/mtd4 &

===== The FPGA uses LSB SPI transfer for configuration, and the linux driver bitreverses everything in the buffer instead of setting the SPI LSB transfer bit, which is a bit odd.

/Ulf

Thanks, Michal