Hi Andreas,

We have been shipping 2011.03 for some time and internally are planning to use 2012.07 when it is released since we are tracking the top of trunk. We have made a lot of improvements since then and added a lot of new capabilities.

-Aaron

On 07/05/2012 06:18 AM, Andreas Bießmann wrote:

Dear sayed zahid,

On 05.07.2012 13:42, sayed zahid wrote:

...

Hi , I have basic knowledge of porting u-boot to a new board. But now i have got a task to port u-boot on cavium mips based board. As i know that mips architecture is already supported, but will it support all cavium octeon mips variants?

I guess no. I had to work with some non mainlined version of u-boot provided by cavium a year ago (or something around that). At that time we got a

---8<--- abiessmann@azuregos % git grep U_BOOT_VERSION include/version.h include/version.h:#define U_BOOT_VERSION "U-Boot 1.1.1" --->8---

:(

AFAIR there was a lot of basic stuff missing in mainline U-Boot to get Cavium mips64 running at that time so I didn't start to port these proprietary changes (well it would not have been accepted the way it was written down; beside that cavium forced us to sign a NDA for that piece of GPL code).

I have not followed their open source efforts since then, but I saw a lot of work on the linux-mips list by David Daney and I think Aaron Williams is working for Cavium too and is on this list. Maybe one of them can comment?

...

Please put some light on this. I would be glad if you guys, give my some idea to where to start with or suggest some documents which will help me understand and port on a new architecture

You should not hesitate to contact their support and ask for proper mainline support of u-boot ;)

Enough Cavium bashing.

Regarding your question for documents, I'm sorry I can not help you here. I recommend start working with some mainlined device which is comparable, get firm with the architecture, try to find commonalities, get the new cpu in u-boot step by step.

Best regards

Andreas Bießmann

Hi Andreas,

I would love to see OCTEON support in the mainline as well, though I am not sure how I should go about this since it is a substantial amount of code. Fortunately most of the changes are not to the common code, and many of the common code changes are feature enhancements portable to other platforms as well.

One big problem I have is I have no way of testing any of my stuff on non-Cavium boards or CPUs so testing my changes and making sure I didn't break anything is rather difficult. I am also not too familiar with git or how to actually submit the patches with it though I can work with our Linux kernel developer on this.

When I redid U-Boot using the up-to-date code I made a large effort to keep our stuff separate from the rest of the U-Boot code so much of our stuff is fairly well isolated. I placed most of our code under arch/mips/cpu/octeon and arch/mips/include/asm/arch-octeon (though some of the files in arch/mips/include/asm also had to be changed). Most of the rest of the code is in board/octeon/xxx though there's a fair amount in board/octeon/common.

Most major changes to the common code were done by marking the common code functions as "weak" and replacing them elsewhere.

A few of the files in U-Boot are also shared by some other utilities, such as our code for loading and executing simple executive applications and initializing DDR memory.

One constantly moving target in U-Boot is the simple executive code. We support writing se applications which are basically ELF executables that run directly on the OCTEON processor without an operating system underneath. These can be run in parallel with other applications on different cores along with Linux. The main thing involved here is the memory management. We use a "named block" memory scheme which is used by the Linux kernel and simple executive applications to find and share various data structures such as the flat device tree (BE and LE versions), temporary load address, linux reserved address, PCI console memory block and some JTAG information. We need this because we can have multiple applications running on different cores.

U-Boot itself always runs on core 0 but needs to be aware of the other CPU cores (currently up to 32).

We have little in common with the other MIPS processors since we have to support 64-bit mode. To do this we compile it with our own toolchain using the N32 ABI. I believe most of our changes are now in the mainline GCC as well now.

We also do not use arch/mips/lib/board.c since our version is so different. I will be the first to admit that the code needs to be cleaned up since board_init_f is huge.

We have done some interesting things with our U-Boot image. Our U-Boot binary can be executed at any memory address as long as it's 4MB aligned (or 64K aligned when running from flash). We have a single binary image that can be booted directly from RAM (when booting over eJTAG or PCI/PCIe), out of L2 cache and out of flash. By doing this we can also store two copies of U-Boot in flash (the same binary image) to support a failsafe and normal bootloader.

2702: common board code 28,867: support for 33 different boards 13,568: configuration .h files for all of our boards 2,215: board_octeon.c (replacement for board.c) 24,004: .c files in arch/mips/cpu/octeon/ 10,147: .c files in arch/mips/cpu/octeon/commands/ 155,994: .c files from our simple executive which are linked to U-Boot. This includes files for various errata, networking support, clock, memory management used by our simple executive environment as well as code to deal with many of the modules in all of our various chips. 10,166: .c files for OCTEON-specific commands, some of which could be merged into the core code. 279,576: .h files for our simple executive. Some of these files are very large because they include generated header files for all of the register definitions for all of our chips (including different revisions).

The most common change we have to make to any PCI drivers is to support the appropriate address mapping. In our case the virtual address needs to be translated to a PCI DMA address, which may be different than a physical address. Fortunately the mechanisms are available in the PCI driver to support this. The drivers also need to be extended to be able to pass 64-bit addresses to the devices.

For devices not on the PCI bus (i.e. USB EHCI) we have to use a different mechanism in order to always pass the physical address.

All accesses to registers have to go through wrappers (which are usually already in place) since all of our non-PCI device registers are always mapped into various 64-bit address spaces.

Here's a list of changes from a quick glance:

Added and changed drivers: - TI tmp42x temperature sensor - NX SAA56004X temperature sensor (I think I wrote one of the other temp sensor drivers as well) - Power driver for ispPAC chip - OCTEON MDIO driver - OCTEON USB driver for OCTEON I and OCTEON - USB clock init code for OCTEON II USB EHCI interface (slight modifications to EHCI driver for OCTEON II to handle virtual memory mapping and 64-bit addressing) - Intel E1000 driver - minor change to support 64-bit addressing and virtual to PCI DMA addressing. - OCTEON GPIO driver - OCTEON MMC/SD/SDHC/SDXC driver - OCTEON SPI driver - Fixed AHCI SATA driver so it compiles and works - Modified NAND driver in order to support large NAND chips, also added support for multi-bit ECC - Fixed CFI NOR flash driver to properly handle 8-bit mode. - Minor changes to PCI driver

Drivers not under drivers/ but under arch/mips/cpu/octeon: - Watchdog - compact flash (this also requires a rather hacked up version of the cmd_ide.c file in order to support multiple CF devices properly) - ethernet (including management, SGMII, XAUI, SPI and some other interface types) - PCI driver - PCI console driver (for OCTEON NIC/PCI target boards) - serial - I2C - MIPS micro assembler (required for watchdog support)

lib changes: - Made md5, sha1, sha256 and crc32 functions weak since we supply OCTEON replacement functions that take advantage of encryption/hash/CRC instructions to reduce code size - Updated lzma code to latest - Added bch.c for handling multi-bit ECC. - Enabled and/or added a few more string functions - Made a few other functions "weak"

Common/other changes: - Minor changes to various commands so that if an address of 0 is supplied by the user it will use the default load address (typically 0x20000000 for us) - Some environment changes in order to support the environment being in RAM when the bootloader is started. - RAM environment support - Support for user-defined functions to be called whenever setenv is called to hook environment variable changes. - MTD and flash filesystem support for NAND > 4GiB to match Linux kernel. - Support for prompt to be created dynamically and also defined in an environment variable (including in hush shell) - Minor change to DHCP code to allow for custom options - Support for DHCP option to set TFTP server IP - Added commands for bunzip and unlzma - Minor change to mmc command to enable some additional functionality - Minor change to dlmalloc.c init code so to not zero out memory in our simulator case. - Minor changes to usb code to handle some buggy hardware (i.e. SanDisk Cruzer similar to Linux). - Support for go command, elf and some other commands to call a function so we can perform clean-up before handing execution off (i.e. shutting down networking support, freeing resources).

Octeon specific commands: - bootloaderupdate - bootloadervalidate - bootoct (boots simple executive applications) - bootoctlinux (boots the OCTEON Linux kernel) - base64 - sets 64-bit base address for md64, etc. - cmp64/cp64/loop64/md64/mm64 (64-bit versions of cp/md/mm), also added .d for 64-bit accesses - tftp (alias for tftpboot since so many customers rely on tftp) - read64/read64b/read64l/read64s - read a 64-bit memory address (used to read registers) - write64/write64b/write64l/write64s - writes a 64-bit memory address (used to write registers) - octnand - OCTEON-specific NAND read/write routines used for writing bootloaders, etc. to NAND - qlm - print/modify our QLM interface JTAG settings (a QLM is an I/O interface that can be SGMII/XAUI/ILK/PCIe and various other high-speed interface types). - octreginfo - prints out all of the CP0 registers and TLB entries. - namedprint - prints out all named memory blocks - freeprint - prints out all free memory blocks - namedalloc - allocate a named memory block - octwd - configure OCTEON watchdog support - tlv_eeprom - display or modify the contents of a board-specific I2C EEPROM used by Cavium boards which contains board identification and configuration information. - eraseenv - erases the environment flash block - bootstage3 - searches for a stage 3 bootloader, used when booting from MMC/SD. - octbootbus - Prints all of the timing parameters on our boot bus or allows the timing parameters to be changed for various devices

-Aaron

On 07/06/2012 01:19 AM, Andreas Bießmann wrote:

Dear Aaron Williams,

On 06.07.2012 01:52, Aaron Williams wrote:

...

Hi Zahid,

I am in charge of U-Boot for OCTEON MIPS. I have not pushed the changes back upstream since the amount of code is enormous (over 430K lines of code!). Granted, a huge percentage of that is from generated register files but it is still a huge amount of code. Just the DRAM initialization code is 9600 lines of code for DDR2 and DDR3 support.

We also do things that no other architecture does with U-Boot such as we always run in TLB mapped memory. The code is available under GPL but you need to contact support@cavium.com. With the TLB mapping it no longer matters where U-Boot is loaded in memory. The same U-Boot binary image can start executing at any 64K flash boundary in the first 4MB of flash (so we support the same image as a failsafe and standard bootloader), any 4MB boundary in RAM (when booted over PCI or eJTAG) and out of L2 cache (our top of trunk copies itself from flash to the L2 cache very early on to speed up memory initialization). The TLB mapping also allows U-Boot to be copied to the very top of memory since KSEG0 is rather limited to only 256MB. This is essential since we can support many GB of RAM which otherwise requires 64-bit addressing.

I need to push some of my changes back upstream since I have added some drivers for some temperature sensors, power monitors, fixed the AHCI driver and added a few features over the standard U-Boot (such as dynamically generated prompt support).

I would really like to see the whole stuff in mainline in favor of out of tree patches maintained only inside Cavium. Mainline U-Boot may also benefit from your generic changes and you will get reviews for free.

Best regards

Andreas Bießmann

Hi Daniel,

My comments are inline.

On 07/09/2012 02:11 PM, Daniel Schwierzeck wrote:

Hi Aaron,

2012/7/6 Aaron Williams Aaron.Williams@cavium.com:

...

Hi Andreas,

I would love to see OCTEON support in the mainline as well, though I am not sure how I should go about this since it is a substantial amount of code. Fortunately most of the changes are not to the common code, and many of the common code changes are feature enhancements portable to other platforms as well.

we should do this step by step. My suggestions are:

1. send a patch series with only the changes on MIPS common code. I

assume that you updated most of the header files with the current ones from linux? I am currently working on extending the support for MIPS32 CPUs thus we should unify the common MIPS code base at first.

I made a number of additions to the header files. For the most part I did not update them from Linux but modified the existing ones. I tried to keep my changes minimal. As I said I use board_octeon.c instead of board.c. I have wrapped all of our changes with #ifdef CONFIG_OCTEON.

It should be fairly easy to do this.

2. send a patch series with only basic support for Octeon and provide

a toolchain for free download. This allows us to do build tests.

GCC as of version 4.7 should work. Note that for OCTEON we compile using the N32 ABI instead of O32. Doing "basic" OCTEON support will not be easy since there is a huge dependence on our SDK.

3. send patches for all minimum required Octeon drivers and common

code changes to make mainline U-Boot working on real hardware

The only required drivers are the MDIO drivers. Most of the other drivers, i.e. Ethernet, PCI, I2C, serial, flash, NAND and watchdog are located under arch/mips/cpu/octeon. I don't know if it makes sense to move these into the drivers tree since these are specific to OCTEON and some (i.e. the Ethernet driver and NAND) have a strong dependency on our SDK.

4. send patches for additional drivers and features

...

One big problem I have is I have no way of testing any of my stuff on non-Cavium boards or CPUs so testing my changes and making sure I didn't break anything is rather difficult.

I think that is no big problem. I can test MIPS specific changes on MIPS32 machines and QEMU. Changes to common code are reviewed by and tested by the according maintainers. At least you have to do compile tests as described here: http://www.denx.de/wiki/view/U-Boot/Patches#Notes

I'll try this. I have been enforcing the U-Boot coding standard and recently we switched our SDK to follow that (though our SDK does not honor the 80 column limit).

...

I am also not too familiar with git or how to actually submit the patches with it though I can work with our Linux kernel developer on this.

When I redid U-Boot using the up-to-date code I made a large effort to keep our stuff separate from the rest of the U-Boot code so much of our stuff is fairly well isolated. I placed most of our code under arch/mips/cpu/octeon and arch/mips/include/asm/arch-octeon (though some of the files in arch/mips/include/asm also had to be changed). Most of the rest of the code is in board/octeon/xxx though there's a fair amount in board/octeon/common.

Sounds good. Maybe you should use board/cavium/xxx instead because the directories in board/ reflect either the board name or the vendor name.

We do that. We use board/octeon/xxx for everything right now including all of our vendors since the boards all have dependencies on board/octeon/common. Because we had a problem with vendors just modifying an existing board and complaining when we did a new release I created a script to assist in adding a new board which uses this directory. I would rather use octeon instead of cavium since we also have several ARM based chips with more to follow.

<snip>

...

We have little in common with the other MIPS processors since we have to support 64-bit mode. To do this we compile it with our own toolchain using the N32 ABI. I believe most of our changes are now in the mainline GCC as well now.

We also do not use arch/mips/lib/board.c since our version is so different. I will be the first to admit that the code needs to be cleaned up since board_init_f is huge.

actually that is not a problem. What about arch/mips/lib/bootm.c?

Best regards, Daniel

The only change I made to bootm.c is that for OCTEON we don't have uncached memory. On the OCTEON processor, all DRAM is cached (and the L1 data and L2 caches are coherent).

I know that some of the code will need to be cleaned up. Our board_octeon.c has grown into a big mess that could use some serious clean-up.

We also have added some tools for U-Boot. One tool generates a special header for U-Boot which contains a CRC and other information.

Unfortunately I will not have a lot of time in the near future since I need to do some work on Valgrind. I can try and create a patch for the header files and bootm.c, though.

-Aaron