[U-Boot] Using Uboot to communicate a PCI device.
Dear sir, I am using U-boot as a Boot loader for booting Cavium Octeon Processor. My current requirement is to implement Octeon as the PCI bus arbiter i.e. Octeon as a PCI host. Octeon will be responsible for bringing up other processor like Wintegra Winpath Network processor i.e in turn boot the Winpath NP. Octeon will be connected to Winpath over a 32-bit PCI bus. *How can i develop an Uboot application which communicates through Octeon over the PCI bus to Wintegra device such that it transfers data to Winpath device memory address?* *What are the prerequisites i need to write a U-Boot Application to communicate over a PCI bus to talk to a PCI device?* According to my understanding, i need a Winpath Device Driver that needs to be at U-Boot location to communicate to the device over a PCI bus which i need to add to the /Drivers folder path in U-boot Code. Assuming that i have the Driver code for Winpath Device.. how should i proceed further to initiate communication over a PCI bus? I am new to PCI and U-boot, Please guide me through the procedure. If you have any example wherein U-boot is used to talk to a *PCI device*i.e. do a read/write to device's memory location, then please share that example with me. If you have any Internet link which explains similar example then please do share with me. Thanks in advance. Regards, Darshan
Hi Darshan,
Let me summarize your hardware and request: - Cavium Octeon host-processor - Wintegra Winpath Network slave processors - Devices connected via the PCI bus - You want to create a U-Boot application to boot the processors and then communicate over the PCI bus.
However, before we discuss U-Boot coding, it would be useful if you could provide a more detailed system-level description, so that we can provide suggestions. For example;
- I assume the Cavium processor boots U-Boot, and performs some initialization of the NPs, and then boots its OS. What is the OS?
Does the Cavium instance of U-Boot have to boot the NPs? Can the Cavium processor U-Boot initialize the PCI interfaces on the NPs and then boot its OS, and then the Cavium OS can boot the NPs.
U-Boot is a bootloader. More complex features/control may be easier to deal with inside a complete OS such as Linux.
- What OS will the NPs be running? Is there any need for the NPs to run U-Boot? Can the NPs have their OS image copied into their local memory, and then the processors boot directly into their OS - no bootloader necessary? Can the Cavium processor perform all device initialization that U-Boot running on an NP would normally perform?
- Does the NP need to communicate with the Cavium via the PCI bus, or would it be just as efficient for them to communicate over the network?
Ira Snyder has worked on a PCI-to-PCI network driver that might work for your processors, however, I'd need to look at the datasheets for the Cavium and NPs to see if they have the right features for this type of interface.
Try and get the system-level design described first, and then the software requirements for the lower-levels will become clearer, or others will suggest solutions that you may not have considered.
Cheers, Dave
Hi Dave, Thanks for the reply. Cavium's U-boot version included has been modified to support multicore OCTEON processors. The 'bootoct' command has been added, which supports the booting of an Octeon simple executive application. Let me answer your questions now. Octeon Software Development Kit: Octeon Bootloader*I assume the Cavium processor boots U-Boot, and performs some initialization of the NPs, and then boots its OS. What is the OS?* Cavium Octeon Processors are multicore network processors which have two modes for executing a application. Simple Executive Mode: Its the hardware abstraction mode through which an user application can perform Network layer processing stuff by accessing various h/w unit of the Octeon NP.They are faster and have less overhead. This is called Fast-path Processing. Linux(Or other OS) Mode: In this mode Linux can be installed and User applications can be developed over it. This is called Slow-path processing. In my case I dont have any OS on the board hence i need to do it at Simple executive level itself. The U-boot is responsible for loading the aforementioned applications into the Octeon Memory space. But i have heard that Uboot(inside Octeon) can be modified to communicate to a device. So that's why my question "What all the prerequisites i need for the PCI device at Octeon side to add it into u-boot application?". * Does the Cavium instance of U-Boot have to boot the NPs? Can the Cavium processor U-Boot initialize the PCI interfaces on the NPs and then boot its OS, and then the Cavium OS can boot the NPs.?* Since No OS such as Linux exists, Cavium's U-Boot will initialize the processor itself (As a Simple Executive) along with its PCI interface initialization is also done. Now, how should i go ahead in talking to the Winpath NP for doing a task. For eg, Copying the Winpath Boot Image onto the NP via the PCI interface?
*What OS will the NPs be running? Is there any need for the NPs to run U-Boot? Can the NPs have their OS image copied into their local memory, and then the processors boot directly into their OS - no bootloader necessary? Can the Cavium processor perform all device initialization that U-Boot running on an NP would normally perform?* Winpath also has a similar Hardware Abstraction (similar to Simple Executive as in Octeon). The Winpath has a similar U-boot*-like* bootloader to boot up the processor... since it doesn't have tftp or flash interface for copying the bootloader image... Cavium has to do this task via its PCI interface. Cavium's task is to load Wintegra NP's Bootloader into the memory space of Wintegra so that WinPath boots itself and further all the applications that Winpath has to run shall be "loaded" to Winpath into its memory addresses. The bottomline is Winapth NP does not have direct access to TFTP/Flash for loading the applications or even the bootloader, it has to do it via the Cavium on the PCI interface.
*Does the NP need to communicate with the Cavium via the PCI bus, or would it be just as efficient for them to communicate over the network?* Since Cavium's Octeon and Wintegra's Winpath are 'sitting' on the same board as two chips they cant communicate over the network.
I hope i am much clearer this time. Thanks a lot for the prompt response. :) Darshan
2009/7/18 dwh@ovro.caltech.edu
Hi Darshan,
Let me summarize your hardware and request:
- Cavium Octeon host-processor
- Wintegra Winpath Network slave processors
- Devices connected via the PCI bus
- You want to create a U-Boot application to boot the
processors and then communicate over the PCI bus.
However, before we discuss U-Boot coding, it would be useful if you could provide a more detailed system-level description, so that we can provide suggestions. For example;
- I assume the Cavium processor boots U-Boot, and performs
some initialization of the NPs, and then boots its OS. What is the OS?
Does the Cavium instance of U-Boot have to boot the NPs? Can the Cavium processor U-Boot initialize the PCI interfaces on the NPs and then boot its OS, and then the Cavium OS can boot the NPs.
U-Boot is a bootloader. More complex features/control may be easier to deal with inside a complete OS such as Linux.
- What OS will the NPs be running? Is there any need for
the NPs to run U-Boot? Can the NPs have their OS image copied into their local memory, and then the processors boot directly into their OS - no bootloader necessary? Can the Cavium processor perform all device initialization that U-Boot running on an NP would normally perform?
- Does the NP need to communicate with the Cavium via
the PCI bus, or would it be just as efficient for them to communicate over the network?
Ira Snyder has worked on a PCI-to-PCI network driver that might work for your processors, however, I'd need to look at the datasheets for the Cavium and NPs to see if they have the right features for this type of interface.
Try and get the system-level design described first, and then the software requirements for the lower-levels will become clearer, or others will suggest solutions that you may not have considered.
Cheers, Dave
Hi Darshan,
Cavium's U-boot version included has been modified to support multicore OCTEON processors. The 'bootoct' command has been added, which supports the booting of an Octeon simple executive application.
Ok. Since the processor is already supported in U-Boot, your life is much easier.
In my case I dont have any OS on the board
Ok.
U-boot is responsible for loading applications into the Octeon Memory space.
Ok.
But i have heard that Uboot(inside Octeon) can be modified to communicate to a device.
If the other processors are in the address space of the Octeon processor, or if an Octeon PCI window can be moved to overlay each of the NPs, then host read/write accesses can be used to initialize the slave NPs. There is no additional 'communication' code needed.
Since No OS such as Linux exists, Cavium's U-Boot will initialize the processor itself (As a Simple Executive) along with its PCI interface initialization is also done. Now, how should i go ahead in talking to the Winpath NP for doing a task. For eg, Copying the Winpath Boot Image onto the NP via the PCI interface?
Once the NP PCI interface is configured and enabled, you should be able to use read/write accesses on your host that hit the PCI addresses corresponding to the NP.
As an analogy, we have a single x86 host processor in a cPCI interface with 15 PowerPC boards plugged into the backplane. The host BIOS initializes the PCI interfaces on the PowerPC boards and enables them. On the PowerPCs, if U-Boot has not been installed, a default PCI window is setup, and if U-Boot is installed a custom setting is used (the local-side PCI initialization completes before the x86 BIOS enables the interfaces). As far as the x86 host is concerned, each PowerPC appears as at least two windows in the PCI space; one window is a set of PowerPC control registers, and another window points to an arbitrary address in the PowerPC memory map. Manipulation of a control register changes the base address, so the window can be moved around.
Your NPs should have something similar. The host's U-Boot can initialize the PCI interfaces on the NPs, and then later your U-Boot 'NP boot' command (U-boot application) can use the PCI interface registers to move the window around while copying application code into the NP memory space. You would need to have access via PCI to NP control registers too, so that you can release the NP core from reset and allow it to boot from the application code your host just copied into its address space.
Based on your current understanding of the processors, does any of this sound possible?
The Winpath has a similar U-boot*-like* bootloader to boot up the processor... since it doesn't have tftp or flash interface for copying the bootloader image... Cavium has to do this task via its PCI interface. Cavium's task is to load Wintegra NP's Bootloader into the memory space of Wintegra so that WinPath boots itself and further all the applications that Winpath has to run shall be "loaded" to Winpath into its memory addresses. The bottomline is Winapth NP does not have direct access to TFTP/Flash for loading the applications or even the bootloader, it has to do it via the Cavium on the PCI interface.
Ok. This is should be no problem.
These devices are 'network processors'. Do they have a direct network connection, or do they get all their network data via the host processor?
The host can copy all data into the NP memory space, however, once the NP is booted, you will want some form of commincations; which is what you are asking. However, there are more questions:
- Cavium host boots U-Boot - NP boots Winpath NP bootloader
Do you have access to the source code for the NP bootloader so that you can implement both ends of the communication path?
Depending on the complexity needed, the communications can be as simple as an interlock to indicate to the NP that a new application is needed, or it can be as complex as an emulated serial-port-over-PCI and network-over-PCI as demonstrated in Ira's drivers.
Since Cavium's Octeon and Wintegra's Winpath are 'sitting' on the same board as two chips they cant communicate over the network.
Why not? It doesn't matter where they are sitting if they both have a network interface. However, I assume you mean they do not have an external connection via the network.
I hope i am much clearer this time.
We're getting there :)
Is there any documentation on the boards you are working on that is available online? At a minimum links to the Cavium and NP data sheets would be useful.
Cheers, Dave
Hi Dave, I am really happy for your co-operation in making me understand things which i completely don't have an idea for. I am getting some clarity into this requirement after looking at your answers and explanations. Let me tell you what i am able to understand from these explanations at this juncture. 1. Since Octeon's u-boot has the capabilities of initialize Octeon's PCI (Assign bus address ranges used to configure devices found on the PCI bus), we can talk to a PCI device on the other end provided we have info about their memory addressing etc. *Having said this, what all (info) i need to have from the PCI device (Winpath NP) so that i am able to bring it up. I mean do i need to have some device driver of NP so that i am able to read/write into its Config space/Memory space?* I am really sorry if my questions are too silly. I am new to u-boot and PCI hence such questions.
Now let me answer the questions, you have asked me. *The host's U-Boot can initialize the PCI interfaces on the NPs, and then later your U-Boot 'NP boot' command (U-boot application) can use the PCI interface registers to move the window around while copying application code into the NP memory space. You would need to have access via PCI to NP control registers too, so that you can release the NP core from reset and allow it to boot from the application code your host just copied into its address space. Based on your current understanding of the processors, does any of this sound possible? *I am not able to apprehend the very first sentence you have quoted. The host's U-boot can initialize its own end of the PCI interface. How can it initialize the PCI interface of the NP? Please, can you give me more clarity on this? You mention, "You would need to have access via PCI to NP control registers too, so that you can release the NP core from reset and allow it to boot from the application code your host just copied into its address space*.*" In order to execute such task, do i need a program code (NP device driver) present at my end (inside my U-boot's driver folder) so that i will be able to write an u-boot application('NP Boot')?
*Do you have access to the source code for the NP bootloader so that you can implement both ends of the communication path?* Can i conclude after reading the above sentence written from you that i need to write some code (which involves modification in the NP bootloader) to complete the 'NP boot' initiated from the other end? Also, can i conclude that in order to achieve my objective of Booting an NP(PCI device) from Octeon (PCI host) involves work at both the ends?
*Is there any documentation on the boards you are working on that is available online? At a minimum links to the Cavium and NP data sheets would be useful.* I am afraid such datasheets wont be available speaking details like PIN configuration etc.Check this link a primitive one ;-) ( http://www.caviumnetworks.com/OCTEON-Plus_CN58XX.html)
2009/7/18 dwh@ovro.caltech.edu
Hi Darshan,
Cavium's U-boot version included has been modified to support multicore OCTEON processors. The 'bootoct' command has been added, which supports the booting of an Octeon simple executive application.
Ok. Since the processor is already supported in U-Boot, your life is much easier.
In my case I dont have any OS on the board
Ok.
U-boot is responsible for loading applications into the Octeon Memory space.
Ok.
But i have heard that Uboot(inside Octeon) can be modified to communicate to a device.
If the other processors are in the address space of the Octeon processor, or if an Octeon PCI window can be moved to overlay each of the NPs, then host read/write accesses can be used to initialize the slave NPs. There is no additional 'communication' code needed.
Since No OS such as Linux exists, Cavium's U-Boot will initialize the processor itself (As a Simple Executive) along with its PCI interface initialization is also done. Now, how should i go ahead in talking to the Winpath NP for doing a task. For eg, Copying the Winpath Boot Image onto the NP via the PCI interface?
Once the NP PCI interface is configured and enabled, you should be able to use read/write accesses on your host that hit the PCI addresses corresponding to the NP.
As an analogy, we have a single x86 host processor in a cPCI interface with 15 PowerPC boards plugged into the backplane. The host BIOS initializes the PCI interfaces on the PowerPC boards and enables them. On the PowerPCs, if U-Boot has not been installed, a default PCI window is setup, and if U-Boot is installed a custom setting is used (the local-side PCI initialization completes before the x86 BIOS enables the interfaces). As far as the x86 host is concerned, each PowerPC appears as at least two windows in the PCI space; one window is a set of PowerPC control registers, and another window points to an arbitrary address in the PowerPC memory map. Manipulation of a control register changes the base address, so the window can be moved around.
Your NPs should have something similar. The host's U-Boot can initialize the PCI interfaces on the NPs, and then later your U-Boot 'NP boot' command (U-boot application) can use the PCI interface registers to move the window around while copying application code into the NP memory space. You would need to have access via PCI to NP control registers too, so that you can release the NP core from reset and allow it to boot from the application code your host just copied into its address space.
Based on your current understanding of the processors, does any of this sound possible?
The Winpath has a similar U-boot*-like* bootloader to boot up the processor... since it doesn't have tftp or flash interface for copying the bootloader image... Cavium has to do this task via its PCI interface. Cavium's task is to load Wintegra NP's Bootloader into the memory space of Wintegra so that WinPath boots itself and further all the applications that Winpath has to run shall be "loaded" to Winpath into its memory addresses. The bottomline is Winapth NP does not have direct access to TFTP/Flash for loading the applications or even the bootloader, it has to do it via the Cavium on the PCI interface.
Ok. This is should be no problem.
These devices are 'network processors'. Do they have a direct network connection, or do they get all their network data via the host processor?
The host can copy all data into the NP memory space, however, once the NP is booted, you will want some form of commincations; which is what you are asking. However, there are more questions:
- Cavium host boots U-Boot
- NP boots Winpath NP bootloader
Do you have access to the source code for the NP bootloader so that you can implement both ends of the communication path?
Depending on the complexity needed, the communications can be as simple as an interlock to indicate to the NP that a new application is needed, or it can be as complex as an emulated serial-port-over-PCI and network-over-PCI as demonstrated in Ira's drivers.
Since Cavium's Octeon and Wintegra's Winpath are 'sitting' on the same board as two chips they cant communicate over the network.
Why not? It doesn't matter where they are sitting if they both have a network interface. However, I assume you mean they do not have an external connection via the network.
I hope i am much clearer this time.
We're getting there :)
Is there any documentation on the boards you are working on that is available online? At a minimum links to the Cavium and NP data sheets would be useful.
Cheers, Dave
Hi Darshan,
- Since Octeon's u-boot has the capabilities of initialize
Octeon's PCI (Assign bus address ranges used to configure devices found on the PCI bus)
Right. The host CPU can configure the PCI interface on the NP.
Having said this, what all (info) i need to have from the PCI device (Winpath NP) so that i am able to bring it up. I mean do i need to have some device driver of NP so that i am able to read/write into its Config space/Memory space?
The host CPU must be able to issue configuration space read/writes, since that is what it uses to setup devices. The host CPU will generally also support memory and I/O read/writes. The user manual will tell you how to generate the configuration space read/write, whereas the memory map configuration will generally determine whether a memory or I/O transaction is generated. These questions are totally independent of U-Boot and require that you read the user manual - the trick is knowing what to look for - which you now do.
I am really sorry if my questions are too silly. I am new to u-boot and PCI hence such questions.
They're not silly, just inexperienced.
Now let me answer the questions, you have asked me. *The host's U-Boot can initialize the PCI interfaces on the NPs, and then later your U-Boot 'NP boot' command (U-boot application) can use the PCI interface registers to move the window around while copying application code into the NP memory space. You would need to have access via PCI to NP control registers too, so that you can release the NP core from reset and allow it to boot from the application code your host just copied into its address space. Based on your current understanding of the processors, does any of this sound possible? *I am not able to apprehend the very first sentence you have quoted. The host's U-boot can initialize its own end of the PCI interface. How can it initialize the PCI interface of the NP? Please, can you give me more clarity on this?
Initialization of a PCI interface will require actions from both the slave side (NP) of the bridge and on the host side. The activity on the host side is standardized by the PCI spec, i.e., there is a standard sequence used to initialize PCI devices. However, the configuration of the slave side of the NP PCI interface needs to be performed to describe how many base-address-registers (BARs) and what they point to. For that, you need to read the user manual for the NP, and look at the PCI initialization (its possible that its hardcoded, but you'll have to find that out).
You mention, "You would need to have access via PCI to NP control registers too, so that you can release the NP core from reset and allow it to boot from the application code your host just copied into its address space*.*" In order to execute such task, do i need a program code (NP device driver) present at my end (inside my U-boot's driver folder) so that i will be able to write an u-boot application('NP Boot')?
At some point you will have to write some code. But first, you should be able to manually perform most of the operations needed to boot an NP processor.
Once you figure that out, you can post an RFC (request for comments) on how you plan to proceed so others can provide feedback. Its quite possible that once you know what you want to do, that something already exists, or something similar that can be generalized exists.
*Do you have access to the source code for the NP bootloader so that you can implement both ends of the communication path?* Can i conclude after reading the above sentence written from you that i need to write some code (which involves modification in the NP bootloader) to complete the 'NP boot' initiated from the other end?
Sure, its not magic. If you write a custom communications protocol at one end, then you need to write the code at the other end.
The board hardware engineer should have figured out how the two devices were planning on communicating. Perhaps you are lucky and that in addition to the PCI interfaces being routed together, the serial ports on the cores are connected (if the processors have such things). In that case, the host might be able to control the slaves via serial commands.
You have not provided sufficient information to say much more.
Also, can i conclude that in order to achieve my objective of Booting an NP(PCI device) from Octeon (PCI host) involves work at both the ends?
Maybe. Again, insufficient information.
*Is there any documentation on the boards you are working on that is available online? At a minimum links to the Cavium and NP data sheets would be useful.* I am afraid such datasheets wont be available speaking details like PIN configuration etc.Check this link a primitive one ;-) ( http://www.caviumnetworks.com/OCTEON-Plus_CN58XX.html)
- This isn't a user manual, its a block diagram and product request interface. - What about the processors connected via the PCI bus? - What about the actual wiring on the board?
One huge advantage of using open-source software is to use the combined knowledge of the open-source community. However, its a two-way street. If you can't provide easily accessible low-level documentation, you're not going to get much help.
Processors have a huge amount of configuration possible, and there are a lot of power-on pin strappings that hardwire some features of the processor on boot. If your board was designed without consideration for how the processor bootup was to proceed, then you could be in trouble.
What you need to do is read the processor documentation for both the host and slave processors, and look at the board design.
You need to discuss this board design with the original hardware engineer to determine how they expected the power-on-sequence and boot sequences to proceed.
I understand you are learning, so will help provide feedback where I can.
Cheers, Dave
Dear Darshan Vasisht,
In message 49797f780907172331s4e6e9121w122812e2ccc3c77f@mail.gmail.com you wrote:
Cavium's U-boot version included has been modified to support multicore OCTEON processors. The 'bootoct' command has been added, which supports the booting of an Octeon simple executive application.
It would have been really nice if such development had not been done behind closed doors, and if design decisions had been discussed here on the mailinglist.
I guess this could have saved you a lot of effort whenever you try to push this code upstream.
Best regards,
Wolfgang Denk
participants (3)
-
Darshan Vasisht -
dwh＠ovro.caltech.edu -
Wolfgang Denk