[U-Boot] RFC: Testing U-Boot Part 1
Hi,
Summary: I am quite keen on improving the test infrastructure in U-Boot. I would like to have a test suite that can run in a minute or two on a Linux PC and test all non-platform code.
Detail ====== We can break the U-Boot code base into two parts:
1. Platform code, which is SOC-specifc. At present this is the CPU init (arch/xxx/cpu/...) and SOC-specific drivers (mostly in the drivers directory). There is also a small amount of generic CPU code in arch/xxx/lib and some board-specific code/drivers (e.g. drivers not within the SOC).
2. Portable/Generic U-Boot code, which is cross-platform. This includes many drivers, the various commands, file system support, things like the MMC, USB and network stacks, etc.
My focus in this email the the second part of the code base - all the code which is not platform-specific, but can still have bugs.
Proposal ======== To a large extent, testing of this part of the code base could simply be built on one or more of the available platforms. We could then run the tests on that platform, and announce that the code base works fine on that platform. Obviously the platform needs to support all possible features of U-Boot.
However, this approach fails to take advantage of two useful properties of this code:
- It is cross-platform, and even runs on x86 - It is standalone, requiring no OS libraries to run
For speed, debugging and convenience, it would be nice to run U-Boot under a generic Linux environment on a workstation, and test all the generic non-platform code. The basic problem with this is that the non-platform code requires the platform code to operate. Even the x86 platform code is designed for standalone operation on a particular x86 board, and is not suitable for running under x86 Linux.
To get around this I propose that we create a new ‘native’ architecture. We write code in ‘arch/native’ which can run under Linux. Since all the non-platform code will happily run under this new ‘architecture’, we can then write tests which run quickly under x86 Linux (or another Linux for that matter). This U-Boot 'architecture' should build natively on any 32/64-bit Linux machine since it just uses standard Linux system calls. Calls to Linux would be entirely within this arch/native subdirectory.
Benefit ======= What will this buy us? Well we can do things like:
- Create a test SPI flash device, which is file-backed. Use this to write a test of the cmd_sf layer by issuing a series of commands and making sure that the correct SPI flash contents is obtained
- Create a test MMC or IDE device, backed by a file. Use this to issue ext2 and fat commands to manipulate the filesystem. Then loopback mount the file and check from Linux that U-Boot did the right thing
- Create a test Ethernet device with a mocked remote host attached. Use this to issue network commands like bootp and tftp and check that the correct results are obtained.
Ultimately (one day) we might have a set of unit tests for U-Boot which can run to very quickly check that a patch does not break the core U-Boot code.
Comments ======== At this early stage I am looking for comments on the concept - how useful it is and how best to implement it.
Regards, Simon
Dear Simon,
Am 25.08.2011 14:58, schrieb Simon Glass:
Hi,
Summary: I am quite keen on improving the test infrastructure in U-Boot. I would like to have a test suite that can run in a minute or two on a Linux PC and test all non-platform code.
<snip>
To get around this I propose that we create a new ‘native’ architecture. We write code in ‘arch/native’ which can run under Linux. Since all the non-platform code will happily run under this new ‘architecture’, we can then write tests which run quickly under x86 Linux (or another Linux for that matter). This U-Boot 'architecture' should build natively on any 32/64-bit Linux machine since it just uses standard Linux system calls. Calls to Linux would be entirely within this arch/native subdirectory.
why don't use some unit testing framework like cunit, or ceedling (which can do HW mocks easily)?
regards
Andreas Bießmann
On Thursday, August 25, 2011 09:56:02 Andreas Bießmann wrote:
Am 25.08.2011 14:58, schrieb Simon Glass:
Summary: I am quite keen on improving the test infrastructure in U-Boot. I would like to have a test suite that can run in a minute or two on a Linux PC and test all non-platform code.
<snip>
To get around this I propose that we create a new ‘native’ architecture. We write code in ‘arch/native’ which can run under Linux. Since all the non-platform code will happily run under this new ‘architecture’, we can then write tests which run quickly under x86 Linux (or another Linux for that matter). This U-Boot 'architecture' should build natively on any 32/64-bit Linux machine since it just uses standard Linux system calls. Calls to Linux would be entirely within this arch/native subdirectory.
why don't use some unit testing framework like cunit, or ceedling (which can do HW mocks easily)?
these testing frameworks wont make any difference to what Simon is proposing. he is focusing on getting u-boot to build & run on your desktop machine. after that is done, we can talk about the actual tests and harnesses (although anything that requires ruby should immediately be disqualified imo :P). -mike
On Thursday, August 25, 2011 04:56:39 PM Mike Frysinger wrote:
On Thursday, August 25, 2011 09:56:02 Andreas Bießmann wrote:
Am 25.08.2011 14:58, schrieb Simon Glass:
Summary: I am quite keen on improving the test infrastructure in U-Boot. I would like to have a test suite that can run in a minute or two on a Linux PC and test all non-platform code.
<snip>
To get around this I propose that we create a new ‘native’ architecture. We write code in ‘arch/native’ which can run under Linux. Since all the non-platform code will happily run under this new ‘architecture’, we can then write tests which run quickly under x86 Linux (or another Linux for that matter). This U-Boot 'architecture' should build natively on any 32/64-bit Linux machine since it just uses standard Linux system calls. Calls to Linux would be entirely within this arch/native subdirectory.
why don't use some unit testing framework like cunit, or ceedling (which can do HW mocks easily)?
these testing frameworks wont make any difference to what Simon is proposing. he is focusing on getting u-boot to build & run on your desktop machine. after that is done, we can talk about the actual tests and harnesses (although anything that requires ruby should immediately be disqualified imo :P). -mike
Definitelly agree with the RUBY part.
On Thursday, August 25, 2011 02:58:00 PM Simon Glass wrote:
Hi,
Summary: I am quite keen on improving the test infrastructure in U-Boot. I would like to have a test suite that can run in a minute or two on a Linux PC and test all non-platform code.
Detail
We can break the U-Boot code base into two parts:
- Platform code, which is SOC-specifc. At present this is the CPU
init (arch/xxx/cpu/...) and SOC-specific drivers (mostly in the drivers directory). There is also a small amount of generic CPU code in arch/xxx/lib and some board-specific code/drivers (e.g. drivers not within the SOC).
- Portable/Generic U-Boot code, which is cross-platform. This
includes many drivers, the various commands, file system support, things like the MMC, USB and network stacks, etc.
My focus in this email the the second part of the code base - all the code which is not platform-specific, but can still have bugs.
Proposal
To a large extent, testing of this part of the code base could simply be built on one or more of the available platforms. We could then run the tests on that platform, and announce that the code base works fine on that platform. Obviously the platform needs to support all possible features of U-Boot.
However, this approach fails to take advantage of two useful properties of this code:
- It is cross-platform, and even runs on x86
- It is standalone, requiring no OS libraries to run
For speed, debugging and convenience, it would be nice to run U-Boot under a generic Linux environment on a workstation, and test all the generic non-platform code. The basic problem with this is that the non-platform code requires the platform code to operate. Even the x86 platform code is designed for standalone operation on a particular x86 board, and is not suitable for running under x86 Linux.
To get around this I propose that we create a new ‘native’ architecture. We write code in ‘arch/native’ which can run under Linux. Since all the non-platform code will happily run under this new ‘architecture’, we can then write tests which run quickly under x86 Linux (or another Linux for that matter). This U-Boot 'architecture' should build natively on any 32/64-bit Linux machine since it just uses standard Linux system calls. Calls to Linux would be entirely within this arch/native subdirectory.
Benefit
What will this buy us? Well we can do things like:
- Create a test SPI flash device, which is file-backed. Use this to
write a test of the cmd_sf layer by issuing a series of commands and making sure that the correct SPI flash contents is obtained
- Create a test MMC or IDE device, backed by a file. Use this to issue
ext2 and fat commands to manipulate the filesystem. Then loopback mount the file and check from Linux that U-Boot did the right thing
- Create a test Ethernet device with a mocked remote host attached.
Use this to issue network commands like bootp and tftp and check that the correct results are obtained.
Ultimately (one day) we might have a set of unit tests for U-Boot which can run to very quickly check that a patch does not break the core U-Boot code.
Comments
At this early stage I am looking for comments on the concept - how useful it is and how best to implement it.
Hi Simon,
sounds just awesome. I was planning on doing a similar thing, but "from the other side". I'm sure you're aware of the POST test framework. I was planning to either extend it or write new self-standing implementation that'd allow debuging platform-dependent drivers.
I can CC you on that if you have some interest.
Regards, Simon _______________________________________________ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
On Thursday, August 25, 2011 08:58:00 Simon Glass wrote:
Proposal
for people who might be familiar with the barebox boot loader, Simon is basically proposing the same thing as barebox's "sandbox" target.
to avoid confusion/fragmentation in this area, and since the name is a large part bike shedding, i propose we adopt the same name that barebox has. that way we can sanely use the same term when comparing/contrasting.
from the barebox website: Sandbox
If you develop features for Barebox, you can use the 'sandbox' target which compiles Barebox as a POSIX application in the Linux userspace: it can be started like a normal command and even has network access (tun/tap). Files from the local filesytem can be used to simulate devices.
Comments
At this early stage I am looking for comments on the concept - how useful it is and how best to implement it.
i certainly think it's a worthwhile exercise, especially if it gets us to the point where we can do `make check` and have confidence in our common code changes not being completely f-ed up. -mike
On Thu, Aug 25, 2011 at 8:01 AM, Mike Frysinger vapier@gentoo.org wrote:
On Thursday, August 25, 2011 08:58:00 Simon Glass wrote:
Proposal
for people who might be familiar with the barebox boot loader, Simon is basically proposing the same thing as barebox's "sandbox" target.
to avoid confusion/fragmentation in this area, and since the name is a large part bike shedding, i propose we adopt the same name that barebox has. that way we can sanely use the same term when comparing/contrasting.
from the barebox website: Sandbox
If you develop features for Barebox, you can use the 'sandbox' target which compiles Barebox as a POSIX application in the Linux userspace: it can be started like a normal command and even has network access (tun/tap). Files from the local filesytem can be used to simulate devices.
Makes good sense.
Comments
At this early stage I am looking for comments on the concept - how useful it is and how best to implement it.
i certainly think it's a worthwhile exercise, especially if it gets us to the point where we can do `make check` and have confidence in our common code changes not being completely f-ed up.
Indeed, I am very excited about this as well. :)
Thanks, Anton
-mike
U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
Hi Simon, Mike
On Fri, Aug 26, 2011 at 1:01 AM, Mike Frysinger vapier@gentoo.org wrote:
On Thursday, August 25, 2011 08:58:00 Simon Glass wrote:
Proposal
for people who might be familiar with the barebox boot loader, Simon is basically proposing the same thing as barebox's "sandbox" target.
to avoid confusion/fragmentation in this area, and since the name is a large part bike shedding, i propose we adopt the same name that barebox has. that way we can sanely use the same term when comparing/contrasting.
Yes, I agree that naming the target 'sandbox' is a good idea
Comments
At this early stage I am looking for comments on the concept - how useful it is and how best to implement it.
i certainly think it's a worthwhile exercise, especially if it gets us to the point where we can do `make check` and have confidence in our common code changes not being completely f-ed up.
Indead - And I think it will serve as further impetus to arch-neutralise code outside /arch/
Regards,
Graeme
Hi,
Thanks all for comments. It sounds like people are keen on the idea so far as it goes. I will work on a patch set complete enough to bring up a U-Boot prompt, allowing typing of 'help' and with a special segfault feature for anything else.
Before I do this, and to avoid me redoing work later:
1. What should I call the architecture? I have so far called it 'native'.
2. What should I call the vendor (board/xxx)? 'test' or 'sandbox'?
3. What should I call the board? Is that 'sandbox'?
4. When I create a driver, like the serial test driver, should that be serial_test.c, test_serial.c, sandbox_serial or something else?
5. I need an arch/xxx/lib/board.c. Rather than try to create a new minimal one, I think I should copy an existing one and change it as little as possible (in the light of Graeme's comments and to avoid yet another version of this file). Thoughts?
Regarding the actual test framework, I think this can come later. Clearly we can do unit tests at the file level, or more integration-like tests such as I listed in my email. But I know very little about Ruby (little girl up the road?)
Summary of suggestions so that I don't flood this thread with replies:
Andreas Bießmann: why don't use some unit testing framework like cunit, or ceedling (which can do HW mocks easily)?
- the test framework is TBD - as Mike commented this is something else - another level that can be piled on later.
Marek Vasut: sounds just awesome. I was planning on doing a similar thing, but "from the other side". I'm sure you're aware of the POST test framework. I was planning to either extend it or write new self-standing implementation that'd allow debuging platform-dependent drivers.
- sounds great to me - perhaps we could have a standard test for SPI flash, another for NAND, another for USB, etc.
Mike Frysinger: i certainly think it's a worthwhile exercise, especially if it gets us to the point where we can do `make check` and have confidence in our common code changes not being completely f-ed up.
- that is my hope, particularly if 'make check' is fast.
Wolfgang Denk: In a later step it might even be possible to implement some "pass through" mode to acces actual devices / drivers in some way. Think of using libusb to talk to USB devices.
- Yes - that would allow checking of more layers of the USB stack. But I even see some value in just mocking out the whole USB stack and providing a pretend USB memory stick as a block device..
Wolfgang Denk: I'm not sure what you mean by "a mocked remote host". We should be able to send and receive packets from a real network interface as well.
- I mean that the tftp command will 'obtain' a file when it asks for one, although the actual Ethernet layer is mocked and doesn't actually go out on the wire. Imagine an Ethernet driver which has a half-baked tftp server in it. Yes I also see value in actually using machine interfaces since the testing can be more thorough.
Wolfgang Denk: Some parts of what you are looking for can be done by running U-Boot images in a QEMU based emulated envrionment. So far we have only a few targets in the tree that actually support this. I wish we had more of them.
...I am aware that this is a different thing from what you propose here, but I would like to state that we should not forget this option - anybody who wants to work in that area will be more than welcome, too.
-Yes, both are valuable. Maybe one day we will get a s/w model of an SOC when it is released. For now, this sort of thing is quite a bit of work. There are loads of automated ways to test things- ones I can think of are:
1. real hardware, with some sort of control board which can push buttons, produce/access Ethernet/USB traffic, camera to watch the LCD, LCD to driver the camera, etc.
2. test code which runs on real hardware either at the board level or at the U-Boot command level. Requires a user to watch / driver things
3. test code which runs on faked hardware (this is what I am proposing here). Basically the opposite of 2.
4. test code could run on real hardware, but is actually run on QEMU etc., with special hooks in the simulator for injecting failure, handling the other end of ethernet//USB, etc..A lot of work but really cool.
5. unit tests for each file, where anything that it calls is mocked and replaced with something which can inject failure, etc. We should have more of this anyway. People often don't like writing it though :-(
Graeme Russ: Indead - And I think it will serve as further impetus to arch-neutralise code outside /arch/
- Yes - let's hope so.
Regards, Simon
Hi Simon,
On Fri, Aug 26, 2011 at 1:32 PM, Simon Glass sjg@chromium.org wrote:
Hi,
Thanks all for comments. It sounds like people are keen on the idea so far as it goes. I will work on a patch set complete enough to bring up a U-Boot prompt, allowing typing of 'help' and with a special segfault feature for anything else.
Before I do this, and to avoid me redoing work later:
- What should I call the architecture? I have so far called it 'native'.
sandbox
- What should I call the vendor (board/xxx)? 'test' or 'sandbox'?
sandbox
- What should I call the board? Is that 'sandbox'?
sandbox
- When I create a driver, like the serial test driver, should that be
serial_test.c, test_serial.c, sandbox_serial or something else?
I guess you'll have /drivers/serial/sandbox.c, /drivers/net/sandbox.c etc.
/include/configs/sandbox.h will need to include defaults for how these devices are configured. For example, you may want to have the sandbox serial go to /dev/ttyS0 or /dev/ttyS1
Make sure that printf() goes through U-Boot printf() not the host's libc
And have you dealt with putc() and getc() hooking so that the U-Boot stdio can go to either the hosts stdio or a serial port?
- I need an arch/xxx/lib/board.c. Rather than try to create a new
minimal one, I think I should copy an existing one and change it as little as possible (in the light of Graeme's comments and to avoid yet another version of this file). Thoughts?
Make this board an example of a New World Order ;) - Seriously, take arch/x86/lib/board.c as it has arrays for both init_f and init_r. If you look at board_init_f it has very little other than the loop. But board_init_r is a mess - It starts out as a loop and finished being a sequence of function calls. Implement board_init_r so it consists of only a processing loop - Debatable as to whether or not to but the while(1) { main_loop;} as a final 'init' function or whether to have this after the init loop is processed
I imagine that sandbox will have no _f init functions as there will be no relocation anyway. Still, put in board_init_f but have an empty sequence.
[snip]
Wolfgang Denk: I'm not sure what you mean by "a mocked remote host". We should be able to send and receive packets from a real network interface as well.
- I mean that the tftp command will 'obtain' a file when it asks for
one, although the actual Ethernet layer is mocked and doesn't actually go out on the wire. Imagine an Ethernet driver which has a half-baked tftp server in it. Yes I also see value in actually using machine interfaces since the testing can be more thorough.
Yes, using a real interface (loopback even) would be good as you can then also test that your host's tftp server is working
[snip]
Regards,
Graeme
On Friday, August 26, 2011 00:36:15 Graeme Russ wrote:
On Fri, Aug 26, 2011 at 1:32 PM, Simon Glass wrote:
- When I create a driver, like the serial test driver, should that be
serial_test.c, test_serial.c, sandbox_serial or something else?
I guess you'll have /drivers/serial/sandbox.c, /drivers/net/sandbox.c etc.
/include/configs/sandbox.h will need to include defaults for how these devices are configured. For example, you may want to have the sandbox serial go to /dev/ttyS0 or /dev/ttyS1
since we get a main() entry point, we can make these into runtime flags
Make sure that printf() goes through U-Boot printf() not the host's libc
u-boot already takes care of this by running the linker directly. it is the compiler driver (i.e. `gcc`) that adds the implicit -lc and friends.
And have you dealt with putc() and getc() hooking so that the U-Boot stdio can go to either the hosts stdio or a serial port?
that would be the problem of the sandbox serial driver, and i dont think it'd be that hard. simply use read/write syscalls directly :). -mike
Hi Mike,
On Fri, Aug 26, 2011 at 1:59 PM, Mike Frysinger vapier@gentoo.org wrote:
On Friday, August 26, 2011 00:36:15 Graeme Russ wrote:
On Fri, Aug 26, 2011 at 1:32 PM, Simon Glass wrote:
- When I create a driver, like the serial test driver, should that be
serial_test.c, test_serial.c, sandbox_serial or something else?
I guess you'll have /drivers/serial/sandbox.c, /drivers/net/sandbox.c etc.
/include/configs/sandbox.h will need to include defaults for how these devices are configured. For example, you may want to have the sandbox serial go to /dev/ttyS0 or /dev/ttyS1
since we get a main() entry point, we can make these into runtime flags
Yes that's the plan.
Make sure that printf() goes through U-Boot printf() not the host's libc
u-boot already takes care of this by running the linker directly. it is the compiler driver (i.e. `gcc`) that adds the implicit -lc and friends.
My plan is actually to have a minimal link script and mostly do things as the native compiler intends.
And have you dealt with putc() and getc() hooking so that the U-Boot stdio can go to either the hosts stdio or a serial port?
that would be the problem of the sandbox serial driver, and i dont think it'd be that hard. simply use read/write syscalls directly :).
Yes - I am thinking of an os.c file which contains the OS interface - it will include Linux headers and no common.h, so avoid conflicts.
Regards, Simon
-mike
On Thursday, August 25, 2011 23:32:38 Simon Glass wrote:
- What should I call the architecture? I have so far called it 'native'.
- What should I call the vendor (board/xxx)? 'test' or 'sandbox'?
- What should I call the board? Is that 'sandbox'?
as Graeme said, just call them all "sandbox"
- When I create a driver, like the serial test driver, should that be
serial_test.c, test_serial.c, sandbox_serial or something else?
i think it depends on its function. if the serial driver actually goes to std{in,err,out}, then perhaps "serial/sandbox_stdio.c". let's not assume we'll only ever have one pseudo driver that we can use under the sandbox :).
Wolfgang Denk: I'm not sure what you mean by "a mocked remote host". We should be able to send and receive packets from a real network interface as well.
- I mean that the tftp command will 'obtain' a file when it asks for
one, although the actual Ethernet layer is mocked and doesn't actually go out on the wire. Imagine an Ethernet driver which has a half-baked tftp server in it. Yes I also see value in actually using machine interfaces since the testing can be more thorough.
why not just build on top of tun/tap ? then we do get "real" network traffic, and you dont have to write your own tftp server because you can simply use the same exact one on your development machine that the board would connect to. -mike
Hi Mike,
On Fri, Aug 26, 2011 at 1:55 PM, Mike Frysinger vapier@gentoo.org wrote:
On Thursday, August 25, 2011 23:32:38 Simon Glass wrote:
- What should I call the architecture? I have so far called it 'native'.
- What should I call the vendor (board/xxx)? 'test' or 'sandbox'?
- What should I call the board? Is that 'sandbox'?
as Graeme said, just call them all "sandbox"
OK, sandbox it is.
- When I create a driver, like the serial test driver, should that be
serial_test.c, test_serial.c, sandbox_serial or something else?
i think it depends on its function. if the serial driver actually goes to std{in,err,out}, then perhaps "serial/sandbox_stdio.c". let's not assume we'll only ever have one pseudo driver that we can use under the sandbox :).
OK well I suppose I can start with sandbox and we can see where it takes us.
Wolfgang Denk: I'm not sure what you mean by "a mocked remote host". We should be able to send and receive packets from a real network interface as well.
- I mean that the tftp command will 'obtain' a file when it asks for
one, although the actual Ethernet layer is mocked and doesn't actually go out on the wire. Imagine an Ethernet driver which has a half-baked tftp server in it. Yes I also see value in actually using machine interfaces since the testing can be more thorough.
why not just build on top of tun/tap ? then we do get "real" network traffic, and you dont have to write your own tftp server because you can simply use the same exact one on your development machine that the board would connect to. -mike
Because then you need to set up a real tftp server. It's fine to do what you suggest, but if possible it would be nice to have self-contained tests also, so long as it isn't too much work.
Regards, Simon
Hi Simon,
On 27/08/11 10:25, Simon Glass wrote:
Hi Mike,
On Fri, Aug 26, 2011 at 1:55 PM, Mike Frysinger vapier@gentoo.org wrote:
On Thursday, August 25, 2011 23:32:38 Simon Glass wrote:
[snip]
- I mean that the tftp command will 'obtain' a file when it asks for
one, although the actual Ethernet layer is mocked and doesn't actually go out on the wire. Imagine an Ethernet driver which has a half-baked tftp server in it. Yes I also see value in actually using machine interfaces since the testing can be more thorough.
why not just build on top of tun/tap ? then we do get "real" network traffic, and you dont have to write your own tftp server because you can simply use the same exact one on your development machine that the board would connect to. -mike
Because then you need to set up a real tftp server. It's fine to do what you suggest, but if possible it would be nice to have self-contained tests also, so long as it isn't too much work.
I don't consider having to set up a tftp server as a bad thing - Quite the opposite really. There is plenty of network code in U-Boot that will benefit from testing under the sandbox target because it is much easier to debug. And there will be minimal impact on U-Boot code (just a sandbox 'Ethernet' driver is all that will be needed)
I am reminded of when the R&D department that developed to eNET board were doing the firmware development before the first prototypes were available - The created a HAL which used pcap from memory. I wasn't part of that team, so I really don't know the details...
Regards,
Graeme
Hi Graeme,
On Fri, Aug 26, 2011 at 7:23 PM, Graeme Russ graeme.russ@gmail.com wrote:
Hi Simon,
On 27/08/11 10:25, Simon Glass wrote:
Hi Mike,
On Fri, Aug 26, 2011 at 1:55 PM, Mike Frysinger vapier@gentoo.org wrote:
On Thursday, August 25, 2011 23:32:38 Simon Glass wrote:
[snip]
- I mean that the tftp command will 'obtain' a file when it asks for
one, although the actual Ethernet layer is mocked and doesn't actually go out on the wire. Imagine an Ethernet driver which has a half-baked tftp server in it. Yes I also see value in actually using machine interfaces since the testing can be more thorough.
why not just build on top of tun/tap ? then we do get "real" network traffic, and you dont have to write your own tftp server because you can simply use the same exact one on your development machine that the board would connect to. -mike
Because then you need to set up a real tftp server. It's fine to do what you suggest, but if possible it would be nice to have self-contained tests also, so long as it isn't too much work.
I don't consider having to set up a tftp server as a bad thing - Quite the opposite really. There is plenty of network code in U-Boot that will benefit from testing under the sandbox target because it is much easier to debug. And there will be minimal impact on U-Boot code (just a sandbox 'Ethernet' driver is all that will be needed)
I am reminded of when the R&D department that developed to eNET board were doing the firmware development before the first prototypes were available - The created a HAL which used pcap from memory. I wasn't part of that team, so I really don't know the details...
Yes, sounds good, will put you down for the network testing part :-)
I will work on a basic patch set and try to get it to the list early next week.
Regards, Simon
Regards,
Graeme
Hi Simon,
On Tue, Aug 30, 2011 at 8:04 AM, Simon Glass sjg@chromium.org wrote:
Hi Graeme,
On Fri, Aug 26, 2011 at 7:23 PM, Graeme Russ graeme.russ@gmail.com wrote:
Hi Simon,
On 27/08/11 10:25, Simon Glass wrote:
Hi Mike,
On Fri, Aug 26, 2011 at 1:55 PM, Mike Frysinger vapier@gentoo.org wrote:
On Thursday, August 25, 2011 23:32:38 Simon Glass wrote:
[snip]
I don't consider having to set up a tftp server as a bad thing - Quite the opposite really. There is plenty of network code in U-Boot that will benefit from testing under the sandbox target because it is much easier to debug. And there will be minimal impact on U-Boot code (just a sandbox 'Ethernet' driver is all that will be needed)
I am reminded of when the R&D department that developed to eNET board were doing the firmware development before the first prototypes were available - The created a HAL which used pcap from memory. I wasn't part of that team, so I really don't know the details...
Yes, sounds good, will put you down for the network testing part :-)
Arghh - Rope, meet neck. Neck, meet rope ;)
I will work on a basic patch set and try to get it to the list early next week.
Sounds good
Regards,
Graeme
Dear Simon Glass,
In message CAPnjgZ3ZqFhZgtCAUSRngU3qB5ajxO9FthkmyuWurAWYkEvAHg@mail.gmail.com you wrote:
Summary: I am quite keen on improving the test infrastructure in U-Boot. I would like to have a test suite that can run in a minute or two on a Linux PC and test all non-platform code.
I highly appreaciate such efforts!
For speed, debugging and convenience, it would be nice to run U-Boot under a generic Linux environment on a workstation, and test all the generic non-platform code. The basic problem with this is that the non-platform code requires the platform code to operate. Even the x86 platform code is designed for standalone operation on a particular x86 board, and is not suitable for running under x86 Linux.
Some parts of what you are looking for can be done by running U-Boot images in a QEMU based emulated envrionment. So far we have only a few targets in the tree that actually support this. I wish we had more of them.
I am aware that this is a different thing from what you propose here, but I would like to state that we should not forget this option - anybody who wants to work in that area will be more than welcome, too.
To get around this I propose that we create a new native architecture. We write code in arch/native which can run under Linux. Since all the non-platform code will happily run under this new architecture, we can then write tests which run quickly under x86 Linux (or another Linux for that matter). This U-Boot 'architecture' should build natively on any 32/64-bit Linux machine since it just uses standard Linux system calls. Calls to Linux would be entirely within this arch/native subdirectory.
As Mike already pointed out, this is what BB has as "sandbox" configuration.
- Create a test SPI flash device, which is file-backed. Use this to
write a test of the cmd_sf layer by issuing a series of commands and making sure that the correct SPI flash contents is obtained
- Create a test MMC or IDE device, backed by a file. Use this to issue
ext2 and fat commands to manipulate the filesystem. Then loopback mount the file and check from Linux that U-Boot did the right thing
In a later step it might even be possible to implement some "pass through" mode to acces actual devices / drivers in some way. Think of using libusb to talk to USB devices.
- Create a test Ethernet device with a mocked remote host attached.
I'm not sure what you mean by "a mocked remote host". We should be able to send and receive packets from a real network interface as well.
At this early stage I am looking for comments on the concept - how useful it is and how best to implement it.
Any step in this direction, how inclomplete it may be in the beginning, is highly appreciated. It may make sense to look over the fence and study how BBis doing this.
Best regards,
Wolfgang Denk
On Thursday, August 25, 2011 16:21:51 Wolfgang Denk wrote:
Simon Glass wrote:
For speed, debugging and convenience, it would be nice to run U-Boot under a generic Linux environment on a workstation, and test all the generic non-platform code. The basic problem with this is that the non-platform code requires the platform code to operate. Even the x86 platform code is designed for standalone operation on a particular x86 board, and is not suitable for running under x86 Linux.
Some parts of what you are looking for can be done by running U-Boot images in a QEMU based emulated envrionment. So far we have only a few targets in the tree that actually support this. I wish we had more of them.
I am aware that this is a different thing from what you propose here, but I would like to state that we should not forget this option - anybody who wants to work in that area will be more than welcome, too.
i do a lot of testing using the GNU sim and Blackfin boards. the goal there is that the GNU sim is able to execute all of the Blackfin builds *unmodified*, so i dont need sep configs. as you say, both methods are very useful to have. -mike
participants (7)
-
Andreas Bießmann -
Anton Staaf -
Graeme Russ -
Marek Vasut -
Mike Frysinger -
Simon Glass -
Wolfgang Denk