Hi,
Thanks all for comments. It sounds like people are keen on the idea so far as it goes. I will work on a patch set complete enough to bring up a U-Boot prompt, allowing typing of 'help' and with a special segfault feature for anything else.
Before I do this, and to avoid me redoing work later:
1. What should I call the architecture? I have so far called it 'native'.
2. What should I call the vendor (board/xxx)? 'test' or 'sandbox'?
3. What should I call the board? Is that 'sandbox'?
4. When I create a driver, like the serial test driver, should that be serial_test.c, test_serial.c, sandbox_serial or something else?
5. I need an arch/xxx/lib/board.c. Rather than try to create a new minimal one, I think I should copy an existing one and change it as little as possible (in the light of Graeme's comments and to avoid yet another version of this file). Thoughts?
Regarding the actual test framework, I think this can come later. Clearly we can do unit tests at the file level, or more integration-like tests such as I listed in my email. But I know very little about Ruby (little girl up the road?)
Summary of suggestions so that I don't flood this thread with replies:
Andreas Bießmann: why don't use some unit testing framework like cunit, or ceedling (which can do HW mocks easily)?
- the test framework is TBD - as Mike commented this is something else - another level that can be piled on later.
Marek Vasut: sounds just awesome. I was planning on doing a similar thing, but "from the other side". I'm sure you're aware of the POST test framework. I was planning to either extend it or write new self-standing implementation that'd allow debuging platform-dependent drivers.
- sounds great to me - perhaps we could have a standard test for SPI flash, another for NAND, another for USB, etc.
Mike Frysinger: i certainly think it's a worthwhile exercise, especially if it gets us to the point where we can do `make check` and have confidence in our common code changes not being completely f-ed up.
- that is my hope, particularly if 'make check' is fast.
Wolfgang Denk: In a later step it might even be possible to implement some "pass through" mode to acces actual devices / drivers in some way. Think of using libusb to talk to USB devices.
- Yes - that would allow checking of more layers of the USB stack. But I even see some value in just mocking out the whole USB stack and providing a pretend USB memory stick as a block device..
Wolfgang Denk: I'm not sure what you mean by "a mocked remote host". We should be able to send and receive packets from a real network interface as well.
- I mean that the tftp command will 'obtain' a file when it asks for one, although the actual Ethernet layer is mocked and doesn't actually go out on the wire. Imagine an Ethernet driver which has a half-baked tftp server in it. Yes I also see value in actually using machine interfaces since the testing can be more thorough.
Wolfgang Denk: Some parts of what you are looking for can be done by running U-Boot images in a QEMU based emulated envrionment. So far we have only a few targets in the tree that actually support this. I wish we had more of them.
...I am aware that this is a different thing from what you propose here, but I would like to state that we should not forget this option - anybody who wants to work in that area will be more than welcome, too.
-Yes, both are valuable. Maybe one day we will get a s/w model of an SOC when it is released. For now, this sort of thing is quite a bit of work. There are loads of automated ways to test things- ones I can think of are:
1. real hardware, with some sort of control board which can push buttons, produce/access Ethernet/USB traffic, camera to watch the LCD, LCD to driver the camera, etc.
2. test code which runs on real hardware either at the board level or at the U-Boot command level. Requires a user to watch / driver things
3. test code which runs on faked hardware (this is what I am proposing here). Basically the opposite of 2.
4. test code could run on real hardware, but is actually run on QEMU etc., with special hooks in the simulator for injecting failure, handling the other end of ethernet//USB, etc..A lot of work but really cool.
5. unit tests for each file, where anything that it calls is mocked and replaced with something which can inject failure, etc. We should have more of this anyway. People often don't like writing it though :-(
Graeme Russ: Indead - And I think it will serve as further impetus to arch-neutralise code outside /arch/
- Yes - let's hope so.
Regards, Simon