Hi Niklaus,
Niklaus Giger wrote:
Hi
As I consider testing as an important part to ensure high code quality for any product. It should form part of the global development process.
- When adding a new board or feature to U-Boot running tests to ensure that
it works as advertised should be mandatory but not time consuming.
Yes.
- If one has a board up and running, I consider it adviseable to ensure that
it always compiles the actual state of the U-Boot code and passes all the tests.
Yes!
- It would be nice to have somewhere a central repository where everybody
(specially the maintainers) could see whether the different boards pass all the tests or not.
Yes. Something to consider: Rockbox has a distributed build/report framework that we might be able to use or at least learn from (Daniel Stenberg is a u-boot user/contributor).
http://www.rockbox.org/wiki/BuildClient -8<-------------------------------------------------------------------->8- Why not distcc
distcc is a great and fun project, but unsuitable for use by us. Since distcc still preprocesses each file locally, hand over the file to the remote servers to get build and then gets the object file back, it requires slow compiles and fast networks to be really efficient.
In contrast we have relatively fast compiles and a slow network, to the extent that transferring the file over and getting an object file back is slower than simply compiling it locally. This is of course also due to the somewhat slow network between our build servers. -8<--------------------------------------------------------------------->8-
To achieve the points 1) and 2) I am working on a solution which should achieve these goals if you meet the following requirements
- you have a spare board to run the tests.
- you have some HW to switch on/off the power (There are solutions that switch 4 or 8 outlet for 100 or 200 Euros).
Jon Diekema has the NP-0201D and likes it (the temperature probe isn't useful). Looks like the HP-02H (220vAC) would be the European version.
http://www.synaccess-net.com/remote_power_products.php/1/8 - $200 NP-02 (simpler)
http://www.synaccess-net.com/remote_power_products.php/1/36 - $245 NP-0201D (adds current, etc)
- you have system running a GNU/Linux. (My system runs Debian Lenny) which has some background processing power.
- you have (at least weekly) some time to look after the results and report/fix the problems.
- have 1 to 4 hours to setup a testing environment.
I would be interested what you think about the points raised above and whether following the instructions given under http://ngiger.dyndns.org/duts_v2/doc/index.html and http://ngiger.dyndns.org/hudson.doc/doc/index.html allows you to achieve above mentioned goals 1) and 2).
Background:
In 2004 - 2006 I run for a while a buildbot for the Xenomai project.
In the last two months I used my pre-existing Ruby scripts and the DENX Universal Test Suite (http://www.denx.de/wiki/DUTS/DUTSDocs) to rewrite into DUTS-v2 (http://ngiger.dyndns.org/duts_v2/doc/index.html). Quite some effort did go into writing unit tests for test suite. Easy adaption to various environments was a important design factor.
Ruby - yes! I didn't realize how much you've already done. TCL - meh, I have used it but never warmed up to it.
My previous experiences with the continuos integrations tools http://buildbot.net/trac and http://cruisecontrol.sourceforge.net/ made me have a look at https://hudson.dev.java.net/.
Hudson proved to be really simple to setup and administer. My experiences are documented at http://ngiger.dyndns.org/hudson.doc/doc/index.html.
I've heard good things about hudson, have not done anything in it or buildbot. I've looked at cruisecontrol XML files (minor hacking) - writing programs (build scripts) in a markup language is evil (hudson apparently fixes this by hiding the XML).
Future extensions:
a) Use the same infrastructure not only for U-Boot but also for related projects like xenomai, RT-preempt, ltp.
b) To achieve goal 3) I could not find any project around the linux kernel which publishes in an automated way test results.
See above http://www.rockbox.org/wiki/BuildClient (perl :-/ since I'm being a language snob tonight).
If I don't receive other suggestions I plan the following steps:
- Create a XML file with test test results containings
- version of the test suite/scripts
- information about the target like
- HW configuration (CPU, RAM, devices)
- SW version/configuration
- name and result of each teststep
- performance information (e.g. min/max/avg, histogram) for parameters like latency, throughput
- ???
- Send XML file at end of test-run to a special mailing list
- Process mailing lists to feed all the XML into a database
- Web-Frontend to this database using ??? (Ruby On Rails comes to my mind)
Any comments to these points?
I've been wanting to do this for my own use (regression testing for the FDT command handling, board support regression testing) but have not made time to do it. :-( I appreciate you blazing a path.
Next steps:
- Listen to your suggestions
- Fix bugs reported
- Move code from my SVN-repository to git.denx.de ?
- Open wiki entries for points 1 - 3 ?
Yes to all four points.
Best regards
Niklaus
Thanks! gvb