On Thu, Nov 07, 2013 at 08:26:57PM +0100, Wolfgang Denk wrote:
Dear Tom,
In message 20131107133159.GR5925@bill-the-cat you wrote:
I feel this is the hard part of the problem, and what we're glossing over. What has to be tested by the board maintainer? What are we going to leave to their discretion? Will am335x_evm not count if I don't dig up the NOR cape for it?
Good question. Eventually this is something that develops over time.
Intially, we might be satisfied with a very basic "it works" message, which may just mean that this specific version booted on the actual hardware.
In the long run, we might provide a more detailed questionaire to the reported. I could for example imagine a tool that parses the board's config file and then provides some checkboxes - if there is NOR flash configured on the board, ask if NOR has been tested; similar for network, MMC, USB, ... other features.
One day we might even have more developers using automatic test tools so we could generate information on a per-command base.
I know that I'm just dreaming, but we should try to just be open for any such future extensions, even if we start really small now.
I think we all agree that _any_ kind of test information will be better than none.
What we need to be careful of here is making sure whatever we grow is both useful and not overly complicated. What I honestly wonder about is automated testing for commands (crc32 pops to mind only because I just fixed things) but otherwise having things broken down into a front end where people select what they did "Booted a ___ into ___ via ___", provide some output from a command (maybe add just a touch more info to 'version') and cover non-boot testing with copy/paste'able drop-downs.
I know automated testing is The Thing, but given N frameworks, everyone of them has issues because frankly, every SoC family has its own quirks about how boot and load and what is and is not even feasible, especially for the bootloader.