On 7/2/19 6:04 PM, Troy Benjegerdes wrote:
On Jun 22, 2019, at 2:43 PM, Marek Vasut marex@denx.de wrote:
On 6/22/19 9:12 PM, Heinrich Schuchardt wrote:
On 6/22/19 8:15 PM, Simon Glass wrote:
Hi,
On Sat, 22 Jun 2019 at 16:10, Andreas Färber afaerber@suse.de wrote:
Hi Simon,
Am 22.06.19 um 16:55 schrieb Simon Glass:
I'd like to better understand the benefits of the 3-month timeline.
It takes time to learn about a release, package and build it, test it on various hardware, investigate and report errors, wait for feedback and fixes, rinse and repeat with the next -rc. Many people don't do this as their main job.
If we shorten the release cycle, newer boards will get out faster (which is good) but the overall quality of boards not actively worked on (because they were working good enough before) will decay, which is bad. The only way to counteract that would be to automatically test on real hardware rather than just building, and doing that for all these masses of boards seems unrealistic.
Here I think you are talking about distributions. But why not just take every second release?
I have certain had the experience of getting a board our of the cupboard and finding that the latest U-Boot doesn't work, nor the one before, nor the three before that.
Are we actually seeing an improvement in regressions? I feel that testing is the only way to get that.
Perhaps we should select a small subset of boards which do get tested, and actually have custodians build/test on those for every rc?
What I have been doing before all my recent pull requests is to boot both an arm32 (Orange Pi) and and an aarch64 (Pine A64 LTS) board via bootefi and GRUB. To make this easier I am using a Raspberry with a relay board and a Tizen SD-Wire card (https://wiki.tizen.org/SDWire) controlling the system under test, cf https://pbs.twimg.com/media/D5ugi3iX4AAh1bn.jpg:large What would be needed is scripts to automate the testing including all the Python tests.
It would make sense to have such test automation for all of our architectures similar to what Kernel CI (https://kernelci.org/) does.
So who's gonna set it up and host it ?
I just got the infrastructure going to do this for the HiFive Unleashed (RiscV port), but that’s only one board right now.
I’d propose that one of the responsibilities of being a custodian/ maintainer for a board and/or arch is a commitment to run a *simple* automated testing framework on a set of boards.
I’ve looked into KenrelCI enough to see that it seems rather complex to get up and running. We need a dead-simple setup (a few debian packages? A container? An SDcard image for a BeagleBone?) that can collect serial console output and power cycle a board.
My proposal would be that you take a few exceptionally weird, disparate, boards and try to implement dead-simple CI around those. You'll start hitting all kinds of weird issues in the process and the dead-simple will quickly start turning into complicated :)
You can, however, explore the U-Boot test.py, tbot, or whatever other test frameworks there are and try to come up with some easy setup howto.
Eventually maybe we should have a Tizen SDWire or something like that, however that requires some real money for board development since I can’t seem to find a source for where I can buy an SDWire.
Not all boards boot from SD cards though, some boot from hard-to-recover non-replaceable boot media.
With the HiFive Unleashed in SiFive’s test lab, we use OpenOCD for JTAG, all I need is one USB cable and I can load U-boot via JTAG, and boot a recovery image, and reload the SDcard, so the SDwire is not really necessary for boards that have easy JTAG setup.
btw booting from JTAG and cold-booting from boot media may trigger different bugs. You most certainly want to install the bootloader and try both cold and warm boot.