On 29/08/2023 17:37, Andrejs Cainikovs wrote:
On 29/08/2023 17:04, Tom Rini wrote:
Interesting. How exactly are you using these CI images? We have tools/buildman/requirements.txt now to cover the newly-added modules in CI, as we call pip on that. But, it's not being used in the prepopulate the pip cache stage in the Dockerfile. And, I'm not immune to the idea that we should instead be installing the distro package here.
Tom, I got the full picture right after I sent you my previous email. This particular file, tools/buildman/requirements.txt in used in *CI*, right.
Well, in this case my patch doesn't make much sense. Or it does.
I'd like to propose to install packages to runner image instead. I see the reason why it might be convenient to have packages installed to runner for particular jobs via .gitlab.yml - a bit less maintenance of Docker image, which would need to be rebuilt every time there's new dependency. However, if you think about pipeline execution times, on a global scale, that's a waste. As an example, I see four occasions of:
pip install -r tools/buildman/requirements.txt
Twice in world build stage and twice in testsuites stage.
If we take another example, packages from py/requirements.txt are installed 37 (if I counted correctly) times during entire testsuite stage [2]. And I see you run them a lot [1].
Since you are maintainer of this, I will leave this up to you. But if you're up for this proposal, I can do this change by myself. Of course, this would imply rebuilding Docker image and pushing to registry from your side, to keep stuff synchronized.
[1]: https://source.denx.de/u-boot/u-boot/-/pipelines/17556 [2]: https://source.denx.de/u-boot/u-boot/-/jobs/685270#L366
Regards, Andrejs.
I see in Dockerfile test/py/requirements.txt and doc/sphinx/requirements.txt, but not tools/buildman/requirements.txt, so whatever is mentioned there is not part of the image on CI runners.
Following is the part of *our own* .gitlab.yml file, which uses a public Docker image (docker.io/trini/u-boot-gitlab-ci-runner:jammy-20230624-20Jul2023, as of now):
verdin-am62: stage: build script: | ... tools/buildman/buildman -o /tmp -P -E "verdin-am62" -M ...
And this is how it fails in our CI without this change:
Building current source for 3 boards (3 threads, 14 jobs per thread) arm: + verdin-am62_r5 +binman: Unknown entry type 'ti-board-config' in node '/binman/board-cfg/ti-board-config' (expected etype/ti_board_config.py, error 'No module named 'jsonschema'' +make[1]: *** [Makefile:1108: .binman_stamp] Error 1 +make: *** [Makefile:177: sub-make] Error 2