[PATCH 1/5] Azure: Correct comment about the strategy in the world matrix build
At this point noting that we have a split in our job similar to TravisCI (which we have not used in years) isn't helpful, and is also not true anymore either. Instead, explain that we split the world up in to 10 jobs as that's the maximum we can have going in parallel on the free tier of Azure.
Signed-off-by: Tom Rini trini@konsulko.com --- .azure-pipelines.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/.azure-pipelines.yml b/.azure-pipelines.yml index 27f69583c655..f662f41872fd 100644 --- a/.azure-pipelines.yml +++ b/.azure-pipelines.yml @@ -473,8 +473,8 @@ stages: pool: vmImage: $(ubuntu_vm) strategy: - # Use almost the same target division in .travis.yml, only merged - # 3 small build jobs (arc/microblaze/xtensa) into one. + # We split the world up in to 10 jobs as we can have at most 10 + # parallel jobs going on the free tier of Azure. matrix: am33xx_at91_kirkwood_mvebu_omap: BUILDMAN: "am33xx at91_kirkwood mvebu omap -x siemens"
Instead of defining BUILDMAN to the value we'll build in each part of the matrix job, define a variable with that name and have it list what to build. This will allow us to reference these multiple times consistently later on.
Signed-off-by: Tom Rini trini@konsulko.com --- .azure-pipelines.yml | 31 +++++++++++++++++++++---------- 1 file changed, 21 insertions(+), 10 deletions(-)
diff --git a/.azure-pipelines.yml b/.azure-pipelines.yml index f662f41872fd..6d3bdcee8aac 100644 --- a/.azure-pipelines.yml +++ b/.azure-pipelines.yml @@ -8,6 +8,17 @@ variables: # since our $(ci_runner_image) user is not root. container_option: -u 0 work_dir: /u + # We define all of these as variables so we can easily reference them twice + am33xx_at91_kirkwood_mvebu_omap: "am33xx at91_kirkwood mvebu omap -x siemens" + amlogic_bcm_boundary_engicam_siemens_technexion_oradex: "amlogic bcm boundary engicam siemens technexion toradex -x mips" + arm_nxp_minus_imx: "freescale -x powerpc,m68k,imx,mx" + imx: "mx imx -x boundary,engicam,technexion,toradex" + rk: "rk" + sunxi: "sunxi" + powerpc: "powerpc" + arm_catch_all: "arm -x aarch64,am33xx,at91,bcm,ls1,kirkwood,mvebu,omap,rk,siemens,mx,sunxi,technexion,toradex" + aarch64_catch_all: "aarch64 -x amlogic,bcm,engicam,imx,ls1,ls2,lx216,mvebu,rk,siemens,sunxi,toradex" + everything_but_arm_and_powerpc: "-x arm,powerpc"
stages: - stage: testsuites @@ -477,25 +488,25 @@ stages: # parallel jobs going on the free tier of Azure. matrix: am33xx_at91_kirkwood_mvebu_omap: - BUILDMAN: "am33xx at91_kirkwood mvebu omap -x siemens" + BUILDMAN: $(am33xx_at91_kirkwood_mvebu_omap) amlogic_bcm_boundary_engicam_siemens_technexion_oradex: - BUILDMAN: "amlogic bcm boundary engicam siemens technexion toradex -x mips" + BUILDMAN: $(amlogic_bcm_boundary_engicam_siemens_technexion_oradex) arm_nxp_minus_imx: - BUILDMAN: "freescale -x powerpc,m68k,imx,mx" + BUILDMAN: $(arm_nxp_minus_imx) imx: - BUILDMAN: "mx imx -x boundary,engicam,technexion,toradex" + BUILDMAN: $(imx) rk: - BUILDMAN: "rk" + BUILDMAN: $(rk) sunxi: - BUILDMAN: "sunxi" + BUILDMAN: $(sunxi) powerpc: - BUILDMAN: "powerpc" + BUILDMAN: $(powerpc) arm_catch_all: - BUILDMAN: "arm -x aarch64,am33xx,at91,bcm,ls1,kirkwood,mvebu,omap,rk,siemens,mx,sunxi,technexion,toradex" + BUILDMAN: $(arm_catch_all) aarch64_catch_all: - BUILDMAN: "aarch64 -x amlogic,bcm,engicam,imx,ls1,ls2,lx216,mvebu,rk,siemens,sunxi,toradex" + BUILDMAN: $(aarch64_catch_all) everything_but_arm_and_powerpc: - BUILDMAN: "-x arm,powerpc" + BUILDMAN: $(everything_but_arm_and_powerpc) steps: - script: | cat << EOF > build.sh
In order to get the list of boards that will be done in a "dry run" build we need to have something listed and not just an exclude list. Populate the job with all architecture directories except arm and powerpc.
Signed-off-by: Tom Rini trini@konsulko.com --- .azure-pipelines.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.azure-pipelines.yml b/.azure-pipelines.yml index 6d3bdcee8aac..2c2d0809cab1 100644 --- a/.azure-pipelines.yml +++ b/.azure-pipelines.yml @@ -18,7 +18,7 @@ variables: powerpc: "powerpc" arm_catch_all: "arm -x aarch64,am33xx,at91,bcm,ls1,kirkwood,mvebu,omap,rk,siemens,mx,sunxi,technexion,toradex" aarch64_catch_all: "aarch64 -x amlogic,bcm,engicam,imx,ls1,ls2,lx216,mvebu,rk,siemens,sunxi,toradex" - everything_but_arm_and_powerpc: "-x arm,powerpc" + everything_but_arm_and_powerpc: "arc m68k microblaze mips nios2 riscv sandbox sh x86 xtensa -x arm,powerpc"
stages: - stage: testsuites
As part of commit 9aeac898da66 ("Azure: Rework build the world jobs") I made a few mistakes. An errant '_' meant that we built neither at91 nor kirkwood platforms. Further, the non-freescale (NXP) "LS1xxx" platforms were also not being built. Adjust some jobs to have these be built again.
Fixes: 9aeac898da66 ("Azure: Rework build the world jobs") Signed-off-by: Tom Rini trini@konsulko.com --- Cc: Jiaxun Yang jiaxun.yang@flygoat.com --- .azure-pipelines.yml | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
diff --git a/.azure-pipelines.yml b/.azure-pipelines.yml index 2c2d0809cab1..116ce903b067 100644 --- a/.azure-pipelines.yml +++ b/.azure-pipelines.yml @@ -9,9 +9,9 @@ variables: container_option: -u 0 work_dir: /u # We define all of these as variables so we can easily reference them twice - am33xx_at91_kirkwood_mvebu_omap: "am33xx at91_kirkwood mvebu omap -x siemens" + am33xx_kirkwood_ls1_mvebu_omap: "am33xx kirkwood ls1 mvebu omap -x siemens,freescale" amlogic_bcm_boundary_engicam_siemens_technexion_oradex: "amlogic bcm boundary engicam siemens technexion toradex -x mips" - arm_nxp_minus_imx: "freescale -x powerpc,m68k,imx,mx" + arm_nxp_minus_imx_and_at91: "at91 freescale -x powerpc,m68k,imx,mx" imx: "mx imx -x boundary,engicam,technexion,toradex" rk: "rk" sunxi: "sunxi" @@ -487,12 +487,12 @@ stages: # We split the world up in to 10 jobs as we can have at most 10 # parallel jobs going on the free tier of Azure. matrix: - am33xx_at91_kirkwood_mvebu_omap: - BUILDMAN: $(am33xx_at91_kirkwood_mvebu_omap) + am33xx_kirkwood_ls1_mvebu_omap: + BUILDMAN: $(am33xx_kirkwood_ls1_mvebu_omap) amlogic_bcm_boundary_engicam_siemens_technexion_oradex: BUILDMAN: $(amlogic_bcm_boundary_engicam_siemens_technexion_oradex) - arm_nxp_minus_imx: - BUILDMAN: $(arm_nxp_minus_imx) + arm_nxp_minus_imx_and_at91: + BUILDMAN: $(arm_nxp_minus_imx_and_at91) imx: BUILDMAN: $(imx) rk:
Now that we have each stage of the world build using variables to define what it will attempt to build, and that we have added in missing machines, add a job to make sure that we would always be building everything.
Signed-off-by: Tom Rini trini@konsulko.com --- .azure-pipelines.yml | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+)
diff --git a/.azure-pipelines.yml b/.azure-pipelines.yml index 116ce903b067..c43bb51066a5 100644 --- a/.azure-pipelines.yml +++ b/.azure-pipelines.yml @@ -196,6 +196,34 @@ stages: steps: - script: make pip
+ - job: count_built_machines + displayName: 'Ensure we build all possible machines' + pool: + vmImage: $(ubuntu_vm) + container: + image: $(ci_runner_image) + options: $(container_option) + steps: + - script: | + BMANARGS="-o /tmp --dry-run -v" + # First get the total number of boards + total=$(tools/buildman/buildman ${BMANARGS} | grep "Total boards to build for each commit" | cut -d ' ' -f 8) + # Now build up the list of what each job built. + built="$(tools/buildman/buildman ${BMANARGS} $(am33xx_kirkwood_ls1_mvebu_omap) | grep '^ ')" + built="$built $(tools/buildman/buildman ${BMANARGS} $(amlogic_bcm_boundary_engicam_siemens_technexion_oradex) | grep '^ ')" + built="$built $(tools/buildman/buildman ${BMANARGS} $(arm_nxp_minus_imx_and_at91) | grep '^ ')" + built="$built $(tools/buildman/buildman ${BMANARGS} $(imx) | grep '^ ')" + built="$built $(tools/buildman/buildman ${BMANARGS} $(rk) | grep '^ ')" + built="$built $(tools/buildman/buildman ${BMANARGS} $(sunxi) | grep '^ ')" + built="$built $(tools/buildman/buildman ${BMANARGS} $(powerpc) | grep '^ ')" + built="$built $(tools/buildman/buildman ${BMANARGS} $(arm_catch_all) | grep '^ ')" + built="$built $(tools/buildman/buildman ${BMANARGS} $(aarch64_catch_all) | grep '^ ')" + built="$built $(tools/buildman/buildman ${BMANARGS} $(everything_but_arm_and_powerpc) | grep '^ ')" + # Finally see how many machines that is. + actual=$(tools/buildman/buildman ${BMANARGS} $built | grep "Total boards to build for each commit" | cut -d ' ' -f 8) + echo We would build a total of $actual out of $total platforms this CI run + [ $actual -eq $total ] && exit 0 || exit 1 + - job: create_test_py_wrapper_script displayName: 'Create and stage a wrapper for test.py runs' pool:
On Tue, 18 Jun 2024 08:18:47 -0600, Tom Rini wrote:
At this point noting that we have a split in our job similar to TravisCI (which we have not used in years) isn't helpful, and is also not true anymore either. Instead, explain that we split the world up in to 10 jobs as that's the maximum we can have going in parallel on the free tier of Azure.
[...]
Applied to u-boot/next, thanks!
participants (1)
-
Tom Rini