On Sun, Nov 20, 2011 at 11:15 PM, Simon Glass sjg@chromium.org wrote:
Hi Mike,
On Sun, Nov 20, 2011 at 1:23 PM, Mike Frysinger vapier@gentoo.org wrote:
On Thursday 03 November 2011 03:28:29 Andy Fleming wrote:
you don't need those semicolons. also, this is not as good as it should be: if you're running 10 jobs in parallel, you fork 10, and then you wait for all of them to finish before forking another set of 10.
what you could do is something like: JOB_IDX=0 JOB_IDX_FIRST=0 BUILD_NBUILDS=1
... foreach target ... ; do build_target & pids[$(( JOB_IDX++ ))]=$! if [ $(( JOB_IDX - JOB_IDX_FIRST )) -ge ${BUILD_NBUILDS} ] ; then wait ${pids[$(( JOB_IDX_FIRST++ ))]} fi done wait
this isn't perfect as it assumes the first job launched will always finish first, but it's a lot closer than the current code.
Since all the jobs are launched at the same time this is not really a valid assumption is it? IMO on this point it's good enough as it is for now...
Mike's idea led me to a slightly crazier one that (I think) works. Basically, I have each build_targets invocation create a file named LOG/build_N when it is done, where N is the value of TOTAL_CNT when it is invoked. Then I keep track of the oldest uncompleted build #. When we hit our build limit, the script scans for LOG/build_N, for N from oldest to TOTAL_CNT, and updates the count of outstanding builds, as appropriate (and moves the index of the oldest). If the scanning completes without freeing up room for another build, it waits for a second, and then re-scans.
I tried this on my 24-core system with BUILD_NBUILDS set to 8 and BUILD_NCPUS set to 4, and pegged the system at ~24-30 the whole time.
I will note that it only shaved about a minute off the 15 minutes it took to build all 85xx. I'm still debugging a bit (I want to try 16 builds/2 cores, and clean up a bit), but I figured I'd ask what people thought of this approach.
My next patch will also address Mike's comments. If I'm feeling inspired, I might address some of your suggestions (I think I can use the build-completion files to coalesce some of the debug output).
Andy