On Mon, Nov 21, 2011 at 1:01 PM, Mike Frysinger vapier@gentoo.org wrote:
On Monday 21 November 2011 01:28:50 Andy Fleming wrote:
On Sun, Nov 20, 2011 at 11:15 PM, Simon Glass wrote:
On Sun, Nov 20, 2011 at 1:23 PM, Mike Frysinger wrote:
On Thursday 03 November 2011 03:28:29 Andy Fleming wrote:
you don't need those semicolons. also, this is not as good as it should be: if you're running 10 jobs in parallel, you fork 10, and then you wait for all of them to finish before forking another set of 10.
what you could do is something like: JOB_IDX=0 JOB_IDX_FIRST=0 BUILD_NBUILDS=1
... foreach target ... ; do build_target & pids[$(( JOB_IDX++ ))]=$! if [ $(( JOB_IDX - JOB_IDX_FIRST )) -ge ${BUILD_NBUILDS} ] ; then wait ${pids[$(( JOB_IDX_FIRST++ ))]} fi done wait
this isn't perfect as it assumes the first job launched will always finish first, but it's a lot closer than the current code.
Since all the jobs are launched at the same time this is not really a valid assumption is it? IMO on this point it's good enough as it is for now...
Mike's idea led me to a slightly crazier one that (I think) works. Basically, I have each build_targets invocation create a file named LOG/build_N when it is done, where N is the value of TOTAL_CNT when it is invoked. Then I keep track of the oldest uncompleted build #. When we hit our build limit, the script scans for LOG/build_N, for N from oldest to TOTAL_CNT, and updates the count of outstanding builds, as appropriate (and moves the index of the oldest). If the scanning completes without freeing up room for another build, it waits for a second, and then re-scans.
sounds like it would work, but i'd want to see how much code it takes to actually implement this
I tried this on my 24-core system with BUILD_NBUILDS set to 8 and BUILD_NCPUS set to 4, and pegged the system at ~24-30 the whole time.
I will note that it only shaved about a minute off the 15 minutes it took to build all 85xx.
compare to what ? your original patch, or my tweaked version ?
Compared to my original. However, I think that may be due to my configuration. 8 builds with -j4 makes was 14, but 16 builds with -j2 resulted in just over 10 minutes. I'm doing more testing.
Andy