Hi Mike,
On Mon, Nov 21, 2011 at 11:00 AM, Mike Frysinger vapier@gentoo.org wrote:
On Monday 21 November 2011 00:15:21 Simon Glass wrote:
On Sun, Nov 20, 2011 at 1:23 PM, Mike Frysinger wrote:
On Thursday 03 November 2011 03:28:29 Andy Fleming wrote:
- # Limit number of parallel builds
- if [ ${CURRENT_COUNT} -gt ${BUILD_NBUILDS} ] ; then
- CURRENT_COUNT=0;
- wait;
fi
you don't need those semicolons. also, this is not as good as it should be: if you're running 10 jobs in parallel, you fork 10, and then you wait for all of them to finish before forking another set of 10.
what you could do is something like: JOB_IDX=0 JOB_IDX_FIRST=0 BUILD_NBUILDS=1
... foreach target ... ; do build_target & pids[$(( JOB_IDX++ ))]=$! if [ $(( JOB_IDX - JOB_IDX_FIRST )) -ge ${BUILD_NBUILDS} ] ; then wait ${pids[$(( JOB_IDX_FIRST++ ))]} fi done wait
this isn't perfect as it assumes the first job launched will always finish first, but it's a lot closer than the current code.
Since all the jobs are launched at the same time this is not really a valid assumption is it? IMO on this point it's good enough as it is for now...
my point was, it's about as valid as you can get in bash. you can't do non- blocking wait's in bash, nor can you wait on a list of pids and have it return only one at a time. the launching-jobs-at-the-same-time isn't really the worse part: builds can vary greatly in terms of how much code they actually compile. some boards are very small while others are quite fat.
so your choices are: - make a general "good enough" assumption (which i did here) - have the main thread manually poll every child through some IPC mech such as looking for files the children touch when they're done, and *then* do the wait explicitly on that child
You missed one:
- write it in Python :-)
(which I might try if patman is accepted)
the problem with the latter is that the code tends to get much more complicated, and the main thread ends up doing quite a bit of busy work which kills CPU cycles. my original assumption ends up keeping the CPU's fairly loaded i think, and is fairly simplistic code. -mike
It should be easy enough in Python to just maintain 20 threads (or whatever) and start the next job when a thread becomes idle.
Regards, Simon