For those used to MAKEALL, buildman seems strange. Add some notes to ease the transition.

Signed-off-by: Simon Glass sjg@chromium.org ---

Changes in v5: None Changes in v4: None Changes in v3: None Changes in v2: - Minor changes to the text

tools/buildman/README | 92 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 92 insertions(+)

diff --git a/tools/buildman/README b/tools/buildman/README index a5d181c..1c919af 100644 --- a/tools/buildman/README +++ b/tools/buildman/README @@ -3,6 +3,8 @@ # SPDX-License-Identifier: GPL-2.0+ #

+(Please read 'How to change from MAKEALL' if you are used to that tool) + What is this? =============

@@ -663,6 +665,96 @@ Other options Buildman has various other command line options. Try --help to see them.

+How to change from MAKEALL +========================== + +Buildman includes most of the features of MAKEALL and is generally faster +and easier to use. In particular it builds entire branches: if a particular +commit introduces an error in a particular board, buildman can easily show +you this, even if a later commit fixes that error. + +The reasons to deprecate MAKEALL are: +- We don't want to maintain two build systems +- Buildman is typically faster +- Buildman has a lot more features + +But still, many people will be sad to lose MAKEALL. If you are used to +MAKEALL, here are a few pointers. + +First you need to set up your tool chains - see the 'Setting up' section +for details. Once you have your required toolchain(s) detected then you are +ready to go. + +Buildman works on entire branches, so the normal use is: + + ./tools/buildman/buildman -b <branch_name> <list of things to build> + +followed by (afterwards, or perhaps concurrently in another terminal): + + ./tools/buildman/buildman -b <branch_name> -s <list of things to build> + +to see the results of the build. Rather than showing you all the output, +buildman just shows a summary, with red indicating that a commit introduced +an error and green indicating that a commit fixed an error. Use the -e +flag to see the full errors. + +You don't need to stick around on that branch while buildman is running. It +checks out its own copy of the source code, so you can change branches, +add commits, etc. without affecting the build in progress. + +The <list of things to build> can include board names, architectures or the +like. There are no flags to disambiguate since ambiguities are rare. Using +the examples from MAKEALL: + +Examples: + - build all Power Architecture boards: + MAKEALL -a powerpc + MAKEALL --arch powerpc + MAKEALL powerpc + ** buildman -b <branch> powerpc + - build all PowerPC boards manufactured by vendor "esd": + MAKEALL -a powerpc -v esd + ** buildman -b <branch> esd + - build all PowerPC boards manufactured either by "keymile" or "siemens": + MAKEALL -a powerpc -v keymile -v siemens + ** buildman -b <branch> keymile siemens + - build all Freescale boards with MPC83xx CPUs, plus all 4xx boards: + MAKEALL -c mpc83xx -v freescale 4xx + ** buildman -b <branch> mpc83xx freescale 4xx + +Buildman automatically tries to use all the CPUs in your machine. If you +are building a lot of boards it will use one thread for every CPU core +it detects in your machine. This is like MAKEALL's BUILD_NBUILDS option. +You can use the -T flag to change the number of threads. If you are only +building a few boards, buildman will automatically run make with the -j +flag to increase the number of concurrent make tasks. It isn't normally +that helpful to fiddle with this option, but if you use the BUILD_NCPUS +option in MAKEALL then -j is the equivalent in buildman. + +Buildman puts its output in ../<branch_name> by default but you can change +this with the -o option. Buildman normally does out-of-tree builds: use -i +to disable that if you really want to. But be careful that once you have +used -i you pollute buildman's copies of the source tree, and you will need +to remove the build directory (normally ../<branch_name>) to run buildman +in normal mode (without -i). + +Buildman doesn't keep the output result normally, but use the -k option to +do this. + +Please read 'Theory of Operation' a few times as it will make a lot of +things clearer. + +Some options you might like are: + + -B shows which functions are growing/shrinking in which commit - great + for finding code bloat. + -S shows image sizes for each commit (just an overall summary) + -u shows boards that you haven't built yet + --step 0 will build just the upstream commit and the last commit of your + branch. This is often a quick sanity check that your branch doesn't + break anything. But note this does not check bisectability! + + TODO ====

The builder.py file is getting too long, so split out some code.

Signed-off-by: Simon Glass sjg@chromium.org ---

Changes in v5: None Changes in v4: - Fix missing import in builder.py

Changes in v3: - Add new patch to move BuilderThread code to its own file

Changes in v2: None

tools/buildman/builder.py | 439 +--------------------------------------- tools/buildman/builderthread.py | 434 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 442 insertions(+), 431 deletions(-) create mode 100644 tools/buildman/builderthread.py

diff --git a/tools/buildman/builder.py b/tools/buildman/builder.py index d2b72d5..e2da0eb 100644 --- a/tools/buildman/builder.py +++ b/tools/buildman/builder.py @@ -6,7 +6,6 @@ #

import collections -import errno from datetime import datetime, timedelta import glob import os @@ -15,9 +14,9 @@ import Queue import shutil import string import sys -import threading import time

+import builderthread import command import gitutil import terminal @@ -97,428 +96,6 @@ OUTCOME_OK, OUTCOME_WARNING, OUTCOME_ERROR, OUTCOME_UNKNOWN = range(4) trans_valid_chars = string.maketrans("/: ", "---")

-def Mkdir(dirname): - """Make a directory if it doesn't already exist. - - Args: - dirname: Directory to create - """ - try: - os.mkdir(dirname) - except OSError as err: - if err.errno == errno.EEXIST: - pass - else: - raise - -class BuilderJob: - """Holds information about a job to be performed by a thread - - Members: - board: Board object to build - commits: List of commit options to build. - """ - def __init__(self): - self.board = None - self.commits = [] - - -class ResultThread(threading.Thread): - """This thread processes results from builder threads. - - It simply passes the results on to the builder. There is only one - result thread, and this helps to serialise the build output. - """ - def __init__(self, builder): - """Set up a new result thread - - Args: - builder: Builder which will be sent each result - """ - threading.Thread.__init__(self) - self.builder = builder - - def run(self): - """Called to start up the result thread. - - We collect the next result job and pass it on to the build. - """ - while True: - result = self.builder.out_queue.get() - self.builder.ProcessResult(result) - self.builder.out_queue.task_done() - - -class BuilderThread(threading.Thread): - """This thread builds U-Boot for a particular board. - - An input queue provides each new job. We run 'make' to build U-Boot - and then pass the results on to the output queue. - - Members: - builder: The builder which contains information we might need - thread_num: Our thread number (0-n-1), used to decide on a - temporary directory - """ - def __init__(self, builder, thread_num): - """Set up a new builder thread""" - threading.Thread.__init__(self) - self.builder = builder - self.thread_num = thread_num - - def Make(self, commit, brd, stage, cwd, *args, **kwargs): - """Run 'make' on a particular commit and board. - - The source code will already be checked out, so the 'commit' - argument is only for information. - - Args: - commit: Commit object that is being built - brd: Board object that is being built - stage: Stage of the build. Valid stages are: - distclean - can be called to clean source - config - called to configure for a board - build - the main make invocation - it does the build - args: A list of arguments to pass to 'make' - kwargs: A list of keyword arguments to pass to command.RunPipe() - - Returns: - CommandResult object - """ - return self.builder.do_make(commit, brd, stage, cwd, *args, - **kwargs) - - def RunCommit(self, commit_upto, brd, work_dir, do_config, force_build, - force_build_failures): - """Build a particular commit. - - If the build is already done, and we are not forcing a build, we skip - the build and just return the previously-saved results. - - Args: - commit_upto: Commit number to build (0...n-1) - brd: Board object to build - work_dir: Directory to which the source will be checked out - do_config: True to run a make <board>_defconfig on the source - force_build: Force a build even if one was previously done - force_build_failures: Force a bulid if the previous result showed - failure - - Returns: - tuple containing: - - CommandResult object containing the results of the build - - boolean indicating whether 'make config' is still needed - """ - # Create a default result - it will be overwritte by the call to - # self.Make() below, in the event that we do a build. - result = command.CommandResult() - result.return_code = 0 - if self.builder.in_tree: - out_dir = work_dir - else: - out_dir = os.path.join(work_dir, 'build') - - # Check if the job was already completed last time - done_file = self.builder.GetDoneFile(commit_upto, brd.target) - result.already_done = os.path.exists(done_file) - will_build = (force_build or force_build_failures or - not result.already_done) - if result.already_done and will_build: - # Get the return code from that build and use it - with open(done_file, 'r') as fd: - result.return_code = int(fd.readline()) - err_file = self.builder.GetErrFile(commit_upto, brd.target) - if os.path.exists(err_file) and os.stat(err_file).st_size: - result.stderr = 'bad' - elif not force_build: - # The build passed, so no need to build it again - will_build = False - - if will_build: - # We are going to have to build it. First, get a toolchain - if not self.toolchain: - try: - self.toolchain = self.builder.toolchains.Select(brd.arch) - except ValueError as err: - result.return_code = 10 - result.stdout = '' - result.stderr = str(err) - # TODO(sjg@chromium.org): This gets swallowed, but needs - # to be reported. - - if self.toolchain: - # Checkout the right commit - if self.builder.commits: - commit = self.builder.commits[commit_upto] - if self.builder.checkout: - git_dir = os.path.join(work_dir, '.git') - gitutil.Checkout(commit.hash, git_dir, work_dir, - force=True) - else: - commit = 'current' - - # Set up the environment and command line - env = self.toolchain.MakeEnvironment() - Mkdir(out_dir) - args = [] - cwd = work_dir - if not self.builder.in_tree: - if commit_upto is None: - # In this case we are building in the original source - # directory (i.e. the current directory where buildman - # is invoked. The output directory is set to this - # thread's selected work directory. - # - # Symlinks can confuse U-Boot's Makefile since - # we may use '..' in our path, so remove them. - work_dir = os.path.realpath(work_dir) - args.append('O=%s/build' % work_dir) - cwd = None - else: - args.append('O=build') - args.append('-s') - if self.builder.num_jobs is not None: - args.extend(['-j', str(self.builder.num_jobs)]) - config_args = ['%s_defconfig' % brd.target] - config_out = '' - args.extend(self.builder.toolchains.GetMakeArguments(brd)) - - # If we need to reconfigure, do that now - if do_config: - result = self.Make(commit, brd, 'distclean', cwd, - 'distclean', *args, env=env) - result = self.Make(commit, brd, 'config', cwd, - *(args + config_args), env=env) - config_out = result.combined - do_config = False # No need to configure next time - if result.return_code == 0: - result = self.Make(commit, brd, 'build', cwd, *args, - env=env) - result.stdout = config_out + result.stdout - else: - result.return_code = 1 - result.stderr = 'No tool chain for %s\n' % brd.arch - result.already_done = False - - result.toolchain = self.toolchain - result.brd = brd - result.commit_upto = commit_upto - result.out_dir = out_dir - return result, do_config - - def _WriteResult(self, result, keep_outputs): - """Write a built result to the output directory. - - Args: - result: CommandResult object containing result to write - keep_outputs: True to store the output binaries, False - to delete them - """ - # Fatal error - if result.return_code < 0: - return - - # Aborted? - if result.stderr and 'No child processes' in result.stderr: - return - - if result.already_done: - return - - # Write the output and stderr - output_dir = self.builder._GetOutputDir(result.commit_upto) - Mkdir(output_dir) - build_dir = self.builder.GetBuildDir(result.commit_upto, - result.brd.target) - Mkdir(build_dir) - - outfile = os.path.join(build_dir, 'log') - with open(outfile, 'w') as fd: - if result.stdout: - fd.write(result.stdout) - - errfile = self.builder.GetErrFile(result.commit_upto, - result.brd.target) - if result.stderr: - with open(errfile, 'w') as fd: - fd.write(result.stderr) - elif os.path.exists(errfile): - os.remove(errfile) - - if result.toolchain: - # Write the build result and toolchain information. - done_file = self.builder.GetDoneFile(result.commit_upto, - result.brd.target) - with open(done_file, 'w') as fd: - fd.write('%s' % result.return_code) - with open(os.path.join(build_dir, 'toolchain'), 'w') as fd: - print >>fd, 'gcc', result.toolchain.gcc - print >>fd, 'path', result.toolchain.path - print >>fd, 'cross', result.toolchain.cross - print >>fd, 'arch', result.toolchain.arch - fd.write('%s' % result.return_code) - - with open(os.path.join(build_dir, 'toolchain'), 'w') as fd: - print >>fd, 'gcc', result.toolchain.gcc - print >>fd, 'path', result.toolchain.path - - # Write out the image and function size information and an objdump - env = result.toolchain.MakeEnvironment() - lines = [] - for fname in ['u-boot', 'spl/u-boot-spl']: - cmd = ['%snm' % self.toolchain.cross, '--size-sort', fname] - nm_result = command.RunPipe([cmd], capture=True, - capture_stderr=True, cwd=result.out_dir, - raise_on_error=False, env=env) - if nm_result.stdout: - nm = self.builder.GetFuncSizesFile(result.commit_upto, - result.brd.target, fname) - with open(nm, 'w') as fd: - print >>fd, nm_result.stdout, - - cmd = ['%sobjdump' % self.toolchain.cross, '-h', fname] - dump_result = command.RunPipe([cmd], capture=True, - capture_stderr=True, cwd=result.out_dir, - raise_on_error=False, env=env) - rodata_size = '' - if dump_result.stdout: - objdump = self.builder.GetObjdumpFile(result.commit_upto, - result.brd.target, fname) - with open(objdump, 'w') as fd: - print >>fd, dump_result.stdout, - for line in dump_result.stdout.splitlines(): - fields = line.split() - if len(fields) > 5 and fields[1] == '.rodata': - rodata_size = fields[2] - - cmd = ['%ssize' % self.toolchain.cross, fname] - size_result = command.RunPipe([cmd], capture=True, - capture_stderr=True, cwd=result.out_dir, - raise_on_error=False, env=env) - if size_result.stdout: - lines.append(size_result.stdout.splitlines()[1] + ' ' + - rodata_size) - - # Write out the image sizes file. This is similar to the output - # of binutil's 'size' utility, but it omits the header line and - # adds an additional hex value at the end of each line for the - # rodata size - if len(lines): - sizes = self.builder.GetSizesFile(result.commit_upto, - result.brd.target) - with open(sizes, 'w') as fd: - print >>fd, '\n'.join(lines) - - # Now write the actual build output - if keep_outputs: - patterns = ['u-boot', '*.bin', 'u-boot.dtb', '*.map', - 'include/autoconf.mk', 'spl/u-boot-spl', - 'spl/u-boot-spl.bin'] - for pattern in patterns: - file_list = glob.glob(os.path.join(result.out_dir, pattern)) - for fname in file_list: - shutil.copy(fname, build_dir) - - - def RunJob(self, job): - """Run a single job - - A job consists of a building a list of commits for a particular board. - - Args: - job: Job to build - """ - brd = job.board - work_dir = self.builder.GetThreadDir(self.thread_num) - self.toolchain = None - if job.commits: - # Run 'make board_defconfig' on the first commit - do_config = True - commit_upto = 0 - force_build = False - for commit_upto in range(0, len(job.commits), job.step): - result, request_config = self.RunCommit(commit_upto, brd, - work_dir, do_config, - force_build or self.builder.force_build, - self.builder.force_build_failures) - failed = result.return_code or result.stderr - did_config = do_config - if failed and not do_config: - # If our incremental build failed, try building again - # with a reconfig. - if self.builder.force_config_on_failure: - result, request_config = self.RunCommit(commit_upto, - brd, work_dir, True, True, False) - did_config = True - if not self.builder.force_reconfig: - do_config = request_config - - # If we built that commit, then config is done. But if we got - # an warning, reconfig next time to force it to build the same - # files that created warnings this time. Otherwise an - # incremental build may not build the same file, and we will - # think that the warning has gone away. - # We could avoid this by using -Werror everywhere... - # For errors, the problem doesn't happen, since presumably - # the build stopped and didn't generate output, so will retry - # that file next time. So we could detect warnings and deal - # with them specially here. For now, we just reconfigure if - # anything goes work. - # Of course this is substantially slower if there are build - # errors/warnings (e.g. 2-3x slower even if only 10% of builds - # have problems). - if (failed and not result.already_done and not did_config and - self.builder.force_config_on_failure): - # If this build failed, try the next one with a - # reconfigure. - # Sometimes if the board_config.h file changes it can mess - # with dependencies, and we get: - # make: *** No rule to make target `include/autoconf.mk', - # needed by `depend'. - do_config = True - force_build = True - else: - force_build = False - if self.builder.force_config_on_failure: - if failed: - do_config = True - result.commit_upto = commit_upto - if result.return_code < 0: - raise ValueError('Interrupt') - - # We have the build results, so output the result - self._WriteResult(result, job.keep_outputs) - self.builder.out_queue.put(result) - else: - # Just build the currently checked-out build - result, request_config = self.RunCommit(None, brd, work_dir, True, - True, self.builder.force_build_failures) - result.commit_upto = 0 - self._WriteResult(result, job.keep_outputs) - self.builder.out_queue.put(result) - - def run(self): - """Our thread's run function - - This thread picks a job from the queue, runs it, and then goes to the - next job. - """ - alive = True - while True: - job = self.builder.queue.get() - if self.builder.active and alive: - self.RunJob(job) - ''' - try: - if self.builder.active and alive: - self.RunJob(job) - except Exception as err: - alive = False - print err - ''' - self.builder.queue.task_done() - - class Builder: """Class for building U-Boot for a particular commit.

@@ -639,13 +216,13 @@ class Builder: self.queue = Queue.Queue() self.out_queue = Queue.Queue() for i in range(self.num_threads): - t = BuilderThread(self, i) + t = builderthread.BuilderThread(self, i) t.setDaemon(True) t.start() self.threads.append(t)

self.last_line_len = 0 - t = ResultThread(self) + t = builderthread.ResultThread(self) t.setDaemon(True) t.start() self.threads.append(t) @@ -1385,7 +962,7 @@ class Builder: for self.commit_upto in range(self.commit_count): self.SelectCommit(commits[self.commit_upto]) self.SelectOutputDir() - Mkdir(self.output_dir) + builderthread.Mkdir(self.output_dir)

self.BuildBoardsForCommit(board_selected, keep_outputs) board_dict, err_lines = self.GetResultSummary() @@ -1413,7 +990,7 @@ class Builder: setup_git: True to set up a git repo clone """ thread_dir = self.GetThreadDir(thread_num) - Mkdir(thread_dir) + builderthread.Mkdir(thread_dir) git_dir = os.path.join(thread_dir, '.git')

# Clone the repo if it doesn't already exist @@ -1436,7 +1013,7 @@ class Builder: max_threads: Maximum number of threads we expect to need. setup_git: True to set up a git repo clone """ - Mkdir(self._working_dir) + builderthread.Mkdir(self._working_dir) for thread in range(max_threads): self._PrepareThread(thread, setup_git)

@@ -1469,7 +1046,7 @@ class Builder: self.commits = commits

self.ResetResultSummary(board_selected) - Mkdir(self.base_dir) + builderthread.Mkdir(self.base_dir) self._PrepareWorkingSpace(min(self.num_threads, len(board_selected)), commits is not None) self._PrepareOutputSpace() @@ -1478,7 +1055,7 @@ class Builder:

# Create jobs to build all commits for each board for brd in board_selected.itervalues(): - job = BuilderJob() + job = builderthread.BuilderJob() job.board = brd job.commits = commits job.keep_outputs = keep_outputs diff --git a/tools/buildman/builderthread.py b/tools/buildman/builderthread.py new file mode 100644 index 0000000..32297e2 --- /dev/null +++ b/tools/buildman/builderthread.py @@ -0,0 +1,434 @@ +# Copyright (c) 2014 Google, Inc +# +# SPDX-License-Identifier: GPL-2.0+ +# + +import errno +import glob +import os +import shutil +import threading + +import command +import gitutil + +def Mkdir(dirname): + """Make a directory if it doesn't already exist. + + Args: + dirname: Directory to create + """ + try: + os.mkdir(dirname) + except OSError as err: + if err.errno == errno.EEXIST: + pass + else: + raise + +class BuilderJob: + """Holds information about a job to be performed by a thread + + Members: + board: Board object to build + commits: List of commit options to build. + """ + def __init__(self): + self.board = None + self.commits = [] + + +class ResultThread(threading.Thread): + """This thread processes results from builder threads. + + It simply passes the results on to the builder. There is only one + result thread, and this helps to serialise the build output. + """ + def __init__(self, builder): + """Set up a new result thread + + Args: + builder: Builder which will be sent each result + """ + threading.Thread.__init__(self) + self.builder = builder + + def run(self): + """Called to start up the result thread. + + We collect the next result job and pass it on to the build. + """ + while True: + result = self.builder.out_queue.get() + self.builder.ProcessResult(result) + self.builder.out_queue.task_done() + + +class BuilderThread(threading.Thread): + """This thread builds U-Boot for a particular board. + + An input queue provides each new job. We run 'make' to build U-Boot + and then pass the results on to the output queue. + + Members: + builder: The builder which contains information we might need + thread_num: Our thread number (0-n-1), used to decide on a + temporary directory + """ + def __init__(self, builder, thread_num): + """Set up a new builder thread""" + threading.Thread.__init__(self) + self.builder = builder + self.thread_num = thread_num + + def Make(self, commit, brd, stage, cwd, *args, **kwargs): + """Run 'make' on a particular commit and board. + + The source code will already be checked out, so the 'commit' + argument is only for information. + + Args: + commit: Commit object that is being built + brd: Board object that is being built + stage: Stage of the build. Valid stages are: + distclean - can be called to clean source + config - called to configure for a board + build - the main make invocation - it does the build + args: A list of arguments to pass to 'make' + kwargs: A list of keyword arguments to pass to command.RunPipe() + + Returns: + CommandResult object + """ + return self.builder.do_make(commit, brd, stage, cwd, *args, + **kwargs) + + def RunCommit(self, commit_upto, brd, work_dir, do_config, force_build, + force_build_failures): + """Build a particular commit. + + If the build is already done, and we are not forcing a build, we skip + the build and just return the previously-saved results. + + Args: + commit_upto: Commit number to build (0...n-1) + brd: Board object to build + work_dir: Directory to which the source will be checked out + do_config: True to run a make <board>_defconfig on the source + force_build: Force a build even if one was previously done + force_build_failures: Force a bulid if the previous result showed + failure + + Returns: + tuple containing: + - CommandResult object containing the results of the build + - boolean indicating whether 'make config' is still needed + """ + # Create a default result - it will be overwritte by the call to + # self.Make() below, in the event that we do a build. + result = command.CommandResult() + result.return_code = 0 + if self.builder.in_tree: + out_dir = work_dir + else: + out_dir = os.path.join(work_dir, 'build') + + # Check if the job was already completed last time + done_file = self.builder.GetDoneFile(commit_upto, brd.target) + result.already_done = os.path.exists(done_file) + will_build = (force_build or force_build_failures or + not result.already_done) + if result.already_done and will_build: + # Get the return code from that build and use it + with open(done_file, 'r') as fd: + result.return_code = int(fd.readline()) + err_file = self.builder.GetErrFile(commit_upto, brd.target) + if os.path.exists(err_file) and os.stat(err_file).st_size: + result.stderr = 'bad' + elif not force_build: + # The build passed, so no need to build it again + will_build = False + + if will_build: + # We are going to have to build it. First, get a toolchain + if not self.toolchain: + try: + self.toolchain = self.builder.toolchains.Select(brd.arch) + except ValueError as err: + result.return_code = 10 + result.stdout = '' + result.stderr = str(err) + # TODO(sjg@chromium.org): This gets swallowed, but needs + # to be reported. + + if self.toolchain: + # Checkout the right commit + if self.builder.commits: + commit = self.builder.commits[commit_upto] + if self.builder.checkout: + git_dir = os.path.join(work_dir, '.git') + gitutil.Checkout(commit.hash, git_dir, work_dir, + force=True) + else: + commit = 'current' + + # Set up the environment and command line + env = self.toolchain.MakeEnvironment() + Mkdir(out_dir) + args = [] + cwd = work_dir + if not self.builder.in_tree: + if commit_upto is None: + # In this case we are building in the original source + # directory (i.e. the current directory where buildman + # is invoked. The output directory is set to this + # thread's selected work directory. + # + # Symlinks can confuse U-Boot's Makefile since + # we may use '..' in our path, so remove them. + work_dir = os.path.realpath(work_dir) + args.append('O=%s/build' % work_dir) + cwd = None + else: + args.append('O=build') + args.append('-s') + if self.builder.num_jobs is not None: + args.extend(['-j', str(self.builder.num_jobs)]) + config_args = ['%s_defconfig' % brd.target] + config_out = '' + args.extend(self.builder.toolchains.GetMakeArguments(brd)) + + # If we need to reconfigure, do that now + if do_config: + result = self.Make(commit, brd, 'distclean', cwd, + 'distclean', *args, env=env) + result = self.Make(commit, brd, 'config', cwd, + *(args + config_args), env=env) + config_out = result.combined + do_config = False # No need to configure next time + if result.return_code == 0: + result = self.Make(commit, brd, 'build', cwd, *args, + env=env) + result.stdout = config_out + result.stdout + else: + result.return_code = 1 + result.stderr = 'No tool chain for %s\n' % brd.arch + result.already_done = False + + result.toolchain = self.toolchain + result.brd = brd + result.commit_upto = commit_upto + result.out_dir = out_dir + return result, do_config + + def _WriteResult(self, result, keep_outputs): + """Write a built result to the output directory. + + Args: + result: CommandResult object containing result to write + keep_outputs: True to store the output binaries, False + to delete them + """ + # Fatal error + if result.return_code < 0: + return + + # Aborted? + if result.stderr and 'No child processes' in result.stderr: + return + + if result.already_done: + return + + # Write the output and stderr + output_dir = self.builder._GetOutputDir(result.commit_upto) + Mkdir(output_dir) + build_dir = self.builder.GetBuildDir(result.commit_upto, + result.brd.target) + Mkdir(build_dir) + + outfile = os.path.join(build_dir, 'log') + with open(outfile, 'w') as fd: + if result.stdout: + fd.write(result.stdout) + + errfile = self.builder.GetErrFile(result.commit_upto, + result.brd.target) + if result.stderr: + with open(errfile, 'w') as fd: + fd.write(result.stderr) + elif os.path.exists(errfile): + os.remove(errfile) + + if result.toolchain: + # Write the build result and toolchain information. + done_file = self.builder.GetDoneFile(result.commit_upto, + result.brd.target) + with open(done_file, 'w') as fd: + fd.write('%s' % result.return_code) + with open(os.path.join(build_dir, 'toolchain'), 'w') as fd: + print >>fd, 'gcc', result.toolchain.gcc + print >>fd, 'path', result.toolchain.path + print >>fd, 'cross', result.toolchain.cross + print >>fd, 'arch', result.toolchain.arch + fd.write('%s' % result.return_code) + + with open(os.path.join(build_dir, 'toolchain'), 'w') as fd: + print >>fd, 'gcc', result.toolchain.gcc + print >>fd, 'path', result.toolchain.path + + # Write out the image and function size information and an objdump + env = result.toolchain.MakeEnvironment() + lines = [] + for fname in ['u-boot', 'spl/u-boot-spl']: + cmd = ['%snm' % self.toolchain.cross, '--size-sort', fname] + nm_result = command.RunPipe([cmd], capture=True, + capture_stderr=True, cwd=result.out_dir, + raise_on_error=False, env=env) + if nm_result.stdout: + nm = self.builder.GetFuncSizesFile(result.commit_upto, + result.brd.target, fname) + with open(nm, 'w') as fd: + print >>fd, nm_result.stdout, + + cmd = ['%sobjdump' % self.toolchain.cross, '-h', fname] + dump_result = command.RunPipe([cmd], capture=True, + capture_stderr=True, cwd=result.out_dir, + raise_on_error=False, env=env) + rodata_size = '' + if dump_result.stdout: + objdump = self.builder.GetObjdumpFile(result.commit_upto, + result.brd.target, fname) + with open(objdump, 'w') as fd: + print >>fd, dump_result.stdout, + for line in dump_result.stdout.splitlines(): + fields = line.split() + if len(fields) > 5 and fields[1] == '.rodata': + rodata_size = fields[2] + + cmd = ['%ssize' % self.toolchain.cross, fname] + size_result = command.RunPipe([cmd], capture=True, + capture_stderr=True, cwd=result.out_dir, + raise_on_error=False, env=env) + if size_result.stdout: + lines.append(size_result.stdout.splitlines()[1] + ' ' + + rodata_size) + + # Write out the image sizes file. This is similar to the output + # of binutil's 'size' utility, but it omits the header line and + # adds an additional hex value at the end of each line for the + # rodata size + if len(lines): + sizes = self.builder.GetSizesFile(result.commit_upto, + result.brd.target) + with open(sizes, 'w') as fd: + print >>fd, '\n'.join(lines) + + # Now write the actual build output + if keep_outputs: + patterns = ['u-boot', '*.bin', 'u-boot.dtb', '*.map', + 'include/autoconf.mk', 'spl/u-boot-spl', + 'spl/u-boot-spl.bin'] + for pattern in patterns: + file_list = glob.glob(os.path.join(result.out_dir, pattern)) + for fname in file_list: + shutil.copy(fname, build_dir) + + + def RunJob(self, job): + """Run a single job + + A job consists of a building a list of commits for a particular board. + + Args: + job: Job to build + """ + brd = job.board + work_dir = self.builder.GetThreadDir(self.thread_num) + self.toolchain = None + if job.commits: + # Run 'make board_defconfig' on the first commit + do_config = True + commit_upto = 0 + force_build = False + for commit_upto in range(0, len(job.commits), job.step): + result, request_config = self.RunCommit(commit_upto, brd, + work_dir, do_config, + force_build or self.builder.force_build, + self.builder.force_build_failures) + failed = result.return_code or result.stderr + did_config = do_config + if failed and not do_config: + # If our incremental build failed, try building again + # with a reconfig. + if self.builder.force_config_on_failure: + result, request_config = self.RunCommit(commit_upto, + brd, work_dir, True, True, False) + did_config = True + if not self.builder.force_reconfig: + do_config = request_config + + # If we built that commit, then config is done. But if we got + # an warning, reconfig next time to force it to build the same + # files that created warnings this time. Otherwise an + # incremental build may not build the same file, and we will + # think that the warning has gone away. + # We could avoid this by using -Werror everywhere... + # For errors, the problem doesn't happen, since presumably + # the build stopped and didn't generate output, so will retry + # that file next time. So we could detect warnings and deal + # with them specially here. For now, we just reconfigure if + # anything goes work. + # Of course this is substantially slower if there are build + # errors/warnings (e.g. 2-3x slower even if only 10% of builds + # have problems). + if (failed and not result.already_done and not did_config and + self.builder.force_config_on_failure): + # If this build failed, try the next one with a + # reconfigure. + # Sometimes if the board_config.h file changes it can mess + # with dependencies, and we get: + # make: *** No rule to make target `include/autoconf.mk', + # needed by `depend'. + do_config = True + force_build = True + else: + force_build = False + if self.builder.force_config_on_failure: + if failed: + do_config = True + result.commit_upto = commit_upto + if result.return_code < 0: + raise ValueError('Interrupt') + + # We have the build results, so output the result + self._WriteResult(result, job.keep_outputs) + self.builder.out_queue.put(result) + else: + # Just build the currently checked-out build + result, request_config = self.RunCommit(None, brd, work_dir, True, + True, self.builder.force_build_failures) + result.commit_upto = 0 + self._WriteResult(result, job.keep_outputs) + self.builder.out_queue.put(result) + + def run(self): + """Our thread's run function + + This thread picks a job from the queue, runs it, and then goes to the + next job. + """ + alive = True + while True: + job = self.builder.queue.get() + if self.builder.active and alive: + self.RunJob(job) + ''' + try: + if self.builder.active and alive: + self.RunJob(job) + except Exception as err: + alive = False + print err + ''' + self.builder.queue.task_done()

We need the output options to be available in several places. It's a pain to pass them into each function. Make them properties of the builder and add a single function to set them up. At the same time, add a function which produces summary output using these options.

Signed-off-by: Simon Glass sjg@chromium.org ---

Changes in v5: - Update test for output options

Changes in v4: None Changes in v3: - Add new patch to refactor output options

Changes in v2: None

tools/buildman/builder.py | 46 +++++++++++++++++++++++++++------------------- tools/buildman/control.py | 8 ++++---- tools/buildman/test.py | 6 +++--- 3 files changed, 34 insertions(+), 26 deletions(-)

diff --git a/tools/buildman/builder.py b/tools/buildman/builder.py index e2da0eb..a6c43e0 100644 --- a/tools/buildman/builder.py +++ b/tools/buildman/builder.py @@ -235,6 +235,20 @@ class Builder: for t in self.threads: del t

+ def SetDisplayOptions(self, show_errors=False, show_sizes=False, + show_detail=False, show_bloat=False): + """Setup display options for the builder. + + show_errors: True to show summarised error/warning info + show_sizes: Show size deltas + show_detail: Show detail for each board + show_bloat: Show detail for each function + """ + self._show_errors = show_errors + self._show_sizes = show_sizes + self._show_detail = show_detail + self._show_bloat = show_bloat + def _AddTimestamp(self): """Add a new timestamp to the list and record the build period.

@@ -891,9 +905,18 @@ class Builder: print "Boards not built (%d): %s" % (len(not_built), ', '.join(not_built))

+ def ProduceResultSummary(self, commit_upto, commits, board_selected): + board_dict, err_lines = self.GetResultSummary(board_selected, + commit_upto, read_func_sizes=self._show_bloat) + if commits: + msg = '%02d: %s' % (commit_upto + 1, + commits[commit_upto].subject) + print self.col.Color(self.col.BLUE, msg) + self.PrintResultSummary(board_selected, board_dict, + err_lines if self._show_errors else [], + self._show_sizes, self._show_detail, self._show_bloat)

- def ShowSummary(self, commits, board_selected, show_errors, show_sizes, - show_detail, show_bloat): + def ShowSummary(self, commits, board_selected): """Show a build summary for U-Boot for a given board list.

Reset the result summary, then repeatedly call GetResultSummary on @@ -902,27 +925,13 @@ class Builder: Args: commit: Commit objects to summarise board_selected: Dict containing boards to summarise - show_errors: Show errors that occured - show_sizes: Show size deltas - show_detail: Show detail for each board - show_bloat: Show detail for each function """ self.commit_count = len(commits) if commits else 1 self.commits = commits self.ResetResultSummary(board_selected)

for commit_upto in range(0, self.commit_count, self._step): - board_dict, err_lines = self.GetResultSummary(board_selected, - commit_upto, read_func_sizes=show_bloat) - if commits: - msg = '%02d: %s' % (commit_upto + 1, - commits[commit_upto].subject) - else: - msg = 'current' - print self.col.Color(self.col.BLUE, msg) - self.PrintResultSummary(board_selected, board_dict, - err_lines if show_errors else [], show_sizes, show_detail, - show_bloat) + self.ProduceResultSummary(commit_upto, commits, board_selected)

def SetupBuild(self, board_selected, commits): @@ -1032,14 +1041,13 @@ class Builder: if dirname not in dir_list: shutil.rmtree(dirname)

- def BuildBoards(self, commits, board_selected, show_errors, keep_outputs): + def BuildBoards(self, commits, board_selected, keep_outputs): """Build all commits for a list of boards

Args: commits: List of commits to be build, each a Commit object boards_selected: Dict of selected boards, key is target name, value is Board object - show_errors: True to show summarised error/warning info keep_outputs: True to save build output files """ self.commit_count = len(commits) if commits else 1 diff --git a/tools/buildman/control.py b/tools/buildman/control.py index 12a9699..d8fa74b 100644 --- a/tools/buildman/control.py +++ b/tools/buildman/control.py @@ -207,13 +207,13 @@ def DoBuildman(options, args): print GetActionSummary(options.summary, commits, board_selected, options)

+ builder.SetDisplayOptions(options.show_errors, options.show_sizes, + options.show_detail, options.show_bloat) if options.summary: # We can't show function sizes without board details at present if options.show_bloat: options.show_detail = True - builder.ShowSummary(commits, board_selected, - options.show_errors, options.show_sizes, - options.show_detail, options.show_bloat) + builder.ShowSummary(commits, board_selected) else: builder.BuildBoards(commits, board_selected, - options.show_errors, options.keep_outputs) + options.keep_outputs) diff --git a/tools/buildman/test.py b/tools/buildman/test.py index 068784a..c6923af 100644 --- a/tools/buildman/test.py +++ b/tools/buildman/test.py @@ -137,9 +137,9 @@ class TestBuild(unittest.TestCase): board_selected = self.boards.GetSelectedDict()

#build.BuildCommits(self.commits, board_selected, False) - build.BuildBoards(self.commits, board_selected, False, False) - build.ShowSummary(self.commits, board_selected, True, False, - False, False) + build.BuildBoards(self.commits, board_selected, False) + build.SetDisplayOptions(show_errors=True); + build.ShowSummary(self.commits, board_selected)

def _testGit(self): """Test basic builder operation by building a branch"""

The non-incremental build method is no longer used, so remove it.

Signed-off-by: Simon Glass sjg@chromium.org ---

Changes in v5: None Changes in v4: - Add new patch to remove unused non-incremental build method code

Changes in v3: None Changes in v2: None

tools/buildman/builder.py | 34 ---------------------------------- tools/buildman/test.py | 1 - 2 files changed, 35 deletions(-)

diff --git a/tools/buildman/builder.py b/tools/buildman/builder.py index dd7f5a4..22a24b1 100644 --- a/tools/buildman/builder.py +++ b/tools/buildman/builder.py @@ -955,40 +955,6 @@ class Builder: self.upto = self.warned = self.fail = 0 self._timestamps = collections.deque()

- def BuildBoardsForCommit(self, board_selected, keep_outputs): - """Build all boards for a single commit""" - self.SetupBuild(board_selected) - self.count = len(board_selected) - for brd in board_selected.itervalues(): - job = BuilderJob() - job.board = brd - job.commits = None - job.keep_outputs = keep_outputs - self.queue.put(brd) - - self.queue.join() - self.out_queue.join() - print - self.ClearLine(0) - - def BuildCommits(self, commits, board_selected, show_errors, keep_outputs): - """Build all boards for all commits (non-incremental)""" - self.commit_count = len(commits) - - self.ResetResultSummary(board_selected) - for self.commit_upto in range(self.commit_count): - self.SelectCommit(commits[self.commit_upto]) - self.SelectOutputDir() - builderthread.Mkdir(self.output_dir) - - self.BuildBoardsForCommit(board_selected, keep_outputs) - board_dict, err_lines = self.GetResultSummary() - self.PrintResultSummary(board_selected, board_dict, - err_lines if show_errors else []) - - if self.already_done: - print '%d builds already done' % self.already_done - def GetThreadDir(self, thread_num): """Get the directory path to the working dir for a thread.

diff --git a/tools/buildman/test.py b/tools/buildman/test.py index a724d83..502c9b4 100644 --- a/tools/buildman/test.py +++ b/tools/buildman/test.py @@ -136,7 +136,6 @@ class TestBuild(unittest.TestCase): build.do_make = self.Make board_selected = self.boards.GetSelectedDict()

- #build.BuildCommits(self.commits, board_selected, False) build.BuildBoards(self.commits, board_selected, keep_outputs=False, verbose=False) build.SetDisplayOptions(show_errors=True);

If buildman finds no problems it prints nothing. This can be a bit confusing, so add a message that all is well.

Signed-off-by: Simon Glass sjg@chromium.org ---

Changes in v5: None Changes in v4: - Add new patch to show a message when there are no errors

Changes in v3: None Changes in v2: None

tools/buildman/builder.py | 6 ++++++ 1 file changed, 6 insertions(+)

diff --git a/tools/buildman/builder.py b/tools/buildman/builder.py index 22a24b1..3a156ae 100644 --- a/tools/buildman/builder.py +++ b/tools/buildman/builder.py @@ -891,10 +891,13 @@ class Builder: self.col.MAGENTA) for arch, target_list in arch_list.iteritems(): print '%10s: %s' % (arch, target_list) + self._error_lines += 1 if better_err: print self.col.Color(self.col.GREEN, '\n'.join(better_err)) + self._error_lines += 1 if worse_err: print self.col.Color(self.col.RED, '\n'.join(worse_err)) + self._error_lines += 1

if show_sizes: self.PrintSizeSummary(board_selected, board_dict, show_detail, @@ -937,9 +940,12 @@ class Builder: self.commit_count = len(commits) if commits else 1 self.commits = commits self.ResetResultSummary(board_selected) + self._error_lines = 0

for commit_upto in range(0, self.commit_count, self._step): self.ProduceResultSummary(commit_upto, commits, board_selected) + if not self._error_lines: + print self.col.Color(self.col.GREEN, '(no errors to report)')

def SetupBuild(self, board_selected, commits):

Currently buildman allows a list of boards to build to be specified on the command line. The list can include specific board names, architecture, SOC and so on.

At present the list of boards is dealt with in an 'OR' fashion, and there is no way to specify something like 'arm & freescale', meaning boards with ARM architecture but only those made by Freescale. This would exclude the PowerPC boards made by Freescale.

Support an '&' operator on the command line to permit this. Ensure that arguments can be specified in a single string to permit easy shell quoting.

Suggested-by: York Sun yorksun@freescale.com Signed-off-by: Simon Glass sjg@chromium.org ---

Changes in v5: - Add new patch to add an 'and' operator for board selection

Changes in v4: None Changes in v3: None Changes in v2: None

tools/buildman/README | 9 +++ tools/buildman/board.py | 144 ++++++++++++++++++++++++++++++++++++++++++------ tools/buildman/test.py | 48 ++++++++++++++++ 3 files changed, 184 insertions(+), 17 deletions(-)

diff --git a/tools/buildman/README b/tools/buildman/README index aaf0a10..d4e8404 100644 --- a/tools/buildman/README +++ b/tools/buildman/README @@ -108,6 +108,15 @@ behaviour is a superset of exact or substring matching. Examples are: * '^tegra[23]0$' All boards with either Tegra20 or Tegra30 SoC * 'powerpc' All PowerPC boards

+While the default is to OR the terms together, you can also make use of +the '&' operator to limit the selection: + +* 'freescale & arm sandbox' All Freescale boards with ARM architecture, + plus sandbox + +It is convenient to use the -n option to see whaat will be built based on +the subset given. + Buildman does not store intermediate object files. It optionally copies the binary output into a directory when a build is successful. Size information is always recorded. It needs a fair bit of disk space to work, diff --git a/tools/buildman/board.py b/tools/buildman/board.py index 7bcc932..a333287 100644 --- a/tools/buildman/board.py +++ b/tools/buildman/board.py @@ -5,6 +5,72 @@

import re

+class Expr: + """A single regular expression for matching boards to build""" + + def __init__(self, expr): + """Set up a new Expr object. + + Args: + expr: String cotaining regular expression to store + """ + self._expr = expr + self._re = re.compile(expr) + + def Matches(self, props): + """Check if any of the properties match the regular expression. + + Args: + props: List of properties to check + Returns: + True if any of the properties match the regular expression + """ + for prop in props: + if self._re.match(prop): + return True + return False + + def __str__(self): + return self._expr + +class Term: + """A list of expressions each of which must match with properties. + + This provides a list of 'AND' expressions, meaning that each must + match the board properties for that board to be built. + """ + def __init__(self): + self._expr_list = [] + self._board_count = 0 + + def AddExpr(self, expr): + """Add an Expr object to the list to check. + + Args: + expr: New Expr object to add to the list of those that must + match for a board to be built. + """ + self._expr_list.append(Expr(expr)) + + def __str__(self): + """Return some sort of useful string describing the term""" + return '&'.join([str(expr) for expr in self._expr_list]) + + def Matches(self, props): + """Check if any of the properties match this term + + Each of the expressions in the term is checked. All must match. + + Args: + props: List of properties to check + Returns: + True if all of the expressions in the Term match, else False + """ + for expr in self._expr_list: + if not expr.Matches(props): + return False + return True + class Board: """A particular board that we can build""" def __init__(self, status, arch, cpu, soc, vendor, board_name, target, options): @@ -124,6 +190,55 @@ class Boards: """ return [board.target for board in self._boards if board.build_it]

+ def _BuildTerms(self, args): + """Convert command line arguments to a list of terms. + + This deals with parsing of the arguments. It handles the '&' + operator, which joins several expressions into a single Term. + + For example: + ['arm & freescale sandbox', 'tegra'] + + will produce 3 Terms containing expressions as follows: + arm, freescale + sandbox + tegra + + The first Term has two expressions, both of which must match for + a board to be selected. + + Args: + args: List of command line arguments + Returns: + A list of Term objects + """ + syms = [] + for arg in args: + for word in arg.split(): + sym_build = [] + for term in word.split('&'): + if term: + sym_build.append(term) + sym_build.append('&') + syms += sym_build[:-1] + terms = [] + term = None + oper = None + for sym in syms: + if sym == '&': + oper = sym + elif oper: + term.AddExpr(sym) + oper = None + else: + if term: + terms.append(term) + term = Term() + term.AddExpr(sym) + if term: + terms.append(term) + return terms + def SelectBoards(self, args): """Mark boards selected based on args

@@ -137,26 +252,21 @@ class Boards: due to each argument, arranged by argument. """ result = {} - argres = {} - for arg in args: - result[arg] = 0 - argres[arg] = re.compile(arg) + terms = self._BuildTerms(args) + result['all'] = 0 + for term in terms: + result[str(term)] = 0

for board in self._boards: - if args: - for arg in args: - argre = argres[arg] - match = False - for prop in board.props: - match = argre.match(prop) - if match: - break - if match: - if not board.build_it: - board.build_it = True - result[arg] += 1 - result['all'] += 1 + if terms: + match = False + for term in terms: + if term.Matches(board.props): + board.build_it = True + result[str(term)] += 1 + result['all'] += 1 + break else: board.build_it = True result['all'] += 1 diff --git a/tools/buildman/test.py b/tools/buildman/test.py index 502c9b4..a51c942 100644 --- a/tools/buildman/test.py +++ b/tools/buildman/test.py @@ -165,5 +165,53 @@ class TestBuild(unittest.TestCase): args = ['tegra20'] control.DoBuildman(options, args)

+ def testBoardSingle(self): + """Test single board selection""" + self.assertEqual(self.boards.SelectBoards(['sandbox']), + {'all': 1, 'sandbox': 1}) + + def testBoardArch(self): + """Test single board selection""" + self.assertEqual(self.boards.SelectBoards(['arm']), + {'all': 2, 'arm': 2}) + + def testBoardArchSingle(self): + """Test single board selection""" + self.assertEqual(self.boards.SelectBoards(['arm sandbox']), + {'all': 3, 'arm': 2, 'sandbox' : 1}) + + def testBoardArchSingleMultiWord(self): + """Test single board selection""" + self.assertEqual(self.boards.SelectBoards(['arm', 'sandbox']), + {'all': 3, 'arm': 2, 'sandbox' : 1}) + + def testBoardSingleAnd(self): + """Test single board selection""" + self.assertEqual(self.boards.SelectBoards(['Tester & arm']), + {'all': 2, 'Tester&arm': 2}) + + def testBoardTwoAnd(self): + """Test single board selection""" + self.assertEqual(self.boards.SelectBoards(['Tester', '&', 'arm', + 'Tester' '&', 'powerpc', + 'sandbox']), + {'all': 5, 'Tester&powerpc': 2, 'Tester&arm': 2, + 'sandbox' : 1}) + + def testBoardAll(self): + """Test single board selection""" + self.assertEqual(self.boards.SelectBoards([]), {'all': 5}) + + def testBoardRegularExpression(self): + """Test single board selection""" + self.assertEqual(self.boards.SelectBoards(['T.*r&^Po']), + {'T.*r&^Po': 2, 'all': 2}) + + def testBoardDuplicate(self): + """Test single board selection""" + self.assertEqual(self.boards.SelectBoards(['sandbox sandbox', + 'sandbox']), + {'all': 1, 'sandbox': 1}) + if __name__ == "__main__": unittest.main()