[U-Boot] [PATCH 00/11] buildman: Make the tool friendlier for first-time users

This makes a few minor improvements to buildman to make it work more easiler for first-time users:
- Improve progress and warning messages when fetching toolchains - Fix a bug where toolchain paths can be overwritten when fetching - Note at the top of the help how to get started
Also this series removes MAKEALL. Since buildman has been around for 3 years it may be time to do this. If not, we can leave it for now.
Simon Glass (11): buildman: Tidy up the README a little buildman: Automatically create a config file if needed buildman: Fix the 'help' test to use the correct path buildman: Allow the toolchain error to be suppressed buildman: Improve the toolchain progress/error output buildman: Fix a typo in TestSettingsHasPath() buildman: Drop the toolchain error when downloading toolchains buildman: Avoid overwriting existing toolchain entries buildman: Add a quick-start note Drop the MAKEALL tool Drop references to MAKEALL in the documentation
MAKEALL | 850 ------------------------------------- README | 35 +- board/Barix/ipam390/README.ipam390 | 12 +- doc/README.sha1 | 5 +- tools/buildman/README | 67 ++- tools/buildman/bsettings.py | 64 +++ tools/buildman/control.py | 26 +- tools/buildman/func_test.py | 2 +- tools/buildman/toolchain.py | 73 +++- 9 files changed, 194 insertions(+), 940 deletions(-) delete mode 100755 MAKEALL

Tidy up some problems found by a recent review.
Signed-off-by: Simon Glass sjg@chromium.org ---
tools/buildman/README | 53 +++++++++++++++++++++++++++++---------------------- 1 file changed, 30 insertions(+), 23 deletions(-)
diff --git a/tools/buildman/README b/tools/buildman/README index 26755c5..cd672cf 100644 --- a/tools/buildman/README +++ b/tools/buildman/README @@ -22,16 +22,14 @@ help for anyone working with >10 patches at a time. Caveats =======
-Buildman is still in its infancy. It is already a very useful tool, but -expect to find problems and send patches. - Buildman can be stopped and restarted, in which case it will continue where it left off. This should happen cleanly and without side-effects. If not, it is a bug, for which a patch would be welcome.
Buildman gets so tied up in its work that it can ignore the outside world. You may need to press Ctrl-C several times to quit it. Also it will print -out various exceptions when stopped. +out various exceptions when stopped. You may have to kill it since the +Ctrl-C handling is somewhat broken.
Theory of Operation @@ -46,6 +44,13 @@ warnings and binaries if you ask for them) is stored in output directories, which you can look at while the build is progressing, or when it is finished.
+Buildman is designed to build entire git branches, i.e. muliple commits. It +can be run repeatedly on the same branch. In this case it will automatically +rebuild commits which have changed (and remove its old results for that +commit). It is possible to build a branch for one board, then later build it +for another board. If you want buildman to re-build a commit it has already +built (e.g. because of a toolchain update), use the -f flag. + Buildman produces a concise summary of which boards succeeded and failed. It shows which commit introduced which board failure using a simple red/green colour coding. Full error information can be requested, in which @@ -420,10 +425,7 @@ Tool chain test: OK
Or download them all from kernel.org and move them to /toolchains directory,
-$ for i in aarch64 arm avr32 i386 m68k microblaze mips or32 powerpc sparc - do - ./tools/buildman/buildman --fetch-arch $i - done +$ ./tools/buildman/buildman --fetch-arch all $ sudo mkdir -p /toolchains $ sudo mv ~/.buildman-toolchains/*/* /toolchains/
@@ -440,8 +442,8 @@ nios2: http://sourcery.mentor.com/public/gnu_toolchain/nios2-linux-gnu/ sh: http://sourcery.mentor.com/public/gnu_toolchain/sh-linux-gnu/ renesas-4.4-200-sh-linux-gnu-i686-pc-linux-gnu.tar.bz2
-Note openrisc kernel.org toolchain is out of date, download latest one from -http://opencores.org/or1k/OpenRISC_GNU_tool_chain#Prebuilt_versions, eg: +Note openrisc kernel.org toolchain is out of date. Download the latest one from +http://opencores.org/or1k/OpenRISC_GNU_tool_chain#Prebuilt_versions - eg: ftp://ocuser:ocuser@openrisc.opencores.org/toolchain/gcc-or1k-elf-4.8.1-x86.tar.bz2.
Buildman should now be set up to use your new toolchain. @@ -521,7 +523,7 @@ Building 18 commits for 1059 boards (4 threads, 1 job per thread) This means that it is building 19062 board/commit combinations. So far it has managed to successfully build 528. Another 36 have built with warnings, and 124 more didn't build at all. Buildman expects to complete the process -in an hour and 15 minutes. Use this time to buy a faster computer. +in around an hour and a quarter. Use this time to buy a faster computer.
To find out how the build went, ask for a summary with -s. You can do this @@ -556,7 +558,8 @@ the build is still in progress so many boards are not built yet (use -u to see which ones). But still we can see a few failures. The galaxy5200_LOWBOOT never builds correctly. This could be a problem with our toolchain, or it could be a bug in the upstream. The good news is that we probably don't need -to blame our commits. The bad news is it isn't tested on that board. +to blame our commits. The bad news is that our commits are not tested on that +board.
Commit 12 broke lubbock. That's what the '+ lubbock' means. The failure is never fixed by a later commit, or you would see lubbock again, in green, @@ -585,19 +588,20 @@ So the problem is in lcd.c, due to missing cache operations. This information should be enough to work out what that commit is doing to break these boards. (In this case pxa did not have cache operations defined).
-If you see error lines marked with - that means that the errors were fixed +If you see error lines marked with '-', that means that the errors were fixed by that commit. Sometimes commits can be in the wrong order, so that a breakage is introduced for a few commits and fixed by later commits. This shows up clearly with buildman. You can then reorder the commits and try again.
-At commit 16, the error moves - you can see that the old error at line 120 +At commit 16, the error moves: you can see that the old error at line 120 is fixed, but there is a new one at line 126. This is probably only because we added some code and moved the broken line further down the file.
If many boards have the same error, then -e will display the error only once. This makes the output as concise as possible. To see which boards have -each error, use -l. +each error, use -l. So it is safe to omit the board name - you will not get +lots of repeated output for every board.
Buildman tries to distinguish warnings from errors, and shows warning lines separately with a 'w' prefix. @@ -619,8 +623,8 @@ The full build output in this case is available in:
sizes: Shows image size information.
-It is possible to get the build output there also. Use the -k option for -this. In that case you will also see some output files, like: +It is possible to get the build binary output there also. Use the -k option +for this. In that case you will also see some output files, like:
System.map toolchain u-boot u-boot.bin u-boot.map autoconf.mk (also SPL versions u-boot-spl and u-boot-spl.bin if available) @@ -631,7 +635,7 @@ Checking Image Sizes
A key requirement for U-Boot is that you keep code/data size to a minimum. Where a new feature increases this noticeably it should normally be put -behind a CONFIG flag so that boards can leave it off and keep the image +behind a CONFIG flag so that boards can leave it disabled and keep the image size more or less the same with each new release.
To check the impact of your commits on image size, use -S. For example: @@ -670,12 +674,13 @@ A useful option is --step which lets you skip some commits. For example --step 2 will show the image sizes for only every 2nd commit (so it will compare the image sizes of the 1st, 3rd, 5th... commits). You can also use --step 0 which will compare only the first and last commits. This is useful -for an overview of how your entire series affects code size. +for an overview of how your entire series affects code size. It will build +only the upstream commit and your final branch commit.
You can also use -d to see a detailed size breakdown for each board. This list is sorted in order from largest growth to largest reduction.
-It is possible to go a little further with the -B option (--bloat). This +It is even possible to go a little further with the -B option (--bloat). This shows where U-Boot has bloated, breaking the size change down to the function level. Example output is below:
@@ -798,9 +803,9 @@ $ ./tools/buildman/buildman -b us-mem4 -sSdB ...
-This shows that commit 19 has increased text size for arm (although only one -board was built) and by 96 bytes for powerpc. This increase was offset in both -cases by reductions in rodata and data/bss. +This shows that commit 19 has reduced codesize for arm slightly and increased +it for powerpc. This increase was offset in by reductions in rodata and +data/bss.
Shown below the summary lines are the sizes for each board. Below each board are the sizes for each function. This information starts with: @@ -1063,6 +1068,8 @@ access to log files. Also it would be nice if buildman could 'hunt' for problems, perhaps by building a few boards for each arch, or checking commits for changed files and building only boards which use those files.
+A specific problem to fix is that Ctrl-C does not exit buildman cleanly when +multiple builder threads are active.
Credits =======

On Sun, Jul 03, 2016 at 03:14:17PM -0600, Simon Glass wrote:
Tidy up some problems found by a recent review.
Signed-off-by: Simon Glass sjg@chromium.org
Reviewed-by: Tom Rini trini@konsulko.com

If there is no ~/.buildman file, buildman currently complains and exists. To make things a little more friendly, create an empty one automatically. This will not allow things to be built, but --fetch-arch can be used to handle that.
Signed-off-by: Simon Glass sjg@chromium.org ---
tools/buildman/bsettings.py | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+)
diff --git a/tools/buildman/bsettings.py b/tools/buildman/bsettings.py index b361469..6f8fdfd 100644 --- a/tools/buildman/bsettings.py +++ b/tools/buildman/bsettings.py @@ -22,6 +22,9 @@ def Setup(fname=''): config_fname = fname if config_fname == '': config_fname = '%s/.buildman' % os.getenv('HOME') + if not os.path.exists(config_fname): + print 'No config file found ~/.buildman\nCreating one...\n' + CreateBuildmanConfigFile(config_fname) if config_fname: settings.read(config_fname)
@@ -53,3 +56,43 @@ def SetItem(section, tag, value): if config_fname is not None: with open(config_fname, 'w') as fd: settings.write(fd) + +def CreateBuildmanConfigFile(config_fname): + """Creates a new config file with no tool chain information. + + Args: + config_fname: Config filename to create + + Returns: + None + """ + try: + f = open(config_fname, 'w') + except IOError: + print "Couldn't create buildman config file '%s'\n" % config_fname + raise + + print >>f, '''[toolchain] +# name = path +# e.g. x86 = /opt/gcc-4.6.3-nolibc/x86_64-linux + +[toolchain-prefix] +# name = path to prefix +# e.g. x86 = /opt/gcc-4.6.3-nolibc/x86_64-linux/bin/x86_64-linux- + +[toolchain-alias] +# arch = alias +# Indicates which toolchain should be used to build for that arch +x86 = i386 +blackfin = bfin +nds32 = nds32le +openrisc = or1k + +[make-flags] +# Special flags to pass to 'make' for certain boards, e.g. to pass a test +# flag and build tag to snapper boards: +# snapper-boards=ENABLE_AT91_TEST=1 +# snapper9260=${snapper-boards} BUILD_TAG=442 +# snapper9g45=${snapper-boards} BUILD_TAG=443 +''' + f.close();

On 07/03/2016 03:14 PM, Simon Glass wrote:
If there is no ~/.buildman file, buildman currently complains and exists. To make things a little more friendly, create an empty one automatically. This will not allow things to be built, but --fetch-arch can be used to handle that.
Is it worth pointing out that caveat in the message that's printed?
Is that caveat true e.g. in the case where the user already has a suitable compiler installed in (the default?) /usr/bin?

When buildman is run via a symlink, this test fails. Fix it to work the same way as buildman itself.
Signed-off-by: Simon Glass sjg@chromium.org ---
tools/buildman/func_test.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tools/buildman/func_test.py b/tools/buildman/func_test.py index a0bd46c..d439e17 100644 --- a/tools/buildman/func_test.py +++ b/tools/buildman/func_test.py @@ -180,7 +180,7 @@ class TestFunctional(unittest.TestCase): self._base_dir = tempfile.mkdtemp() self._git_dir = os.path.join(self._base_dir, 'src') self._buildman_pathname = sys.argv[0] - self._buildman_dir = os.path.dirname(sys.argv[0]) + self._buildman_dir = os.path.dirname(os.path.realpath(sys.argv[0])) command.test_result = self._HandleCommand self.setupToolchains() self._toolchains.Add('arm-gcc', test=False)

When there are no toolchains a warning is printed. But in some cases this is confusing, such as when the user is fetching new toolchains.
Adjust the function to supress the warning in this case.
Signed-off-by: Simon Glass sjg@chromium.org ---
tools/buildman/toolchain.py | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-)
diff --git a/tools/buildman/toolchain.py b/tools/buildman/toolchain.py index 3993db3..f6ab295 100644 --- a/tools/buildman/toolchain.py +++ b/tools/buildman/toolchain.py @@ -167,9 +167,12 @@ class Toolchains: self.paths = [] self._make_flags = dict(bsettings.GetItems('make-flags'))
- def GetPathList(self): + def GetPathList(self, show_warning=True): """Get a list of available toolchain paths
+ Args: + show_warning: True to show a warning if there are no tool chains. + Returns: List of strings, each a path to a toolchain mentioned in the [toolchain] section of the settings file. @@ -188,9 +191,14 @@ class Toolchains: paths.append(value) return paths
- def GetSettings(self): - self.prefixes = bsettings.GetItems('toolchain-prefix') - self.paths += self.GetPathList() + def GetSettings(self, show_warning=True): + """Get toolchain settings from the settings file. + + Args: + show_warning: True to show a warning if there are no tool chains. + """ + self.prefixes = bsettings.GetItems('toolchain-prefix') + self.paths += self.GetPathList(show_warning)
def Add(self, fname, test=True, verbose=False, priority=PRIORITY_CALC, arch=None): @@ -479,7 +487,7 @@ class Toolchains: Returns: True if the path is in settings, False if not """ - paths = self.GetPathList() + paths = self.GetPathList(False) return path in paths
def ListArchs(self):

On 07/03/2016 03:14 PM, Simon Glass wrote:
When there are no toolchains a warning is printed. But in some cases this is confusing, such as when the user is fetching new toolchains.
Adjust the function to supress the warning in this case.
diff --git a/tools/buildman/toolchain.py b/tools/buildman/toolchain.py
- def GetPathList(self):
def GetPathList(self, show_warning=True): """Get a list of available toolchain paths
Args:
show_warning: True to show a warning if there are no tool chains.
I don't see the new parameter being used anywhere?

Use colour to make it easier to see what is going on. Also print a message before downloading a new toolchain. Mention --fetch-arch in the message that is shown when there are no available toolchains, since this is the quickest way to resolve the problem.
Signed-off-by: Simon Glass sjg@chromium.org ---
tools/buildman/control.py | 9 ++++++--- tools/buildman/toolchain.py | 24 +++++++++++++++--------- 2 files changed, 21 insertions(+), 12 deletions(-)
diff --git a/tools/buildman/control.py b/tools/buildman/control.py index aeb128a..46053d8 100644 --- a/tools/buildman/control.py +++ b/tools/buildman/control.py @@ -107,6 +107,7 @@ def DoBuildman(options, args, toolchains=None, make_func=None, boards=None, return 0
gitutil.Setup() + col = terminal.Color()
options.git_dir = os.path.join(options.git, '.git')
@@ -122,14 +123,17 @@ def DoBuildman(options, args, toolchains=None, make_func=None, boards=None, if options.fetch_arch: if options.fetch_arch == 'list': sorted_list = toolchains.ListArchs() - print 'Available architectures: %s\n' % ' '.join(sorted_list) + print col.Color(col.BLUE, 'Available architectures: %s\n' % + ' '.join(sorted_list)) return 0 else: fetch_arch = options.fetch_arch if fetch_arch == 'all': fetch_arch = ','.join(toolchains.ListArchs()) - print 'Downloading toolchains: %s\n' % fetch_arch + print col.Color(col.CYAN, '\nDownloading toolchains: %s' % + fetch_arch) for arch in fetch_arch.split(','): + print ret = toolchains.FetchAndInstall(arch) if ret: return ret @@ -138,7 +142,6 @@ def DoBuildman(options, args, toolchains=None, make_func=None, boards=None, # Work out how many commits to build. We want to build everything on the # branch. We also build the upstream commit as a control so we can see # problems introduced by the first commit on the branch. - col = terminal.Color() count = options.count has_range = options.branch and '..' in options.branch if count == -1: diff --git a/tools/buildman/toolchain.py b/tools/buildman/toolchain.py index f6ab295..6f39a2b 100644 --- a/tools/buildman/toolchain.py +++ b/tools/buildman/toolchain.py @@ -13,6 +13,7 @@ import urllib2
import bsettings import command +import terminal
(PRIORITY_FULL_PREFIX, PRIORITY_PREFIX_GCC, PRIORITY_PREFIX_GCC_PATH, PRIORITY_CALC) = range(4) @@ -179,9 +180,11 @@ class Toolchains: """ toolchains = bsettings.GetItems('toolchain') if not toolchains: - print ('Warning: No tool chains - please add a [toolchain] section' - ' to your buildman config file %s. See README for details' % - bsettings.config_fname) + print ("Warning: No tool chains. Please run 'buildman " + "--fetch-arch all' to download all available toolchains, or " + "add a [toolchain] section to your buildman config file " + "%s. See README for details" % + bsettings.config_fname)
paths = [] for name, value in toolchains: @@ -294,7 +297,9 @@ class Toolchains:
def List(self): """List out the selected toolchains for each architecture""" - print 'List of available toolchains (%d):' % len(self.toolchains) + col = terminal.Color() + print col.Color(col.BLUE, 'List of available toolchains (%d):' % + len(self.toolchains)) if len(self.toolchains): for key, value in sorted(self.toolchains.iteritems()): print '%-10s: %s' % (key, value.gcc) @@ -509,6 +514,8 @@ class Toolchains: Architecture to fetch, or 'list' to list """ # Fist get the URL for this architecture + col = terminal.Color() + print col.Color(col.BLUE, "Downloading toolchain for arch '%s'" % arch) url = self.LocateArchUrl(arch) if not url: print ("Cannot find toolchain for arch '%s' - use 'list' to list" % @@ -523,7 +530,7 @@ class Toolchains: tmpdir, tarfile = self.Download(url) if not tarfile: return 1 - print 'Unpacking to: %s' % dest, + print col.Color(col.GREEN, 'Unpacking to: %s' % dest), sys.stdout.flush() path = self.Unpack(tarfile, dest) os.remove(tarfile) @@ -531,16 +538,15 @@ class Toolchains: print
# Check that the toolchain works - print 'Testing' + print col.Color(col.GREEN, 'Testing') dirpath = os.path.join(dest, path) compiler_fname_list = self.ScanPath(dirpath, True) if not compiler_fname_list: print 'Could not locate C compiler - fetch failed.' return 1 if len(compiler_fname_list) != 1: - print ('Internal error, ambiguous toolchains: %s' % - (', '.join(compiler_fname))) - return 1 + print col.Color(col.RED, 'Warning, ambiguous toolchains: %s' % + ', '.join(compiler_fname_list)) toolchain = Toolchain(compiler_fname_list[0], True, True)
# Make sure that it will be found by buildman

On Sun, Jul 03, 2016 at 03:14:21PM -0600, Simon Glass wrote:
Use colour to make it easier to see what is going on. Also print a message before downloading a new toolchain. Mention --fetch-arch in the message that is shown when there are no available toolchains, since this is the quickest way to resolve the problem.
Signed-off-by: Simon Glass sjg@chromium.org
Reviewed-by: Tom Rini trini@konsulko.com

The function comment should say 'buildman'. Fix it.
Signed-off-by: Simon Glass sjg@chromium.org ---
tools/buildman/toolchain.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tools/buildman/toolchain.py b/tools/buildman/toolchain.py index 6f39a2b..584c9f2 100644 --- a/tools/buildman/toolchain.py +++ b/tools/buildman/toolchain.py @@ -487,7 +487,7 @@ class Toolchains: return stdout.splitlines()[0][:-1]
def TestSettingsHasPath(self, path): - """Check if builmand will find this toolchain + """Check if buildman will find this toolchain
Returns: True if the path is in settings, False if not

It doesn't make sense to complain about missing toolchains when the --fetch-arch option is being used. The user is presumably aware that there is a toolchain problem and is actively correcting it by running with this option.
Refactor the code to avoid printing this confusing message.
Signed-off-by: Simon Glass sjg@chromium.org ---
tools/buildman/control.py | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-)
diff --git a/tools/buildman/control.py b/tools/buildman/control.py index 46053d8..b86d7b3 100644 --- a/tools/buildman/control.py +++ b/tools/buildman/control.py @@ -111,14 +111,9 @@ def DoBuildman(options, args, toolchains=None, make_func=None, boards=None,
options.git_dir = os.path.join(options.git, '.git')
- if not toolchains: + no_toolchains = toolchains is None + if no_toolchains: toolchains = toolchain.Toolchains() - toolchains.GetSettings() - toolchains.Scan(options.list_tool_chains) - if options.list_tool_chains: - toolchains.List() - print - return 0
if options.fetch_arch: if options.fetch_arch == 'list': @@ -139,6 +134,14 @@ def DoBuildman(options, args, toolchains=None, make_func=None, boards=None, return ret return 0
+ if no_toolchains: + toolchains.GetSettings() + toolchains.Scan(options.list_tool_chains) + if options.list_tool_chains: + toolchains.List() + print + return 0 + # Work out how many commits to build. We want to build everything on the # branch. We also build the upstream commit as a control so we can see # problems introduced by the first commit on the branch.

The current code for setting up the toolchain config always writes the new paths to an item called 'toolchain'. This means that it will overwrite any existing toolchain item with the same name. In practice, this means that:
buildman --fetch-arch all
will fetch all toolchains, but only the path of the final one will be added to the config. This normally works out OK, since most toolchains are the same version (e.g. gcc 4.9) and will be found on the same path. But it is not correct and toolchains for archs which don't use the same version will not function as expected.
Adjust the code to use unique names for each toolchain path entry.
Signed-off-by: Simon Glass sjg@chromium.org ---
tools/buildman/bsettings.py | 21 +++++++++++++++++++++ tools/buildman/toolchain.py | 29 +++++++++++++++++++++++++---- 2 files changed, 46 insertions(+), 4 deletions(-)
diff --git a/tools/buildman/bsettings.py b/tools/buildman/bsettings.py index 6f8fdfd..655fef8 100644 --- a/tools/buildman/bsettings.py +++ b/tools/buildman/bsettings.py @@ -47,6 +47,27 @@ def GetItems(section): except: raise
+def GetItemsAsDict(section): + """Get the items from a section of the config. + + Args: + section: name of section to retrieve + + Returns: + Dict: + key: name of item + value: value of item + """ + try: + items = {} + for item in settings.items(section): + items[item[0]] = item[1] + return items + except ConfigParser.NoSectionError as e: + return {} + except: + raise + def SetItem(section, tag, value): """Set an item and write it back to the settings file""" global settings diff --git a/tools/buildman/toolchain.py b/tools/buildman/toolchain.py index 584c9f2..293b297 100644 --- a/tools/buildman/toolchain.py +++ b/tools/buildman/toolchain.py @@ -550,9 +550,30 @@ class Toolchains: toolchain = Toolchain(compiler_fname_list[0], True, True)
# Make sure that it will be found by buildman - if not self.TestSettingsHasPath(dirpath): - print ("Adding 'download' to config file '%s'" % - bsettings.config_fname) + if self.TestSettingsHasPath(dirpath): + print col.Color(col.GREEN, 'Reusing existing config item') + else: tools_dir = os.path.dirname(dirpath) - bsettings.SetItem('toolchain', 'download', '%s/*' % tools_dir) + self.AddToolchainSetting('%s/*' % tools_dir) return 0 + + def AddToolchainSetting(self, tools_path): + """Add a new toolchain setting to the [toolchain] section. + + Be careful to avoid overwriting an existing entry. + + Args: + tools_path: New tools patch to add (normally ends with "/*") + """ + col = terminal.Color() + items = bsettings.GetItemsAsDict('toolchain') + for upto in xrange(100): + name = 'download%d' % upto + if name not in items: + break + if upto == 100: + print 'Internal error: too many items in [toolchain]' + return 1 + print col.Color(col.YELLOW, "Adding '%s' to config file '%s'" % + (name, bsettings.config_fname)) + bsettings.SetItem('toolchain', name, tools_path)

On 07/03/2016 03:14 PM, Simon Glass wrote:
The current code for setting up the toolchain config always writes the new paths to an item called 'toolchain'. This means that it will overwrite any existing toolchain item with the same name. In practice, this means that:
buildman --fetch-arch all
will fetch all toolchains, but only the path of the final one will be added to the config. This normally works out OK, since most toolchains are the same version (e.g. gcc 4.9) and will be found on the same path. But it is not correct and toolchains for archs which don't use the same version will not function as expected.
Adjust the code to use unique names for each toolchain path entry.
diff --git a/tools/buildman/bsettings.py b/tools/buildman/bsettings.py
+def GetItemsAsDict(section):
- """Get the items from a section of the config.
- Args:
section: name of section to retrieve
- Returns:
Dict:
key: name of item
value: value of item
- """
- try:
items = {}
for item in settings.items(section):
items[item[0]] = item[1]
Aren't those last 3 lines equivalent to:
items = dict(settings.items(section))

On Sun, Jul 03, 2016 at 03:14:24PM -0600, Simon Glass wrote:
The current code for setting up the toolchain config always writes the new paths to an item called 'toolchain'. This means that it will overwrite any existing toolchain item with the same name. In practice, this means that:
buildman --fetch-arch all
will fetch all toolchains, but only the path of the final one will be added to the config. This normally works out OK, since most toolchains are the same version (e.g. gcc 4.9) and will be found on the same path. But it is not correct and toolchains for archs which don't use the same version will not function as expected.
Adjust the code to use unique names for each toolchain path entry.
This seems like a lot more work than just adding (and checking for?) : download = /home/trini/.buildman-toolchains/*/*
to the config file.

For those who just want to build a board, it is useful to see a quick hint right at the start of the documentation. Add a few commands showing how to download toolchains and build a board.
Signed-off-by: Simon Glass sjg@chromium.org ---
tools/buildman/README | 14 ++++++++++++++ 1 file changed, 14 insertions(+)
diff --git a/tools/buildman/README b/tools/buildman/README index cd672cf..25800fb 100644 --- a/tools/buildman/README +++ b/tools/buildman/README @@ -5,6 +5,20 @@
(Please read 'How to change from MAKEALL' if you are used to that tool)
+Quick-start +=========== + +If you just want to quickly set up buildman so you can build something (for +example Raspberry Pi 2), and don't mind downloading lots of stuff: + + cd /path/to/u-boot + PATH=$PATH:`pwd`/tools/buildman + buildman --fetch-arch all + buildman -k rpi_2 + ls ../current/rpi_2 + # u-boot.bin is the output image + + What is this? =============

On 07/03/2016 03:14 PM, Simon Glass wrote:
For those who just want to build a board, it is useful to see a quick hint right at the start of the documentation. Add a few commands showing how to download toolchains and build a board.
diff --git a/tools/buildman/README b/tools/buildman/README
+If you just want to quickly set up buildman so you can build something (for +example Raspberry Pi 2), and don't mind downloading lots of stuff:
- cd /path/to/u-boot
- PATH=$PATH:`pwd`/tools/buildman
- buildman --fetch-arch all
- buildman -k rpi_2
- ls ../current/rpi_2
- # u-boot.bin is the output image
For just an rpi_2 build, perhaps "--fetch-arch arm" would be quicker, although I suppose then you'd have to make the text more complicated by pointing out that it only downloads an ARM toolchain and so the user would need to --fetch-arch again for anything else.

On Thu, Jul 07, 2016 at 12:23:37PM -0600, Stephen Warren wrote:
On 07/03/2016 03:14 PM, Simon Glass wrote:
For those who just want to build a board, it is useful to see a quick hint right at the start of the documentation. Add a few commands showing how to download toolchains and build a board.
diff --git a/tools/buildman/README b/tools/buildman/README
+If you just want to quickly set up buildman so you can build something (for +example Raspberry Pi 2), and don't mind downloading lots of stuff:
- cd /path/to/u-boot
- PATH=$PATH:`pwd`/tools/buildman
- buildman --fetch-arch all
- buildman -k rpi_2
- ls ../current/rpi_2
- # u-boot.bin is the output image
For just an rpi_2 build, perhaps "--fetch-arch arm" would be quicker, although I suppose then you'd have to make the text more complicated by pointing out that it only downloads an ARM toolchain and so the user would need to --fetch-arch again for anything else.
I think for a "quick start" section it's OK to have it just download ARM and if they then fail when building aarch64 or powerpc or whatever, it's reasonable to assume the user will come back to read more of the documentation (or at least dig into --fetch-arch).

Buildman has been around for 3 years now. It has had a lot of use and testing. Perhaps it is time to remove MAKEALL.
Signed-off-by: Simon Glass sjg@chromium.org ---
MAKEALL | 850 ---------------------------------------------------------------- 1 file changed, 850 deletions(-) delete mode 100755 MAKEALL
diff --git a/MAKEALL b/MAKEALL deleted file mode 100755 index 7e42f10..0000000 --- a/MAKEALL +++ /dev/null @@ -1,850 +0,0 @@ -#!/bin/bash -# Tool mainly for U-Boot Quality Assurance: build one or more board -# configurations with minimal verbosity, showing only warnings and -# errors. -# -# SPDX-License-Identifier: GPL-2.0+ - -usage() -{ - # if exiting with 0, write to stdout, else write to stderr - local ret=${1:-0} - [ "${ret}" -eq 1 ] && exec 1>&2 - cat <<-EOF - Usage: MAKEALL [options] [--] [boards-to-build] - - Options: - -a ARCH, --arch ARCH Build all boards with arch ARCH - -c CPU, --cpu CPU Build all boards with cpu CPU - -v VENDOR, --vendor VENDOR Build all boards with vendor VENDOR - -s SOC, --soc SOC Build all boards with soc SOC - -b BOARD, --board BOARD Build all boards with board name BOARD - -l, --list List all targets to be built - -m, --maintainers List all targets and maintainer email - -M, --mails List all targets and all affilated emails - -C, --check Enable build checking - -n, --continue Continue (skip boards already built) - -r, --rebuild-errors Rebuild any boards that errored - -h, --help This help output - - Selections by these options are logically ANDed; if the same option - is used repeatedly, such selections are ORed. So "-v FOO -v BAR" - will select all configurations where the vendor is either FOO or - BAR. Any additional arguments specified on the command line are - always build additionally. See the boards.cfg file for more info. - - If no boards are specified, then the default is "powerpc". - - Environment variables: - BUILD_NCPUS number of parallel make jobs (default: auto) - CROSS_COMPILE cross-compiler toolchain prefix (default: "") - CROSS_COMPILE_<ARCH> cross-compiler toolchain prefix for - architecture "ARCH". Substitute "ARCH" for any - supported architecture (default: "") - MAKEALL_LOGDIR output all logs to here (default: ./LOG/) - BUILD_DIR output build directory (default: ./) - BUILD_NBUILDS number of parallel targets (default: 1) - - Examples: - - build all Power Architecture boards: - MAKEALL -a powerpc - MAKEALL --arch powerpc - MAKEALL powerpc - - build all PowerPC boards manufactured by vendor "esd": - MAKEALL -a powerpc -v esd - - build all PowerPC boards manufactured either by "keymile" or "siemens": - MAKEALL -a powerpc -v keymile -v siemens - - build all Freescale boards with MPC83xx CPUs, plus all 4xx boards: - MAKEALL -c mpc83xx -v freescale 4xx - EOF - exit ${ret} -} - -deprecation() { - echo "** Note: MAKEALL is deprecated - please use buildman instead" - echo "** See tools/buildman/README for details" - echo -} - -deprecation - -SHORT_OPTS="ha:c:v:s:b:lmMCnr" -LONG_OPTS="help,arch:,cpu:,vendor:,soc:,board:,list,maintainers,mails,check,continue,rebuild-errors" - -# Option processing based on util-linux-2.13/getopt-parse.bash - -# Note that we use `"$@"' to let each command-line parameter expand to a -# separate word. The quotes around `$@' are essential! -# We need TEMP as the `eval set --' would nuke the return value of -# getopt. -TEMP=`getopt -o ${SHORT_OPTS} --long ${LONG_OPTS} \ - -n 'MAKEALL' -- "$@"` - -[ $? != 0 ] && usage 1 - -# Note the quotes around `$TEMP': they are essential! -eval set -- "$TEMP" - -SELECTED='' -ONLY_LIST='' -PRINT_MAINTS='' -MAINTAINERS_ONLY='' -CONTINUE='' -REBUILD_ERRORS='' - -while true ; do - case "$1" in - -a|--arch) - # echo "Option ARCH: argument `$2'" - if [ "$opt_a" ] ; then - opt_a="${opt_a%)} || $2 == "$2")" - else - opt_a="($2 == "$2")" - fi - SELECTED='y' - shift 2 ;; - -c|--cpu) - # echo "Option CPU: argument `$2'" - if [ "$opt_c" ] ; then - opt_c="${opt_c%)} || $3 == "$2" || $3 ~ /$2:/)" - else - opt_c="($3 == "$2" || $3 ~ /$2:/)" - fi - SELECTED='y' - shift 2 ;; - -s|--soc) - # echo "Option SoC: argument `$2'" - if [ "$opt_s" ] ; then - opt_s="${opt_s%)} || $4 == "$2" || $4 ~ /$2/)" - else - opt_s="($4 == "$2" || $4 ~ /$2/)" - fi - SELECTED='y' - shift 2 ;; - -v|--vendor) - # echo "Option VENDOR: argument `$2'" - if [ "$opt_v" ] ; then - opt_v="${opt_v%)} || $5 == "$2")" - else - opt_v="($5 == "$2")" - fi - SELECTED='y' - shift 2 ;; - -b|--board) - # echo "Option BOARD: argument `$2'" - if [ "$opt_b" ] ; then - opt_b="${opt_b%)} || $6 == "$2" || $7 == "$2")" - else - # We need to check the 7th field too - # for boards whose 6th field is "-" - opt_b="($6 == "$2" || $7 == "$2")" - fi - SELECTED='y' - shift 2 ;; - -C|--check) - CHECK='C=1' - shift ;; - -n|--continue) - CONTINUE='y' - shift ;; - -r|--rebuild-errors) - REBUILD_ERRORS='y' - shift ;; - -l|--list) - ONLY_LIST='y' - shift ;; - -m|--maintainers) - ONLY_LIST='y' - PRINT_MAINTS='y' - MAINTAINERS_ONLY='y' - shift ;; - -M|--mails) - ONLY_LIST='y' - PRINT_MAINTS='y' - shift ;; - -h|--help) - usage ;; - --) - shift ; break ;; - *) - echo "Internal error!" >&2 ; exit 1 ;; - esac -done - -GNU_MAKE=$(scripts/show-gnu-make) || { - echo "GNU Make not found" >&2 - exit 1 -} - -# echo "Remaining arguments:" -# for arg do echo '--> '"`$arg'" ; done - -tools/genboardscfg.py || { - echo "Failed to generate boards.cfg" >&2 - exit 1 -} - -FILTER="$1 !~ /^#/" -[ "$opt_a" ] && FILTER="${FILTER} && $opt_a" -[ "$opt_c" ] && FILTER="${FILTER} && $opt_c" -[ "$opt_s" ] && FILTER="${FILTER} && $opt_s" -[ "$opt_v" ] && FILTER="${FILTER} && $opt_v" -[ "$opt_b" ] && FILTER="${FILTER} && $opt_b" - -if [ "$SELECTED" ] ; then - SELECTED=$(awk '('"$FILTER"') { print $7 }' boards.cfg) - - # Make sure some boards from boards.cfg are actually found - if [ -z "$SELECTED" ] ; then - echo "Error: No boards selected, invalid arguments" - exit 1 - fi -fi - -######################################################################### - -# Print statistics when we exit -trap exit 1 2 3 15 -trap print_stats 0 - -# Determine number of CPU cores if no default was set -: ${BUILD_NCPUS:="`getconf _NPROCESSORS_ONLN`"} - -if [ "$BUILD_NCPUS" -gt 1 ] -then - JOBS="-j $((BUILD_NCPUS + 1))" -else - JOBS="" -fi - -if [ "${MAKEALL_LOGDIR}" ] ; then - LOG_DIR=${MAKEALL_LOGDIR} -else - LOG_DIR="LOG" -fi - -: ${BUILD_NBUILDS:=1} -BUILD_MANY=0 - -if [ "${BUILD_NBUILDS}" -gt 1 ] ; then - BUILD_MANY=1 - : ${BUILD_DIR:=./build} - mkdir -p "${BUILD_DIR}/ERR" - find "${BUILD_DIR}/ERR/" -type f -exec rm -f {} + -fi - -: ${BUILD_DIR:=.} - -OUTPUT_PREFIX="${BUILD_DIR}" - -[ -d ${LOG_DIR} ] || mkdir "${LOG_DIR}" || exit 1 -if [ "$CONTINUE" != 'y' -a "$REBUILD_ERRORS" != 'y' ] ; then - find "${LOG_DIR}/" -type f -exec rm -f {} + -fi - -LIST="" - -# Keep track of the number of builds and errors -ERR_CNT=0 -ERR_LIST="" -WRN_CNT=0 -WRN_LIST="" -TOTAL_CNT=0 -SKIP_CNT=0 -CURRENT_CNT=0 -OLDEST_IDX=1 -RC=0 - -# Helper funcs for parsing boards.cfg -targets_by_field() -{ - field=$1 - regexp=$2 - - awk '($1 !~ /^#/ && $'"$field"' ~ /^'"$regexp"'$/) { print $7 }' \ - boards.cfg -} - -targets_by_arch() { targets_by_field 2 "$@" ; } -targets_by_cpu() { targets_by_field 3 "$@" ; targets_by_field 3 "$@:.*" ; } -targets_by_soc() { targets_by_field 4 "$@" ; } - -######################################################################### -## MPC5xx Systems -######################################################################### - -LIST_5xx="$(targets_by_cpu mpc5xx)" - -######################################################################### -## MPC5xxx Systems -######################################################################### - -LIST_5xxx="$(targets_by_cpu mpc5xxx)" - -######################################################################### -## MPC512x Systems -######################################################################### - -LIST_512x="$(targets_by_cpu mpc512x)" - -######################################################################### -## MPC8xx Systems -######################################################################### - -LIST_8xx="$(targets_by_cpu mpc8xx)" - -######################################################################### -## PPC4xx Systems -######################################################################### - -LIST_4xx="$(targets_by_cpu ppc4xx)" - -######################################################################### -## MPC8260 Systems (includes 8250, 8255 etc.) -######################################################################### - -LIST_8260="$(targets_by_cpu mpc8260)" - -######################################################################### -## MPC83xx Systems (includes 8349, etc.) -######################################################################### - -LIST_83xx="$(targets_by_cpu mpc83xx)" - -######################################################################### -## MPC85xx Systems (includes 8540, 8560 etc.) -######################################################################### - -LIST_85xx="$(targets_by_cpu mpc85xx)" - -######################################################################### -## MPC86xx Systems -######################################################################### - -LIST_86xx="$(targets_by_cpu mpc86xx)" - -######################################################################### -## PowerPC groups -######################################################################### - -LIST_TSEC=" \ - ${LIST_83xx} \ - ${LIST_85xx} \ - ${LIST_86xx} \ -" - -LIST_powerpc=" \ - ${LIST_5xx} \ - ${LIST_512x} \ - ${LIST_5xxx} \ - ${LIST_8xx} \ - ${LIST_824x} \ - ${LIST_8260} \ - ${LIST_83xx} \ - ${LIST_85xx} \ - ${LIST_86xx} \ - ${LIST_4xx} \ -" - -# Alias "ppc" -> "powerpc" to not break compatibility with older scripts -# still using "ppc" instead of "powerpc" -LIST_ppc=" \ - ${LIST_powerpc} \ -" - -######################################################################### -## StrongARM Systems -######################################################################### - -LIST_SA="$(targets_by_cpu sa1100)" - -######################################################################### -## ARM7 Systems -######################################################################### - -LIST_ARM7="$(targets_by_cpu arm720t)" - -######################################################################### -## ARM9 Systems -######################################################################### - -LIST_ARM9="$(targets_by_cpu arm920t) \ - $(targets_by_cpu arm926ejs) \ - $(targets_by_cpu arm946es) \ -" - -######################################################################### -## ARM11 Systems -######################################################################### -LIST_ARM11="$(targets_by_cpu arm1136) \ - $(targets_by_cpu arm1176) \ -" - -######################################################################### -## ARMV7 Systems -######################################################################### - -LIST_ARMV7="$(targets_by_cpu armv7)" - -######################################################################### -## ARMV8 Systems -######################################################################### - -LIST_ARMV8="$(targets_by_cpu armv8)" - -######################################################################### -## AT91 Systems -######################################################################### - -LIST_at91="$(targets_by_soc at91)" - -######################################################################### -## Xscale Systems -######################################################################### - -LIST_pxa="$(targets_by_cpu pxa)" - -######################################################################### -## SPEAr Systems -######################################################################### - -LIST_spear="$(targets_by_soc spear)" - -######################################################################### -## ARM groups -######################################################################### - -LIST_arm="$(targets_by_arch arm | \ - for ARMV8_TARGET in $LIST_ARMV8; \ - do sed "/$ARMV8_TARGET/d"; \ - done) \ -" - -######################################################################### -## MIPS Systems (default = big endian) -######################################################################### - -LIST_mips="$(targets_by_arch mips)" - -######################################################################### -## OpenRISC Systems -######################################################################### - -LIST_openrisc="$(targets_by_arch openrisc)" - -######################################################################### -## x86 Systems -######################################################################### - -LIST_x86="$(targets_by_arch x86)" - -######################################################################### -## Nios-II Systems -######################################################################### - -LIST_nios2="$(targets_by_arch nios2)" - -######################################################################### -## MicroBlaze Systems -######################################################################### - -LIST_microblaze="$(targets_by_arch microblaze)" - -######################################################################### -## ColdFire Systems -######################################################################### - -LIST_m68k="$(targets_by_arch m68k)" -LIST_coldfire=${LIST_m68k} - -######################################################################### -## AVR32 Systems -######################################################################### - -LIST_avr32="$(targets_by_arch avr32)" - -######################################################################### -## Blackfin Systems -######################################################################### - -LIST_blackfin="$(targets_by_arch blackfin)" - -######################################################################### -## SH Systems -######################################################################### - -LIST_sh2="$(targets_by_cpu sh2)" -LIST_sh3="$(targets_by_cpu sh3)" -LIST_sh4="$(targets_by_cpu sh4)" - -LIST_sh="$(targets_by_arch sh)" - -######################################################################### -## SPARC Systems -######################################################################### - -LIST_sparc="$(targets_by_arch sparc)" - -######################################################################### -## NDS32 Systems -######################################################################### - -LIST_nds32="$(targets_by_arch nds32)" - -######################################################################### -## ARC Systems -######################################################################### - -LIST_arc="$(targets_by_arch arc)" - -#----------------------------------------------------------------------- - -get_target_location() { - local target=$1 - local BOARD_NAME="" - local CONFIG_NAME="" - local board="" - local vendor="" - - # Automatic mode - local line=`awk '$7 == "'"$target"'" { print $0 }' boards.cfg` - if [ -z "${line}" ] ; then echo "" ; return ; fi - - set ${line} - - CONFIG_NAME="${7%_defconfig}" - - [ "${BOARD_NAME}" ] || BOARD_NAME="${7%_defconfig}" - - if [ $# -gt 5 ]; then - if [ "$6" = "-" ] ; then - board=${BOARD_NAME} - else - board="$6" - fi - fi - - [ $# -gt 4 ] && [ "$5" != "-" ] && vendor="$5" - [ $# -gt 6 ] && [ "$8" != "-" ] && { - tmp="${8%:*}" - if [ "$tmp" ] ; then - CONFIG_NAME="$tmp" - fi - } - - # Assign board directory to BOARDIR variable - if [ "${vendor}" == "-" ] ; then - BOARDDIR=${board} - else - BOARDDIR=${vendor}/${board} - fi - - echo "${CONFIG_NAME}:${BOARDDIR}:${BOARD_NAME}" -} - -get_target_maintainers() { - local name=`echo $1 | cut -d : -f 3` - - local line=`awk '$7 == "'"$target"'" { print $0 }' boards.cfg` - if [ -z "${line}" ]; then - echo "" - return ; - fi - - local mails=`echo ${line} | cut -d ' ' -f 9- | sed -e 's/[^<]*<//' -e 's/>.*</ /' -e 's/>[^>]*$//'` - [ "$mails" == "-" ] && mails="" - echo "$mails" -} - -get_target_arch() { - local target=$1 - - awk '$7 == "'$target'" { print $2 }' boards.cfg -} - -list_target() { - if [ "$PRINT_MAINTS" != 'y' ] ; then - echo "$1" - return - fi - - echo -n "$1:" - - local loc=`get_target_location $1` - - if [ -z "${loc}" ] ; then echo "ERROR" ; return ; fi - - local maintainers_result=`get_target_maintainers ${loc} | tr " " "\n"` - - if [ "$MAINTAINERS_ONLY" != 'y' ] ; then - - local dir=`echo ${loc} | cut -d ":" -f 2` - local cfg=`echo ${loc} | cut -d ":" -f 1` - local git_result=`git log --format=%aE board/${dir} \ - include/configs/${cfg}.h | grep "@"` - local git_result_recent=`echo ${git_result} | tr " " "\n" | \ - head -n 3` - local git_result_top=`echo ${git_result} | tr " " "\n" | \ - sort | uniq -c | sort -nr | head -n 3 | \ - sed "s/^ +[0-9]+ +//"` - - echo -e "$git_result_recent\n$git_result_top\n$maintainers_result" | \ - sort -u | tr "\n" " " | sed "s/ $//" ; - else - echo -e "$maintainers_result" | sort -u | tr "\n" " " | \ - sed "s/ $//" ; - fi - - echo "" -} - -# Each finished build will have a file called ${donep}${n}, -# where n is the index of the build. Each build -# we've already noted as finished will have ${skipp}${n}. -# The code managing the build process will use this information -# to ensure that only BUILD_NBUILDS builds are in flight at once -donep="${LOG_DIR}/._done_" -skipp="${LOG_DIR}/._skip_" - -build_target_killed() { - echo "Aborted $target build." - # Remove the logs for this board since it was aborted - rm -f ${LOG_DIR}/$target.MAKELOG ${LOG_DIR}/$target.ERR - exit -} - -build_target() { - target=$1 - build_idx=$2 - - if [ "$ONLY_LIST" == 'y' ] ; then - list_target ${target} - return - fi - - if [ $BUILD_MANY == 1 ] ; then - output_dir="${OUTPUT_PREFIX}/${target}" - mkdir -p "${output_dir}" - trap build_target_killed TERM - else - output_dir="${OUTPUT_PREFIX}" - fi - - target_arch=$(get_target_arch ${target}) - eval cross_toolchain=$CROSS_COMPILE_`echo $target_arch | tr '[:lower:]' '[:upper:]'` - if [ "${cross_toolchain}" ] ; then - MAKE="$GNU_MAKE CROSS_COMPILE=${cross_toolchain}" - elif [ "${CROSS_COMPILE}" ] ; then - MAKE="$GNU_MAKE CROSS_COMPILE=${CROSS_COMPILE}" - else - MAKE=$GNU_MAKE - fi - - if [ "${output_dir}" != "." ] ; then - MAKE="${MAKE} O=${output_dir}" - fi - - ${MAKE} mrproper >/dev/null - - echo "Building ${target} board..." - ${MAKE} -s ${target}_defconfig >/dev/null - - ${MAKE} ${JOBS} ${CHECK} all \ - >${LOG_DIR}/$target.MAKELOG 2> ${LOG_DIR}/$target.ERR - - # Check for 'make' errors - if [ ${PIPESTATUS[0]} -ne 0 ] ; then - RC=1 - fi - - OBJS=${output_dir}/u-boot - if [ -e ${output_dir}/spl/u-boot-spl ]; then - OBJS="${OBJS} ${output_dir}/spl/u-boot-spl" - fi - - ${CROSS_COMPILE}size ${OBJS} | tee -a ${LOG_DIR}/$target.MAKELOG - - if [ $BUILD_MANY == 1 ] ; then - trap - TERM - - ${MAKE} -s clean - - if [ -s ${LOG_DIR}/${target}.ERR ] ; then - cp ${LOG_DIR}/${target}.ERR ${OUTPUT_PREFIX}/ERR/${target} - else - rm ${LOG_DIR}/${target}.ERR - fi - else - if [ -s ${LOG_DIR}/${target}.ERR ] ; then - if grep -iw error ${LOG_DIR}/${target}.ERR ; then - : $(( ERR_CNT += 1 )) - ERR_LIST="${ERR_LIST} $target" - else - : $(( WRN_CNT += 1 )) - WRN_LIST="${WRN_LIST} $target" - fi - else - rm ${LOG_DIR}/${target}.ERR - fi - fi - - [ -e "${LOG_DIR}/${target}.ERR" ] && cat "${LOG_DIR}/${target}.ERR" - - touch "${donep}${build_idx}" -} - -manage_builds() { - search_idx=${OLDEST_IDX} - if [ "$ONLY_LIST" == 'y' ] ; then return ; fi - - while true; do - if [ -e "${donep}${search_idx}" ] ; then - : $(( CURRENT_CNT-- )) - [ ${OLDEST_IDX} -eq ${search_idx} ] && - : $(( OLDEST_IDX++ )) - - # Only want to count it once - rm -f "${donep}${search_idx}" - touch "${skipp}${search_idx}" - elif [ -e "${skipp}${search_idx}" ] ; then - [ ${OLDEST_IDX} -eq ${search_idx} ] && - : $(( OLDEST_IDX++ )) - fi - : $(( search_idx++ )) - if [ ${search_idx} -gt ${TOTAL_CNT} ] ; then - if [ ${CURRENT_CNT} -ge ${BUILD_NBUILDS} ] ; then - search_idx=${OLDEST_IDX} - sleep 1 - else - break - fi - fi - done -} - -build_targets() { - for t in "$@" ; do - # If a LIST_xxx var exists, use it. But avoid variable - # expansion in the eval when a board name contains certain - # characters that the shell interprets. - case ${t} in - *[-+=]*) list= ;; - *) list=$(eval echo '${LIST_'$t'}') ;; - esac - if [ -n "${list}" ] ; then - build_targets ${list} - else - : $((TOTAL_CNT += 1)) - : $((CURRENT_CNT += 1)) - rm -f "${donep}${TOTAL_CNT}" - rm -f "${skipp}${TOTAL_CNT}" - if [ "$CONTINUE" = 'y' -a -e ${LOG_DIR}/$t.MAKELOG ] ; then - : $((SKIP_CNT += 1)) - touch "${donep}${TOTAL_CNT}" - elif [ "$REBUILD_ERRORS" = 'y' -a ! -e ${LOG_DIR}/$t.ERR ] ; then - : $((SKIP_CNT += 1)) - touch "${donep}${TOTAL_CNT}" - else - if [ $BUILD_MANY == 1 ] ; then - build_target ${t} ${TOTAL_CNT} & - else - CUR_TGT="${t}" - build_target ${t} ${TOTAL_CNT} - CUR_TGT='' - fi - fi - fi - - # We maintain a running count of all the builds we have done. - # Each finished build will have a file called ${donep}${n}, - # where n is the index of the build. Each build - # we've already noted as finished will have ${skipp}${n}. - # We track the current index via TOTAL_CNT, and the oldest - # index. When we exceed the maximum number of parallel builds, - # We look from oldest to current for builds that have completed, - # and update the current count and oldest index as appropriate. - # If we've gone through the entire list, wait a second, and - # reprocess the entire list until we find a build that has - # completed - if [ ${CURRENT_CNT} -ge ${BUILD_NBUILDS} ] ; then - manage_builds - fi - done -} - -#----------------------------------------------------------------------- - -kill_children() { - local OS=$(uname -s) - local children="" - case "${OS}" in - "Darwin") - # Mac OS X is known to have BSD style ps - local pgid=$(ps -p $$ -o pgid | sed -e "/PGID/d") - children=$(ps -g $pgid -o pid | sed -e "/PID|$$|$pgid/d") - ;; - *) - # everything else tries the GNU style - local pgid=$(ps -p $$ --no-headers -o "%r" | tr -d ' ') - children=$(pgrep -g $pgid | sed -e "/$$|$pgid/d") - ;; - esac - - kill $children 2> /dev/null - wait $children 2> /dev/null - - exit -} - -print_stats() { - if [ "$ONLY_LIST" == 'y' ] ; then return ; fi - - # Only count boards that completed - : $((TOTAL_CNT = `find ${skipp}* 2> /dev/null | wc -l`)) - - rm -f ${donep}* ${skipp}* - - if [ $BUILD_MANY == 1 ] && [ -e "${OUTPUT_PREFIX}/ERR" ] ; then - ERR_LIST=`grep -riwl error ${OUTPUT_PREFIX}/ERR/` - ERR_LIST=`for f in $ERR_LIST ; do echo -n " $(basename $f)" ; done` - ERR_CNT=`echo $ERR_LIST | wc -w | awk '{print $1}'` - WRN_LIST=`grep -riwL error ${OUTPUT_PREFIX}/ERR/` - WRN_LIST=`for f in $WRN_LIST ; do echo -n " $(basename $f)" ; done` - WRN_CNT=`echo $WRN_LIST | wc -w | awk '{print $1}'` - else - # Remove the logs for any board that was interrupted - rm -f ${LOG_DIR}/${CUR_TGT}.MAKELOG ${LOG_DIR}/${CUR_TGT}.ERR - fi - - : $((TOTAL_CNT -= ${SKIP_CNT})) - echo "" - echo "--------------------- SUMMARY ----------------------------" - if [ "$CONTINUE" = 'y' -o "$REBUILD_ERRORS" = 'y' ] ; then - echo "Boards skipped: ${SKIP_CNT}" - fi - echo "Boards compiled: ${TOTAL_CNT}" - if [ ${ERR_CNT} -gt 0 ] ; then - echo "Boards with errors: ${ERR_CNT} (${ERR_LIST} )" - fi - if [ ${WRN_CNT} -gt 0 ] ; then - echo "Boards with warnings but no errors: ${WRN_CNT} (${WRN_LIST} )" - fi - echo "----------------------------------------------------------" - - if [ $BUILD_MANY == 1 ] ; then - kill_children - fi - - deprecation - - exit $RC -} - -#----------------------------------------------------------------------- - -# Build target groups selected by options, plus any command line args -set -- ${SELECTED} "$@" -# run PowerPC by default -[ $# = 0 ] && set -- powerpc -build_targets "$@" -wait

On Sun, Jul 03, 2016 at 03:14:26PM -0600, Simon Glass wrote:
Buildman has been around for 3 years now. It has had a lot of use and testing. Perhaps it is time to remove MAKEALL.
Signed-off-by: Simon Glass sjg@chromium.org
Reviewed-by: Tom Rini trini@konsulko.com

It is confusing to mention MAKEALL when it is not the normal way of building U-Boot anymore. Update the documentation to suit.
Signed-off-by: Simon Glass sjg@chromium.org ---
README | 35 ++++++----------------------------- board/Barix/ipam390/README.ipam390 | 12 ++++++------ doc/README.sha1 | 5 +++-- 3 files changed, 15 insertions(+), 37 deletions(-)
diff --git a/README b/README index 26d5ad2..704a370 100644 --- a/README +++ b/README @@ -2753,7 +2753,7 @@ CBFS (Coreboot Filesystem) support with a special header) as build targets. By defining CONFIG_BUILD_TARGET in the SoC / board header, this special image will be automatically built upon calling - make / MAKEALL. + make / buildman.
CONFIG_IDENT_STRING
@@ -5078,33 +5078,10 @@ official or latest in the git repository) version of U-Boot sources. But before you submit such a patch, please verify that your modifi- cation did not break existing code. At least make sure that *ALL* of the supported boards compile WITHOUT ANY compiler warnings. To do so, -just run the "MAKEALL" script, which will configure and build U-Boot -for ALL supported system. Be warned, this will take a while. You can -select which (cross) compiler to use by passing a `CROSS_COMPILE' -environment variable to the script, i. e. to use the ELDK cross tools -you can type - - CROSS_COMPILE=ppc_8xx- MAKEALL - -or to build on a native PowerPC system you can type - - CROSS_COMPILE=' ' MAKEALL - -When using the MAKEALL script, the default behaviour is to build -U-Boot in the source directory. This location can be changed by -setting the BUILD_DIR environment variable. Also, for each target -built, the MAKEALL script saves two log files (<target>.ERR and -<target>.MAKEALL) in the <source dir>/LOG directory. This default -location can be changed by setting the MAKEALL_LOGDIR environment -variable. For example: - - export BUILD_DIR=/tmp/build - export MAKEALL_LOGDIR=/tmp/log - CROSS_COMPILE=ppc_8xx- MAKEALL - -With the above settings build objects are saved in the /tmp/build, -log files are saved in the /tmp/log and the source tree remains clean -during the whole build process. +just run the buildman script (tools/buildman/buildman), which will +configure and build U-Boot for ALL supported system. Be warned, this +will take a while. Please see the buildman README, or run 'buildman -H' +for documentation.
See also "U-Boot Porting Guide" below. @@ -6560,7 +6537,7 @@ it:
Notes:
-* Before sending the patch, run the MAKEALL script on your patched +* Before sending the patch, run the buildman script on your patched source tree and make sure that no errors or warnings are reported for any of the boards.
diff --git a/board/Barix/ipam390/README.ipam390 b/board/Barix/ipam390/README.ipam390 index 5c45fca..be09280 100644 --- a/board/Barix/ipam390/README.ipam390 +++ b/board/Barix/ipam390/README.ipam390 @@ -31,9 +31,9 @@ loading the next image from a different media, etc).
Compilation =========== -run "./MAKEALL ipam390" in the u-boot source tree. -Once this build completes you will have a u-boot.ais file that needs to -be written to the nand flash. +run "tools/buildman/buildman -k ipam390" in the u-boot source tree. +Once this build completes you will have a ../current/ipam390/u-boot.ais file +that needs to be written to the nand flash.
Flashing the images to NAND ========================== @@ -71,13 +71,13 @@ here[1] to create an uboot-uart-ais.bin file - cd to the u-boot source tree
- compile the u-boot for the ipam390 board: -$ ./MAKEALL ipam390 +$ tools/buildman/buildman -k ipam390
-> Now we shall have u-boot.bin
- Create u-boot-uart-ais.bin -$ mono HexAIS_OMAP-L138.exe -entrypoint 0xC1080000 -ini -ipam390-ais-uart.cfg -o ./uboot-uart-ais.bin ./u-boot.bin@0xC1080000; +$ mono HexAIS_OMAP-L138.exe -entrypoint 0xC1080000 -ini ipam390-ais-uart.cfg \ + -o ../current/ipam390/uboot-uart-ais.bin ./u-boot.bin@0xC1080000;
Note: The ipam390-ais-uart.cfg is found in the board directory for the ipam390 board, u-boot:/board/Barix/ipam390/ipam390-ais-uart.cfg diff --git a/doc/README.sha1 b/doc/README.sha1 index f6cca40..f178f37 100644 --- a/doc/README.sha1 +++ b/doc/README.sha1 @@ -51,7 +51,8 @@ c) now calculate the SHA1 sum from the memoryrange and write
Now you have a U-Boot-Image for the pcs440ep board with the correct SHA1 sum.
-If you do a "./MAKEALL pcs440ep" or a "make all" to get the U-Boot image, -the correct SHA1 sum will be automagically included in the U-Boot image. +If you do a "buildman -k pcs440ep" or a "make all" to get the U-Boot image, +which will be found in ../current/ipam390/ - the correct SHA1 sum will be +automagically included in the U-Boot image.
Heiko Schocher, 11 Jul 2007

On Sun, Jul 03, 2016 at 03:14:27PM -0600, Simon Glass wrote:
It is confusing to mention MAKEALL when it is not the normal way of building U-Boot anymore. Update the documentation to suit.
Signed-off-by: Simon Glass sjg@chromium.org
Reviewed-by: Tom Rini trini@konsulko.com

On 07/03/2016 03:14 PM, Simon Glass wrote:
This makes a few minor improvements to buildman to make it work more easiler for first-time users:
- Improve progress and warning messages when fetching toolchains
- Fix a bug where toolchain paths can be overwritten when fetching
- Note at the top of the help how to get started
Also this series removes MAKEALL. Since buildman has been around for 3 years it may be time to do this. If not, we can leave it for now.
Sounds great:-)
BTW, one problem I've come across with buildman recently is that it re-orders warnings/errors from the compiler. In particular, it seems to separate what it thinks are warning from errors (perhaps stdout/stderr split??) but I'm not sure it always gets it right, and I think elides duplicate lines when doing so. When running buildman repeatedly while doing development, and experiencing a compile error, this makes it extremely difficult to interpret some of the compiler's multi-line messages. Is it possible to have an option that turns off all the output processing, and just shows stderr/out mixed up as they would appear "interactively" when manually invoked from a shell, with no post-processing? Occasionally I have to fall back to a manual "make" invocation due to this.

On Thu, Jul 07, 2016 at 12:27:53PM -0600, Stephen Warren wrote:
On 07/03/2016 03:14 PM, Simon Glass wrote:
This makes a few minor improvements to buildman to make it work more easiler for first-time users:
- Improve progress and warning messages when fetching toolchains
- Fix a bug where toolchain paths can be overwritten when fetching
- Note at the top of the help how to get started
Also this series removes MAKEALL. Since buildman has been around for 3 years it may be time to do this. If not, we can leave it for now.
Sounds great:-)
BTW, one problem I've come across with buildman recently is that it re-orders warnings/errors from the compiler. In particular, it seems to separate what it thinks are warning from errors (perhaps stdout/stderr split??) but I'm not sure it always gets it right, and I think elides duplicate lines when doing so. When running buildman repeatedly while doing development, and experiencing a compile error, this makes it extremely difficult to interpret some of the compiler's multi-line messages. Is it possible to have an option that turns off all the output processing, and just shows stderr/out mixed up as they would appear "interactively" when manually invoked from a shell, with no post-processing? Occasionally I have to fall back to a manual "make" invocation due to this.
Yes, some way to turn off this coelescing of messages would be good or maybe put the messages in a file (per board) in order?

On Sun, Jul 03, 2016 at 03:14:16PM -0600, Simon Glass wrote:
This makes a few minor improvements to buildman to make it work more easiler for first-time users:
- Improve progress and warning messages when fetching toolchains
- Fix a bug where toolchain paths can be overwritten when fetching
- Note at the top of the help how to get started
Also this series removes MAKEALL. Since buildman has been around for 3 years it may be time to do this. If not, we can leave it for now.
Thanks a lot for doing this. A few comments to follow.
participants (3)
-
Simon Glass
-
Stephen Warren
-
Tom Rini