[U-Boot] [PATCH] Don't apply: tools: add a tool to move automatically CONFIGs from headers to defconfigs
This tool can move CONFIG macros from C headers (include/configs/*.h) to defconfigs (configs/*_defconfig) all over the boards.
There are tons of CONFIGs in U-boot and moving them by hand is absolutely tool painful. I wrote this script for my local use, so this patch might not clean enough to be applied to the code base. It might contain some bugs.
Please do not apply this patch!
This patch is here because Simon requested me to share it. See the discussion in here: http://lists.denx.de/pipermail/u-boot/2015-January/201556.html
Usage:
[1] Please run "git status" and make sure your git repository is clean.
[2] Describe the CONFIG option you want move in the file "~/.moveconfig"
Say, you want to move CONFIG_CMD_NAND, for example. In this case, the content of "~/.moveconfig" should be like this:
$ cat .moveconfig CONFIG_CMD_NAND bool n y
- The first field is the name of the CONFIG. - The second field is the type of the CONFIG such as "bool", "int", "hex", etc. - The third value is the default value. The value that is same as the default is omitted in each defconfig. If the type of the CONFIG is bool, the default value must be either 'y' or 'n'. - The forth field shows whether the CONFIG depends on CONFIG_SPL_BUILD. For example, CONFIG_CMD_* makes sense only on the main-image. (depends on !SPL_BUILD) In this case, specify 'y' in the forth fields. If the CONFIG should appear also on SPL, specify 'n' here.
[4] Run "tools/moveconfig.py"
The log will be displayed like this
$ tools/moveconfig.py Moving CONFIG_CMD_NAND (type: bool, default: n, no_spl: y) ... (jobs: 8) ms7750se : (default) davinci_sonata : (default) tk71 : y db-mv784mp-gp : (default) cm_t3517 : y P2041RDB_SDCARD : y ...
The left-hand side field is the board[defconfig] name and the colon is followed by the value of the CONFIG.
At the last stage, you will be asked like this:
Clean up CONFIG_CMD_NAND in headers? [y/n]:
If you say 'y' here, the defines in C headers will be removed.
Enjoy!
Signed-off-by: Masahiro Yamada yamada.m@jp.panasonic.com Cc: Simon Glass sjg@chromium.org Cc: Alexey Brodkin abrodkin@synopsys.com ---
tools/moveconfig.py | 439 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 439 insertions(+) create mode 100755 tools/moveconfig.py
diff --git a/tools/moveconfig.py b/tools/moveconfig.py new file mode 100755 index 0000000..cb7f7ea --- /dev/null +++ b/tools/moveconfig.py @@ -0,0 +1,439 @@ +#!/usr/bin/env python +# +# Author: Masahiro Yamada yamada.m@jp.panasonic.com +# +# SPDX-License-Identifier: GPL-2.0+ +# + +""" +Move the specified config option to Kconfig +""" + +import errno +import fnmatch +import multiprocessing +import optparse +import os +import re +import shutil +import subprocess +import sys +import tempfile +import time + +SHOW_GNU_MAKE = 'scripts/show-gnu-make' +SLEEP_TIME=0.03 + +CROSS_COMPILE = { + 'aarch64': 'aarch64-linux-', + 'arm': 'arm-unknown-linux-gnueabi-', + 'avr32': 'avr32-linux-', + 'blackfin': 'bfin-elf-', + 'm68k': 'm68k-linux-', + 'microblaze': 'microblaze-linux-', + 'mips': 'mips-linux-', + 'nds32': 'nds32le-linux-', + 'nios2': 'nios2-linux-', + 'openrisc': 'or32-linux-', + 'powerpc': 'powerpc-linux-', + 'sh': 'sh4-gentoo-linux-gnu-', + 'sparc': 'sparc-linux-', + 'x86': 'i386-linux-' +} + +STATE_IDLE = 0 +STATE_DEFCONFIG = 1 +STATE_SILENTOLDCONFIG = 2 + +### helper functions ### +def get_devnull(): + """Get the file object of '/dev/null' device.""" + try: + devnull = subprocess.DEVNULL # py3k + except AttributeError: + devnull = open(os.devnull, 'wb') + return devnull + +def check_top_directory(): + """Exit if we are not at the top of source directory.""" + for f in ('README', 'Licenses'): + if not os.path.exists(f): + sys.exit('Please run at the top of source directory.') + +def get_make_cmd(): + """Get the command name of GNU Make.""" + process = subprocess.Popen([SHOW_GNU_MAKE], stdout=subprocess.PIPE) + ret = process.communicate() + if process.returncode: + sys.exit('GNU Make not found') + return ret[0].rstrip() + +def cleanup_header(header_path, patterns): + with open(header_path) as f: + lines = f.readlines() + + matched = [] + for i, line in enumerate(lines): + for pattern in patterns: + m = pattern.search(line) + if m: + print '%s: %s: %s' % (header_path, i + 1, line), + matched.append(i) + break + + if not matched: + return + + with open(header_path, 'w') as f: + for i, line in enumerate(lines): + if not i in matched: + f.write(line) + +def cleanup_headers(config): + while True: + choice = raw_input('Clean up %s in headers? [y/n]: ' % config).lower() + print choice + if choice == 'y' or choice == 'n': + break + + if choice != 'y': + return + + patterns = [] + patterns.append(re.compile(r'#\s*define\s+%s\W' % config)) + patterns.append(re.compile(r'#\s*undef\s+%s\W' % config)) + + for (dirpath, dirnames, filenames) in os.walk('include'): + for filename in filenames: + if not fnmatch.fnmatch(filename, '*~'): + cleanup_header(os.path.join(dirpath, filename), patterns) + +### classes ### +class KconfigParser: + + """A parser of .config file. + + Each line of the output should have the form of: + Status, Arch, CPU, SoC, Vendor, Board, Target, Options, Maintainers + Most of them are extracted from .config file. + MAINTAINERS files are also consulted for Status and Maintainers fields. + """ + + re_arch = re.compile(r'CONFIG_SYS_ARCH="(.*)"') + re_cpu = re.compile(r'CONFIG_SYS_CPU="(.*)"') + PREFIX = { '.': '+', 'spl': 'S', 'tpl': 'T' } + + def __init__(self, build_dir, config_attr): + """Create a new .config perser. + + Arguments: + build_dir: Build directory where .config is located + output: File object which the result is written to + """ + self.build_dir = build_dir + self.config = config_attr['config'] + self.type = config_attr['type'] + self.default = config_attr['default'] + if config_attr['no_spl_support'] == 'y': + self.no_spl_support = True + else: + self.no_spl_support = False + self.re = re.compile(r'%s=(.*)' % self.config) + + def get_cross_compile(self): + """Parse .config file and return CROSS_COMPILE + + Arguments: + defconfig: Board (defconfig) name + """ + arch = '' + cpu = '' + dotconfig = os.path.join(self.build_dir, '.config') + for line in open(dotconfig): + m = self.re_arch.match(line) + if m: + arch = m.group(1) + continue + m = self.re_cpu.match(line) + if m: + cpu = m.group(1) + + assert arch, 'Error: arch is not defined in %s' % defconfig + + # fix-up for aarch64 + if arch == 'arm' and cpu == 'armv8': + arch = 'aarch64' + + return CROSS_COMPILE.get(arch, '') + + def update_defconfig(self, defconfig): + values = [] + output_lines = [] + prefixes = {} + + if self.no_spl_support: + images = ('.') + else: + images = ('.', 'spl', 'tpl') + + + for img in images: + autoconf = os.path.join(self.build_dir, img, 'include', + 'autoconf.mk') + + if not os.path.exists(autoconf): + values.append('') + continue + + if self.type == 'bool': + value = 'n' + else: + value = '(undef)' + + for line in open(autoconf): + m = self.re.match(line) + if m: + value = m.group(1) + break + + if value == self.default: + value = '(default)' + + values.append(value) + os.remove(autoconf) + + if value == '(undef)' or value == '(default)': + continue + + if self.type == 'bool' and value == 'n': + output_line = '# %s is not set' % (self.config) + else: + output_line = '%s=%s' % (self.config, value) + if output_line in output_lines: + prefixes[output_line] += self.PREFIX[img] + else: + output_lines.append(output_line) + prefixes[output_line] = self.PREFIX[img] + + output = defconfig[:-len('_defconfig')].ljust(37) + ': ' + for value in values: + output += value.ljust(12) + + print output.strip() + + with open(os.path.join('configs', defconfig), 'a') as f: + for line in output_lines: + if prefixes[line] != '+': + line = prefixes[line] + ':' + line + f.write(line + '\n') + +class Slot: + + """A slot to store a subprocess. + + Each instance of this class handles one subprocess. + This class is useful to control multiple processes + for faster processing. + """ + + def __init__(self, config_attr, devnull, make_cmd): + """Create a new slot. + + Arguments: + output: File object which the result is written to + """ + self.build_dir = tempfile.mkdtemp() + self.devnull = devnull + self.make_cmd = (make_cmd, 'O=' + self.build_dir) + self.parser = KconfigParser(self.build_dir, config_attr) + self.state = STATE_IDLE + + def __del__(self): + """Delete the working directory""" + if self.state != STATE_IDLE: + while self.ps.poll() == None: + pass + shutil.rmtree(self.build_dir) + + def add(self, defconfig): + """Add a new subprocess to the slot. + + Fails if the slot is occupied, that is, the current subprocess + is still running. + + Arguments: + defconfig: Board (defconfig) name + + Returns: + Return True on success or False on fail + """ + if self.state != STATE_IDLE: + return False + cmd = list(self.make_cmd) + cmd.append(defconfig) + self.ps = subprocess.Popen(cmd, stdout=self.devnull) + self.defconfig = defconfig + self.state = STATE_DEFCONFIG + return True + + def poll(self): + """Check if the subprocess is running and invoke the .config + parser if the subprocess is terminated. + + Returns: + Return True if the subprocess is terminated, False otherwise + """ + if self.state == STATE_IDLE: + return True + + if self.ps.poll() == None: + return False + + if self.ps.poll() != 0: + sys.exit("failed to process '%s'" % self.defconfig) + + if self.state == STATE_SILENTOLDCONFIG: + self.parser.update_defconfig(self.defconfig) + self.state = STATE_IDLE + return True + + cross_compile = self.parser.get_cross_compile() + cmd = list(self.make_cmd) + if cross_compile: + cmd.append('CROSS_COMPILE=%s' % cross_compile) + cmd.append('silentoldconfig') + self.ps = subprocess.Popen(cmd, stdout=self.devnull) + self.state = STATE_SILENTOLDCONFIG + return False + +class Slots: + + """Controller of the array of subprocess slots.""" + + def __init__(self, config_attr, jobs): + """Create a new slots controller. + + Arguments: + jobs: A number of slots to instantiate + """ + self.slots = [] + devnull = get_devnull() + make_cmd = get_make_cmd() + for i in range(jobs): + self.slots.append(Slot(config_attr, devnull, make_cmd)) + + def add(self, defconfig): + """Add a new subprocess if a vacant slot is available. + + Arguments: + defconfig: Board (defconfig) name + + Returns: + Return True on success or False on fail + """ + for slot in self.slots: + if slot.add(defconfig): + return True + return False + + def available(self): + """Check if there is a vacant slot. + + Returns: + Return True if a vacant slot is found, False if all slots are full + """ + for slot in self.slots: + if slot.poll(): + return True + return False + + def empty(self): + """Check if all slots are vacant. + + Returns: + Return True if all slots are vacant, False if at least one slot + is running + """ + ret = True + for slot in self.slots: + if not slot.poll(): + ret = False + return ret + +def move_config(config_attr, jobs=1): + check_top_directory() + print 'Moving %s (type: %s, default: %s, no_spl: %s) ... (jobs: %d)' % ( + config_attr['config'], + config_attr['type'], + config_attr['default'], + config_attr['no_spl_support'], + jobs) + + # All the defconfig files to be processed + defconfigs = [] + for (dirpath, dirnames, filenames) in os.walk('configs'): + dirpath = dirpath[len('configs') + 1:] + for filename in fnmatch.filter(filenames, '*_defconfig'): + if fnmatch.fnmatch(filename, '.*'): + continue + defconfigs.append(os.path.join(dirpath, filename)) + + slots = Slots(config_attr, jobs) + + # Main loop to process defconfig files: + # Add a new subprocess into a vacant slot. + # Sleep if there is no available slot. + for defconfig in defconfigs: + while not slots.add(defconfig): + while not slots.available(): + # No available slot: sleep for a while + time.sleep(SLEEP_TIME) + + # wait until all the subprocesses finish + while not slots.empty(): + time.sleep(SLEEP_TIME) + + cleanup_headers(config_attr['config']) + +def main(): + try: + cpu_count = multiprocessing.cpu_count() + except NotImplementedError: + cpu_count = 1 + + parser = optparse.OptionParser() + # Add options here + parser.add_option('-j', '--jobs', type='int', default=cpu_count, + help='the number of jobs to run simultaneously') + parser.usage += ' config type default no_spl_support' + (options, args) = parser.parse_args() + + args_key = ('config', 'type', 'default', 'no_spl_support') + config_attr = {} + + if len(args) >= len(args_key): + saved_attr = '' + for i, key in enumerate(args_key): + config_attr[key] = args[i] + saved_attr = saved_attr + ' %s' % args[i] + with open('.moveconfig', 'w') as f: + f.write(saved_attr) + elif os.path.exists('.moveconfig'): + f = open('.moveconfig') + try: + saved_attr = f.readline().split() + for i, key in enumerate(args_key): + config_attr[key] = saved_attr[i] + except: + sys.exit('%s: broken format' % '.moveconfig') + else: + parser.print_usage() + sys.exit(1) + + if not config_attr['config'].startswith('CONFIG_'): + config_attr['config'] = 'CONFIG_' + config_attr['config'] + + move_config(config_attr, options.jobs) + +if __name__ == '__main__': + main()
Hi Masahiro,
On 19 January 2015 at 05:12, Masahiro Yamada yamada.m@jp.panasonic.com wrote:
This tool can move CONFIG macros from C headers (include/configs/*.h) to defconfigs (configs/*_defconfig) all over the boards.
There are tons of CONFIGs in U-boot and moving them by hand is absolutely tool painful. I wrote this script for my local use, so this patch might not clean enough to be applied to the code base. It might contain some bugs.
Please do not apply this patch!
This patch is here because Simon requested me to share it. See the discussion in here: http://lists.denx.de/pipermail/u-boot/2015-January/201556.html
Usage:
[1] Please run "git status" and make sure your git repository is clean.
[2] Describe the CONFIG option you want move in the file "~/.moveconfig"
Say, you want to move CONFIG_CMD_NAND, for example. In this case, the content of "~/.moveconfig" should be like this:
$ cat .moveconfig CONFIG_CMD_NAND bool n y
- The first field is the name of the CONFIG.
- The second field is the type of the CONFIG such as "bool", "int", "hex", etc.
- The third value is the default value. The value that is same as the default is omitted in each defconfig. If the type of the CONFIG is bool, the default value must be either 'y' or 'n'.
- The forth field shows whether the CONFIG depends on CONFIG_SPL_BUILD. For example, CONFIG_CMD_* makes sense only on the main-image. (depends on !SPL_BUILD) In this case, specify 'y' in the forth fields. If the CONFIG should appear also on SPL, specify 'n' here.
[4] Run "tools/moveconfig.py"
The log will be displayed like this
$ tools/moveconfig.py Moving CONFIG_CMD_NAND (type: bool, default: n, no_spl: y) ... (jobs: 8) ms7750se : (default) davinci_sonata : (default) tk71 : y db-mv784mp-gp : (default) cm_t3517 : y P2041RDB_SDCARD : y ...
The left-hand side field is the board[defconfig] name and the colon is followed by the value of the CONFIG.
At the last stage, you will be asked like this:
Clean up CONFIG_CMD_NAND in headers? [y/n]:
If you say 'y' here, the defines in C headers will be removed.
Enjoy!
Just a note to say that I used this tool to handle the BOOTSTAGE patch. It worked very nicely, thank you.!
I don't think it would take much to get this into shape for applying. What do you think?
Signed-off-by: Masahiro Yamada yamada.m@jp.panasonic.com Cc: Simon Glass sjg@chromium.org Cc: Alexey Brodkin abrodkin@synopsys.com
tools/moveconfig.py | 439 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 439 insertions(+) create mode 100755 tools/moveconfig.py
Regards, Simon
Hi Simon, Sorry for my late reply.
2015-03-04 9:18 GMT+09:00 Simon Glass sjg@chromium.org:
Hi Masahiro,
On 19 January 2015 at 05:12, Masahiro Yamada yamada.m@jp.panasonic.com wrote:
This tool can move CONFIG macros from C headers (include/configs/*.h) to defconfigs (configs/*_defconfig) all over the boards.
There are tons of CONFIGs in U-boot and moving them by hand is absolutely tool painful. I wrote this script for my local use, so this patch might not clean enough to be applied to the code base. It might contain some bugs.
Please do not apply this patch!
This patch is here because Simon requested me to share it. See the discussion in here: http://lists.denx.de/pipermail/u-boot/2015-January/201556.html
Usage:
[1] Please run "git status" and make sure your git repository is clean.
[2] Describe the CONFIG option you want move in the file "~/.moveconfig"
Say, you want to move CONFIG_CMD_NAND, for example. In this case, the content of "~/.moveconfig" should be like this:
$ cat .moveconfig CONFIG_CMD_NAND bool n y
- The first field is the name of the CONFIG.
- The second field is the type of the CONFIG such as "bool", "int", "hex", etc.
- The third value is the default value. The value that is same as the default is omitted in each defconfig. If the type of the CONFIG is bool, the default value must be either 'y' or 'n'.
- The forth field shows whether the CONFIG depends on CONFIG_SPL_BUILD. For example, CONFIG_CMD_* makes sense only on the main-image. (depends on !SPL_BUILD) In this case, specify 'y' in the forth fields. If the CONFIG should appear also on SPL, specify 'n' here.
[4] Run "tools/moveconfig.py"
The log will be displayed like this
$ tools/moveconfig.py Moving CONFIG_CMD_NAND (type: bool, default: n, no_spl: y) ... (jobs: 8) ms7750se : (default) davinci_sonata : (default) tk71 : y db-mv784mp-gp : (default) cm_t3517 : y P2041RDB_SDCARD : y ...
The left-hand side field is the board[defconfig] name and the colon is followed by the value of the CONFIG.
At the last stage, you will be asked like this:
Clean up CONFIG_CMD_NAND in headers? [y/n]:
If you say 'y' here, the defines in C headers will be removed.
Enjoy!
Just a note to say that I used this tool to handle the BOOTSTAGE patch. It worked very nicely, thank you.!
I don't think it would take much to get this into shape for applying. What do you think?
OK, I will do it.
Best Regards Masahiro Yamada
The existing target won't actually build the needed .mk file that is expected by the rest of the script. It seems that silentoldconfig does not actually cause this to be built any longer.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com --- This patch is based on: "Don't apply: tools: add a tool to move automatically CONFIGs from headers to defconfigs" from the list.
tools/moveconfig.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index cb7f7ea..30dc4f6 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -301,7 +301,7 @@ class Slot: cmd = list(self.make_cmd) if cross_compile: cmd.append('CROSS_COMPILE=%s' % cross_compile) - cmd.append('silentoldconfig') + cmd.append('include/autoconf.mk') self.ps = subprocess.Popen(cmd, stdout=self.devnull) self.state = STATE_SILENTOLDCONFIG return False
The arc architecture is supported by U-Boot, so add a mapping here for it as well.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com ---
tools/moveconfig.py | 1 + 1 file changed, 1 insertion(+)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index 30dc4f6..c81f32c 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -26,6 +26,7 @@ SLEEP_TIME=0.03
CROSS_COMPILE = { 'aarch64': 'aarch64-linux-', + 'arc': 'arc-linux-', 'arm': 'arm-unknown-linux-gnueabi-', 'avr32': 'avr32-linux-', 'blackfin': 'bfin-elf-',
Some compilers are hard to come by or have so few boards they are not worth messing with for this tool. Provide a list that need manual intervention and continue moving the bulk of boards.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com ---
tools/moveconfig.py | 29 +++++++++++++++++++++++++++-- 1 file changed, 27 insertions(+), 2 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index c81f32c..ab53476 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -17,6 +17,7 @@ import os import re import shutil import subprocess +from subprocess import PIPE import sys import tempfile import time @@ -277,6 +278,13 @@ class Slot: self.state = STATE_DEFCONFIG return True
+ def defconfig_error(self, errmsg): + output = self.defconfig[:-len('_defconfig')].ljust(37) + ': ' + print output + errmsg + + """Save a list of targets that have to be checked by hand""" + open('moveconfig.failed', 'a+').write("%s\n" % self.defconfig) + def poll(self): """Check if the subprocess is running and invoke the .config parser if the subprocess is terminated. @@ -291,7 +299,13 @@ class Slot: return False
if self.ps.poll() != 0: - sys.exit("failed to process '%s'" % self.defconfig) + errmsg = 'ERROR - build error' + errout = self.ps.stderr.read() + if errout.find('gcc: command not found') != -1: + errmsg = 'ERROR - compiler not found' + self.defconfig_error(errmsg) + self.state = STATE_IDLE + return True
if self.state == STATE_SILENTOLDCONFIG: self.parser.update_defconfig(self.defconfig) @@ -303,7 +317,7 @@ class Slot: if cross_compile: cmd.append('CROSS_COMPILE=%s' % cross_compile) cmd.append('include/autoconf.mk') - self.ps = subprocess.Popen(cmd, stdout=self.devnull) + self.ps = subprocess.Popen(cmd, stdout=self.devnull, stderr=PIPE) self.state = STATE_SILENTOLDCONFIG return False
@@ -363,6 +377,11 @@ class Slots:
def move_config(config_attr, jobs=1): check_top_directory() + + """Clean up any previous log of failed moves""" + if os.path.exists('moveconfig.failed'): + os.remove('moveconfig.failed') + print 'Moving %s (type: %s, default: %s, no_spl: %s) ... (jobs: %d)' % ( config_attr['config'], config_attr['type'], @@ -396,6 +415,12 @@ def move_config(config_attr, jobs=1):
cleanup_headers(config_attr['config'])
+ if os.path.exists('moveconfig.failed'): + print '!!! Some boards were not processed; move the config manually.' + print '!!! The list of failed boards are saved in moveconfig.failed' + print + print open('moveconfig.failed', 'r').read() + def main(): try: cpu_count = multiprocessing.cpu_count()
The main image autoconf.mk certainly had better exist. I'm not sure about SPL and TPL, but at least this one. Check for this and error if missing.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com ---
tools/moveconfig.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index ab53476..7635186 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -184,6 +184,8 @@ class KconfigParser: 'autoconf.mk')
if not os.path.exists(autoconf): + if img == '.': + return True values.append('') continue
@@ -228,6 +230,7 @@ class KconfigParser: if prefixes[line] != '+': line = prefixes[line] + ':' + line f.write(line + '\n') + return False
class Slot:
@@ -308,7 +311,8 @@ class Slot: return True
if self.state == STATE_SILENTOLDCONFIG: - self.parser.update_defconfig(self.defconfig) + if self.parser.update_defconfig(self.defconfig): + self.defconfig_error('ERROR - autoconf.mk not found') self.state = STATE_IDLE return True
This will ensure that the order of the defconfig entries will always match that of the Kconfig files. After one slightly painful (but still early in the process) pass over all boards, this should keep the defconfigs clean from here on.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com ---
tools/moveconfig.py | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index 7635186..02ec976 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -46,6 +46,7 @@ CROSS_COMPILE = { STATE_IDLE = 0 STATE_DEFCONFIG = 1 STATE_SILENTOLDCONFIG = 2 +STATE_SAVEDEFCONFIG = 3
### helper functions ### def get_devnull(): @@ -150,8 +151,8 @@ class KconfigParser: """ arch = '' cpu = '' - dotconfig = os.path.join(self.build_dir, '.config') - for line in open(dotconfig): + self.dotconfig = os.path.join(self.build_dir, '.config') + for line in open(self.dotconfig): m = self.re_arch.match(line) if m: arch = m.group(1) @@ -225,7 +226,7 @@ class KconfigParser:
print output.strip()
- with open(os.path.join('configs', defconfig), 'a') as f: + with open(os.path.join(self.dotconfig), 'a') as f: for line in output_lines: if prefixes[line] != '+': line = prefixes[line] + ':' + line @@ -313,6 +314,23 @@ class Slot: if self.state == STATE_SILENTOLDCONFIG: if self.parser.update_defconfig(self.defconfig): self.defconfig_error('ERROR - autoconf.mk not found') + self.state = STATE_IDLE + return True + + """Save off the defconfig in a consistent way""" + cmd = list(self.make_cmd) + cmd.append('savedefconfig') + self.ps = subprocess.Popen(cmd, stdout=self.devnull, + stderr=self.devnull) + self.state = STATE_SAVEDEFCONFIG + return False + + if self.state == STATE_SAVEDEFCONFIG: + defconfig_path = os.path.join(self.build_dir, 'defconfig') + if not os.path.exists(defconfig_path): + self.defconfig_error('ERROR - defconfig not updated') + shutil.move(defconfig_path, + os.path.join('configs', self.defconfig)) self.state = STATE_IDLE return True
The existing target won't actually build the needed .mk file that is expected by the rest of the script. It seems that silentoldconfig does not actually cause this to be built any longer.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com --- This patch is based on: "Don't apply: tools: add a tool to move automatically CONFIGs from headers to defconfigs" from the list.
Changes in v2: None
tools/moveconfig.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index cb7f7ea..30dc4f6 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -301,7 +301,7 @@ class Slot: cmd = list(self.make_cmd) if cross_compile: cmd.append('CROSS_COMPILE=%s' % cross_compile) - cmd.append('silentoldconfig') + cmd.append('include/autoconf.mk') self.ps = subprocess.Popen(cmd, stdout=self.devnull) self.state = STATE_SILENTOLDCONFIG return False
The arc architecture is supported by U-Boot, so add a mapping here for it as well.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com ---
Changes in v2: None
tools/moveconfig.py | 1 + 1 file changed, 1 insertion(+)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index 30dc4f6..c81f32c 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -26,6 +26,7 @@ SLEEP_TIME=0.03
CROSS_COMPILE = { 'aarch64': 'aarch64-linux-', + 'arc': 'arc-linux-', 'arm': 'arm-unknown-linux-gnueabi-', 'avr32': 'avr32-linux-', 'blackfin': 'bfin-elf-',
Hi Joe.
2015-05-12 2:23 GMT+09:00 Joe Hershberger joe.hershberger@ni.com:
The arc architecture is supported by U-Boot, so add a mapping here for it as well.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
Changes in v2: None
tools/moveconfig.py | 1 + 1 file changed, 1 insertion(+)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index 30dc4f6..c81f32c 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -26,6 +26,7 @@ SLEEP_TIME=0.03
CROSS_COMPILE = { 'aarch64': 'aarch64-linux-',
- 'arc': 'arc-linux-', 'arm': 'arm-unknown-linux-gnueabi-', 'avr32': 'avr32-linux-', 'blackfin': 'bfin-elf-',
IIRC, I intentionally did not add it because the default ARC CROSS_COMPILE is defined by arch/arc/config.mk (Looks like ARC wants to choose little/big endian variants depending on the configuration, although it might not be an issue for this tool.)
Better to override it?
Hi Masahiro-san,
On Tue, May 12, 2015 at 9:14 PM, Masahiro Yamada yamada.masahiro@socionext.com wrote:
Hi Joe.
2015-05-12 2:23 GMT+09:00 Joe Hershberger joe.hershberger@ni.com:
The arc architecture is supported by U-Boot, so add a mapping here for it as well.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
Changes in v2: None
tools/moveconfig.py | 1 + 1 file changed, 1 insertion(+)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index 30dc4f6..c81f32c 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -26,6 +26,7 @@ SLEEP_TIME=0.03
CROSS_COMPILE = { 'aarch64': 'aarch64-linux-',
- 'arc': 'arc-linux-', 'arm': 'arm-unknown-linux-gnueabi-', 'avr32': 'avr32-linux-', 'blackfin': 'bfin-elf-',
IIRC, I intentionally did not add it because the default ARC CROSS_COMPILE is defined by arch/arc/config.mk (Looks like ARC wants to choose little/big endian variants depending on the configuration, although it might not be an issue for this tool.)
Thanks for pointing that out.
Better to override it?
I am successfully building, at least to get the config output, without switching to a big-endian compiler. I can build the other compiler too if you prefer not to map this.
Cheers, -Joe
Some compilers are hard to come by or have so few boards they are not worth messing with for this tool. Provide a list that need manual intervention and continue moving the bulk of boards.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
---
Changes in v2: -Print which compiler is missing
tools/moveconfig.py | 35 ++++++++++++++++++++++++++++++----- 1 file changed, 30 insertions(+), 5 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index c81f32c..315f4be 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -17,6 +17,7 @@ import os import re import shutil import subprocess +from subprocess import PIPE import sys import tempfile import time @@ -277,6 +278,13 @@ class Slot: self.state = STATE_DEFCONFIG return True
+ def defconfig_error(self, errmsg): + output = self.defconfig[:-len('_defconfig')].ljust(37) + ': ' + print output + errmsg + + """Save a list of targets that have to be checked by hand""" + open('moveconfig.failed', 'a+').write("%s\n" % self.defconfig) + def poll(self): """Check if the subprocess is running and invoke the .config parser if the subprocess is terminated. @@ -291,19 +299,25 @@ class Slot: return False
if self.ps.poll() != 0: - sys.exit("failed to process '%s'" % self.defconfig) + errmsg = 'ERROR - build error' + errout = self.ps.stderr.read() + if errout.find('gcc: command not found') != -1: + errmsg = 'ERROR - compiler not found (%s)' % self.cross_compile + self.defconfig_error(errmsg) + self.state = STATE_IDLE + return True
if self.state == STATE_SILENTOLDCONFIG: self.parser.update_defconfig(self.defconfig) self.state = STATE_IDLE return True
- cross_compile = self.parser.get_cross_compile() + self.cross_compile = self.parser.get_cross_compile() cmd = list(self.make_cmd) - if cross_compile: - cmd.append('CROSS_COMPILE=%s' % cross_compile) + if self.cross_compile: + cmd.append('CROSS_COMPILE=%s' % self.cross_compile) cmd.append('include/autoconf.mk') - self.ps = subprocess.Popen(cmd, stdout=self.devnull) + self.ps = subprocess.Popen(cmd, stdout=self.devnull, stderr=PIPE) self.state = STATE_SILENTOLDCONFIG return False
@@ -363,6 +377,7 @@ class Slots:
def move_config(config_attr, jobs=1): check_top_directory() + print 'Moving %s (type: %s, default: %s, no_spl: %s) ... (jobs: %d)' % ( config_attr['config'], config_attr['type'], @@ -379,6 +394,10 @@ def move_config(config_attr, jobs=1): continue defconfigs.append(os.path.join(dirpath, filename))
+ """Clean up any previous log of failed moves""" + if os.path.exists('moveconfig.failed'): + os.remove('moveconfig.failed') + slots = Slots(config_attr, jobs)
# Main loop to process defconfig files: @@ -396,6 +415,12 @@ def move_config(config_attr, jobs=1):
cleanup_headers(config_attr['config'])
+ if os.path.exists('moveconfig.failed'): + print '!!! Some boards were not processed; move the config manually.' + print '!!! The list of failed boards are saved in moveconfig.failed' + print + print open('moveconfig.failed', 'r').read() + def main(): try: cpu_count = multiprocessing.cpu_count()
The main image autoconf.mk certainly had better exist. I'm not sure about SPL and TPL, but at least this one. Check for this and error if missing.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com ---
Changes in v2: None
tools/moveconfig.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index 315f4be..2e5f2ce 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -184,6 +184,8 @@ class KconfigParser: 'autoconf.mk')
if not os.path.exists(autoconf): + if img == '.': + return True values.append('') continue
@@ -228,6 +230,7 @@ class KconfigParser: if prefixes[line] != '+': line = prefixes[line] + ':' + line f.write(line + '\n') + return False
class Slot:
@@ -308,7 +311,8 @@ class Slot: return True
if self.state == STATE_SILENTOLDCONFIG: - self.parser.update_defconfig(self.defconfig) + if self.parser.update_defconfig(self.defconfig): + self.defconfig_error('ERROR - autoconf.mk not found') self.state = STATE_IDLE return True
This will ensure that the order of the defconfig entries will always match that of the Kconfig files. After one slightly painful (but still early in the process) pass over all boards, this should keep the defconfigs clean from here on.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com ---
Changes in v2: None
tools/moveconfig.py | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index 2e5f2ce..5572be9 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -46,6 +46,7 @@ CROSS_COMPILE = { STATE_IDLE = 0 STATE_DEFCONFIG = 1 STATE_SILENTOLDCONFIG = 2 +STATE_SAVEDEFCONFIG = 3
### helper functions ### def get_devnull(): @@ -150,8 +151,8 @@ class KconfigParser: """ arch = '' cpu = '' - dotconfig = os.path.join(self.build_dir, '.config') - for line in open(dotconfig): + self.dotconfig = os.path.join(self.build_dir, '.config') + for line in open(self.dotconfig): m = self.re_arch.match(line) if m: arch = m.group(1) @@ -225,7 +226,7 @@ class KconfigParser:
print output.strip()
- with open(os.path.join('configs', defconfig), 'a') as f: + with open(os.path.join(self.dotconfig), 'a') as f: for line in output_lines: if prefixes[line] != '+': line = prefixes[line] + ':' + line @@ -313,6 +314,23 @@ class Slot: if self.state == STATE_SILENTOLDCONFIG: if self.parser.update_defconfig(self.defconfig): self.defconfig_error('ERROR - autoconf.mk not found') + self.state = STATE_IDLE + return True + + """Save off the defconfig in a consistent way""" + cmd = list(self.make_cmd) + cmd.append('savedefconfig') + self.ps = subprocess.Popen(cmd, stdout=self.devnull, + stderr=self.devnull) + self.state = STATE_SAVEDEFCONFIG + return False + + if self.state == STATE_SAVEDEFCONFIG: + defconfig_path = os.path.join(self.build_dir, 'defconfig') + if not os.path.exists(defconfig_path): + self.defconfig_error('ERROR - defconfig not updated') + shutil.move(defconfig_path, + os.path.join('configs', self.defconfig)) self.state = STATE_IDLE return True
This is helpful to re-attempt to move failed boards from a previous run without starting over.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com ---
Changes in v2: None
tools/moveconfig.py | 31 ++++++++++++++++++------------- 1 file changed, 18 insertions(+), 13 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index 5572be9..ad9dbac 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -397,7 +397,7 @@ class Slots: ret = False return ret
-def move_config(config_attr, jobs=1): +def move_config(config_attr, options): check_top_directory()
print 'Moving %s (type: %s, default: %s, no_spl: %s) ... (jobs: %d)' % ( @@ -405,22 +405,25 @@ def move_config(config_attr, jobs=1): config_attr['type'], config_attr['default'], config_attr['no_spl_support'], - jobs) - - # All the defconfig files to be processed - defconfigs = [] - for (dirpath, dirnames, filenames) in os.walk('configs'): - dirpath = dirpath[len('configs') + 1:] - for filename in fnmatch.filter(filenames, '*_defconfig'): - if fnmatch.fnmatch(filename, '.*'): - continue - defconfigs.append(os.path.join(dirpath, filename)) + options.jobs) + + if options.defconfigs: + defconfigs = [line.strip() for line in open(options.defconfigs, 'r')] + else: + # All the defconfig files to be processed + defconfigs = [] + for (dirpath, dirnames, filenames) in os.walk('configs'): + dirpath = dirpath[len('configs') + 1:] + for filename in fnmatch.filter(filenames, '*_defconfig'): + if fnmatch.fnmatch(filename, '.*'): + continue + defconfigs.append(os.path.join(dirpath, filename))
"""Clean up any previous log of failed moves""" if os.path.exists('moveconfig.failed'): os.remove('moveconfig.failed')
- slots = Slots(config_attr, jobs) + slots = Slots(config_attr, options.jobs)
# Main loop to process defconfig files: # Add a new subprocess into a vacant slot. @@ -453,6 +456,8 @@ def main(): # Add options here parser.add_option('-j', '--jobs', type='int', default=cpu_count, help='the number of jobs to run simultaneously') + parser.add_option('-d', '--defconfigs', type='string', + help='a file containing a list of defconfigs to move') parser.usage += ' config type default no_spl_support' (options, args) = parser.parse_args()
@@ -481,7 +486,7 @@ def main(): if not config_attr['config'].startswith('CONFIG_'): config_attr['config'] = 'CONFIG_' + config_attr['config']
- move_config(config_attr, options.jobs) + move_config(config_attr, options)
if __name__ == '__main__': main()
In some cases the build for the autoconf breaks. This outputs the errors following the status so that action can be taken without building again manually.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com ---
Changes in v2: None
tools/moveconfig.py | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index ad9dbac..b5c3157 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -242,7 +242,7 @@ class Slot: for faster processing. """
- def __init__(self, config_attr, devnull, make_cmd): + def __init__(self, config_attr, devnull, make_cmd, options): """Create a new slot.
Arguments: @@ -251,6 +251,7 @@ class Slot: self.build_dir = tempfile.mkdtemp() self.devnull = devnull self.make_cmd = (make_cmd, 'O=' + self.build_dir) + self.options = options self.parser = KconfigParser(self.build_dir, config_attr) self.state = STATE_IDLE
@@ -308,6 +309,8 @@ class Slot: if errout.find('gcc: command not found') != -1: errmsg = 'ERROR - compiler not found (%s)' % self.cross_compile self.defconfig_error(errmsg) + if self.options.verbose: + print errout self.state = STATE_IDLE return True
@@ -347,7 +350,7 @@ class Slots:
"""Controller of the array of subprocess slots."""
- def __init__(self, config_attr, jobs): + def __init__(self, config_attr, options): """Create a new slots controller.
Arguments: @@ -356,8 +359,8 @@ class Slots: self.slots = [] devnull = get_devnull() make_cmd = get_make_cmd() - for i in range(jobs): - self.slots.append(Slot(config_attr, devnull, make_cmd)) + for i in range(options.jobs): + self.slots.append(Slot(config_attr, devnull, make_cmd, options))
def add(self, defconfig): """Add a new subprocess if a vacant slot is available. @@ -423,7 +426,7 @@ def move_config(config_attr, options): if os.path.exists('moveconfig.failed'): os.remove('moveconfig.failed')
- slots = Slots(config_attr, options.jobs) + slots = Slots(config_attr, options)
# Main loop to process defconfig files: # Add a new subprocess into a vacant slot. @@ -458,6 +461,9 @@ def main(): help='the number of jobs to run simultaneously') parser.add_option('-d', '--defconfigs', type='string', help='a file containing a list of defconfigs to move') + parser.add_option('-v', '--verbose', dest='verbose', + action='store_true', default=False, + help='show any build errors as boards are build') parser.usage += ' config type default no_spl_support' (options, args) = parser.parse_args()
The existing target won't actually build the needed .mk file that is expected by the rest of the script. It seems that silentoldconfig does not actually cause this to be built any longer.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com --- This patch is based on: "Don't apply: tools: add a tool to move automatically CONFIGs from headers to defconfigs" from the list.
Also dropped the patch that adds an arc cross-tool mapping.
Changes in v3: None Changes in v2: None
tools/moveconfig.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index cb7f7ea..30dc4f6 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -301,7 +301,7 @@ class Slot: cmd = list(self.make_cmd) if cross_compile: cmd.append('CROSS_COMPILE=%s' % cross_compile) - cmd.append('silentoldconfig') + cmd.append('include/autoconf.mk') self.ps = subprocess.Popen(cmd, stdout=self.devnull) self.state = STATE_SILENTOLDCONFIG return False
Some compilers are hard to come by or have so few boards they are not worth messing with for this tool. Provide a list that need manual intervention and continue moving the bulk of boards.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
---
Changes in v3: None Changes in v2: -Print which compiler is missing
tools/moveconfig.py | 35 ++++++++++++++++++++++++++++++----- 1 file changed, 30 insertions(+), 5 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index 30dc4f6..d9ae859 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -17,6 +17,7 @@ import os import re import shutil import subprocess +from subprocess import PIPE import sys import tempfile import time @@ -276,6 +277,13 @@ class Slot: self.state = STATE_DEFCONFIG return True
+ def defconfig_error(self, errmsg): + output = self.defconfig[:-len('_defconfig')].ljust(37) + ': ' + print output + errmsg + + """Save a list of targets that have to be checked by hand""" + open('moveconfig.failed', 'a+').write("%s\n" % self.defconfig) + def poll(self): """Check if the subprocess is running and invoke the .config parser if the subprocess is terminated. @@ -290,19 +298,25 @@ class Slot: return False
if self.ps.poll() != 0: - sys.exit("failed to process '%s'" % self.defconfig) + errmsg = 'ERROR - build error' + errout = self.ps.stderr.read() + if errout.find('gcc: command not found') != -1: + errmsg = 'ERROR - compiler not found (%s)' % self.cross_compile + self.defconfig_error(errmsg) + self.state = STATE_IDLE + return True
if self.state == STATE_SILENTOLDCONFIG: self.parser.update_defconfig(self.defconfig) self.state = STATE_IDLE return True
- cross_compile = self.parser.get_cross_compile() + self.cross_compile = self.parser.get_cross_compile() cmd = list(self.make_cmd) - if cross_compile: - cmd.append('CROSS_COMPILE=%s' % cross_compile) + if self.cross_compile: + cmd.append('CROSS_COMPILE=%s' % self.cross_compile) cmd.append('include/autoconf.mk') - self.ps = subprocess.Popen(cmd, stdout=self.devnull) + self.ps = subprocess.Popen(cmd, stdout=self.devnull, stderr=PIPE) self.state = STATE_SILENTOLDCONFIG return False
@@ -362,6 +376,7 @@ class Slots:
def move_config(config_attr, jobs=1): check_top_directory() + print 'Moving %s (type: %s, default: %s, no_spl: %s) ... (jobs: %d)' % ( config_attr['config'], config_attr['type'], @@ -378,6 +393,10 @@ def move_config(config_attr, jobs=1): continue defconfigs.append(os.path.join(dirpath, filename))
+ """Clean up any previous log of failed moves""" + if os.path.exists('moveconfig.failed'): + os.remove('moveconfig.failed') + slots = Slots(config_attr, jobs)
# Main loop to process defconfig files: @@ -395,6 +414,12 @@ def move_config(config_attr, jobs=1):
cleanup_headers(config_attr['config'])
+ if os.path.exists('moveconfig.failed'): + print '!!! Some boards were not processed; move the config manually.' + print '!!! The list of failed boards are saved in moveconfig.failed' + print + print open('moveconfig.failed', 'r').read() + def main(): try: cpu_count = multiprocessing.cpu_count()
2015-05-14 7:28 GMT+09:00 Joe Hershberger joe.hershberger@ni.com:
Some compilers are hard to come by or have so few boards they are not worth messing with for this tool. Provide a list that need manual intervention and continue moving the bulk of boards.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
Changes in v3: None Changes in v2: -Print which compiler is missing
tools/moveconfig.py | 35 ++++++++++++++++++++++++++++++----- 1 file changed, 30 insertions(+), 5 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index 30dc4f6..d9ae859 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -17,6 +17,7 @@ import os import re import shutil import subprocess +from subprocess import PIPE import sys import tempfile import time @@ -276,6 +277,13 @@ class Slot: self.state = STATE_DEFCONFIG return True
- def defconfig_error(self, errmsg):
output = self.defconfig[:-len('_defconfig')].ljust(37) + ': 'print output + errmsg"""Save a list of targets that have to be checked by hand"""open('moveconfig.failed', 'a+').write("%s\n" % self.defconfig)- def poll(self): """Check if the subprocess is running and invoke the .config parser if the subprocess is terminated.
@@ -290,19 +298,25 @@ class Slot: return False
if self.ps.poll() != 0:
sys.exit("failed to process '%s'" % self.defconfig)
errmsg = 'ERROR - build error'errout = self.ps.stderr.read()if errout.find('gcc: command not found') != -1:errmsg = 'ERROR - compiler not found (%s)' % self.cross_compileself.defconfig_error(errmsg)self.state = STATE_IDLEreturn True if self.state == STATE_SILENTOLDCONFIG: self.parser.update_defconfig(self.defconfig) self.state = STATE_IDLE return True
cross_compile = self.parser.get_cross_compile()
self.cross_compile = self.parser.get_cross_compile() cmd = list(self.make_cmd)
if cross_compile:cmd.append('CROSS_COMPILE=%s' % cross_compile)
if self.cross_compile:cmd.append('CROSS_COMPILE=%s' % self.cross_compile) cmd.append('include/autoconf.mk')
self.ps = subprocess.Popen(cmd, stdout=self.devnull)
self.ps = subprocess.Popen(cmd, stdout=self.devnull, stderr=PIPE) self.state = STATE_SILENTOLDCONFIG return False@@ -362,6 +376,7 @@ class Slots:
def move_config(config_attr, jobs=1): check_top_directory()
- print 'Moving %s (type: %s, default: %s, no_spl: %s) ... (jobs: %d)' % ( config_attr['config'], config_attr['type'],
@@ -378,6 +393,10 @@ def move_config(config_attr, jobs=1): continue defconfigs.append(os.path.join(dirpath, filename))
"""Clean up any previous log of failed moves"""
if os.path.exists('moveconfig.failed'):
os.remove('moveconfig.failed')slots = Slots(config_attr, jobs)
# Main loop to process defconfig files:
@@ -395,6 +414,12 @@ def move_config(config_attr, jobs=1):
cleanup_headers(config_attr['config'])
- if os.path.exists('moveconfig.failed'):
print '!!! Some boards were not processed; move the config manually.'print '!!! The list of failed boards are saved in moveconfig.failed'print open('moveconfig.failed', 'r').read()def main(): try: cpu_count = multiprocessing.cpu_count()
I could successfully convert all the defconfigs with my toolchains, but I thought this feature would be helpful.
I merged this feature in my new version with your signed-off-by although the implementation is a bit different.
The main image autoconf.mk certainly had better exist or this tool won't function at all. Check for this and error if missing.
The SPL and TPL don't even exist as separate .config files any more.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com ---
Changes in v3: None Changes in v2: None
tools/moveconfig.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index d9ae859..b4ee0e3 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -183,6 +183,8 @@ class KconfigParser: 'autoconf.mk')
if not os.path.exists(autoconf): + if img == '.': + return True values.append('') continue
@@ -227,6 +229,7 @@ class KconfigParser: if prefixes[line] != '+': line = prefixes[line] + ':' + line f.write(line + '\n') + return False
class Slot:
@@ -307,7 +310,8 @@ class Slot: return True
if self.state == STATE_SILENTOLDCONFIG: - self.parser.update_defconfig(self.defconfig) + if self.parser.update_defconfig(self.defconfig): + self.defconfig_error('ERROR - autoconf.mk not found') self.state = STATE_IDLE return True
This will ensure that the order of the defconfig entries will always match that of the Kconfig files. After one slightly painful (but still early in the process) pass over all boards, this should keep the defconfigs clean from here on.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com ---
Changes in v3: None Changes in v2: None
tools/moveconfig.py | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index b4ee0e3..f2651a9 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -45,6 +45,7 @@ CROSS_COMPILE = { STATE_IDLE = 0 STATE_DEFCONFIG = 1 STATE_SILENTOLDCONFIG = 2 +STATE_SAVEDEFCONFIG = 3
### helper functions ### def get_devnull(): @@ -149,8 +150,8 @@ class KconfigParser: """ arch = '' cpu = '' - dotconfig = os.path.join(self.build_dir, '.config') - for line in open(dotconfig): + self.dotconfig = os.path.join(self.build_dir, '.config') + for line in open(self.dotconfig): m = self.re_arch.match(line) if m: arch = m.group(1) @@ -224,7 +225,7 @@ class KconfigParser:
print output.strip()
- with open(os.path.join('configs', defconfig), 'a') as f: + with open(os.path.join(self.dotconfig), 'a') as f: for line in output_lines: if prefixes[line] != '+': line = prefixes[line] + ':' + line @@ -312,6 +313,23 @@ class Slot: if self.state == STATE_SILENTOLDCONFIG: if self.parser.update_defconfig(self.defconfig): self.defconfig_error('ERROR - autoconf.mk not found') + self.state = STATE_IDLE + return True + + """Save off the defconfig in a consistent way""" + cmd = list(self.make_cmd) + cmd.append('savedefconfig') + self.ps = subprocess.Popen(cmd, stdout=self.devnull, + stderr=self.devnull) + self.state = STATE_SAVEDEFCONFIG + return False + + if self.state == STATE_SAVEDEFCONFIG: + defconfig_path = os.path.join(self.build_dir, 'defconfig') + if not os.path.exists(defconfig_path): + self.defconfig_error('ERROR - defconfig not updated') + shutil.move(defconfig_path, + os.path.join('configs', self.defconfig)) self.state = STATE_IDLE return True
2015-05-14 7:28 GMT+09:00 Joe Hershberger joe.hershberger@ni.com:
This will ensure that the order of the defconfig entries will always match that of the Kconfig files. After one slightly painful (but still early in the process) pass over all boards, this should keep the defconfigs clean from here on.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
I a bit hesitate to always enable this feature because savedefconfig might make git-diff noisier.
Is it possible to make it optional? -s, --savedefconfig or --no-savedefconfig?
Hi Masahiro-san.
On Thu, May 14, 2015 at 10:15 AM, Masahiro Yamada yamada.masahiro@socionext.com wrote:
2015-05-14 7:28 GMT+09:00 Joe Hershberger joe.hershberger@ni.com:
This will ensure that the order of the defconfig entries will always match that of the Kconfig files. After one slightly painful (but still early in the process) pass over all boards, this should keep the defconfigs clean from here on.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
I a bit hesitate to always enable this feature because savedefconfig might make git-diff noisier.
That should never be the case. Its consistent use should always lead to the least noisy diff.
Is it possible to make it optional? -s, --savedefconfig or --no-savedefconfig?
It is possible, but I recommend against it. Its inconsistent use would lead to noisy diffs.
Cheers, -Joe
This is helpful to re-attempt to move failed boards from a previous run without starting over.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
---
Changes in v3: -Fixed command line options order (alphabetize)
Changes in v2: -New for version 2
tools/moveconfig.py | 31 ++++++++++++++++++------------- 1 file changed, 18 insertions(+), 13 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index f2651a9..2688ad1 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -396,7 +396,7 @@ class Slots: ret = False return ret
-def move_config(config_attr, jobs=1): +def move_config(config_attr, options): check_top_directory()
print 'Moving %s (type: %s, default: %s, no_spl: %s) ... (jobs: %d)' % ( @@ -404,22 +404,25 @@ def move_config(config_attr, jobs=1): config_attr['type'], config_attr['default'], config_attr['no_spl_support'], - jobs) - - # All the defconfig files to be processed - defconfigs = [] - for (dirpath, dirnames, filenames) in os.walk('configs'): - dirpath = dirpath[len('configs') + 1:] - for filename in fnmatch.filter(filenames, '*_defconfig'): - if fnmatch.fnmatch(filename, '.*'): - continue - defconfigs.append(os.path.join(dirpath, filename)) + options.jobs) + + if options.defconfigs: + defconfigs = [line.strip() for line in open(options.defconfigs, 'r')] + else: + # All the defconfig files to be processed + defconfigs = [] + for (dirpath, dirnames, filenames) in os.walk('configs'): + dirpath = dirpath[len('configs') + 1:] + for filename in fnmatch.filter(filenames, '*_defconfig'): + if fnmatch.fnmatch(filename, '.*'): + continue + defconfigs.append(os.path.join(dirpath, filename))
"""Clean up any previous log of failed moves""" if os.path.exists('moveconfig.failed'): os.remove('moveconfig.failed')
- slots = Slots(config_attr, jobs) + slots = Slots(config_attr, options.jobs)
# Main loop to process defconfig files: # Add a new subprocess into a vacant slot. @@ -450,6 +453,8 @@ def main():
parser = optparse.OptionParser() # Add options here + parser.add_option('-d', '--defconfigs', type='string', + help='a file containing a list of defconfigs to move') parser.add_option('-j', '--jobs', type='int', default=cpu_count, help='the number of jobs to run simultaneously') parser.usage += ' config type default no_spl_support' @@ -480,7 +485,7 @@ def main(): if not config_attr['config'].startswith('CONFIG_'): config_attr['config'] = 'CONFIG_' + config_attr['config']
- move_config(config_attr, options.jobs) + move_config(config_attr, options)
if __name__ == '__main__': main()
2015-05-14 7:28 GMT+09:00 Joe Hershberger joe.hershberger@ni.com:
This is helpful to re-attempt to move failed boards from a previous run without starting over.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
I like this idea, but I have just submitted a new version.
Would you mind rebasing this patch onto mine, or shall I do it?
In some cases the build for the autoconf breaks. This outputs the errors following the status so that action can be taken without building again manually.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
---
Changes in v3: None Changes in v2: -New for version 2
tools/moveconfig.py | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index 2688ad1..9135e1d 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -241,7 +241,7 @@ class Slot: for faster processing. """
- def __init__(self, config_attr, devnull, make_cmd): + def __init__(self, config_attr, devnull, make_cmd, options): """Create a new slot.
Arguments: @@ -250,6 +250,7 @@ class Slot: self.build_dir = tempfile.mkdtemp() self.devnull = devnull self.make_cmd = (make_cmd, 'O=' + self.build_dir) + self.options = options self.parser = KconfigParser(self.build_dir, config_attr) self.state = STATE_IDLE
@@ -307,6 +308,8 @@ class Slot: if errout.find('gcc: command not found') != -1: errmsg = 'ERROR - compiler not found (%s)' % self.cross_compile self.defconfig_error(errmsg) + if self.options.verbose: + print errout self.state = STATE_IDLE return True
@@ -346,7 +349,7 @@ class Slots:
"""Controller of the array of subprocess slots."""
- def __init__(self, config_attr, jobs): + def __init__(self, config_attr, options): """Create a new slots controller.
Arguments: @@ -355,8 +358,8 @@ class Slots: self.slots = [] devnull = get_devnull() make_cmd = get_make_cmd() - for i in range(jobs): - self.slots.append(Slot(config_attr, devnull, make_cmd)) + for i in range(options.jobs): + self.slots.append(Slot(config_attr, devnull, make_cmd, options))
def add(self, defconfig): """Add a new subprocess if a vacant slot is available. @@ -422,7 +425,7 @@ def move_config(config_attr, options): if os.path.exists('moveconfig.failed'): os.remove('moveconfig.failed')
- slots = Slots(config_attr, options.jobs) + slots = Slots(config_attr, options)
# Main loop to process defconfig files: # Add a new subprocess into a vacant slot. @@ -457,6 +460,9 @@ def main(): help='a file containing a list of defconfigs to move') parser.add_option('-j', '--jobs', type='int', default=cpu_count, help='the number of jobs to run simultaneously') + parser.add_option('-v', '--verbose', dest='verbose', + action='store_true', default=False, + help='show any build errors as boards are build') parser.usage += ' config type default no_spl_support' (options, args) = parser.parse_args()
When moving configs, it is important to know what was defined in the config header even if it duplicates the configs coming from Kconfig.
tools/moveconfig.py now defines KCONFIG_IGNORE_DUPLICATES to prevent the filtering from happening and selecting wrong values for the defconfig.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
---
Changes in v3: -New for version 3
Changes in v2: None
scripts/Makefile.autoconf | 3 ++- tools/moveconfig.py | 1 + 2 files changed, 3 insertions(+), 1 deletion(-)
diff --git a/scripts/Makefile.autoconf b/scripts/Makefile.autoconf index f054081..36bfa17 100644 --- a/scripts/Makefile.autoconf +++ b/scripts/Makefile.autoconf @@ -58,7 +58,8 @@ quiet_cmd_autoconf = GEN $@ $(CPP) $(c_flags) $2 -DDO_DEPS_ONLY -dM $(srctree)/include/common.h > $@.tmp && { \ sed -n -f $(srctree)/tools/scripts/define2mk.sed $@.tmp | \ while read line; do \ - if ! grep -q "$${line%=*}=" include/config/auto.conf; then \ + if [ -n "${KCONFIG_IGNORE_DUPLICATES}" ] || \ + ! grep -q "$${line%=*}=" include/config/auto.conf; then \ echo "$$line"; \ fi \ done > $@; \ diff --git a/tools/moveconfig.py b/tools/moveconfig.py index 9135e1d..97ff597 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -340,6 +340,7 @@ class Slot: cmd = list(self.make_cmd) if self.cross_compile: cmd.append('CROSS_COMPILE=%s' % self.cross_compile) + cmd.append('KCONFIG_IGNORE_DUPLICATES=1') cmd.append('include/autoconf.mk') self.ps = subprocess.Popen(cmd, stdout=self.devnull, stderr=PIPE) self.state = STATE_SILENTOLDCONFIG
2015-05-14 7:28 GMT+09:00 Joe Hershberger joe.hershberger@ni.com:
When moving configs, it is important to know what was defined in the config header even if it duplicates the configs coming from Kconfig.
tools/moveconfig.py now defines KCONFIG_IGNORE_DUPLICATES to prevent the filtering from happening and selecting wrong values for the defconfig.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
Changes in v3: -New for version 3
Changes in v2: None
scripts/Makefile.autoconf | 3 ++- tools/moveconfig.py | 1 + 2 files changed, 3 insertions(+), 1 deletion(-)
diff --git a/scripts/Makefile.autoconf b/scripts/Makefile.autoconf index f054081..36bfa17 100644 --- a/scripts/Makefile.autoconf +++ b/scripts/Makefile.autoconf @@ -58,7 +58,8 @@ quiet_cmd_autoconf = GEN $@ $(CPP) $(c_flags) $2 -DDO_DEPS_ONLY -dM $(srctree)/include/common.h > $@.tmp && { \ sed -n -f $(srctree)/tools/scripts/define2mk.sed $@.tmp | \ while read line; do \
if ! grep -q "$${line%=*}=" include/config/auto.conf; then \
if [ -n "${KCONFIG_IGNORE_DUPLICATES}" ] || \! grep -q "$${line%=*}=" include/config/auto.conf; then \ echo "$$line"; \ fi \ done > $@; \diff --git a/tools/moveconfig.py b/tools/moveconfig.py index 9135e1d..97ff597 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -340,6 +340,7 @@ class Slot: cmd = list(self.make_cmd) if self.cross_compile: cmd.append('CROSS_COMPILE=%s' % self.cross_compile)
cmd.append('KCONFIG_IGNORE_DUPLICATES=1') cmd.append('include/autoconf.mk') self.ps = subprocess.Popen(cmd, stdout=self.devnull, stderr=PIPE) self.state = STATE_SILENTOLDCONFIG
This patch is not clear to me.
Isn't it the same concept as u-boot.cfg instroduced by commit 741e58e0fc8.
Hi Masahiro-san,
On Thu, May 14, 2015 at 10:36 AM, Masahiro Yamada yamada.masahiro@socionext.com wrote:
2015-05-14 7:28 GMT+09:00 Joe Hershberger joe.hershberger@ni.com:
When moving configs, it is important to know what was defined in the config header even if it duplicates the configs coming from Kconfig.
tools/moveconfig.py now defines KCONFIG_IGNORE_DUPLICATES to prevent the filtering from happening and selecting wrong values for the defconfig.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
Changes in v3: -New for version 3
Changes in v2: None
scripts/Makefile.autoconf | 3 ++- tools/moveconfig.py | 1 + 2 files changed, 3 insertions(+), 1 deletion(-)
diff --git a/scripts/Makefile.autoconf b/scripts/Makefile.autoconf index f054081..36bfa17 100644 --- a/scripts/Makefile.autoconf +++ b/scripts/Makefile.autoconf @@ -58,7 +58,8 @@ quiet_cmd_autoconf = GEN $@ $(CPP) $(c_flags) $2 -DDO_DEPS_ONLY -dM $(srctree)/include/common.h > $@.tmp && { \ sed -n -f $(srctree)/tools/scripts/define2mk.sed $@.tmp | \ while read line; do \
if ! grep -q "$${line%=*}=" include/config/auto.conf; then \
if [ -n "${KCONFIG_IGNORE_DUPLICATES}" ] || \! grep -q "$${line%=*}=" include/config/auto.conf; then \ echo "$$line"; \ fi \ done > $@; \diff --git a/tools/moveconfig.py b/tools/moveconfig.py index 9135e1d..97ff597 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -340,6 +340,7 @@ class Slot: cmd = list(self.make_cmd) if self.cross_compile: cmd.append('CROSS_COMPILE=%s' % self.cross_compile)
cmd.append('KCONFIG_IGNORE_DUPLICATES=1') cmd.append('include/autoconf.mk') self.ps = subprocess.Popen(cmd, stdout=self.devnull, stderr=PIPE) self.state = STATE_SILENTOLDCONFIGThis patch is not clear to me.
This was specifically a problem when moving a config that defaulted to true in the Kconfig. That meant that it would be "duplicate removed" from the include since it was already true (by default) in the Kconfig. So it had the effect of disabling that config in every board. This stops the filtering in the case where we are trying to move configs instead of trying to build the board in this mixed config system environment.
Isn't it the same concept as u-boot.cfg instroduced by commit 741e58e0fc8.
I don't believe it is the same thing because this will still include the configs enabled through the config header, but (I think) not those from Kconfig. And anyway, we are evaluating autoconf.mk, not u-boot.cfg.
Cheers, -Joe
Hi Joe,
2015-05-15 3:02 GMT+09:00 Joe Hershberger joe.hershberger@gmail.com:
Hi Masahiro-san,
On Thu, May 14, 2015 at 10:36 AM, Masahiro Yamada yamada.masahiro@socionext.com wrote:
2015-05-14 7:28 GMT+09:00 Joe Hershberger joe.hershberger@ni.com:
When moving configs, it is important to know what was defined in the config header even if it duplicates the configs coming from Kconfig.
tools/moveconfig.py now defines KCONFIG_IGNORE_DUPLICATES to prevent the filtering from happening and selecting wrong values for the defconfig.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
Changes in v3: -New for version 3
Changes in v2: None
scripts/Makefile.autoconf | 3 ++- tools/moveconfig.py | 1 + 2 files changed, 3 insertions(+), 1 deletion(-)
diff --git a/scripts/Makefile.autoconf b/scripts/Makefile.autoconf index f054081..36bfa17 100644 --- a/scripts/Makefile.autoconf +++ b/scripts/Makefile.autoconf @@ -58,7 +58,8 @@ quiet_cmd_autoconf = GEN $@ $(CPP) $(c_flags) $2 -DDO_DEPS_ONLY -dM $(srctree)/include/common.h > $@.tmp && { \ sed -n -f $(srctree)/tools/scripts/define2mk.sed $@.tmp | \ while read line; do \
if ! grep -q "$${line%=*}=" include/config/auto.conf; then \
if [ -n "${KCONFIG_IGNORE_DUPLICATES}" ] || \! grep -q "$${line%=*}=" include/config/auto.conf; then \ echo "$$line"; \ fi \ done > $@; \diff --git a/tools/moveconfig.py b/tools/moveconfig.py index 9135e1d..97ff597 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -340,6 +340,7 @@ class Slot: cmd = list(self.make_cmd) if self.cross_compile: cmd.append('CROSS_COMPILE=%s' % self.cross_compile)
cmd.append('KCONFIG_IGNORE_DUPLICATES=1') cmd.append('include/autoconf.mk') self.ps = subprocess.Popen(cmd, stdout=self.devnull, stderr=PIPE) self.state = STATE_SILENTOLDCONFIGThis patch is not clear to me.
This was specifically a problem when moving a config that defaulted to true in the Kconfig. That meant that it would be "duplicate removed" from the include since it was already true (by default) in the Kconfig. So it had the effect of disabling that config in every board. This stops the filtering in the case where we are trying to move configs instead of trying to build the board in this mixed config system environment.
Uh, make sense.
I was almost forgetting how the "don't apply" version was implemented.
My new one parses .config, defconfig as well as autoconf.mk, so please check if we still need to this hack.
Isn't it the same concept as u-boot.cfg instroduced by commit 741e58e0fc8.
I don't believe it is the same thing because this will still include the configs enabled through the config header, but (I think) not those from Kconfig. And anyway, we are evaluating autoconf.mk, not u-boot.cfg.
Now, I understood your intention.
u-boot.cfg is not useful for this tool, although u-boot.cfg also includes configs from Kconfig.
Hi Masahiro-san,
On Fri, May 15, 2015 at 12:10 AM, Masahiro Yamada yamada.masahiro@socionext.com wrote:
Hi Joe,
2015-05-15 3:02 GMT+09:00 Joe Hershberger joe.hershberger@gmail.com:
Hi Masahiro-san,
On Thu, May 14, 2015 at 10:36 AM, Masahiro Yamada yamada.masahiro@socionext.com wrote:
2015-05-14 7:28 GMT+09:00 Joe Hershberger joe.hershberger@ni.com:
When moving configs, it is important to know what was defined in the config header even if it duplicates the configs coming from Kconfig.
tools/moveconfig.py now defines KCONFIG_IGNORE_DUPLICATES to prevent the filtering from happening and selecting wrong values for the defconfig.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
Changes in v3: -New for version 3
Changes in v2: None
scripts/Makefile.autoconf | 3 ++- tools/moveconfig.py | 1 + 2 files changed, 3 insertions(+), 1 deletion(-)
diff --git a/scripts/Makefile.autoconf b/scripts/Makefile.autoconf index f054081..36bfa17 100644 --- a/scripts/Makefile.autoconf +++ b/scripts/Makefile.autoconf @@ -58,7 +58,8 @@ quiet_cmd_autoconf = GEN $@ $(CPP) $(c_flags) $2 -DDO_DEPS_ONLY -dM $(srctree)/include/common.h > $@.tmp && { \ sed -n -f $(srctree)/tools/scripts/define2mk.sed $@.tmp | \ while read line; do \
if ! grep -q "$${line%=*}=" include/config/auto.conf; then \
if [ -n "${KCONFIG_IGNORE_DUPLICATES}" ] || \! grep -q "$${line%=*}=" include/config/auto.conf; then \ echo "$$line"; \ fi \ done > $@; \diff --git a/tools/moveconfig.py b/tools/moveconfig.py index 9135e1d..97ff597 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -340,6 +340,7 @@ class Slot: cmd = list(self.make_cmd) if self.cross_compile: cmd.append('CROSS_COMPILE=%s' % self.cross_compile)
cmd.append('KCONFIG_IGNORE_DUPLICATES=1') cmd.append('include/autoconf.mk') self.ps = subprocess.Popen(cmd, stdout=self.devnull, stderr=PIPE) self.state = STATE_SILENTOLDCONFIGThis patch is not clear to me.
This was specifically a problem when moving a config that defaulted to true in the Kconfig. That meant that it would be "duplicate removed" from the include since it was already true (by default) in the Kconfig. So it had the effect of disabling that config in every board. This stops the filtering in the case where we are trying to move configs instead of trying to build the board in this mixed config system environment.
Uh, make sense.
I was almost forgetting how the "don't apply" version was implemented.
My new one parses .config, defconfig as well as autoconf.mk, so please check if we still need to this hack.
I checked it out. The same problem exists and the patch is still needed.
Also, I don't believe it is valid to check the .config for things to skip. Just because there is a mention of something does not mean it is the correct value. This is specifically the case for using savedefconfig. For savedefconfig to work properly, the user must add the entry to the Kconfig menu first, then run moveconfig.py. You seem to be expecting that users run moveconfig.py before editing the Kconfigs.
8<snip>8
Cheers, -Joe
Moving configs is a fairly slow process since each board config must pass through a compiler to evaluate the configs. Also, many configs are related in a way that makes sense to atomically move.
Add support to tools/moveconfig.py to read multiple lines from the .moveconfig file and create a parser for each. After compiling the configs, simply run all the parsers on the autoconf.mk to find the state of each of those configs.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
---
Changes in v3: -New for version 3
Changes in v2: None
tools/moveconfig.py | 75 ++++++++++++++++++++++++++++++----------------------- 1 file changed, 42 insertions(+), 33 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index 97ff597..798f717 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -91,9 +91,9 @@ def cleanup_header(header_path, patterns): if not i in matched: f.write(line)
-def cleanup_headers(config): +def cleanup_headers(config_attrs): while True: - choice = raw_input('Clean up %s in headers? [y/n]: ' % config).lower() + choice = raw_input('Clean up headers? [y/n]: ').lower() print choice if choice == 'y' or choice == 'n': break @@ -102,8 +102,9 @@ def cleanup_headers(config): return
patterns = [] - patterns.append(re.compile(r'#\s*define\s+%s\W' % config)) - patterns.append(re.compile(r'#\s*undef\s+%s\W' % config)) + for config_attr in config_attrs: + patterns.append(re.compile(r'#\s*define\s+%s\W' % config_attr['config'])) + patterns.append(re.compile(r'#\s*undef\s+%s\W' % config_attr['config']))
for (dirpath, dirnames, filenames) in os.walk('include'): for filename in filenames: @@ -126,7 +127,7 @@ class KconfigParser: PREFIX = { '.': '+', 'spl': 'S', 'tpl': 'T' }
def __init__(self, build_dir, config_attr): - """Create a new .config perser. + """Create a new .config parser.
Arguments: build_dir: Build directory where .config is located @@ -141,6 +142,7 @@ class KconfigParser: else: self.no_spl_support = False self.re = re.compile(r'%s=(.*)' % self.config) + self.dotconfig = os.path.join(self.build_dir, '.config')
def get_cross_compile(self): """Parse .config file and return CROSS_COMPILE @@ -150,7 +152,6 @@ class KconfigParser: """ arch = '' cpu = '' - self.dotconfig = os.path.join(self.build_dir, '.config') for line in open(self.dotconfig): m = self.re_arch.match(line) if m: @@ -204,7 +205,6 @@ class KconfigParser: value = '(default)'
values.append(value) - os.remove(autoconf)
if value == '(undef)' or value == '(default)': continue @@ -220,6 +220,7 @@ class KconfigParser: prefixes[output_line] = self.PREFIX[img]
output = defconfig[:-len('_defconfig')].ljust(37) + ': ' + output += self.config.replace('CONFIG_','').ljust(18) + ': ' for value in values: output += value.ljust(12)
@@ -241,7 +242,7 @@ class Slot: for faster processing. """
- def __init__(self, config_attr, devnull, make_cmd, options): + def __init__(self, config_attrs, devnull, make_cmd, options): """Create a new slot.
Arguments: @@ -251,7 +252,9 @@ class Slot: self.devnull = devnull self.make_cmd = (make_cmd, 'O=' + self.build_dir) self.options = options - self.parser = KconfigParser(self.build_dir, config_attr) + self.parsers = [] + for config_attr in config_attrs: + self.parsers.append(KconfigParser(self.build_dir, config_attr)) self.state = STATE_IDLE
def __del__(self): @@ -314,10 +317,11 @@ class Slot: return True
if self.state == STATE_SILENTOLDCONFIG: - if self.parser.update_defconfig(self.defconfig): - self.defconfig_error('ERROR - autoconf.mk not found') - self.state = STATE_IDLE - return True + for parser in self.parsers: + if parser.update_defconfig(self.defconfig): + self.defconfig_error('ERROR - autoconf.mk not found') + self.state = STATE_IDLE + return True
"""Save off the defconfig in a consistent way""" cmd = list(self.make_cmd) @@ -336,7 +340,7 @@ class Slot: self.state = STATE_IDLE return True
- self.cross_compile = self.parser.get_cross_compile() + self.cross_compile = self.parsers[0].get_cross_compile() cmd = list(self.make_cmd) if self.cross_compile: cmd.append('CROSS_COMPILE=%s' % self.cross_compile) @@ -350,7 +354,7 @@ class Slots:
"""Controller of the array of subprocess slots."""
- def __init__(self, config_attr, options): + def __init__(self, config_attrs, options): """Create a new slots controller.
Arguments: @@ -360,7 +364,7 @@ class Slots: devnull = get_devnull() make_cmd = get_make_cmd() for i in range(options.jobs): - self.slots.append(Slot(config_attr, devnull, make_cmd, options)) + self.slots.append(Slot(config_attrs, devnull, make_cmd, options))
def add(self, defconfig): """Add a new subprocess if a vacant slot is available. @@ -400,15 +404,16 @@ class Slots: ret = False return ret
-def move_config(config_attr, options): +def move_config(config_attrs, options): check_top_directory()
- print 'Moving %s (type: %s, default: %s, no_spl: %s) ... (jobs: %d)' % ( - config_attr['config'], - config_attr['type'], - config_attr['default'], - config_attr['no_spl_support'], - options.jobs) + for config_attr in config_attrs: + print 'Moving %s (type: %s, default: %s, no_spl: %s)' % ( + config_attr['config'], + config_attr['type'], + config_attr['default'], + config_attr['no_spl_support']) + print '%d jobs...' % options.jobs
if options.defconfigs: defconfigs = [line.strip() for line in open(options.defconfigs, 'r')] @@ -426,7 +431,7 @@ def move_config(config_attr, options): if os.path.exists('moveconfig.failed'): os.remove('moveconfig.failed')
- slots = Slots(config_attr, options) + slots = Slots(config_attrs, options)
# Main loop to process defconfig files: # Add a new subprocess into a vacant slot. @@ -441,7 +446,7 @@ def move_config(config_attr, options): while not slots.empty(): time.sleep(SLEEP_TIME)
- cleanup_headers(config_attr['config']) + cleanup_headers(config_attrs)
if os.path.exists('moveconfig.failed'): print '!!! Some boards were not processed; move the config manually.' @@ -468,31 +473,35 @@ def main(): (options, args) = parser.parse_args()
args_key = ('config', 'type', 'default', 'no_spl_support') - config_attr = {} + config_attrs = []
if len(args) >= len(args_key): saved_attr = '' for i, key in enumerate(args_key): - config_attr[key] = args[i] + config_attrs.append({}) + config_attrs[0][key] = args[i] saved_attr = saved_attr + ' %s' % args[i] with open('.moveconfig', 'w') as f: f.write(saved_attr) elif os.path.exists('.moveconfig'): f = open('.moveconfig') try: - saved_attr = f.readline().split() - for i, key in enumerate(args_key): - config_attr[key] = saved_attr[i] + for j, line in enumerate(f): + config_attrs.append({}) + saved_attr = line.split() + for i, key in enumerate(args_key): + config_attrs[j][key] = saved_attr[i] except: sys.exit('%s: broken format' % '.moveconfig') else: parser.print_usage() sys.exit(1)
- if not config_attr['config'].startswith('CONFIG_'): - config_attr['config'] = 'CONFIG_' + config_attr['config'] + for config_attr in config_attrs: + if not config_attr['config'].startswith('CONFIG_'): + config_attr['config'] = 'CONFIG_' + config_attr['config']
- move_config(config_attr, options) + move_config(config_attrs, options)
if __name__ == '__main__': main()
Hi Joe,
2015-05-14 7:28 GMT+09:00 Joe Hershberger joe.hershberger@ni.com:
Moving configs is a fairly slow process since each board config must pass through a compiler to evaluate the configs. Also, many configs are related in a way that makes sense to atomically move.
Add support to tools/moveconfig.py to read multiple lines from the .moveconfig file and create a parser for each. After compiling the configs, simply run all the parsers on the autoconf.mk to find the state of each of those configs.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
Sorry for leaving this scripts for such a long time. (I have been busy in the Linux side lately.) causing two people to develope a similar feature.
I thought the same thing. My idea was to stop giving config attributes from the command arguments. Instead, the name of the input file is given as the argument.
Hi Masahiro-san,
On Thu, May 14, 2015 at 9:37 AM, Masahiro Yamada yamada.masahiro@socionext.com wrote:
Hi Joe,
2015-05-14 7:28 GMT+09:00 Joe Hershberger joe.hershberger@ni.com:
Moving configs is a fairly slow process since each board config must pass through a compiler to evaluate the configs. Also, many configs are related in a way that makes sense to atomically move.
Add support to tools/moveconfig.py to read multiple lines from the .moveconfig file and create a parser for each. After compiling the configs, simply run all the parsers on the autoconf.mk to find the state of each of those configs.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
Sorry for leaving this scripts for such a long time. (I have been busy in the Linux side lately.) causing two people to develope a similar feature.
Not too big a deal. Hopefully this will be in the tree soon so we can stop managing this in every branch.
I thought the same thing. My idea was to stop giving config attributes from the command arguments. Instead, the name of the input file is given as the argument.
That's fine. I was just trying to maintain your original features.
Cheers, -Joe
This gives a basic idea about progress.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
---
Changes in v3: -New for version 3
Changes in v2: None
tools/moveconfig.py | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index 798f717..bb087d4 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -93,7 +93,7 @@ def cleanup_header(header_path, patterns):
def cleanup_headers(config_attrs): while True: - choice = raw_input('Clean up headers? [y/n]: ').lower() + choice = raw_input('\nClean up headers? [y/n]: ').lower() print choice if choice == 'y' or choice == 'n': break @@ -264,7 +264,7 @@ class Slot: pass shutil.rmtree(self.build_dir)
- def add(self, defconfig): + def add(self, defconfig, num, total): """Add a new subprocess to the slot.
Fails if the slot is occupied, that is, the current subprocess @@ -283,6 +283,8 @@ class Slot: self.ps = subprocess.Popen(cmd, stdout=self.devnull) self.defconfig = defconfig self.state = STATE_DEFCONFIG + self.num = num + self.total = total return True
def defconfig_error(self, errmsg): @@ -323,6 +325,9 @@ class Slot: self.state = STATE_IDLE return True
+ print ' %d defconfigs out of %d\r' % (self.num + 1, self.total), + sys.stdout.flush() + """Save off the defconfig in a consistent way""" cmd = list(self.make_cmd) cmd.append('savedefconfig') @@ -366,7 +371,7 @@ class Slots: for i in range(options.jobs): self.slots.append(Slot(config_attrs, devnull, make_cmd, options))
- def add(self, defconfig): + def add(self, defconfig, num, total): """Add a new subprocess if a vacant slot is available.
Arguments: @@ -376,7 +381,7 @@ class Slots: Return True on success or False on fail """ for slot in self.slots: - if slot.add(defconfig): + if slot.add(defconfig, num, total): return True return False
@@ -436,8 +441,8 @@ def move_config(config_attrs, options): # Main loop to process defconfig files: # Add a new subprocess into a vacant slot. # Sleep if there is no available slot. - for defconfig in defconfigs: - while not slots.add(defconfig): + for i, defconfig in enumerate(defconfigs): + while not slots.add(defconfig, i, len(defconfigs)): while not slots.available(): # No available slot: sleep for a while time.sleep(SLEEP_TIME)
In some case you may want to only cleanup the headers. Make it possible without waiting for all boards to compile.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
---
Changes in v3: -New for version 3
Changes in v2: None
tools/moveconfig.py | 83 ++++++++++++++++++++++++++++------------------------- 1 file changed, 44 insertions(+), 39 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index bb087d4..d63f47f 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -412,48 +412,50 @@ class Slots: def move_config(config_attrs, options): check_top_directory()
- for config_attr in config_attrs: - print 'Moving %s (type: %s, default: %s, no_spl: %s)' % ( - config_attr['config'], - config_attr['type'], - config_attr['default'], - config_attr['no_spl_support']) - print '%d jobs...' % options.jobs - - if options.defconfigs: - defconfigs = [line.strip() for line in open(options.defconfigs, 'r')] - else: - # All the defconfig files to be processed - defconfigs = [] - for (dirpath, dirnames, filenames) in os.walk('configs'): - dirpath = dirpath[len('configs') + 1:] - for filename in fnmatch.filter(filenames, '*_defconfig'): - if fnmatch.fnmatch(filename, '.*'): - continue - defconfigs.append(os.path.join(dirpath, filename)) - - """Clean up any previous log of failed moves""" - if os.path.exists('moveconfig.failed'): - os.remove('moveconfig.failed') - - slots = Slots(config_attrs, options) - - # Main loop to process defconfig files: - # Add a new subprocess into a vacant slot. - # Sleep if there is no available slot. - for i, defconfig in enumerate(defconfigs): - while not slots.add(defconfig, i, len(defconfigs)): - while not slots.available(): - # No available slot: sleep for a while - time.sleep(SLEEP_TIME) - - # wait until all the subprocesses finish - while not slots.empty(): - time.sleep(SLEEP_TIME) + if not options.clean_only: + for config_attr in config_attrs: + print 'Moving %s (type: %s, default: %s, no_spl: %s)' % ( + config_attr['config'], + config_attr['type'], + config_attr['default'], + config_attr['no_spl_support']) + print '%d jobs...' % options.jobs + + if options.defconfigs: + defconfigs = [line.strip() for line in + open(options.defconfigs, 'r')] + else: + # All the defconfig files to be processed + defconfigs = [] + for (dirpath, dirnames, filenames) in os.walk('configs'): + dirpath = dirpath[len('configs') + 1:] + for filename in fnmatch.filter(filenames, '*_defconfig'): + if fnmatch.fnmatch(filename, '.*'): + continue + defconfigs.append(os.path.join(dirpath, filename)) + + """Clean up any previous log of failed moves""" + if os.path.exists('moveconfig.failed'): + os.remove('moveconfig.failed') + + slots = Slots(config_attrs, options) + + # Main loop to process defconfig files: + # Add a new subprocess into a vacant slot. + # Sleep if there is no available slot. + for i, defconfig in enumerate(defconfigs): + while not slots.add(defconfig, i, len(defconfigs)): + while not slots.available(): + # No available slot: sleep for a while + time.sleep(SLEEP_TIME) + + # wait until all the subprocesses finish + while not slots.empty(): + time.sleep(SLEEP_TIME)
cleanup_headers(config_attrs)
- if os.path.exists('moveconfig.failed'): + if (not options.clean_only) & os.path.exists('moveconfig.failed'): print '!!! Some boards were not processed; move the config manually.' print '!!! The list of failed boards are saved in moveconfig.failed' print @@ -467,6 +469,9 @@ def main():
parser = optparse.OptionParser() # Add options here + parser.add_option('-c', '--clean-only', dest='clean_only', + action='store_true', default=False, + help='only clean the headers') parser.add_option('-d', '--defconfigs', type='string', help='a file containing a list of defconfigs to move') parser.add_option('-j', '--jobs', type='int', default=cpu_count,
2015-05-14 7:28 GMT+09:00 Joe Hershberger joe.hershberger@ni.com:
In some case you may want to only cleanup the headers. Make it possible without waiting for all boards to compile.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
Changes in v3: -New for version 3
Changes in v2: None
tools/moveconfig.py | 83 ++++++++++++++++++++++++++++------------------------- 1 file changed, 44 insertions(+), 39 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index bb087d4..d63f47f 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -412,48 +412,50 @@ class Slots: def move_config(config_attrs, options): check_top_directory()
- for config_attr in config_attrs:
print 'Moving %s (type: %s, default: %s, no_spl: %s)' % (config_attr['config'],config_attr['type'],config_attr['default'],config_attr['no_spl_support'])- print '%d jobs...' % options.jobs
- if options.defconfigs:
defconfigs = [line.strip() for line in open(options.defconfigs, 'r')]- else:
# All the defconfig files to be processeddefconfigs = []for (dirpath, dirnames, filenames) in os.walk('configs'):dirpath = dirpath[len('configs') + 1:]for filename in fnmatch.filter(filenames, '*_defconfig'):if fnmatch.fnmatch(filename, '.*'):continuedefconfigs.append(os.path.join(dirpath, filename))- """Clean up any previous log of failed moves"""
- if os.path.exists('moveconfig.failed'):
os.remove('moveconfig.failed')- slots = Slots(config_attrs, options)
- # Main loop to process defconfig files:
- # Add a new subprocess into a vacant slot.
- # Sleep if there is no available slot.
- for i, defconfig in enumerate(defconfigs):
while not slots.add(defconfig, i, len(defconfigs)):while not slots.available():# No available slot: sleep for a whiletime.sleep(SLEEP_TIME)- # wait until all the subprocesses finish
- while not slots.empty():
time.sleep(SLEEP_TIME)
if not options.clean_only:
for config_attr in config_attrs:print 'Moving %s (type: %s, default: %s, no_spl: %s)' % (config_attr['config'],config_attr['type'],config_attr['default'],config_attr['no_spl_support'])print '%d jobs...' % options.jobsif options.defconfigs:defconfigs = [line.strip() for line inopen(options.defconfigs, 'r')]else:# All the defconfig files to be processeddefconfigs = []for (dirpath, dirnames, filenames) in os.walk('configs'):dirpath = dirpath[len('configs') + 1:]for filename in fnmatch.filter(filenames, '*_defconfig'):if fnmatch.fnmatch(filename, '.*'):continuedefconfigs.append(os.path.join(dirpath, filename))"""Clean up any previous log of failed moves"""if os.path.exists('moveconfig.failed'):os.remove('moveconfig.failed')slots = Slots(config_attrs, options)# Main loop to process defconfig files:# Add a new subprocess into a vacant slot.# Sleep if there is no available slot.for i, defconfig in enumerate(defconfigs):while not slots.add(defconfig, i, len(defconfigs)):while not slots.available():# No available slot: sleep for a whiletime.sleep(SLEEP_TIME)# wait until all the subprocesses finishwhile not slots.empty():time.sleep(SLEEP_TIME)cleanup_headers(config_attrs)
- if os.path.exists('moveconfig.failed'):
- if (not options.clean_only) & os.path.exists('moveconfig.failed'): print '!!! Some boards were not processed; move the config manually.' print '!!! The list of failed boards are saved in moveconfig.failed' print
I am OK with this feature, but this 'if' statement ranges over many code lines.
Perhaps, moving the cleanup_headers to the main function is simpler?
def main(): [snip]
if not options.clean_only: move_config(config_attrs, options)
cleanup_headers(config_attrs)
Hi Masahiro-san,
On Thu, May 14, 2015 at 9:51 AM, Masahiro Yamada yamada.masahiro@socionext.com wrote:
2015-05-14 7:28 GMT+09:00 Joe Hershberger joe.hershberger@ni.com:
In some case you may want to only cleanup the headers. Make it possible without waiting for all boards to compile.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
Changes in v3: -New for version 3
Changes in v2: None
tools/moveconfig.py | 83 ++++++++++++++++++++++++++++------------------------- 1 file changed, 44 insertions(+), 39 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index bb087d4..d63f47f 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -412,48 +412,50 @@ class Slots: def move_config(config_attrs, options): check_top_directory()
- for config_attr in config_attrs:
print 'Moving %s (type: %s, default: %s, no_spl: %s)' % (config_attr['config'],config_attr['type'],config_attr['default'],config_attr['no_spl_support'])- print '%d jobs...' % options.jobs
- if options.defconfigs:
defconfigs = [line.strip() for line in open(options.defconfigs, 'r')]- else:
# All the defconfig files to be processeddefconfigs = []for (dirpath, dirnames, filenames) in os.walk('configs'):dirpath = dirpath[len('configs') + 1:]for filename in fnmatch.filter(filenames, '*_defconfig'):if fnmatch.fnmatch(filename, '.*'):continuedefconfigs.append(os.path.join(dirpath, filename))- """Clean up any previous log of failed moves"""
- if os.path.exists('moveconfig.failed'):
os.remove('moveconfig.failed')- slots = Slots(config_attrs, options)
- # Main loop to process defconfig files:
- # Add a new subprocess into a vacant slot.
- # Sleep if there is no available slot.
- for i, defconfig in enumerate(defconfigs):
while not slots.add(defconfig, i, len(defconfigs)):while not slots.available():# No available slot: sleep for a whiletime.sleep(SLEEP_TIME)- # wait until all the subprocesses finish
- while not slots.empty():
time.sleep(SLEEP_TIME)
if not options.clean_only:
for config_attr in config_attrs:print 'Moving %s (type: %s, default: %s, no_spl: %s)' % (config_attr['config'],config_attr['type'],config_attr['default'],config_attr['no_spl_support'])print '%d jobs...' % options.jobsif options.defconfigs:defconfigs = [line.strip() for line inopen(options.defconfigs, 'r')]else:# All the defconfig files to be processeddefconfigs = []for (dirpath, dirnames, filenames) in os.walk('configs'):dirpath = dirpath[len('configs') + 1:]for filename in fnmatch.filter(filenames, '*_defconfig'):if fnmatch.fnmatch(filename, '.*'):continuedefconfigs.append(os.path.join(dirpath, filename))"""Clean up any previous log of failed moves"""if os.path.exists('moveconfig.failed'):os.remove('moveconfig.failed')slots = Slots(config_attrs, options)# Main loop to process defconfig files:# Add a new subprocess into a vacant slot.# Sleep if there is no available slot.for i, defconfig in enumerate(defconfigs):while not slots.add(defconfig, i, len(defconfigs)):while not slots.available():# No available slot: sleep for a whiletime.sleep(SLEEP_TIME)# wait until all the subprocesses finishwhile not slots.empty():time.sleep(SLEEP_TIME)cleanup_headers(config_attrs)
- if os.path.exists('moveconfig.failed'):
- if (not options.clean_only) & os.path.exists('moveconfig.failed'): print '!!! Some boards were not processed; move the config manually.' print '!!! The list of failed boards are saved in moveconfig.failed' print
I am OK with this feature, but this 'if' statement ranges over many code lines.
Perhaps, moving the cleanup_headers to the main function is simpler?
def main(): [snip]
if not options.clean_only: move_config(config_attrs, options) cleanup_headers(config_attrs)
OK.
-Joe
2015-05-14 7:28 GMT+09:00 Joe Hershberger joe.hershberger@ni.com:
The existing target won't actually build the needed .mk file that is expected by the rest of the script. It seems that silentoldconfig does not actually cause this to be built any longer.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
This patch is based on: "Don't apply: tools: add a tool to move automatically CONFIGs from headers to defconfigs" from the list.
Also dropped the patch that adds an arc cross-tool mapping.
Changes in v3: None Changes in v2: None
tools/moveconfig.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index cb7f7ea..30dc4f6 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -301,7 +301,7 @@ class Slot: cmd = list(self.make_cmd) if cross_compile: cmd.append('CROSS_COMPILE=%s' % cross_compile)
cmd.append('silentoldconfig')
cmd.append('include/autoconf.mk') self.ps = subprocess.Popen(cmd, stdout=self.devnull) self.state = STATE_SILENTOLDCONFIG return False
Yup, the 'Don't apply' version is not working since I removed scripts/multiconfig.sh.
I had locally modified this tool though I tweaked a bit differently.
This will ensure that the order of the defconfig entries will always match that of the Kconfig files. After one slightly painful (but still early in the process) pass over all boards, this should keep the defconfigs clean from here on.
Users must edit the Kconfig first to add the menu entries and then run moveconfig.py to update the defconfig files and the include configs.
As such, moveconfig.py cannot compare against the '.config' contents.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com --- This is based on https://patchwork.ozlabs.org/patch/472591/
Changes in v4: -Rebased series on Masahiro's v2
Changes in v3: None Changes in v2: None
tools/moveconfig.py | 35 ++++++++++++++++++++++++++--------- 1 file changed, 26 insertions(+), 9 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index c39ea95..544f6af 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -46,6 +46,9 @@ should look like this: CONFIG_CMD_USB bool n CONFIG_SYS_TEXT_BASE hex 0x00000000
+Next you must edit the Kconfig to add the menu entries for the configs +you are moving. + And then run this tool giving the file name of the recipe
$ tools/moveconfig.py recipe @@ -192,6 +195,7 @@ CROSS_COMPILE = { STATE_IDLE = 0 STATE_DEFCONFIG = 1 STATE_AUTOCONF = 2 +STATE_SAVEDEFCONFIG = 3
ACTION_MOVE = 0 ACTION_DEFAULT_VALUE = 1 @@ -390,8 +394,7 @@ class KconfigParser:
return CROSS_COMPILE.get(arch, '')
- def parse_one_config(self, config_attr, defconfig_lines, - dotconfig_lines, autoconf_lines): + def parse_one_config(self, config_attr, defconfig_lines, autoconf_lines): """Parse .config, defconfig, include/autoconf.mk for one config.
This function looks for the config options in the lines from @@ -402,7 +405,6 @@ class KconfigParser: config_attr: A dictionary including the name, the type, and the default value of the target config. defconfig_lines: lines from the original defconfig file. - dotconfig_lines: lines from the .config file. autoconf_lines: lines from the include/autoconf.mk file.
Returns: @@ -418,7 +420,7 @@ class KconfigParser: else: default = config + '=' + config_attr['default']
- for line in defconfig_lines + dotconfig_lines: + for line in defconfig_lines: line = line.rstrip() if line.startswith(config + '=') or line == not_set: return (ACTION_ALREADY_EXIST, line) @@ -463,15 +465,12 @@ class KconfigParser: with open(defconfig_path) as f: defconfig_lines = f.readlines()
- with open(dotconfig_path) as f: - dotconfig_lines = f.readlines() - with open(autoconf_path) as f: autoconf_lines = f.readlines()
for config_attr in self.config_attrs: result = self.parse_one_config(config_attr, defconfig_lines, - dotconfig_lines, autoconf_lines) + autoconf_lines) results.append(result)
log = '' @@ -499,7 +498,7 @@ class KconfigParser: print log,
if not self.options.dry_run: - with open(defconfig_path, 'a') as f: + with open(dotconfig_path, 'a') as f: for (action, value) in results: if action == ACTION_MOVE: f.write(value + '\n') @@ -608,6 +607,24 @@ class Slot:
if self.state == STATE_AUTOCONF: self.parser.update_defconfig(self.defconfig) + + """Save off the defconfig in a consistent way""" + cmd = list(self.make_cmd) + cmd.append('savedefconfig') + self.ps = subprocess.Popen(cmd, stdout=self.devnull, + stderr=self.devnull) + self.state = STATE_SAVEDEFCONFIG + return False + + if self.state == STATE_SAVEDEFCONFIG: + defconfig_path = os.path.join(self.build_dir, 'defconfig') + if not os.path.exists(defconfig_path): + print >> sys.stderr, log_msg(self.options.color, + COLOR_LIGHT_RED, + self.defconfig, + 'The defconfig was not updated') + shutil.move(defconfig_path, + os.path.join('configs', self.defconfig)) self.state = STATE_IDLE return True
When moving configs, it is important to know what was defined in the config header even if it duplicates the configs coming from Kconfig.
This is specifically needed for the case where a config is set to default 'y' in the Kconfig. This would previously cause the actual value from the include config to be filtered out, and moveconfig.py would think that it was 'n'... This means that the value that should be 'y' is now (in every defconfig) set to 'not set'.
tools/moveconfig.py now defines KCONFIG_IGNORE_DUPLICATES to prevent the filtering from happening and selecting wrong values for the defconfig.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
---
Changes in v4: None Changes in v3: -New for version 3
Changes in v2: None
scripts/Makefile.autoconf | 3 ++- tools/moveconfig.py | 1 + 2 files changed, 3 insertions(+), 1 deletion(-)
diff --git a/scripts/Makefile.autoconf b/scripts/Makefile.autoconf index f054081..36bfa17 100644 --- a/scripts/Makefile.autoconf +++ b/scripts/Makefile.autoconf @@ -58,7 +58,8 @@ quiet_cmd_autoconf = GEN $@ $(CPP) $(c_flags) $2 -DDO_DEPS_ONLY -dM $(srctree)/include/common.h > $@.tmp && { \ sed -n -f $(srctree)/tools/scripts/define2mk.sed $@.tmp | \ while read line; do \ - if ! grep -q "$${line%=*}=" include/config/auto.conf; then \ + if [ -n "${KCONFIG_IGNORE_DUPLICATES}" ] || \ + ! grep -q "$${line%=*}=" include/config/auto.conf; then \ echo "$$line"; \ fi \ done > $@; \ diff --git a/tools/moveconfig.py b/tools/moveconfig.py index 544f6af..d3009de 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -632,6 +632,7 @@ class Slot: cmd = list(self.make_cmd) if cross_compile: cmd.append('CROSS_COMPILE=%s' % cross_compile) + cmd.append('KCONFIG_IGNORE_DUPLICATES=1') cmd.append('include/config/auto.conf') self.ps = subprocess.Popen(cmd, stdout=self.devnull) self.state = STATE_AUTOCONF
2015-05-16 6:40 GMT+09:00 Joe Hershberger joe.hershberger@ni.com:
When moving configs, it is important to know what was defined in the config header even if it duplicates the configs coming from Kconfig.
This is specifically needed for the case where a config is set to default 'y' in the Kconfig. This would previously cause the actual value from the include config to be filtered out, and moveconfig.py would think that it was 'n'... This means that the value that should be 'y' is now (in every defconfig) set to 'not set'.
tools/moveconfig.py now defines KCONFIG_IGNORE_DUPLICATES to prevent the filtering from happening and selecting wrong values for the defconfig.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
Changes in v4: None Changes in v3: -New for version 3
Changes in v2: None
scripts/Makefile.autoconf | 3 ++- tools/moveconfig.py | 1 + 2 files changed, 3 insertions(+), 1 deletion(-)
diff --git a/scripts/Makefile.autoconf b/scripts/Makefile.autoconf index f054081..36bfa17 100644 --- a/scripts/Makefile.autoconf +++ b/scripts/Makefile.autoconf @@ -58,7 +58,8 @@ quiet_cmd_autoconf = GEN $@ $(CPP) $(c_flags) $2 -DDO_DEPS_ONLY -dM $(srctree)/include/common.h > $@.tmp && { \ sed -n -f $(srctree)/tools/scripts/define2mk.sed $@.tmp | \ while read line; do \
if ! grep -q "$${line%=*}=" include/config/auto.conf; then \
if [ -n "${KCONFIG_IGNORE_DUPLICATES}" ] || \! grep -q "$${line%=*}=" include/config/auto.conf; then \ echo "$$line"; \ fi \ done > $@; \diff --git a/tools/moveconfig.py b/tools/moveconfig.py index 544f6af..d3009de 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -632,6 +632,7 @@ class Slot: cmd = list(self.make_cmd) if cross_compile: cmd.append('CROSS_COMPILE=%s' % cross_compile)
cmd.append('KCONFIG_IGNORE_DUPLICATES=1') cmd.append('include/config/auto.conf') self.ps = subprocess.Popen(cmd, stdout=self.devnull) self.state = STATE_AUTOCONF--
Now, I fully understood your intention. This comes from our work-flow differences.
Because I edited the Kconfig after moving configs that are default to y, so I did not notice this necessity.
Anyway, we should do it beforehand for savedefconfig.
So, this looks good to me.
Acked-by: Masahiro Yamada yamada.masahiro@socionext.com
This is helpful to re-attempt to move failed boards from a previous run without starting over.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
---
Changes in v4: None Changes in v3: -Fixed command line options order (alphabetize)
Changes in v2: -New for version 2
tools/moveconfig.py | 26 ++++++++++++++++++++------ 1 file changed, 20 insertions(+), 6 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index d3009de..3f31e81 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -135,6 +135,9 @@ Available options Surround each portion of the log with escape sequences to display it in color on the terminal.
+ -d, --defconfigs + Specify a file containing a list of defconfigs to move + -n, --dry-run Peform a trial run that does not make any changes. It is useful to see what is going to happen before one actually runs it. @@ -734,12 +737,21 @@ def move_config(config_attrs, options): config_attr['type'], config_attr['default'])
- # All the defconfig files to be processed - defconfigs = [] - for (dirpath, dirnames, filenames) in os.walk('configs'): - dirpath = dirpath[len('configs') + 1:] - for filename in fnmatch.filter(filenames, '*_defconfig'): - defconfigs.append(os.path.join(dirpath, filename)) + if options.defconfigs: + defconfigs = [line.strip() for line in open(options.defconfigs, 'r')] + for i, defconfig in enumerate(defconfigs): + if not defconfig.endswith('_defconfig'): + defconfigs[i] = defconfig + '_defconfig' + if not os.path.exists(os.path.join('configs', defconfigs[i])): + sys.exit('%s - defconfig does not exist. Stopping.' % + defconfigs[i]) + else: + # All the defconfig files to be processed + defconfigs = [] + for (dirpath, dirnames, filenames) in os.walk('configs'): + dirpath = dirpath[len('configs') + 1:] + for filename in fnmatch.filter(filenames, '*_defconfig'): + defconfigs.append(os.path.join(dirpath, filename))
slots = Slots(config_attrs, options)
@@ -840,6 +852,8 @@ def main(): # Add options here parser.add_option('-c', '--color', action='store_true', default=False, help='display the log in color') + parser.add_option('-d', '--defconfigs', type='string', + help='a file containing a list of defconfigs to move') parser.add_option('-n', '--dry-run', action='store_true', default=False, help='perform a trial run (show log with no changes)') parser.add_option('-e', '--exit-on-error', action='store_true',
2015-05-16 6:40 GMT+09:00 Joe Hershberger joe.hershberger@ni.com:
This is helpful to re-attempt to move failed boards from a previous run without starting over.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
Changes in v4: None Changes in v3: -Fixed command line options order (alphabetize)
Changes in v2: -New for version 2
tools/moveconfig.py | 26 ++++++++++++++++++++------ 1 file changed, 20 insertions(+), 6 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index d3009de..3f31e81 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -135,6 +135,9 @@ Available options Surround each portion of the log with escape sequences to display it in color on the terminal.
- -d, --defconfigs
- Specify a file containing a list of defconfigs to move
- -n, --dry-run Peform a trial run that does not make any changes. It is useful to see what is going to happen before one actually runs it.
@@ -734,12 +737,21 @@ def move_config(config_attrs, options): config_attr['type'], config_attr['default'])
- # All the defconfig files to be processed
- defconfigs = []
- for (dirpath, dirnames, filenames) in os.walk('configs'):
dirpath = dirpath[len('configs') + 1:]for filename in fnmatch.filter(filenames, '*_defconfig'):defconfigs.append(os.path.join(dirpath, filename))
- if options.defconfigs:
defconfigs = [line.strip() for line in open(options.defconfigs, 'r')]for i, defconfig in enumerate(defconfigs):if not defconfig.endswith('_defconfig'):defconfigs[i] = defconfig + '_defconfig'if not os.path.exists(os.path.join('configs', defconfigs[i])):sys.exit('%s - defconfig does not exist. Stopping.' %defconfigs[i])
'r' is redundant for open() because read-mode is default.
moveconfig.failed always prefixes _defconfig, so we need not to do so again, I think. (or will we make this file by hand?)
Then, we can drop enumarate and write the error message within 80 columns.
if options.defconfigs: defconfigs = [line.strip() for line in open(options.defconfigs)] for defconfig in defconfigs: if not os.path.exists(os.path.join('configs', defconfig)): sys.exit('%s: defconfig does not exist. Stopping.' % defconfig)
slots = Slots(config_attrs, options)@@ -840,6 +852,8 @@ def main(): # Add options here parser.add_option('-c', '--color', action='store_true', default=False, help='display the log in color')
- parser.add_option('-d', '--defconfigs', type='string',
help='a file containing a list of defconfigs to move')
Hi Masahiro-san,
On Mon, May 18, 2015 at 11:33 PM, Masahiro Yamada yamada.masahiro@socionext.com wrote:
2015-05-16 6:40 GMT+09:00 Joe Hershberger joe.hershberger@ni.com:
This is helpful to re-attempt to move failed boards from a previous run without starting over.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
Changes in v4: None Changes in v3: -Fixed command line options order (alphabetize)
Changes in v2: -New for version 2
tools/moveconfig.py | 26 ++++++++++++++++++++------ 1 file changed, 20 insertions(+), 6 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index d3009de..3f31e81 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -135,6 +135,9 @@ Available options Surround each portion of the log with escape sequences to display it in color on the terminal.
- -d, --defconfigs
- Specify a file containing a list of defconfigs to move
- -n, --dry-run Peform a trial run that does not make any changes. It is useful to see what is going to happen before one actually runs it.
@@ -734,12 +737,21 @@ def move_config(config_attrs, options): config_attr['type'], config_attr['default'])
- # All the defconfig files to be processed
- defconfigs = []
- for (dirpath, dirnames, filenames) in os.walk('configs'):
dirpath = dirpath[len('configs') + 1:]for filename in fnmatch.filter(filenames, '*_defconfig'):defconfigs.append(os.path.join(dirpath, filename))
- if options.defconfigs:
defconfigs = [line.strip() for line in open(options.defconfigs, 'r')]for i, defconfig in enumerate(defconfigs):if not defconfig.endswith('_defconfig'):defconfigs[i] = defconfig + '_defconfig'if not os.path.exists(os.path.join('configs', defconfigs[i])):sys.exit('%s - defconfig does not exist. Stopping.' %defconfigs[i])'r' is redundant for open() because read-mode is default.
OK.
moveconfig.failed always prefixes _defconfig, so we need not to do so again, I think. (or will we make this file by hand?)
I have done both moveconfig.failed as well as a hand-written file for testing certain boards. That's why I added both the append '_defconfig' as well as the check for it actually existing.
Then, we can drop enumarate and write the error message within 80 columns.
if options.defconfigs: defconfigs = [line.strip() for line in open(options.defconfigs)] for defconfig in defconfigs: if not os.path.exists(os.path.join('configs', defconfig)): sys.exit('%s: defconfig does not exist. Stopping.' % defconfig)
I think we should keep the check for endswith('_defconfig').
slots = Slots(config_attrs, options)@@ -840,6 +852,8 @@ def main(): # Add options here parser.add_option('-c', '--color', action='store_true', default=False, help='display the log in color')
- parser.add_option('-d', '--defconfigs', type='string',
help='a file containing a list of defconfigs to move')
Cheers, -Joe
2015-05-20 2:58 GMT+09:00 Joe Hershberger joe.hershberger@gmail.com:
Hi Masahiro-san,
On Mon, May 18, 2015 at 11:33 PM, Masahiro Yamada yamada.masahiro@socionext.com wrote:
2015-05-16 6:40 GMT+09:00 Joe Hershberger joe.hershberger@ni.com:
This is helpful to re-attempt to move failed boards from a previous run without starting over.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
Changes in v4: None Changes in v3: -Fixed command line options order (alphabetize)
Changes in v2: -New for version 2
tools/moveconfig.py | 26 ++++++++++++++++++++------ 1 file changed, 20 insertions(+), 6 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index d3009de..3f31e81 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -135,6 +135,9 @@ Available options Surround each portion of the log with escape sequences to display it in color on the terminal.
- -d, --defconfigs
- Specify a file containing a list of defconfigs to move
- -n, --dry-run Peform a trial run that does not make any changes. It is useful to see what is going to happen before one actually runs it.
@@ -734,12 +737,21 @@ def move_config(config_attrs, options): config_attr['type'], config_attr['default'])
- # All the defconfig files to be processed
- defconfigs = []
- for (dirpath, dirnames, filenames) in os.walk('configs'):
dirpath = dirpath[len('configs') + 1:]for filename in fnmatch.filter(filenames, '*_defconfig'):defconfigs.append(os.path.join(dirpath, filename))
- if options.defconfigs:
defconfigs = [line.strip() for line in open(options.defconfigs, 'r')]for i, defconfig in enumerate(defconfigs):if not defconfig.endswith('_defconfig'):defconfigs[i] = defconfig + '_defconfig'if not os.path.exists(os.path.join('configs', defconfigs[i])):sys.exit('%s - defconfig does not exist. Stopping.' %defconfigs[i])'r' is redundant for open() because read-mode is default.
OK.
moveconfig.failed always prefixes _defconfig, so we need not to do so again, I think. (or will we make this file by hand?)
I have done both moveconfig.failed as well as a hand-written file for testing certain boards. That's why I added both the append '_defconfig' as well as the check for it actually existing.
Then, we can drop enumarate and write the error message within 80 columns.
if options.defconfigs: defconfigs = [line.strip() for line in open(options.defconfigs)] for defconfig in defconfigs: if not os.path.exists(os.path.join('configs', defconfig)): sys.exit('%s: defconfig does not exist. Stopping.' % defconfig)I think we should keep the check for endswith('_defconfig').
OK, then. Let's keep it.
In some case you may want to only cleanup the headers. Make it possible without waiting for all boards to compile.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
---
Changes in v4: None Changes in v3: -New for version 3
Changes in v2: None
tools/moveconfig.py | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index 3f31e81..b6db058 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -146,6 +146,9 @@ Available options Exit immediately if Make exits with a non-zero status while processing a defconfig file.
+ -H, --headers-only + Only cleanup the headers; skip the defconfig processing + -j, --jobs Specify the number of threads to run simultaneously. If not specified, the number of threads is the same as the number of CPU cores. @@ -770,8 +773,6 @@ def move_config(config_attrs, options):
slots.show_failed_boards()
- cleanup_headers(config_attrs, options.dry_run) - def bad_recipe(filename, linenum, msg): """Print error message with the file name and the line number and exit.""" sys.exit("%s: line %d: error : " % (filename, linenum) + msg) @@ -859,6 +860,9 @@ def main(): parser.add_option('-e', '--exit-on-error', action='store_true', default=False, help='exit immediately on any error') + parser.add_option('-H', '--headers-only', dest='cleanup_headers_only', + action='store_true', default=False, + help='only cleanup the headers') parser.add_option('-j', '--jobs', type='int', default=cpu_count, help='the number of jobs to run simultaneously') parser.usage += ' recipe_file\n\n' + \ @@ -879,7 +883,10 @@ def main():
update_cross_compile()
- move_config(config_attrs, options) + if not options.cleanup_headers_only: + move_config(config_attrs, options) + + cleanup_headers(config_attrs, options.dry_run)
if __name__ == '__main__': main()
Hi Joe,
2015-05-16 6:40 GMT+09:00 Joe Hershberger joe.hershberger@ni.com:
In some case you may want to only cleanup the headers. Make it possible without waiting for all boards to compile.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
Changes in v4: None Changes in v3: -New for version 3
Changes in v2: None
tools/moveconfig.py | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index 3f31e81..b6db058 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -146,6 +146,9 @@ Available options Exit immediately if Make exits with a non-zero status while processing a defconfig file.
- -H, --headers-only
- Only cleanup the headers; skip the defconfig processing
- -j, --jobs Specify the number of threads to run simultaneously. If not specified, the number of threads is the same as the number of CPU cores.
@@ -770,8 +773,6 @@ def move_config(config_attrs, options):
slots.show_failed_boards()
- cleanup_headers(config_attrs, options.dry_run)
def bad_recipe(filename, linenum, msg): """Print error message with the file name and the line number and exit.""" sys.exit("%s: line %d: error : " % (filename, linenum) + msg) @@ -859,6 +860,9 @@ def main(): parser.add_option('-e', '--exit-on-error', action='store_true', default=False, help='exit immediately on any error')
- parser.add_option('-H', '--headers-only', dest='cleanup_headers_only',
action='store_true', default=False,help='only cleanup the headers')@@ -879,7 +883,10 @@ def main():
update_cross_compile()
- move_config(config_attrs, options)
- if not options.cleanup_headers_only:
move_config(config_attrs, options)- cleanup_headers(config_attrs, options.dry_run)
if __name__ == '__main__': main()
Could you also move the check_top_directory() call to main() function, above the move_config call.
We should make sure we are at the top directory also for cleaning headers.
Hi Masahiro-san,
On Mon, May 18, 2015 at 9:03 PM, Masahiro Yamada yamada.masahiro@socionext.com wrote:
Hi Joe,
2015-05-16 6:40 GMT+09:00 Joe Hershberger joe.hershberger@ni.com:
In some case you may want to only cleanup the headers. Make it possible without waiting for all boards to compile.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
Changes in v4: None Changes in v3: -New for version 3
Changes in v2: None
tools/moveconfig.py | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index 3f31e81..b6db058 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -146,6 +146,9 @@ Available options Exit immediately if Make exits with a non-zero status while processing a defconfig file.
- -H, --headers-only
- Only cleanup the headers; skip the defconfig processing
- -j, --jobs Specify the number of threads to run simultaneously. If not specified, the number of threads is the same as the number of CPU cores.
@@ -770,8 +773,6 @@ def move_config(config_attrs, options):
slots.show_failed_boards()
- cleanup_headers(config_attrs, options.dry_run)
def bad_recipe(filename, linenum, msg): """Print error message with the file name and the line number and exit.""" sys.exit("%s: line %d: error : " % (filename, linenum) + msg) @@ -859,6 +860,9 @@ def main(): parser.add_option('-e', '--exit-on-error', action='store_true', default=False, help='exit immediately on any error')
- parser.add_option('-H', '--headers-only', dest='cleanup_headers_only',
action='store_true', default=False,help='only cleanup the headers')@@ -879,7 +883,10 @@ def main():
update_cross_compile()
- move_config(config_attrs, options)
- if not options.cleanup_headers_only:
move_config(config_attrs, options)- cleanup_headers(config_attrs, options.dry_run)
if __name__ == '__main__': main()
Could you also move the check_top_directory() call to main() function, above the move_config call.
We should make sure we are at the top directory also for cleaning headers.
Yes... moving it.
-Joe
Some config.h files live in arch and board directories. They will need to be cleaned up as well, so run the same filters there.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
---
Changes in v4: -New for version 4
Changes in v3: None Changes in v2: None
tools/moveconfig.py | 10 ++++++++++ 1 file changed, 10 insertions(+)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index b6db058..25cee21 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -352,6 +352,16 @@ def cleanup_headers(config_attrs, dry_run): if not fnmatch.fnmatch(filename, '*~'): cleanup_one_header(os.path.join(dirpath, filename), patterns, dry_run) + for (dirpath, dirnames, filenames) in os.walk('arch'): + for filename in filenames: + if not fnmatch.fnmatch(filename, '*~'): + cleanup_one_header(os.path.join(dirpath, filename), patterns, + dry_run) + for (dirpath, dirnames, filenames) in os.walk('board'): + for filename in filenames: + if not fnmatch.fnmatch(filename, '*~'): + cleanup_one_header(os.path.join(dirpath, filename), patterns, + dry_run)
### classes ### class KconfigParser:
Hi Joe,
2015-05-16 6:40 GMT+09:00 Joe Hershberger joe.hershberger@ni.com:
Some config.h files live in arch and board directories. They will need to be cleaned up as well, so run the same filters there.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
Changes in v4: -New for version 4
Changes in v3: None Changes in v2: None
tools/moveconfig.py | 10 ++++++++++ 1 file changed, 10 insertions(+)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index b6db058..25cee21 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -352,6 +352,16 @@ def cleanup_headers(config_attrs, dry_run): if not fnmatch.fnmatch(filename, '*~'): cleanup_one_header(os.path.join(dirpath, filename), patterns, dry_run)
- for (dirpath, dirnames, filenames) in os.walk('arch'):
for filename in filenames:if not fnmatch.fnmatch(filename, '*~'):cleanup_one_header(os.path.join(dirpath, filename), patterns,dry_run)- for (dirpath, dirnames, filenames) in os.walk('board'):
for filename in filenames:if not fnmatch.fnmatch(filename, '*~'):cleanup_one_header(os.path.join(dirpath, filename), patterns,dry_run)
To reduce code duplication, can we write like this or something?
for dir in 'include', 'arch', 'board': for (dirpath, dirnames, filenames) in os.walk(dir): for filename in filenames: if not fnmatch.fnmatch(filename, '*~'): cleanup_one_header(os.path.join(dirpath, filename), patterns, dry_run)
On Mon, May 18, 2015 at 8:41 PM, Masahiro Yamada yamada.masahiro@socionext.com wrote:
Hi Joe,
2015-05-16 6:40 GMT+09:00 Joe Hershberger joe.hershberger@ni.com:
Some config.h files live in arch and board directories. They will need to be cleaned up as well, so run the same filters there.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
Changes in v4: -New for version 4
Changes in v3: None Changes in v2: None
tools/moveconfig.py | 10 ++++++++++ 1 file changed, 10 insertions(+)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index b6db058..25cee21 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -352,6 +352,16 @@ def cleanup_headers(config_attrs, dry_run): if not fnmatch.fnmatch(filename, '*~'): cleanup_one_header(os.path.join(dirpath, filename), patterns, dry_run)
- for (dirpath, dirnames, filenames) in os.walk('arch'):
for filename in filenames:if not fnmatch.fnmatch(filename, '*~'):cleanup_one_header(os.path.join(dirpath, filename), patterns,dry_run)- for (dirpath, dirnames, filenames) in os.walk('board'):
for filename in filenames:if not fnmatch.fnmatch(filename, '*~'):cleanup_one_header(os.path.join(dirpath, filename), patterns,dry_run)To reduce code duplication, can we write like this or something?
for dir in 'include', 'arch', 'board': for (dirpath, dirnames, filenames) in os.walk(dir): for filename in filenames: if not fnmatch.fnmatch(filename, '*~'): cleanup_one_header(os.path.join(dirpath, filename), patterns, dry_run)
OK.
This print seems to be redundant and unformatted compared to the next few lines, so remove it.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com ---
Changes in v4: None Changes in v3: None Changes in v2: None
tools/moveconfig.py | 1 - 1 file changed, 1 deletion(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index 25cee21..15b0f2b 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -725,7 +725,6 @@ class Slots: if len(failed_boards) > 0: msg = [ "The following boards were not processed due to error:" ] msg += failed_boards - print msg for line in msg: print >> sys.stderr, color_text(self.options.color, COLOR_LIGHT_RED, line)
Hi Joe
2015-05-16 6:40 GMT+09:00 Joe Hershberger joe.hershberger@ni.com:
This print seems to be redundant and unformatted compared to the next few lines, so remove it.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
Changes in v4: None Changes in v3: None Changes in v2: None
tools/moveconfig.py | 1 - 1 file changed, 1 deletion(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index 25cee21..15b0f2b 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -725,7 +725,6 @@ class Slots: if len(failed_boards) > 0: msg = [ "The following boards were not processed due to error:" ] msg += failed_boards
print msg for line in msg: print >> sys.stderr, color_text(self.options.color, COLOR_LIGHT_RED, line)--
Oops, my mistake.
Acked-by: Masahiro Yamada yamada.masahiro@socionext.com
(Or, because it is a simply bug fix, you can comment on my base patch.)
Hi Masahiro-san,
On Mon, May 18, 2015 at 9:10 PM, Masahiro Yamada yamada.masahiro@socionext.com wrote:
Hi Joe
2015-05-16 6:40 GMT+09:00 Joe Hershberger joe.hershberger@ni.com:
This print seems to be redundant and unformatted compared to the next few lines, so remove it.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
Changes in v4: None Changes in v3: None Changes in v2: None
tools/moveconfig.py | 1 - 1 file changed, 1 deletion(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index 25cee21..15b0f2b 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -725,7 +725,6 @@ class Slots: if len(failed_boards) > 0: msg = [ "The following boards were not processed due to error:" ] msg += failed_boards
print msg for line in msg: print >> sys.stderr, color_text(self.options.color, COLOR_LIGHT_RED, line)--
Oops, my mistake.
Acked-by: Masahiro Yamada yamada.masahiro@socionext.com
(Or, because it is a simply bug fix, you can comment on my base patch.)
I'm dropping this patch and sent a reply to your original patch (not applied to master yet).
-Joe
If boards fail, output that list to a file so that it can easily be passed back into moveconfig.py using the -d option.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com ---
Changes in v4: None Changes in v3: None Changes in v2: None
tools/moveconfig.py | 4 ++++ 1 file changed, 4 insertions(+)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index 15b0f2b..9e923da 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -728,6 +728,10 @@ class Slots: for line in msg: print >> sys.stderr, color_text(self.options.color, COLOR_LIGHT_RED, line) + ffail = open('moveconfig.failed', 'w') + for failed_board in failed_boards: + ffail.write("%s\n" % failed_board) + ffail.close()
def move_config(config_attrs, options): """Move config options to defconfig files.
2015-05-16 6:40 GMT+09:00 Joe Hershberger joe.hershberger@ni.com:
If boards fail, output that list to a file so that it can easily be passed back into moveconfig.py using the -d option.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
Changes in v4: None Changes in v3: None Changes in v2: None
tools/moveconfig.py | 4 ++++ 1 file changed, 4 insertions(+)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index 15b0f2b..9e923da 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -728,6 +728,10 @@ class Slots: for line in msg: print >> sys.stderr, color_text(self.options.color, COLOR_LIGHT_RED, line)
ffail = open('moveconfig.failed', 'w')for failed_board in failed_boards:ffail.write("%s\n" % failed_board)ffail.close()def move_config(config_attrs, options): """Move config options to defconfig files.
If you use with ... as ..., it will automatically close the file on exit.
with open('moveconfig.failed', 'w') as f: for board in failed_boards: f.write(board + '\n')
Hi Masahiro-san,
On Mon, May 18, 2015 at 10:12 PM, Masahiro Yamada yamada.masahiro@socionext.com wrote:
2015-05-16 6:40 GMT+09:00 Joe Hershberger joe.hershberger@ni.com:
If boards fail, output that list to a file so that it can easily be passed back into moveconfig.py using the -d option.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
Changes in v4: None Changes in v3: None Changes in v2: None
tools/moveconfig.py | 4 ++++ 1 file changed, 4 insertions(+)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index 15b0f2b..9e923da 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -728,6 +728,10 @@ class Slots: for line in msg: print >> sys.stderr, color_text(self.options.color, COLOR_LIGHT_RED, line)
ffail = open('moveconfig.failed', 'w')for failed_board in failed_boards:ffail.write("%s\n" % failed_board)ffail.close()def move_config(config_attrs, options): """Move config options to defconfig files.
If you use with ... as ..., it will automatically close the file on exit.
with open('moveconfig.failed', 'w') as f: for board in failed_boards: f.write(board + '\n')
OK.
-Joe
A common case for failed builds is a missing compiler. Print a message for that case to tell the user concisely which compiler was expected that was not found.
This patch also has the effect of not printing build errors any longer. The next patch will add a switch to optionally bring that back.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com ---
Changes in v4: None Changes in v3: None Changes in v2: None
tools/moveconfig.py | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index 9e923da..f986f55 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -166,6 +166,7 @@ import os import re import shutil import subprocess +from subprocess import PIPE import sys import tempfile import time @@ -606,11 +607,14 @@ class Slot: return False
if self.ps.poll() != 0: - + errmsg = 'Failed to process.' + errout = self.ps.stderr.read() + if errout.find('gcc: command not found') != -1: + errmsg = 'Compiler not found (%s)' % self.cross_compile print >> sys.stderr, log_msg(self.options.color, COLOR_LIGHT_RED, self.defconfig, - "failed to process.") + errmsg), if self.options.exit_on_error: sys.exit("Exit on error.") else: @@ -644,13 +648,13 @@ class Slot: self.state = STATE_IDLE return True
- cross_compile = self.parser.get_cross_compile() + self.cross_compile = self.parser.get_cross_compile() cmd = list(self.make_cmd) - if cross_compile: - cmd.append('CROSS_COMPILE=%s' % cross_compile) + if self.cross_compile: + cmd.append('CROSS_COMPILE=%s' % self.cross_compile) cmd.append('KCONFIG_IGNORE_DUPLICATES=1') cmd.append('include/config/auto.conf') - self.ps = subprocess.Popen(cmd, stdout=self.devnull) + self.ps = subprocess.Popen(cmd, stdout=self.devnull, stderr=PIPE) self.state = STATE_AUTOCONF return False
2015-05-16 6:40 GMT+09:00 Joe Hershberger joe.hershberger@ni.com:
A common case for failed builds is a missing compiler. Print a message for that case to tell the user concisely which compiler was expected that was not found.
This patch also has the effect of not printing build errors any longer. The next patch will add a switch to optionally bring that back.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
Changes in v4: None Changes in v3: None Changes in v2: None
tools/moveconfig.py | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index 9e923da..f986f55 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -166,6 +166,7 @@ import os import re import shutil import subprocess +from subprocess import PIPE
Personally I do not prefer from ... import because it disturbs the naming space.
Could you use subprocess.PIPE instead?
import sys import tempfile import time @@ -606,11 +607,14 @@ class Slot: return False
if self.ps.poll() != 0:
errmsg = 'Failed to process.'errout = self.ps.stderr.read()
This throws exception if "make *_defconfig" or "make savedefconfig" fail.
Traceback (most recent call last): File "tools/moveconfig.py", line 924, in <module> main() File "tools/moveconfig.py", line 919, in main move_config(config_attrs, options) File "tools/moveconfig.py", line 794, in move_config while not slots.available(): File "tools/moveconfig.py", line 717, in available if slot.poll(): File "tools/moveconfig.py", line 616, in poll errout = self.ps.stderr.read() AttributeError: 'NoneType' object has no attribute 'read'
Seems better to add PIPE for all the call of subprocess.Popen()
if errout.find('gcc: command not found') != -1:errmsg = 'Compiler not found (%s)' % self.cross_compile
If you do this, should the locale be changed?
Without LANG=C, "command not found" is displayed in Japanese on my computer.
If --verbose is given, we will be able to know the cause of erorr. "missing compiler" is a special case error?
print >> sys.stderr, log_msg(self.options.color, COLOR_LIGHT_RED, self.defconfig,
"failed to process.")
errmsg), if self.options.exit_on_error: sys.exit("Exit on error.") else:@@ -644,13 +648,13 @@ class Slot: self.state = STATE_IDLE return True
cross_compile = self.parser.get_cross_compile()
self.cross_compile = self.parser.get_cross_compile() cmd = list(self.make_cmd)
if cross_compile:cmd.append('CROSS_COMPILE=%s' % cross_compile)
if self.cross_compile:cmd.append('CROSS_COMPILE=%s' % self.cross_compile) cmd.append('KCONFIG_IGNORE_DUPLICATES=1') cmd.append('include/config/auto.conf')
self.ps = subprocess.Popen(cmd, stdout=self.devnull)
self.ps = subprocess.Popen(cmd, stdout=self.devnull, stderr=PIPE) self.state = STATE_AUTOCONF return False
Hi Masahiro-san,
On Mon, May 18, 2015 at 10:23 PM, Masahiro Yamada yamada.masahiro@socionext.com wrote:
2015-05-16 6:40 GMT+09:00 Joe Hershberger joe.hershberger@ni.com:
A common case for failed builds is a missing compiler. Print a message for that case to tell the user concisely which compiler was expected that was not found.
This patch also has the effect of not printing build errors any longer. The next patch will add a switch to optionally bring that back.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
Changes in v4: None Changes in v3: None Changes in v2: None
tools/moveconfig.py | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index 9e923da..f986f55 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -166,6 +166,7 @@ import os import re import shutil import subprocess +from subprocess import PIPE
Personally I do not prefer from ... import because it disturbs the naming space.
Could you use subprocess.PIPE instead?
OK.
import sys import tempfile import time @@ -606,11 +607,14 @@ class Slot: return False
if self.ps.poll() != 0:
errmsg = 'Failed to process.'errout = self.ps.stderr.read()This throws exception if "make *_defconfig" or "make savedefconfig" fail.
Traceback (most recent call last): File "tools/moveconfig.py", line 924, in <module> main() File "tools/moveconfig.py", line 919, in main move_config(config_attrs, options) File "tools/moveconfig.py", line 794, in move_config while not slots.available(): File "tools/moveconfig.py", line 717, in available if slot.poll(): File "tools/moveconfig.py", line 616, in poll errout = self.ps.stderr.read() AttributeError: 'NoneType' object has no attribute 'read'
Seems better to add PIPE for all the call of subprocess.Popen()
OK
if errout.find('gcc: command not found') != -1:errmsg = 'Compiler not found (%s)' % self.cross_compileIf you do this, should the locale be changed?
Without LANG=C, "command not found" is displayed in Japanese on my computer.
If --verbose is given, we will be able to know the cause of erorr. "missing compiler" is a special case error?
That's true, but at least for my use-case before I spent several days getting all tool-chains working, it was nice to know what the build was trying to use for a cross tool-chain in a concise one-liner instead of parsing a bunch of error prints. That's part of why I added the -v flag. It's also helpful (naturally) in getting the compilers all working.
print >> sys.stderr, log_msg(self.options.color, COLOR_LIGHT_RED, self.defconfig,
"failed to process.")
errmsg), if self.options.exit_on_error: sys.exit("Exit on error.") else:@@ -644,13 +648,13 @@ class Slot: self.state = STATE_IDLE return True
cross_compile = self.parser.get_cross_compile()
self.cross_compile = self.parser.get_cross_compile() cmd = list(self.make_cmd)
if cross_compile:cmd.append('CROSS_COMPILE=%s' % cross_compile)
if self.cross_compile:cmd.append('CROSS_COMPILE=%s' % self.cross_compile) cmd.append('KCONFIG_IGNORE_DUPLICATES=1') cmd.append('include/config/auto.conf')
self.ps = subprocess.Popen(cmd, stdout=self.devnull)
self.ps = subprocess.Popen(cmd, stdout=self.devnull, stderr=PIPE) self.state = STATE_AUTOCONF return False-- Best Regards Masahiro Yamada _______________________________________________ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
2015-05-20 2:51 GMT+09:00 Joe Hershberger joe.hershberger@gmail.com:
Hi Masahiro-san,
On Mon, May 18, 2015 at 10:23 PM, Masahiro Yamada yamada.masahiro@socionext.com wrote:
2015-05-16 6:40 GMT+09:00 Joe Hershberger joe.hershberger@ni.com:
A common case for failed builds is a missing compiler. Print a message for that case to tell the user concisely which compiler was expected that was not found.
This patch also has the effect of not printing build errors any longer. The next patch will add a switch to optionally bring that back.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
Changes in v4: None Changes in v3: None Changes in v2: None
tools/moveconfig.py | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index 9e923da..f986f55 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -166,6 +166,7 @@ import os import re import shutil import subprocess +from subprocess import PIPE
Personally I do not prefer from ... import because it disturbs the naming space.
Could you use subprocess.PIPE instead?
OK.
import sys import tempfile import time @@ -606,11 +607,14 @@ class Slot: return False
if self.ps.poll() != 0:
errmsg = 'Failed to process.'errout = self.ps.stderr.read()This throws exception if "make *_defconfig" or "make savedefconfig" fail.
Traceback (most recent call last): File "tools/moveconfig.py", line 924, in <module> main() File "tools/moveconfig.py", line 919, in main move_config(config_attrs, options) File "tools/moveconfig.py", line 794, in move_config while not slots.available(): File "tools/moveconfig.py", line 717, in available if slot.poll(): File "tools/moveconfig.py", line 616, in poll errout = self.ps.stderr.read() AttributeError: 'NoneType' object has no attribute 'read'
Seems better to add PIPE for all the call of subprocess.Popen()
OK
if errout.find('gcc: command not found') != -1:errmsg = 'Compiler not found (%s)' % self.cross_compileIf you do this, should the locale be changed?
Without LANG=C, "command not found" is displayed in Japanese on my computer.
If --verbose is given, we will be able to know the cause of erorr. "missing compiler" is a special case error?
That's true, but at least for my use-case before I spent several days getting all tool-chains working, it was nice to know what the build was trying to use for a cross tool-chain in a concise one-liner instead of parsing a bunch of error prints. That's part of why I added the -v flag. It's also helpful (naturally) in getting the compilers all working.
OK. You pass LANG=C in v5, so it works fine for me.
In some cases the build for the autoconf breaks. This outputs the errors following the status so that action can be taken without building again manually.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
---
Changes in v4: None Changes in v3: None Changes in v2: -New for version 2
tools/moveconfig.py | 9 +++++++++ 1 file changed, 9 insertions(+)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index f986f55..685b47b 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -153,6 +153,9 @@ Available options Specify the number of threads to run simultaneously. If not specified, the number of threads is the same as the number of CPU cores.
+ -v, --verbose + Show any build errors as boards are built + To see the complete list of supported options, run
$ tools/moveconfig.py -h @@ -615,6 +618,9 @@ class Slot: COLOR_LIGHT_RED, self.defconfig, errmsg), + if self.options.verbose: + print >> sys.stderr, color_text(self.options.color, + COLOR_LIGHT_CYAN, errout) if self.options.exit_on_error: sys.exit("Exit on error.") else: @@ -882,6 +888,9 @@ def main(): help='only cleanup the headers') parser.add_option('-j', '--jobs', type='int', default=cpu_count, help='the number of jobs to run simultaneously') + parser.add_option('-v', '--verbose', dest='verbose', + action='store_true', default=False, + help='show any build errors as boards are built') parser.usage += ' recipe_file\n\n' + \ 'The recipe_file should describe config options you want to move.\n' + \ 'Each line should contain config_name, type, default_value\n\n' + \
2015-05-16 6:40 GMT+09:00 Joe Hershberger joe.hershberger@ni.com:
In some cases the build for the autoconf breaks. This outputs the errors following the status so that action can be taken without building again manually.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
Changes in v4: None Changes in v3: None Changes in v2: -New for version 2
tools/moveconfig.py | 9 +++++++++ 1 file changed, 9 insertions(+)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index f986f55..685b47b 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -153,6 +153,9 @@ Available options Specify the number of threads to run simultaneously. If not specified, the number of threads is the same as the number of CPU cores.
- -v, --verbose
- Show any build errors as boards are built
To see the complete list of supported options, run
$ tools/moveconfig.py -h @@ -615,6 +618,9 @@ class Slot: COLOR_LIGHT_RED, self.defconfig, errmsg),
if self.options.verbose:print >> sys.stderr, color_text(self.options.color,COLOR_LIGHT_CYAN, errout) if self.options.exit_on_error: sys.exit("Exit on error.") else:@@ -882,6 +888,9 @@ def main(): help='only cleanup the headers') parser.add_option('-j', '--jobs', type='int', default=cpu_count, help='the number of jobs to run simultaneously')
- parser.add_option('-v', '--verbose', dest='verbose',
action='store_true', default=False,help='show any build errors as boards are built')
dest='verbose' is redundant.
The destination is chosen based on the long option name.
Apart from that, I think this is a nice improvement.
Hi Masahiro-san,
On Mon, May 18, 2015 at 10:25 PM, Masahiro Yamada yamada.masahiro@socionext.com wrote:
2015-05-16 6:40 GMT+09:00 Joe Hershberger joe.hershberger@ni.com:
In some cases the build for the autoconf breaks. This outputs the errors following the status so that action can be taken without building again manually.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
Changes in v4: None Changes in v3: None Changes in v2: -New for version 2
tools/moveconfig.py | 9 +++++++++ 1 file changed, 9 insertions(+)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index f986f55..685b47b 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -153,6 +153,9 @@ Available options Specify the number of threads to run simultaneously. If not specified, the number of threads is the same as the number of CPU cores.
- -v, --verbose
- Show any build errors as boards are built
To see the complete list of supported options, run
$ tools/moveconfig.py -h @@ -615,6 +618,9 @@ class Slot: COLOR_LIGHT_RED, self.defconfig, errmsg),
if self.options.verbose:print >> sys.stderr, color_text(self.options.color,COLOR_LIGHT_CYAN, errout) if self.options.exit_on_error: sys.exit("Exit on error.") else:@@ -882,6 +888,9 @@ def main(): help='only cleanup the headers') parser.add_option('-j', '--jobs', type='int', default=cpu_count, help='the number of jobs to run simultaneously')
- parser.add_option('-v', '--verbose', dest='verbose',
action='store_true', default=False,help='show any build errors as boards are built')dest='verbose' is redundant.
The destination is chosen based on the long option name.
OK. Dropped.
Apart from that, I think this is a nice improvement.
-- Best Regards Masahiro Yamada _______________________________________________ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
This gives a basic idea about progress.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
---
Changes in v4: None Changes in v3: -New for version 3
Changes in v2: None
tools/moveconfig.py | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index 685b47b..fca0197 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -568,7 +568,7 @@ class Slot: pass shutil.rmtree(self.build_dir)
- def add(self, defconfig): + def add(self, defconfig, num, total): """Assign a new subprocess for defconfig and add it to the slot.
If the slot is vacant, create a new subprocess for processing the @@ -588,6 +588,8 @@ class Slot: self.ps = subprocess.Popen(cmd, stdout=self.devnull) self.defconfig = defconfig self.state = STATE_DEFCONFIG + self.num = num + self.total = total return True
def poll(self): @@ -634,6 +636,9 @@ class Slot: if self.state == STATE_AUTOCONF: self.parser.update_defconfig(self.defconfig)
+ print ' %d defconfigs out of %d\r' % (self.num + 1, self.total), + sys.stdout.flush() + """Save off the defconfig in a consistent way""" cmd = list(self.make_cmd) cmd.append('savedefconfig') @@ -688,7 +693,7 @@ class Slots: for i in range(options.jobs): self.slots.append(Slot(config_attrs, options, devnull, make_cmd))
- def add(self, defconfig): + def add(self, defconfig, num, total): """Add a new subprocess if a vacant slot is found.
Arguments: @@ -698,7 +703,7 @@ class Slots: Return True on success or False on failure """ for slot in self.slots: - if slot.add(defconfig): + if slot.add(defconfig, num, total): return True return False
@@ -784,8 +789,8 @@ def move_config(config_attrs, options): # Main loop to process defconfig files: # Add a new subprocess into a vacant slot. # Sleep if there is no available slot. - for defconfig in defconfigs: - while not slots.add(defconfig): + for i, defconfig in enumerate(defconfigs): + while not slots.add(defconfig, i, len(defconfigs)): while not slots.available(): # No available slot: sleep for a while time.sleep(SLEEP_TIME) @@ -794,6 +799,7 @@ def move_config(config_attrs, options): while not slots.empty(): time.sleep(SLEEP_TIME)
+ print '' slots.show_failed_boards()
def bad_recipe(filename, linenum, msg):
2015-05-16 6:40 GMT+09:00 Joe Hershberger joe.hershberger@ni.com:
This gives a basic idea about progress.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
Good idea!
Hi Joe,
2015-05-16 6:40 GMT+09:00 Joe Hershberger joe.hershberger@ni.com:
This will ensure that the order of the defconfig entries will always match that of the Kconfig files. After one slightly painful (but still early in the process) pass over all boards, this should keep the defconfigs clean from here on.
Users must edit the Kconfig first to add the menu entries and then run moveconfig.py to update the defconfig files and the include configs.
As such, moveconfig.py cannot compare against the '.config' contents.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
This feature expects the defconfigs are all clean. Otherwise, savedefconfig would make "git diff" noisier.
It is safer to use "make menuconfig && make savedefconfig" for adding new options, but some people still try to edit the defconfig directly...
Perhaps, should do the global cleanup periodically?
This is based on https://patchwork.ozlabs.org/patch/472591/
Changes in v4: -Rebased series on Masahiro's v2
Changes in v3: None Changes in v2: None
tools/moveconfig.py | 35 ++++++++++++++++++++++++++--------- 1 file changed, 26 insertions(+), 9 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index c39ea95..544f6af 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -46,6 +46,9 @@ should look like this: CONFIG_CMD_USB bool n CONFIG_SYS_TEXT_BASE hex 0x00000000
+Next you must edit the Kconfig to add the menu entries for the configs +you are moving.
And then run this tool giving the file name of the recipe
Uh, I was doing in a different work-flow. (I edit the Kconfig after I move CONFIGs to defconfigs).
But, the Kconfig must be edited beforehand to do savedefconfig.
So, I am OK with this change.
$ tools/moveconfig.py recipe @@ -192,6 +195,7 @@ CROSS_COMPILE = { STATE_IDLE = 0 STATE_DEFCONFIG = 1 STATE_AUTOCONF = 2 +STATE_SAVEDEFCONFIG = 3
ACTION_MOVE = 0 ACTION_DEFAULT_VALUE = 1 @@ -390,8 +394,7 @@ class KconfigParser:
return CROSS_COMPILE.get(arch, '')
- def parse_one_config(self, config_attr, defconfig_lines,
dotconfig_lines, autoconf_lines):
def parse_one_config(self, config_attr, defconfig_lines, autoconf_lines): """Parse .config, defconfig, include/autoconf.mk for one config.
This function looks for the config options in the lines from@@ -402,7 +405,6 @@ class KconfigParser: config_attr: A dictionary including the name, the type, and the default value of the target config. defconfig_lines: lines from the original defconfig file.
dotconfig_lines: lines from the .config file. autoconf_lines: lines from the include/autoconf.mk file. Returns:@@ -418,7 +420,7 @@ class KconfigParser: else: default = config + '=' + config_attr['default']
for line in defconfig_lines + dotconfig_lines:
for line in defconfig_lines: line = line.rstrip() if line.startswith(config + '=') or line == not_set: return (ACTION_ALREADY_EXIST, line)@@ -463,15 +465,12 @@ class KconfigParser: with open(defconfig_path) as f: defconfig_lines = f.readlines()
with open(dotconfig_path) as f:dotconfig_lines = f.readlines()with open(autoconf_path) as f: autoconf_lines = f.readlines() for config_attr in self.config_attrs: result = self.parse_one_config(config_attr, defconfig_lines,dotconfig_lines, autoconf_lines)
autoconf_lines) results.append(result) log = ''
With the change of the work-flow above, we need not parse the .config, so this seems OK.
@@ -499,7 +498,7 @@ class KconfigParser: print log,
if not self.options.dry_run:
with open(defconfig_path, 'a') as f:
with open(dotconfig_path, 'a') as f: for (action, value) in results: if action == ACTION_MOVE: f.write(value + '\n')@@ -608,6 +607,24 @@ class Slot:
if self.state == STATE_AUTOCONF: self.parser.update_defconfig(self.defconfig)
"""Save off the defconfig in a consistent way"""cmd = list(self.make_cmd)cmd.append('savedefconfig')self.ps = subprocess.Popen(cmd, stdout=self.devnull,stderr=self.devnull)self.state = STATE_SAVEDEFCONFIGreturn Falseif self.state == STATE_SAVEDEFCONFIG:defconfig_path = os.path.join(self.build_dir, 'defconfig')if not os.path.exists(defconfig_path):print >> sys.stderr, log_msg(self.options.color,COLOR_LIGHT_RED,self.defconfig,'The defconfig was not updated')
Is this warning message reachable?
The missing defconfig means "make savedefconfig" failed.
That case has been already caught by 'Failed to process.' above, I think.
shutil.move(defconfig_path,os.path.join('configs', self.defconfig)) self.state = STATE_IDLE return True
Hi Masahiro-san,
On Mon, May 18, 2015 at 8:58 PM, Masahiro Yamada yamada.masahiro@socionext.com wrote:
Hi Joe,
2015-05-16 6:40 GMT+09:00 Joe Hershberger joe.hershberger@ni.com:
This will ensure that the order of the defconfig entries will always match that of the Kconfig files. After one slightly painful (but still early in the process) pass over all boards, this should keep the defconfigs clean from here on.
Users must edit the Kconfig first to add the menu entries and then run moveconfig.py to update the defconfig files and the include configs.
As such, moveconfig.py cannot compare against the '.config' contents.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
This feature expects the defconfigs are all clean. Otherwise, savedefconfig would make "git diff" noisier.
It is safer to use "make menuconfig && make savedefconfig" for adding new options, but some people still try to edit the defconfig directly...
Perhaps, should do the global cleanup periodically?
This is based on https://patchwork.ozlabs.org/patch/472591/
Changes in v4: -Rebased series on Masahiro's v2
Changes in v3: None Changes in v2: None
tools/moveconfig.py | 35 ++++++++++++++++++++++++++--------- 1 file changed, 26 insertions(+), 9 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index c39ea95..544f6af 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -46,6 +46,9 @@ should look like this: CONFIG_CMD_USB bool n CONFIG_SYS_TEXT_BASE hex 0x00000000
+Next you must edit the Kconfig to add the menu entries for the configs +you are moving.
And then run this tool giving the file name of the recipe
Uh, I was doing in a different work-flow. (I edit the Kconfig after I move CONFIGs to defconfigs).
But, the Kconfig must be edited beforehand to do savedefconfig.
So, I am OK with this change.
$ tools/moveconfig.py recipe @@ -192,6 +195,7 @@ CROSS_COMPILE = { STATE_IDLE = 0 STATE_DEFCONFIG = 1 STATE_AUTOCONF = 2 +STATE_SAVEDEFCONFIG = 3
ACTION_MOVE = 0 ACTION_DEFAULT_VALUE = 1 @@ -390,8 +394,7 @@ class KconfigParser:
return CROSS_COMPILE.get(arch, '')
- def parse_one_config(self, config_attr, defconfig_lines,
dotconfig_lines, autoconf_lines):
def parse_one_config(self, config_attr, defconfig_lines, autoconf_lines): """Parse .config, defconfig, include/autoconf.mk for one config.
This function looks for the config options in the lines from@@ -402,7 +405,6 @@ class KconfigParser: config_attr: A dictionary including the name, the type, and the default value of the target config. defconfig_lines: lines from the original defconfig file.
dotconfig_lines: lines from the .config file. autoconf_lines: lines from the include/autoconf.mk file. Returns:@@ -418,7 +420,7 @@ class KconfigParser: else: default = config + '=' + config_attr['default']
for line in defconfig_lines + dotconfig_lines:
for line in defconfig_lines: line = line.rstrip() if line.startswith(config + '=') or line == not_set: return (ACTION_ALREADY_EXIST, line)@@ -463,15 +465,12 @@ class KconfigParser: with open(defconfig_path) as f: defconfig_lines = f.readlines()
with open(dotconfig_path) as f:dotconfig_lines = f.readlines()with open(autoconf_path) as f: autoconf_lines = f.readlines() for config_attr in self.config_attrs: result = self.parse_one_config(config_attr, defconfig_lines,dotconfig_lines, autoconf_lines)
autoconf_lines) results.append(result) log = ''With the change of the work-flow above, we need not parse the .config, so this seems OK.
@@ -499,7 +498,7 @@ class KconfigParser: print log,
if not self.options.dry_run:
with open(defconfig_path, 'a') as f:
with open(dotconfig_path, 'a') as f: for (action, value) in results: if action == ACTION_MOVE: f.write(value + '\n')@@ -608,6 +607,24 @@ class Slot:
if self.state == STATE_AUTOCONF: self.parser.update_defconfig(self.defconfig)
"""Save off the defconfig in a consistent way"""cmd = list(self.make_cmd)cmd.append('savedefconfig')self.ps = subprocess.Popen(cmd, stdout=self.devnull,stderr=self.devnull)self.state = STATE_SAVEDEFCONFIGreturn Falseif self.state == STATE_SAVEDEFCONFIG:defconfig_path = os.path.join(self.build_dir, 'defconfig')if not os.path.exists(defconfig_path):print >> sys.stderr, log_msg(self.options.color,COLOR_LIGHT_RED,self.defconfig,'The defconfig was not updated')Is this warning message reachable?
The missing defconfig means "make savedefconfig" failed.
That case has been already caught by 'Failed to process.' above, I think.
OK... dropping that warning.
shutil.move(defconfig_path,os.path.join('configs', self.defconfig)) self.state = STATE_IDLE return True-- Best Regards Masahiro Yamada _______________________________________________ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
Hi Masahiro-san,
On Mon, May 18, 2015 at 8:58 PM, Masahiro Yamada yamada.masahiro@socionext.com wrote:
Hi Joe,
2015-05-16 6:40 GMT+09:00 Joe Hershberger joe.hershberger@ni.com:
This will ensure that the order of the defconfig entries will always match that of the Kconfig files. After one slightly painful (but still early in the process) pass over all boards, this should keep the defconfigs clean from here on.
Users must edit the Kconfig first to add the menu entries and then run moveconfig.py to update the defconfig files and the include configs.
As such, moveconfig.py cannot compare against the '.config' contents.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
This feature expects the defconfigs are all clean. Otherwise, savedefconfig would make "git diff" noisier.
It is safer to use "make menuconfig && make savedefconfig" for adding new options, but some people still try to edit the defconfig directly...
Perhaps, should do the global cleanup periodically?
I agree that until people (both contributors and custodians) get used to this as the best way, we will likely need a few global clean-ups, but I think with the addition of this tool to main-line the instance of this will likely drop sharply. Moving configs is a hassle, so with a tool available I doubt many people will do it manually. That's why it is important that the tool do it the cleanest way. Hopefully for the case where a config is changed on a board, the custodian will notice if it is just added to the bottom of the file. It seems to work out for Linux.
Cheers, -Joe
This will ensure that the order of the defconfig entries will always match that of the Kconfig files. After one slightly painful (but still early in the process) pass over all boards, this should keep the defconfigs clean from here on.
Users must edit the Kconfig first to add the menu entries and then run moveconfig.py to update the defconfig files and the include configs.
As such, moveconfig.py cannot compare against the '.config' contents.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com --- This is based on https://patchwork.ozlabs.org/patch/472591/
Changes in v5: -Removed warning that may never be reached
Changes in v4: -Rebased series on Masahiro's v2
Changes in v3: None Changes in v2: None
tools/moveconfig.py | 30 +++++++++++++++++++++--------- 1 file changed, 21 insertions(+), 9 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index c39ea95..c9984b4 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -46,6 +46,9 @@ should look like this: CONFIG_CMD_USB bool n CONFIG_SYS_TEXT_BASE hex 0x00000000
+Next you must edit the Kconfig to add the menu entries for the configs +you are moving. + And then run this tool giving the file name of the recipe
$ tools/moveconfig.py recipe @@ -192,6 +195,7 @@ CROSS_COMPILE = { STATE_IDLE = 0 STATE_DEFCONFIG = 1 STATE_AUTOCONF = 2 +STATE_SAVEDEFCONFIG = 3
ACTION_MOVE = 0 ACTION_DEFAULT_VALUE = 1 @@ -390,8 +394,7 @@ class KconfigParser:
return CROSS_COMPILE.get(arch, '')
- def parse_one_config(self, config_attr, defconfig_lines, - dotconfig_lines, autoconf_lines): + def parse_one_config(self, config_attr, defconfig_lines, autoconf_lines): """Parse .config, defconfig, include/autoconf.mk for one config.
This function looks for the config options in the lines from @@ -402,7 +405,6 @@ class KconfigParser: config_attr: A dictionary including the name, the type, and the default value of the target config. defconfig_lines: lines from the original defconfig file. - dotconfig_lines: lines from the .config file. autoconf_lines: lines from the include/autoconf.mk file.
Returns: @@ -418,7 +420,7 @@ class KconfigParser: else: default = config + '=' + config_attr['default']
- for line in defconfig_lines + dotconfig_lines: + for line in defconfig_lines: line = line.rstrip() if line.startswith(config + '=') or line == not_set: return (ACTION_ALREADY_EXIST, line) @@ -463,15 +465,12 @@ class KconfigParser: with open(defconfig_path) as f: defconfig_lines = f.readlines()
- with open(dotconfig_path) as f: - dotconfig_lines = f.readlines() - with open(autoconf_path) as f: autoconf_lines = f.readlines()
for config_attr in self.config_attrs: result = self.parse_one_config(config_attr, defconfig_lines, - dotconfig_lines, autoconf_lines) + autoconf_lines) results.append(result)
log = '' @@ -499,7 +498,7 @@ class KconfigParser: print log,
if not self.options.dry_run: - with open(defconfig_path, 'a') as f: + with open(dotconfig_path, 'a') as f: for (action, value) in results: if action == ACTION_MOVE: f.write(value + '\n') @@ -608,6 +607,19 @@ class Slot:
if self.state == STATE_AUTOCONF: self.parser.update_defconfig(self.defconfig) + + """Save off the defconfig in a consistent way""" + cmd = list(self.make_cmd) + cmd.append('savedefconfig') + self.ps = subprocess.Popen(cmd, stdout=self.devnull, + stderr=self.devnull) + self.state = STATE_SAVEDEFCONFIG + return False + + if self.state == STATE_SAVEDEFCONFIG: + defconfig_path = os.path.join(self.build_dir, 'defconfig') + shutil.move(defconfig_path, + os.path.join('configs', self.defconfig)) self.state = STATE_IDLE return True
When moving configs, it is important to know what was defined in the config header even if it duplicates the configs coming from Kconfig.
This is specifically needed for the case where a config is set to default 'y' in the Kconfig. This would previously cause the actual value from the include config to be filtered out, and moveconfig.py would think that it was 'n'... This means that the value that should be 'y' is now (in every defconfig) set to 'not set'.
tools/moveconfig.py now defines KCONFIG_IGNORE_DUPLICATES to prevent the filtering from happening and selecting wrong values for the defconfig.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com Acked-by: Masahiro Yamada yamada.masahiro@socionext.com
---
Changes in v5: None Changes in v4: None Changes in v3: -New for version 3
Changes in v2: None
scripts/Makefile.autoconf | 3 ++- tools/moveconfig.py | 1 + 2 files changed, 3 insertions(+), 1 deletion(-)
diff --git a/scripts/Makefile.autoconf b/scripts/Makefile.autoconf index f054081..36bfa17 100644 --- a/scripts/Makefile.autoconf +++ b/scripts/Makefile.autoconf @@ -58,7 +58,8 @@ quiet_cmd_autoconf = GEN $@ $(CPP) $(c_flags) $2 -DDO_DEPS_ONLY -dM $(srctree)/include/common.h > $@.tmp && { \ sed -n -f $(srctree)/tools/scripts/define2mk.sed $@.tmp | \ while read line; do \ - if ! grep -q "$${line%=*}=" include/config/auto.conf; then \ + if [ -n "${KCONFIG_IGNORE_DUPLICATES}" ] || \ + ! grep -q "$${line%=*}=" include/config/auto.conf; then \ echo "$$line"; \ fi \ done > $@; \ diff --git a/tools/moveconfig.py b/tools/moveconfig.py index c9984b4..dd9434d 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -627,6 +627,7 @@ class Slot: cmd = list(self.make_cmd) if cross_compile: cmd.append('CROSS_COMPILE=%s' % cross_compile) + cmd.append('KCONFIG_IGNORE_DUPLICATES=1') cmd.append('include/config/auto.conf') self.ps = subprocess.Popen(cmd, stdout=self.devnull) self.state = STATE_AUTOCONF
This is helpful to re-attempt to move failed boards from a previous run without starting over.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
---
Changes in v5: -Remove default 'r' mode on open
Changes in v4: None Changes in v3: -Fixed command line options order (alphabetize)
Changes in v2: -New for version 2
tools/moveconfig.py | 26 ++++++++++++++++++++------ 1 file changed, 20 insertions(+), 6 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index dd9434d..e93548c 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -135,6 +135,9 @@ Available options Surround each portion of the log with escape sequences to display it in color on the terminal.
+ -d, --defconfigs + Specify a file containing a list of defconfigs to move + -n, --dry-run Peform a trial run that does not make any changes. It is useful to see what is going to happen before one actually runs it. @@ -729,12 +732,21 @@ def move_config(config_attrs, options): config_attr['type'], config_attr['default'])
- # All the defconfig files to be processed - defconfigs = [] - for (dirpath, dirnames, filenames) in os.walk('configs'): - dirpath = dirpath[len('configs') + 1:] - for filename in fnmatch.filter(filenames, '*_defconfig'): - defconfigs.append(os.path.join(dirpath, filename)) + if options.defconfigs: + defconfigs = [line.strip() for line in open(options.defconfigs)] + for i, defconfig in enumerate(defconfigs): + if not defconfig.endswith('_defconfig'): + defconfigs[i] = defconfig + '_defconfig' + if not os.path.exists(os.path.join('configs', defconfigs[i])): + sys.exit('%s - defconfig does not exist. Stopping.' % + defconfigs[i]) + else: + # All the defconfig files to be processed + defconfigs = [] + for (dirpath, dirnames, filenames) in os.walk('configs'): + dirpath = dirpath[len('configs') + 1:] + for filename in fnmatch.filter(filenames, '*_defconfig'): + defconfigs.append(os.path.join(dirpath, filename))
slots = Slots(config_attrs, options)
@@ -835,6 +847,8 @@ def main(): # Add options here parser.add_option('-c', '--color', action='store_true', default=False, help='display the log in color') + parser.add_option('-d', '--defconfigs', type='string', + help='a file containing a list of defconfigs to move') parser.add_option('-n', '--dry-run', action='store_true', default=False, help='perform a trial run (show log with no changes)') parser.add_option('-e', '--exit-on-error', action='store_true',
In some case you may want to only cleanup the headers. Make it possible without waiting for all boards to compile.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
---
Changes in v5: -Move check_top_directory to main
Changes in v4: None Changes in v3: -New for version 3
Changes in v2: None
tools/moveconfig.py | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index e93548c..561cd9a 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -146,6 +146,9 @@ Available options Exit immediately if Make exits with a non-zero status while processing a defconfig file.
+ -H, --headers-only + Only cleanup the headers; skip the defconfig processing + -j, --jobs Specify the number of threads to run simultaneously. If not specified, the number of threads is the same as the number of CPU cores. @@ -720,8 +723,6 @@ def move_config(config_attrs, options): the type, and the default value of the target config. options: option flags """ - check_top_directory() - if len(config_attrs) == 0: print 'Nothing to do. exit.' sys.exit(0) @@ -765,8 +766,6 @@ def move_config(config_attrs, options):
slots.show_failed_boards()
- cleanup_headers(config_attrs, options.dry_run) - def bad_recipe(filename, linenum, msg): """Print error message with the file name and the line number and exit.""" sys.exit("%s: line %d: error : " % (filename, linenum) + msg) @@ -854,6 +853,9 @@ def main(): parser.add_option('-e', '--exit-on-error', action='store_true', default=False, help='exit immediately on any error') + parser.add_option('-H', '--headers-only', dest='cleanup_headers_only', + action='store_true', default=False, + help='only cleanup the headers') parser.add_option('-j', '--jobs', type='int', default=cpu_count, help='the number of jobs to run simultaneously') parser.usage += ' recipe_file\n\n' + \ @@ -874,7 +876,12 @@ def main():
update_cross_compile()
- move_config(config_attrs, options) + check_top_directory() + + if not options.cleanup_headers_only: + move_config(config_attrs, options) + + cleanup_headers(config_attrs, options.dry_run)
if __name__ == '__main__': main()
Some config.h files live in arch and board directories. They will need to be cleaned up as well, so run the same filters there.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
---
Changes in v5: -Consolidate code to clean up dirs
Changes in v4: -New for version 4
Changes in v3: None Changes in v2: None
tools/moveconfig.py | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index 561cd9a..f38dac1 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -347,11 +347,12 @@ def cleanup_headers(config_attrs, dry_run): patterns.append(re.compile(r'#\s*define\s+%s\W' % config)) patterns.append(re.compile(r'#\s*undef\s+%s\W' % config))
- for (dirpath, dirnames, filenames) in os.walk('include'): - for filename in filenames: - if not fnmatch.fnmatch(filename, '*~'): - cleanup_one_header(os.path.join(dirpath, filename), patterns, - dry_run) + for dir in 'include', 'arch', 'board': + for (dirpath, dirnames, filenames) in os.walk(dir): + for filename in filenames: + if not fnmatch.fnmatch(filename, '*~'): + cleanup_one_header(os.path.join(dirpath, filename), + patterns, dry_run)
### classes ### class KconfigParser:
If boards fail, output that list to a file so that it can easily be passed back into moveconfig.py using the -d option.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
---
Changes in v5: -Changed file handling to use with/as
Changes in v4: None Changes in v3: None Changes in v2: None
tools/moveconfig.py | 4 ++++ 1 file changed, 4 insertions(+)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index f38dac1..8b8eed6 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -716,6 +716,10 @@ class Slots: print >> sys.stderr, color_text(self.options.color, COLOR_LIGHT_RED, line)
+ with open('moveconfig.failed', 'w') as f: + for board in failed_boards: + f.write(board + '\n') + def move_config(config_attrs, options): """Move config options to defconfig files.
A common case for failed builds is a missing compiler. Print a message for that case to tell the user concisely which compiler was expected that was not found.
This patch also has the effect of not printing build errors any longer. The next patch will add a switch to optionally bring that back.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
---
Changes in v5: -Use scoped PIPE instead of importing it -Set LANG=C for calling the compiler when planning to screen-scrape -Use PIPE for all Popen -Make missing compilers yellow
Changes in v4: None Changes in v3: None Changes in v2: None
tools/moveconfig.py | 28 ++++++++++++++++++++-------- 1 file changed, 20 insertions(+), 8 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index 8b8eed6..2fa98e3 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -572,7 +572,8 @@ class Slot: return False cmd = list(self.make_cmd) cmd.append(defconfig) - self.ps = subprocess.Popen(cmd, stdout=self.devnull) + self.ps = subprocess.Popen(cmd, stdout=self.devnull, + stderr=subprocess.PIPE) self.defconfig = defconfig self.state = STATE_DEFCONFIG return True @@ -597,11 +598,18 @@ class Slot: return False
if self.ps.poll() != 0: - + errmsg = 'Failed to process.' + errout = self.ps.stderr.read() + if errout.find('gcc: command not found') != -1: + errmsg = 'Compiler not found (' + errmsg += color_text(self.options.color, COLOR_YELLOW, + self.cross_compile) + errmsg += color_text(self.options.color, COLOR_LIGHT_RED, + ')') print >> sys.stderr, log_msg(self.options.color, COLOR_LIGHT_RED, self.defconfig, - "failed to process.") + errmsg), if self.options.exit_on_error: sys.exit("Exit on error.") else: @@ -619,7 +627,7 @@ class Slot: cmd = list(self.make_cmd) cmd.append('savedefconfig') self.ps = subprocess.Popen(cmd, stdout=self.devnull, - stderr=self.devnull) + stderr=subprocess.PIPE) self.state = STATE_SAVEDEFCONFIG return False
@@ -630,13 +638,17 @@ class Slot: self.state = STATE_IDLE return True
- cross_compile = self.parser.get_cross_compile() + self.cross_compile = self.parser.get_cross_compile() cmd = list(self.make_cmd) - if cross_compile: - cmd.append('CROSS_COMPILE=%s' % cross_compile) + if self.cross_compile: + cmd.append('CROSS_COMPILE=%s' % self.cross_compile) cmd.append('KCONFIG_IGNORE_DUPLICATES=1') cmd.append('include/config/auto.conf') - self.ps = subprocess.Popen(cmd, stdout=self.devnull) + """This will be screen-scraped, so be sure the expected text will be + returned consistently on every machine by setting LANG=C""" + self.ps = subprocess.Popen(cmd, stdout=self.devnull, + env=dict(os.environ, LANG='C'), + stderr=subprocess.PIPE) self.state = STATE_AUTOCONF return False
In some cases the build for the autoconf breaks. This outputs the errors following the status so that action can be taken without building again manually.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
---
Changes in v5: -Remove redundant destination
Changes in v4: None Changes in v3: None Changes in v2: -New for version 2
tools/moveconfig.py | 8 ++++++++ 1 file changed, 8 insertions(+)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index 2fa98e3..a6f1853 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -153,6 +153,9 @@ Available options Specify the number of threads to run simultaneously. If not specified, the number of threads is the same as the number of CPU cores.
+ -v, --verbose + Show any build errors as boards are built + To see the complete list of supported options, run
$ tools/moveconfig.py -h @@ -610,6 +613,9 @@ class Slot: COLOR_LIGHT_RED, self.defconfig, errmsg), + if self.options.verbose: + print >> sys.stderr, color_text(self.options.color, + COLOR_LIGHT_CYAN, errout) if self.options.exit_on_error: sys.exit("Exit on error.") else: @@ -875,6 +881,8 @@ def main(): help='only cleanup the headers') parser.add_option('-j', '--jobs', type='int', default=cpu_count, help='the number of jobs to run simultaneously') + parser.add_option('-v', '--verbose', action='store_true', default=False, + help='show any build errors as boards are built') parser.usage += ' recipe_file\n\n' + \ 'The recipe_file should describe config options you want to move.\n' + \ 'Each line should contain config_name, type, default_value\n\n' + \
This gives a basic idea about progress.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
---
Changes in v5: None Changes in v4: None Changes in v3: -New for version 3
Changes in v2: None
tools/moveconfig.py | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py index a6f1853..a6e7c22 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -558,7 +558,7 @@ class Slot: pass shutil.rmtree(self.build_dir)
- def add(self, defconfig): + def add(self, defconfig, num, total): """Assign a new subprocess for defconfig and add it to the slot.
If the slot is vacant, create a new subprocess for processing the @@ -579,6 +579,8 @@ class Slot: stderr=subprocess.PIPE) self.defconfig = defconfig self.state = STATE_DEFCONFIG + self.num = num + self.total = total return True
def poll(self): @@ -629,6 +631,9 @@ class Slot: if self.state == STATE_AUTOCONF: self.parser.update_defconfig(self.defconfig)
+ print ' %d defconfigs out of %d\r' % (self.num + 1, self.total), + sys.stdout.flush() + """Save off the defconfig in a consistent way""" cmd = list(self.make_cmd) cmd.append('savedefconfig') @@ -682,7 +687,7 @@ class Slots: for i in range(options.jobs): self.slots.append(Slot(config_attrs, options, devnull, make_cmd))
- def add(self, defconfig): + def add(self, defconfig, num, total): """Add a new subprocess if a vacant slot is found.
Arguments: @@ -692,7 +697,7 @@ class Slots: Return True on success or False on failure """ for slot in self.slots: - if slot.add(defconfig): + if slot.add(defconfig, num, total): return True return False
@@ -777,8 +782,8 @@ def move_config(config_attrs, options): # Main loop to process defconfig files: # Add a new subprocess into a vacant slot. # Sleep if there is no available slot. - for defconfig in defconfigs: - while not slots.add(defconfig): + for i, defconfig in enumerate(defconfigs): + while not slots.add(defconfig, i, len(defconfigs)): while not slots.available(): # No available slot: sleep for a while time.sleep(SLEEP_TIME) @@ -787,6 +792,7 @@ def move_config(config_attrs, options): while not slots.empty(): time.sleep(SLEEP_TIME)
+ print '' slots.show_failed_boards()
def bad_recipe(filename, linenum, msg):
Hi Joe,
2015-05-20 3:21 GMT+09:00 Joe Hershberger joe.hershberger@ni.com:
This will ensure that the order of the defconfig entries will always match that of the Kconfig files. After one slightly painful (but still early in the process) pass over all boards, this should keep the defconfigs clean from here on.
Users must edit the Kconfig first to add the menu entries and then run moveconfig.py to update the defconfig files and the include configs.
As such, moveconfig.py cannot compare against the '.config' contents.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
This is based on https://patchwork.ozlabs.org/patch/472591/
Changes in v5: -Removed warning that may never be reached
The whole series of v5 looks good to me.
Tom, looks like this series is delegated to me.
Shall I apply my base patch and Joe's great improvements and then send a pull-req? Or would you do it?
2015-05-20 3:21 GMT+09:00 Joe Hershberger joe.hershberger@ni.com:
This will ensure that the order of the defconfig entries will always match that of the Kconfig files. After one slightly painful (but still early in the process) pass over all boards, this should keep the defconfigs clean from here on.
Users must edit the Kconfig first to add the menu entries and then run moveconfig.py to update the defconfig files and the include configs.
As such, moveconfig.py cannot compare against the '.config' contents.
Signed-off-by: Joe Hershberger joe.hershberger@ni.com
This is based on https://patchwork.ozlabs.org/patch/472591/
Changes in v5: -Removed warning that may never be reached
The series, applied to u-boot-uniphier/misc with my Acked-by. Thanks!
participants (5)
-
Joe Hershberger -
Joe Hershberger -
Masahiro Yamada -
Masahiro Yamada -
Simon Glass