[U-Boot] [PATCH 00/21] patman: Update to support Python 3
This series updates patman to support Python 3:
- Avoid using 'unicode' type directly - Use items() instead of iteritems() - Make sure file I/O uses binary mode where necessary - Change print statements to functions - Use the built-in set() class - Fix up generation of repeated bytes
A few patches for binman are included, but this still requires Python 2.
Simon Glass (21): patman: Update cros_subprocess to use bytearray patman: Convert print statements to Python 3 binman: Convert print statements to Python 3 binman: Don't show errors for failed tests binman: Remove use of Set() patman: Use items() instead of iteritems() binman: Use items() instead of iteritems() tools: binman: Open all binary files in binary mode tools: dtoc: Open all binary files in binary mode patman: Provide a way to get program output in binary mode binman: Use binary mode when compressing data binman: Drop an unused input file binman: Handle repeated bytes for Python 3 Add a simple script to remove boards patman: Support use of stringIO in Python 3 patman: Move unicode helpers to tools patman: Sort series output for repeatabily patman: Avoid unicode type in settings unit tests patman: Adjust functional tests for Python 3 patman: Tidy up a few more unicode conversions patman: Don't require Python 2
tools/binman/binman.py | 26 +++- tools/binman/bsection.py | 7 +- tools/binman/control.py | 8 +- tools/binman/elf.py | 4 +- tools/binman/elf_test.py | 2 +- tools/binman/entry.py | 5 +- tools/binman/etype/blob.py | 2 +- tools/binman/etype/fill.py | 4 +- tools/binman/etype/gbb.py | 2 +- tools/binman/etype/u_boot_spl_bss_pad.py | 2 +- tools/binman/ftest.py | 81 ++++++------ tools/binman/state.py | 7 +- tools/dtoc/fdt.py | 2 +- tools/patman/cros_subprocess.py | 53 +++++--- tools/patman/func_test.py | 41 ++++--- tools/patman/gitutil.py | 16 +-- tools/patman/patman.py | 2 +- tools/patman/series.py | 20 +-- tools/patman/settings.py | 34 ++--- tools/patman/test_util.py | 16 +-- tools/patman/tools.py | 55 ++++++++- tools/rmboard.py | 150 +++++++++++++++++++++++ 22 files changed, 385 insertions(+), 154 deletions(-) create mode 100755 tools/rmboard.py
At present this function uses lists and strings. This does not work so well with Python 3, and testing against '' does not work for a bytearray. Update the code to fix these issues.
Signed-off-by: Simon Glass sjg@chromium.org ---
tools/patman/cros_subprocess.py | 48 ++++++++++++++++++++------------- tools/patman/gitutil.py | 2 ++ 2 files changed, 32 insertions(+), 18 deletions(-)
diff --git a/tools/patman/cros_subprocess.py b/tools/patman/cros_subprocess.py index ebd4300dfd..4de230ac63 100644 --- a/tools/patman/cros_subprocess.py +++ b/tools/patman/cros_subprocess.py @@ -100,6 +100,24 @@ class Popen(subprocess.Popen): if kwargs: raise ValueError("Unit tests do not test extra args - please add tests")
+ def ConvertData(self, data): + """Convert stdout/stderr data to the correct format for output + + Args: + data: Data to convert, or None for '' + + Returns: + Converted data to a unicode string + """ + if data is None: + return '' + else: + try: + data = data.decode('utf-8') + except UnicodeDecodeError: + data = data.decode('utf-8', 'ignore') + return data + def CommunicateFilter(self, output): """Interact with process: Read data from stdout and stderr.
@@ -156,11 +174,11 @@ class Popen(subprocess.Popen): self.stdin.close() if self.stdout: read_set.append(self.stdout) - stdout = [] + stdout = bytearray() if self.stderr and self.stderr != self.stdout: read_set.append(self.stderr) - stderr = [] - combined = [] + stderr = bytearray() + combined = bytearray()
input_offset = 0 while read_set or write_set: @@ -192,12 +210,12 @@ class Popen(subprocess.Popen): data = os.read(self.stdout.fileno(), 1024) except OSError: pass - if data == "": + if not len(data): self.stdout.close() read_set.remove(self.stdout) else: - stdout.append(data) - combined.append(data) + stdout += data + combined += data if output: output(sys.stdout, data) if self.stderr in rlist: @@ -207,25 +225,19 @@ class Popen(subprocess.Popen): data = os.read(self.stderr.fileno(), 1024) except OSError: pass - if data == "": + if not len(data): self.stderr.close() read_set.remove(self.stderr) else: - stderr.append(data) - combined.append(data) + stderr += data + combined += data if output: output(sys.stderr, data)
# All data exchanged. Translate lists into strings. - if stdout is not None: - stdout = ''.join(stdout) - else: - stdout = '' - if stderr is not None: - stderr = ''.join(stderr) - else: - stderr = '' - combined = ''.join(combined) + stdout = self.ConvertData(stdout) + stderr = self.ConvertData(stderr) + combined = self.ConvertData(combined)
# Translate newlines, if requested. We cannot let the file # object do the translation: It is based on stdio, which is diff --git a/tools/patman/gitutil.py b/tools/patman/gitutil.py index 9905bb0bbd..7650b51bd5 100644 --- a/tools/patman/gitutil.py +++ b/tools/patman/gitutil.py @@ -326,6 +326,8 @@ def BuildEmailList(in_list, tag=None, alias=None, raise_on_error=True): result = [] for item in raw: if not item in result: + if type(item) == unicode: + item = item.encode('utf-8') result.append(item) if tag: return ['%s %s%s%s' % (tag, quote, email, quote) for email in result]
Update all print statements to be functions, as required by Python 3.
Signed-off-by: Simon Glass sjg@chromium.org ---
tools/patman/func_test.py | 3 --- tools/patman/gitutil.py | 10 +++++----- tools/patman/test_util.py | 16 +++++++++------- 3 files changed, 14 insertions(+), 15 deletions(-)
diff --git a/tools/patman/func_test.py b/tools/patman/func_test.py index d79e716074..31481157fc 100644 --- a/tools/patman/func_test.py +++ b/tools/patman/func_test.py @@ -159,7 +159,6 @@ class TestFunctional(unittest.TestCase): os.remove(cc_file)
lines = out[0].splitlines() - #print '\n'.join(lines) self.assertEqual('Cleaned %s patches' % len(series.commits), lines[0]) self.assertEqual('Change log missing for v2', lines[1]) self.assertEqual('Change log missing for v3', lines[2]) @@ -223,7 +222,6 @@ Simon Glass (2):
''' lines = open(cover_fname).read().splitlines() - #print '\n'.join(lines) self.assertEqual( 'Subject: [RFC PATCH v3 0/2] test: A test patch series', lines[3]) @@ -231,7 +229,6 @@ Simon Glass (2):
for i, fname in enumerate(args): lines = open(fname).read().splitlines() - #print '\n'.join(lines) subject = [line for line in lines if line.startswith('Subject')] self.assertEqual('Subject: [RFC %d/%d]' % (i + 1, count), subject[0][:18]) diff --git a/tools/patman/gitutil.py b/tools/patman/gitutil.py index 7650b51bd5..11aeb73b74 100644 --- a/tools/patman/gitutil.py +++ b/tools/patman/gitutil.py @@ -397,11 +397,11 @@ def EmailPatches(series, cover_fname, args, dry_run, raise_on_error, cc_fname, git_config_to = command.Output('git', 'config', 'sendemail.to', raise_on_error=False) if not git_config_to: - print ("No recipient.\n" - "Please add something like this to a commit\n" - "Series-to: Fred Bloggs f.blogs@napier.co.nz\n" - "Or do something like this\n" - "git config sendemail.to u-boot@lists.denx.de") + print("No recipient.\n" + "Please add something like this to a commit\n" + "Series-to: Fred Bloggs f.blogs@napier.co.nz\n" + "Or do something like this\n" + "git config sendemail.to u-boot@lists.denx.de") return cc = BuildEmailList(list(set(series.get('cc')) - set(series.get('to'))), '--cc', alias, raise_on_error) diff --git a/tools/patman/test_util.py b/tools/patman/test_util.py index 687d40704a..e462ec8f72 100644 --- a/tools/patman/test_util.py +++ b/tools/patman/test_util.py @@ -3,6 +3,8 @@ # Copyright (c) 2016 Google, Inc #
+from __future__ import print_function + from contextlib import contextmanager import glob import os @@ -54,18 +56,18 @@ def RunTestCoverage(prog, filter_fname, exclude_list, build_dir, required=None): missing_list = required missing_list.difference_update(test_set) if missing_list: - print 'Missing tests for %s' % (', '.join(missing_list)) - print stdout + print('Missing tests for %s' % (', '.join(missing_list))) + print(stdout) ok = False
coverage = lines[-1].split(' ')[-1] ok = True - print coverage + print(coverage) if coverage != '100%': - print stdout - print ("Type 'python-coverage html' to get a report in " - 'htmlcov/index.html') - print 'Coverage error: %s, but should be 100%%' % coverage + print(stdout) + print("Type 'python-coverage html' to get a report in " + 'htmlcov/index.html') + print('Coverage error: %s, but should be 100%%' % coverage) ok = False if not ok: raise ValueError('Test coverage failure')
Update all print statements to be functions, as required by Python 3.
Signed-off-by: Simon Glass sjg@chromium.org ---
tools/binman/binman.py | 14 ++++++++------ tools/binman/control.py | 6 ++++-- 2 files changed, 12 insertions(+), 8 deletions(-)
diff --git a/tools/binman/binman.py b/tools/binman/binman.py index 439908e665..4206d2b0e3 100755 --- a/tools/binman/binman.py +++ b/tools/binman/binman.py @@ -9,6 +9,8 @@
"""See README for more information"""
+from __future__ import print_function + import glob import multiprocessing import os @@ -85,13 +87,13 @@ def RunTests(debug, processes, args): else: suite.run(result)
- print result + print(result) for test, err in result.errors: - print test.id(), err + print(test.id(), err) for test, err in result.failures: - print err, result.failures + print(err, result.failures) if result.errors or result.failures: - print 'binman tests FAILED' + print('binman tests FAILED') return 1 return 0
@@ -143,9 +145,9 @@ def RunBinman(options, args): try: ret_code = control.Binman(options, args) except Exception as e: - print 'binman: %s' % e + print('binman: %s' % e) if options.debug: - print + print() traceback.print_exc() ret_code = 1 return ret_code diff --git a/tools/binman/control.py b/tools/binman/control.py index b32e4e1996..8f7f906860 100644 --- a/tools/binman/control.py +++ b/tools/binman/control.py @@ -5,6 +5,8 @@ # Creates binary images from input files controlled by a description #
+from __future__ import print_function + from collections import OrderedDict import os import sys @@ -134,7 +136,7 @@ def Binman(options, args): del images[name] skip.append(name) if skip and options.verbosity >= 2: - print 'Skipping images: %s' % ', '.join(skip) + print('Skipping images: %s' % ', '.join(skip))
state.Prepare(images, dtb)
@@ -170,7 +172,7 @@ def Binman(options, args): except Exception as e: if options.map: fname = image.WriteMap() - print "Wrote map file '%s' to show errors" % fname + print("Wrote map file '%s' to show errors" % fname) raise image.SetImagePos() if options.update_fdt:
An unfortunate new feature in Python 3.5 causes binman to print errors for non-existent tests, when the test filter is used. Work around this by detecting the unwanted tests and removing them from the result.
Signed-off-by: Simon Glass sjg@chromium.org ---
tools/binman/binman.py | 12 ++++++++++++ 1 file changed, 12 insertions(+)
diff --git a/tools/binman/binman.py b/tools/binman/binman.py index 4206d2b0e3..aad2e9c8bc 100755 --- a/tools/binman/binman.py +++ b/tools/binman/binman.py @@ -87,6 +87,18 @@ def RunTests(debug, processes, args): else: suite.run(result)
+ # Remove errors which just indicate a missing test. Since Python v3.5 If an + # ImportError or AttributeError occurs while traversing name then a + # synthetic test that raises that error when run will be returned. These + # errors are included in the errors accumulated by result.errors. + if test_name: + errors = [] + for test, err in result.errors: + if ("has no attribute '%s'" % test_name) not in err: + errors.append((test, err)) + result.testsRun -= 1 + result.errors = errors + print(result) for test, err in result.errors: print(test.id(), err)
A new built-in set() is used in both Python 2 and 3 now. Move it to use that instead of Set().
Signed-off-by: Simon Glass sjg@chromium.org ---
tools/binman/bsection.py | 3 +-- tools/binman/entry.py | 5 ++--- tools/binman/state.py | 7 +++---- 3 files changed, 6 insertions(+), 9 deletions(-)
diff --git a/tools/binman/bsection.py b/tools/binman/bsection.py index 0ba542ee98..be67ff957e 100644 --- a/tools/binman/bsection.py +++ b/tools/binman/bsection.py @@ -8,7 +8,6 @@ from __future__ import print_function
from collections import OrderedDict -from sets import Set import sys
import fdt_util @@ -109,7 +108,7 @@ class Section(object):
def GetFdtSet(self): """Get the set of device tree files used by this image""" - fdt_set = Set() + fdt_set = set() for entry in self._entries.values(): fdt_set.update(entry.GetFdtSet()) return fdt_set diff --git a/tools/binman/entry.py b/tools/binman/entry.py index 648cfd241f..d842d89dd6 100644 --- a/tools/binman/entry.py +++ b/tools/binman/entry.py @@ -18,7 +18,6 @@ except: have_importlib = False
import os -from sets import Set import sys
import fdt_util @@ -178,8 +177,8 @@ class Entry(object): # It would be better to use isinstance(self, Entry_blob_dtb) here but # we cannot access Entry_blob_dtb if fname and fname.endswith('.dtb'): - return Set([fname]) - return Set() + return set([fname]) + return set()
def ExpandEntries(self): pass diff --git a/tools/binman/state.py b/tools/binman/state.py index d945e4bf65..af9678649c 100644 --- a/tools/binman/state.py +++ b/tools/binman/state.py @@ -7,7 +7,6 @@
import hashlib import re -from sets import Set
import os import tools @@ -24,10 +23,10 @@ entry_args = {} use_fake_dtb = False
# Set of all device tree files references by images -fdt_set = Set() +fdt_set = set()
# Same as above, but excluding the main one -fdt_subset = Set() +fdt_subset = set()
# The DTB which contains the full image information main_dtb = None @@ -136,7 +135,7 @@ def Prepare(images, dtb): main_dtb = dtb fdt_files.clear() fdt_files['u-boot.dtb'] = dtb - fdt_subset = Set() + fdt_subset = set() if not use_fake_dtb: for image in images.values(): fdt_subset.update(image.GetFdtSet())
Python 3 requires this, and Python 2 allows it. Convert the code over to ensure compatibility with Python 3.
Signed-off-by: Simon Glass sjg@chromium.org ---
tools/patman/settings.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tools/patman/settings.py b/tools/patman/settings.py index ea2bc74f75..07bf6a6ea4 100644 --- a/tools/patman/settings.py +++ b/tools/patman/settings.py @@ -163,7 +163,7 @@ class _ProjectConfigParser(ConfigParser.SafeConfigParser): item_dict = dict(top_items) item_dict.update(project_items) return {(self._to_unicode(item), self._to_unicode(val)) - for item, val in item_dict.iteritems()} + for item, val in item_dict.items()}
def ReadGitAliases(fname): """Read a git alias file. This is in the form used by git:
Python 3 requires this, and Python 2 allows it. Convert the code over to ensure compatibility with Python 3.
Signed-off-by: Simon Glass sjg@chromium.org ---
tools/binman/bsection.py | 2 +- tools/binman/control.py | 2 +- tools/binman/elf.py | 4 ++-- tools/binman/etype/gbb.py | 2 +- tools/binman/ftest.py | 2 +- 5 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/tools/binman/bsection.py b/tools/binman/bsection.py index be67ff957e..3ca0592fe1 100644 --- a/tools/binman/bsection.py +++ b/tools/binman/bsection.py @@ -253,7 +253,7 @@ class Section(object): """ for entry in self._entries.values(): offset_dict = entry.GetOffsets() - for name, info in offset_dict.iteritems(): + for name, info in offset_dict.items(): self._SetEntryOffsetSize(name, *info)
def PackEntries(self): diff --git a/tools/binman/control.py b/tools/binman/control.py index 8f7f906860..ce25eb5485 100644 --- a/tools/binman/control.py +++ b/tools/binman/control.py @@ -131,7 +131,7 @@ def Binman(options, args):
if options.image: skip = [] - for name, image in images.iteritems(): + for name, image in images.items(): if name not in options.image: del images[name] skip.append(name) diff --git a/tools/binman/elf.py b/tools/binman/elf.py index 97df8e32c5..828681d76d 100644 --- a/tools/binman/elf.py +++ b/tools/binman/elf.py @@ -59,7 +59,7 @@ def GetSymbols(fname, patterns): flags[1] == 'w')
# Sort dict by address - return OrderedDict(sorted(syms.iteritems(), key=lambda x: x[1].address)) + return OrderedDict(sorted(syms.items(), key=lambda x: x[1].address))
def GetSymbolAddress(fname, sym_name): """Get a value of a symbol from an ELF file @@ -98,7 +98,7 @@ def LookupAndWriteSymbols(elf_fname, entry, section): base = syms.get('__image_copy_start') if not base: return - for name, sym in syms.iteritems(): + for name, sym in syms.items(): if name.startswith('_binman'): msg = ("Section '%s': Symbol '%s'\n in entry '%s'" % (section.GetPath(), name, entry.GetPath())) diff --git a/tools/binman/etype/gbb.py b/tools/binman/etype/gbb.py index 8fe10f4713..a94c0fca9d 100644 --- a/tools/binman/etype/gbb.py +++ b/tools/binman/etype/gbb.py @@ -64,7 +64,7 @@ class Entry_gbb(Entry): self.gbb_flags = 0 flags_node = node.FindNode('flags') if flags_node: - for flag, value in gbb_flag_properties.iteritems(): + for flag, value in gbb_flag_properties.items(): if fdt_util.GetBool(flags_node, flag): self.gbb_flags |= value
diff --git a/tools/binman/ftest.py b/tools/binman/ftest.py index daea1ea138..7cf17526a7 100644 --- a/tools/binman/ftest.py +++ b/tools/binman/ftest.py @@ -214,7 +214,7 @@ class TestFunctional(unittest.TestCase): if verbosity is not None: args.append('-v%d' % verbosity) if entry_args: - for arg, value in entry_args.iteritems(): + for arg, value in entry_args.items(): args.append('-a%s=%s' % (arg, value)) if images: for image in images:
At present some files are opened in text mode despite containing binary data. This works on Python 2 but not always on Python 3, due to unicode problems. BC&D are not my favourite people. Adjust the affected open() statements to use binary mode.
Signed-off-by: Simon Glass sjg@chromium.org
---
tools/binman/ftest.py | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-)
diff --git a/tools/binman/ftest.py b/tools/binman/ftest.py index 7cf17526a7..1111852a40 100644 --- a/tools/binman/ftest.py +++ b/tools/binman/ftest.py @@ -119,11 +119,11 @@ class TestFunctional(unittest.TestCase): TestFunctional._MakeInputFile('refcode.bin', REFCODE_DATA)
# ELF file with a '_dt_ucode_base_size' symbol - with open(self.TestFile('u_boot_ucode_ptr')) as fd: + with open(self.TestFile('u_boot_ucode_ptr'), 'rb') as fd: TestFunctional._MakeInputFile('u-boot', fd.read())
# Intel flash descriptor file - with open(self.TestFile('descriptor.bin')) as fd: + with open(self.TestFile('descriptor.bin'), 'rb') as fd: TestFunctional._MakeInputFile('descriptor.bin', fd.read())
shutil.copytree(self.TestFile('files'), @@ -236,7 +236,7 @@ class TestFunctional(unittest.TestCase): """ tools.PrepareOutputDir(None) dtb = fdt_util.EnsureCompiled(self.TestFile(fname)) - with open(dtb) as fd: + with open(dtb, 'rb') as fd: data = fd.read() TestFunctional._MakeInputFile(outfile, data) tools.FinaliseOutputDir() @@ -317,7 +317,7 @@ class TestFunctional(unittest.TestCase): map_data = fd.read() else: map_data = None - with open(image_fname) as fd: + with open(image_fname, 'rb') as fd: return fd.read(), dtb_data, map_data, out_dtb_fname finally: # Put the test file back @@ -379,7 +379,7 @@ class TestFunctional(unittest.TestCase): Args: Filename of ELF file to use as SPL """ - with open(self.TestFile(src_fname)) as fd: + with open(self.TestFile(src_fname), 'rb') as fd: TestFunctional._MakeInputFile('spl/u-boot-spl', fd.read())
@classmethod @@ -541,7 +541,7 @@ class TestFunctional(unittest.TestCase): self.assertEqual(len(U_BOOT_DATA), image._size) fname = tools.GetOutputFilename('image1.bin') self.assertTrue(os.path.exists(fname)) - with open(fname) as fd: + with open(fname, 'rb') as fd: data = fd.read() self.assertEqual(U_BOOT_DATA, data)
@@ -549,7 +549,7 @@ class TestFunctional(unittest.TestCase): self.assertEqual(3 + len(U_BOOT_DATA) + 5, image._size) fname = tools.GetOutputFilename('image2.bin') self.assertTrue(os.path.exists(fname)) - with open(fname) as fd: + with open(fname, 'rb') as fd: data = fd.read() self.assertEqual(U_BOOT_DATA, data[3:7]) self.assertEqual(chr(0) * 3, data[:3]) @@ -970,7 +970,7 @@ class TestFunctional(unittest.TestCase): """Test that a U-Boot binary without the microcode symbol is detected""" # ELF file without a '_dt_ucode_base_size' symbol try: - with open(self.TestFile('u_boot_no_ucode_ptr')) as fd: + with open(self.TestFile('u_boot_no_ucode_ptr'), 'rb') as fd: TestFunctional._MakeInputFile('u-boot', fd.read())
with self.assertRaises(ValueError) as e: @@ -980,7 +980,7 @@ class TestFunctional(unittest.TestCase):
finally: # Put the original file back - with open(self.TestFile('u_boot_ucode_ptr')) as fd: + with open(self.TestFile('u_boot_ucode_ptr'), 'rb') as fd: TestFunctional._MakeInputFile('u-boot', fd.read())
def testMicrocodeNotInImage(self): @@ -993,7 +993,7 @@ class TestFunctional(unittest.TestCase):
def testWithoutMicrocode(self): """Test that we can cope with an image without microcode (e.g. qemu)""" - with open(self.TestFile('u_boot_no_ucode_ptr')) as fd: + with open(self.TestFile('u_boot_no_ucode_ptr'), 'rb') as fd: TestFunctional._MakeInputFile('u-boot', fd.read()) data, dtb, _, _ = self._DoReadFileDtb('044_x86_optional_ucode.dts', True)
@@ -1357,7 +1357,7 @@ class TestFunctional(unittest.TestCase): fname = pipe_list[0][-1] # Append our GBB data to the file, which will happen every time the # futility command is called. - with open(fname, 'a') as fd: + with open(fname, 'ab') as fd: fd.write(GBB_DATA) return command.CommandResult()
@@ -1431,7 +1431,7 @@ class TestFunctional(unittest.TestCase): def testTpl(self): """Test that an image with TPL and ots device tree can be created""" # ELF file with a '__bss_size' symbol - with open(self.TestFile('bss_data')) as fd: + with open(self.TestFile('bss_data'), 'rb') as fd: TestFunctional._MakeInputFile('tpl/u-boot-tpl', fd.read()) data = self._DoReadFile('078_u_boot_tpl.dts') self.assertEqual(U_BOOT_TPL_DATA + U_BOOT_TPL_DTB_DATA, data) @@ -1693,7 +1693,7 @@ class TestFunctional(unittest.TestCase): u-boot-tpl.dtb with the microcode removed the microcode """ - with open(self.TestFile('u_boot_ucode_ptr')) as fd: + with open(self.TestFile('u_boot_ucode_ptr'), 'rb') as fd: TestFunctional._MakeInputFile('tpl/u-boot-tpl', fd.read()) first, pos_and_size = self._RunMicrocodeTest('093_x86_tpl_ucode.dts', U_BOOT_TPL_NODTB_DATA) @@ -1748,14 +1748,14 @@ class TestFunctional(unittest.TestCase): def testElf(self): """Basic test of ELF entries""" self._SetupSplElf() - with open(self.TestFile('bss_data')) as fd: + with open(self.TestFile('bss_data'), 'rb') as fd: TestFunctional._MakeInputFile('-boot', fd.read()) data = self._DoReadFile('096_elf.dts')
def testElfStripg(self): """Basic test of ELF entries""" self._SetupSplElf() - with open(self.TestFile('bss_data')) as fd: + with open(self.TestFile('bss_data'), 'rb') as fd: TestFunctional._MakeInputFile('-boot', fd.read()) data = self._DoReadFile('097_elf_strip.dts')
Fix an open() statement to use binary mode, so that it works as expected with Python 3.
Signed-off-by: Simon Glass sjg@chromium.org ---
tools/dtoc/fdt.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tools/dtoc/fdt.py b/tools/dtoc/fdt.py index 9ad72f89ec..031b3a0084 100644 --- a/tools/dtoc/fdt.py +++ b/tools/dtoc/fdt.py @@ -469,7 +469,7 @@ class Fdt: if self._fname: self._fname = fdt_util.EnsureCompiled(self._fname)
- with open(self._fname) as fd: + with open(self._fname, 'rb') as fd: self._fdt_obj = libfdt.Fdt(fd.read())
@staticmethod
At present cros_subprocess and the tools library use a string to obtain stdout from a program. This works fine on Python 2. With Python 3 we end up with unicode errors in some cases. Fix this by providing a binary mode, which returns the data as bytes() instead of a string.
Signed-off-by: Simon Glass sjg@chromium.org ---
tools/patman/cros_subprocess.py | 15 +++++++++------ tools/patman/tools.py | 4 ++-- 2 files changed, 11 insertions(+), 8 deletions(-)
diff --git a/tools/patman/cros_subprocess.py b/tools/patman/cros_subprocess.py index 4de230ac63..39d4f21980 100644 --- a/tools/patman/cros_subprocess.py +++ b/tools/patman/cros_subprocess.py @@ -54,7 +54,7 @@ class Popen(subprocess.Popen): """
def __init__(self, args, stdin=None, stdout=PIPE_PTY, stderr=PIPE_PTY, - shell=False, cwd=None, env=None, **kwargs): + shell=False, cwd=None, env=None, binary=False, **kwargs): """Cut-down constructor
Args: @@ -72,6 +72,7 @@ class Popen(subprocess.Popen): """ stdout_pty = None stderr_pty = None + self.binary = binary
if stdout == PIPE_PTY: stdout_pty = pty.openpty() @@ -107,15 +108,17 @@ class Popen(subprocess.Popen): data: Data to convert, or None for ''
Returns: - Converted data to a unicode string + Converted data, normally as a unicode string, but this uses bytes + in binary mode """ if data is None: return '' else: - try: - data = data.decode('utf-8') - except UnicodeDecodeError: - data = data.decode('utf-8', 'ignore') + if not self.binary: + try: + data = data.decode('utf-8') + except UnicodeDecodeError: + data = data.decode('utf-8', 'ignore') return data
def CommunicateFilter(self, output): diff --git a/tools/patman/tools.py b/tools/patman/tools.py index bf099798e6..1df8f2ecd2 100644 --- a/tools/patman/tools.py +++ b/tools/patman/tools.py @@ -167,9 +167,9 @@ def PathHasFile(fname): return True return False
-def Run(name, *args): +def Run(name, *args, **kwargs): try: - return command.Run(name, *args, cwd=outdir, capture=True) + return command.Run(name, *args, cwd=outdir, capture=True, **kwargs) except: if not PathHasFile(name): msg = "Plesae install tool '%s'" % name
The lz4 utility inserts binary data in its output which cannot always be converted to unicode (nor should it be). Fix this by using the new binary mode for program output.
Signed-off-by: Simon Glass sjg@chromium.org ---
tools/binman/etype/blob.py | 2 +- tools/binman/ftest.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/tools/binman/etype/blob.py b/tools/binman/etype/blob.py index ae80bbee53..f56a1f8768 100644 --- a/tools/binman/etype/blob.py +++ b/tools/binman/etype/blob.py @@ -60,7 +60,7 @@ class Entry_blob(Entry): except AttributeError: data = lz4.compress(data) ''' - data = tools.Run('lz4', '-c', self._pathname) + data = tools.Run('lz4', '-c', self._pathname, binary=True) self.SetContents(data) return True
diff --git a/tools/binman/ftest.py b/tools/binman/ftest.py index 1111852a40..4d96933cb4 100644 --- a/tools/binman/ftest.py +++ b/tools/binman/ftest.py @@ -1557,7 +1557,7 @@ class TestFunctional(unittest.TestCase): out = os.path.join(self._indir, 'lz4.tmp') with open(out, 'wb') as fd: fd.write(data) - return tools.Run('lz4', '-dc', out) + return tools.Run('lz4', '-dc', out, binary=True) ''' try: orig = lz4.frame.decompress(data)
Drop this line which is not used or needed.
Signed-off-by: Simon Glass sjg@chromium.org ---
tools/binman/ftest.py | 1 - 1 file changed, 1 deletion(-)
diff --git a/tools/binman/ftest.py b/tools/binman/ftest.py index 4d96933cb4..e1fc9e8e9e 100644 --- a/tools/binman/ftest.py +++ b/tools/binman/ftest.py @@ -291,7 +291,6 @@ class TestFunctional(unittest.TestCase): # Use the compiled test file as the u-boot-dtb input if use_real_dtb: dtb_data = self._SetupDtb(fname) - infile = os.path.join(self._indir, 'u-boot.dtb')
# For testing purposes, make a copy of the DT for SPL and TPL. Add # a node indicating which it is, so aid verification.
The method of multiplying a character by a number works well for creating a repeated string in Python 2. But in Python 3 we need to use bytes() instead, to avoid unicode problems, since 'bytes' is no-longer just an alias of 'str'.
Create a function to handle this detail and call it from the relevant places in binman.
Signed-off-by: Simon Glass sjg@chromium.org ---
tools/binman/bsection.py | 2 +- tools/binman/elf_test.py | 2 +- tools/binman/etype/fill.py | 4 +-- tools/binman/etype/u_boot_spl_bss_pad.py | 2 +- tools/binman/ftest.py | 46 +++++++++++++----------- tools/patman/tools.py | 19 ++++++++++ 6 files changed, 50 insertions(+), 25 deletions(-)
diff --git a/tools/binman/bsection.py b/tools/binman/bsection.py index 3ca0592fe1..03dfa2f805 100644 --- a/tools/binman/bsection.py +++ b/tools/binman/bsection.py @@ -332,7 +332,7 @@ class Section(object):
def GetData(self): """Get the contents of the section""" - section_data = chr(self._pad_byte) * self._size + section_data = tools.GetBytes(self._pad_byte, self._size)
for entry in self._entries.values(): data = entry.GetData() diff --git a/tools/binman/elf_test.py b/tools/binman/elf_test.py index b68530c19b..3d55cb1946 100644 --- a/tools/binman/elf_test.py +++ b/tools/binman/elf_test.py @@ -122,7 +122,7 @@ class TestElf(unittest.TestCase): section = FakeSection(sym_value=None) elf_fname = os.path.join(binman_dir, 'test', 'u_boot_binman_syms') syms = elf.LookupAndWriteSymbols(elf_fname, entry, section) - self.assertEqual(chr(255) * 16 + 'a' * 4, entry.data) + self.assertEqual(tools.GetBytes(255, 16) + 'a' * 4, entry.data)
def testDebug(self): """Check that enabling debug in the elf module produced debug output""" diff --git a/tools/binman/etype/fill.py b/tools/binman/etype/fill.py index dcfe978a5b..68efe42ec0 100644 --- a/tools/binman/etype/fill.py +++ b/tools/binman/etype/fill.py @@ -5,7 +5,7 @@
from entry import Entry import fdt_util - +import tools
class Entry_fill(Entry): """An entry which is filled to a particular byte value @@ -28,5 +28,5 @@ class Entry_fill(Entry): self.fill_value = fdt_util.GetByte(self._node, 'fill-byte', 0)
def ObtainContents(self): - self.SetContents(chr(self.fill_value) * self.size) + self.SetContents(tools.GetBytes(self.fill_value, self.size)) return True diff --git a/tools/binman/etype/u_boot_spl_bss_pad.py b/tools/binman/etype/u_boot_spl_bss_pad.py index 00b7ac5004..66a296a6f8 100644 --- a/tools/binman/etype/u_boot_spl_bss_pad.py +++ b/tools/binman/etype/u_boot_spl_bss_pad.py @@ -38,5 +38,5 @@ class Entry_u_boot_spl_bss_pad(Entry_blob): bss_size = elf.GetSymbolAddress(fname, '__bss_size') if not bss_size: self.Raise('Expected __bss_size symbol in spl/u-boot-spl') - self.SetContents(chr(0) * bss_size) + self.SetContents(tools.GetBytes(0, bss_size)) return True diff --git a/tools/binman/ftest.py b/tools/binman/ftest.py index e1fc9e8e9e..971fade343 100644 --- a/tools/binman/ftest.py +++ b/tools/binman/ftest.py @@ -551,8 +551,8 @@ class TestFunctional(unittest.TestCase): with open(fname, 'rb') as fd: data = fd.read() self.assertEqual(U_BOOT_DATA, data[3:7]) - self.assertEqual(chr(0) * 3, data[:3]) - self.assertEqual(chr(0) * 5, data[7:]) + self.assertEqual(tools.GetBytes(0, 3), data[:3]) + self.assertEqual(tools.GetBytes(0, 5), data[7:])
def testBadAlign(self): """Test that an invalid alignment value is detected""" @@ -731,7 +731,8 @@ class TestFunctional(unittest.TestCase): """Test that the image pad byte can be specified""" self._SetupSplElf() data = self._DoReadFile('021_image_pad.dts') - self.assertEqual(U_BOOT_SPL_DATA + (chr(0xff) * 1) + U_BOOT_DATA, data) + self.assertEqual(U_BOOT_SPL_DATA + tools.GetBytes(0xff, 1) + + U_BOOT_DATA, data)
def testImageName(self): """Test that image files can be named""" @@ -754,8 +755,8 @@ class TestFunctional(unittest.TestCase): """Test that entries can be sorted""" self._SetupSplElf() data = self._DoReadFile('024_sorted.dts') - self.assertEqual(chr(0) * 1 + U_BOOT_SPL_DATA + chr(0) * 2 + - U_BOOT_DATA, data) + self.assertEqual(tools.GetBytes(0, 1) + U_BOOT_SPL_DATA + + tools.GetBytes(0, 2) + U_BOOT_DATA, data)
def testPackZeroOffset(self): """Test that an entry at offset 0 is not given a new offset""" @@ -797,8 +798,8 @@ class TestFunctional(unittest.TestCase): """Test that a basic x86 ROM can be created""" self._SetupSplElf() data = self._DoReadFile('029_x86-rom.dts') - self.assertEqual(U_BOOT_DATA + chr(0) * 7 + U_BOOT_SPL_DATA + - chr(0) * 2, data) + self.assertEqual(U_BOOT_DATA + tools.GetBytes(0, 7) + U_BOOT_SPL_DATA + + tools.GetBytes(0, 2), data)
def testPackX86RomMeNoDesc(self): """Test that an invalid Intel descriptor entry is detected""" @@ -1005,7 +1006,7 @@ class TestFunctional(unittest.TestCase):
used_len = len(U_BOOT_NODTB_DATA) + fdt_len third = data[used_len:] - self.assertEqual(chr(0) * (0x200 - used_len), third) + self.assertEqual(tools.GetBytes(0, 0x200 - used_len), third)
def testUnknownPosSize(self): """Test that microcode must be placed within the image""" @@ -1034,7 +1035,8 @@ class TestFunctional(unittest.TestCase): # ELF file with a '__bss_size' symbol self._SetupSplElf() data = self._DoReadFile('047_spl_bss_pad.dts') - self.assertEqual(U_BOOT_SPL_DATA + (chr(0) * 10) + U_BOOT_DATA, data) + self.assertEqual(U_BOOT_SPL_DATA + tools.GetBytes(0, 10) + U_BOOT_DATA, + data)
def testSplBssPadMissing(self): """Test that a missing symbol is detected""" @@ -1108,9 +1110,9 @@ class TestFunctional(unittest.TestCase): self._SetupSplElf('u_boot_binman_syms') data = self._DoReadFile('053_symbols.dts') sym_values = struct.pack('<LQL', 0x24 + 0, 0x24 + 24, 0x24 + 20) - expected = (sym_values + U_BOOT_SPL_DATA[16:] + chr(0xff) + - U_BOOT_DATA + - sym_values + U_BOOT_SPL_DATA[16:]) + expected = (sym_values + U_BOOT_SPL_DATA[16:] + + tools.GetBytes(0xff, 1) + U_BOOT_DATA + sym_values + + U_BOOT_SPL_DATA[16:]) self.assertEqual(expected, data)
def testPackUnitAddress(self): @@ -1280,8 +1282,8 @@ class TestFunctional(unittest.TestCase): } data, _, _, _ = self._DoReadFileDtb('066_text.dts', entry_args=entry_args) - expected = (TEXT_DATA + chr(0) * (8 - len(TEXT_DATA)) + TEXT_DATA2 + - TEXT_DATA3 + 'some text') + expected = (TEXT_DATA + tools.GetBytes(0, 8 - len(TEXT_DATA)) + + TEXT_DATA2 + TEXT_DATA3 + 'some text') self.assertEqual(expected, data)
def testEntryDocs(self): @@ -1340,7 +1342,7 @@ class TestFunctional(unittest.TestCase): def testFill(self): """Test for an fill entry type""" data = self._DoReadFile('069_fill.dts') - expected = 8 * chr(0xff) + 8 * chr(0) + expected = tools.GetBytes(0xff, 8) + tools.GetBytes(0, 8) self.assertEqual(expected, data)
def testFillNoSize(self): @@ -1370,7 +1372,8 @@ class TestFunctional(unittest.TestCase): data, _, _, _ = self._DoReadFileDtb('071_gbb.dts', entry_args=entry_args)
# Since futility - expected = GBB_DATA + GBB_DATA + 8 * chr(0) + (0x2180 - 16) * chr(0) + expected = (GBB_DATA + GBB_DATA + tools.GetBytes(0, 8) + + tools.GetBytes(0, 0x2180 - 16)) self.assertEqual(expected, data)
def testGbbTooSmall(self): @@ -1445,7 +1448,7 @@ class TestFunctional(unittest.TestCase): def testFillZero(self): """Test for an fill entry type with a size of 0""" data = self._DoReadFile('080_fill_empty.dts') - self.assertEqual(chr(0) * 16, data) + self.assertEqual(tools.GetBytes(0, 16), data)
def testTextMissing(self): """Test for a text entry type where there is no text""" @@ -1796,9 +1799,12 @@ class TestFunctional(unittest.TestCase): 0000002c 00000000 00000004 u-boot ''', map_data) self.assertEqual(data, - 4 * chr(0x26) + U_BOOT_DATA + 12 * chr(0x21) + - 4 * chr(0x26) + U_BOOT_DATA + 12 * chr(0x61) + - 4 * chr(0x26) + U_BOOT_DATA + 8 * chr(0x26)) + tools.GetBytes(0x26, 4) + U_BOOT_DATA + + tools.GetBytes(0x21, 12) + + tools.GetBytes(0x26, 4) + U_BOOT_DATA + + tools.GetBytes(0x61, 12) + + tools.GetBytes(0x26, 4) + U_BOOT_DATA + + tools.GetBytes(0x26, 8))
if __name__ == "__main__": diff --git a/tools/patman/tools.py b/tools/patman/tools.py index 1df8f2ecd2..0ad0fb9705 100644 --- a/tools/patman/tools.py +++ b/tools/patman/tools.py @@ -7,6 +7,7 @@ import command import glob import os import shutil +import sys import tempfile
import tout @@ -239,3 +240,21 @@ def WriteFile(fname, data): #(fname, len(data), len(data))) with open(Filename(fname), 'wb') as fd: fd.write(data) + +def GetBytes(byte, size): + """Get a string of bytes of a given size + + This handles the unfortunate different between Python 2 and Python 2. + + Args: + byte: Numeric byte value to use + size: Size of bytes/string to return + + Returns: + A bytes type with 'byte' repeated 'size' times + """ + if sys.version_info[0] >= 3: + data = bytes([byte]) * size + else: + data = chr(byte) * size + return data
This script attempts to create a git commit which removes a single board. It is quite fallible and everything it does needs checking. But it can help speed up the process.
Signed-off-by: Simon Glass sjg@chromium.org ---
tools/rmboard.py | 150 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 150 insertions(+) create mode 100755 tools/rmboard.py
diff --git a/tools/rmboard.py b/tools/rmboard.py new file mode 100755 index 0000000000..17952f795d --- /dev/null +++ b/tools/rmboard.py @@ -0,0 +1,150 @@ +#! /usr/bin/python +# SPDX-License-Identifier: GPL-2.0+ +# Copyright 2019 Google LLC +# + +""" +Script to remove boards + +Usage: + rmboard.py <board_name>... + +A single commit is created for each board removed. + +Some boards may depend on files provided by another and this will cause +problems, generally the removal of files which should not be removed. + +This script works by: + - Looking through the MAINTAINERS files which mention a board to find out + what files the board uses + - Looking through the Kconfig files which mention a board to find one that + needs to have material removed + +Search for ## to update the commit message manually. +""" + +from __future__ import print_function + +import glob +import os +import re +import sys + +# Bring in the patman libraries +our_path = os.path.dirname(os.path.realpath(__file__)) +sys.path.append(os.path.join(our_path, '../tools/patman')) + +import command + +def rm_kconfig_include(path): + """Remove a path from Kconfig files + + This function finds the given path in a 'source' statement in a Kconfig + file and removes that line from the file. This is needed because the path + is going to be removed, so any reference to it will cause a problem with + Kconfig parsing. + + The changes are made locally and then added to the git staging area. + + Args: + path: Path to search for and remove + """ + cmd = ['git', 'grep', path] + stdout = command.RunPipe([cmd], capture=True, raise_on_error=False).stdout + if not stdout: + return + fname = stdout.split(':')[0] + + print("Fixing up '%s' to remove reference to '%s'" % (fname, path)) + cmd = ['sed', '-i', '|%s|d' % path, fname] + stdout = command.RunPipe([cmd], capture=True).stdout + + cmd = ['git', 'add', fname] + stdout = command.RunPipe([cmd], capture=True).stdout + +def rm_board(board): + """Create a commit which removes a single board + + This looks up the MAINTAINERS file to file files that need to be removed, + then removes pieces from the Kconfig files that mention the board. + + + Args: + board: Board name to remove + """ + + # Find all MAINTAINERS and Kconfig files which mention the board + cmd = ['git', 'grep', '-l', board] + stdout = command.RunPipe([cmd], capture=True).stdout + maintain = [] + kconfig = [] + for line in stdout.splitlines(): + line = line.strip() + if 'MAINTAINERS' in line: + if line not in maintain: + maintain.append(line) + elif 'Kconfig' in line: + kconfig.append(line) + paths = [] + cc = [] + + # Look through the MAINTAINERS file to find things to remove + for fname in maintain: + with open(fname) as fd: + for line in fd: + line = line.strip() + fields = re.split('[ \t]', line, 1) + if len(fields) == 2: + if fields[0] == 'M:': + cc.append(fields[1]) + elif fields[0] == 'F:': + paths.append(fields[1].strip()) + + # Expannd any wildcards in the MAINTAINRERS file + real = [] + for path in paths: + if path[-1] == '/': + path = path[:-1] + if '*' in path: + globbed = glob.glob(path) + print("Expanded '%s' to '%s'" % (path, globbed)) + real += globbed + else: + real.append(path) + + # Search for Kconfig files in the resulting list. Remove any 'source' lines + # which reference Kconfig files we want to remove + for path in real: + cmd = ['find', path] + stdout = (command.RunPipe([cmd], capture=True, raise_on_error=False). + stdout) + for fname in stdout.splitlines(): + if fname.endswith('Kconfig'): + rm_kconfig_include(fname) + + # Remove unwanted files + cmd = ['git', 'rm', '-r'] + real + stdout = command.RunPipe([cmd], capture=True).stdout + + ## Change the messages as needed + msg = '''arm: Remove %s board + +This board has not been converted to CONFIG_DM_MMC by the deadline. +Remove it. + +''' % board + for name in cc: + msg += 'Patch-cc: %s\n' % name + + # Create the commit + cmd = ['git', 'commit', '-s', '-m', msg] + stdout = command.RunPipe([cmd], capture=True).stdout + + # Check if the board is mentioned anywhere else. The user will need to deal + # with this + cmd = ['git', 'grep', '-il', board] + print(command.RunPipe([cmd], capture=True, raise_on_error=False).stdout) + print(' '.join(cmd)) + +for board in sys.argv[1:]: + rm_board(board)
On Wed, May 15, 2019 at 9:54 AM Simon Glass sjg@chromium.org wrote:
This script attempts to create a git commit which removes a single board. It is quite fallible and everything it does needs checking. But it can help speed up the process.
Signed-off-by: Simon Glass sjg@chromium.org
Did you mean to include this file in the series. It seems unrelated to the other patman/binman changes.
tools/rmboard.py | 150 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 150 insertions(+) create mode 100755 tools/rmboard.py
diff --git a/tools/rmboard.py b/tools/rmboard.py new file mode 100755 index 0000000000..17952f795d --- /dev/null +++ b/tools/rmboard.py @@ -0,0 +1,150 @@ +#! /usr/bin/python +# SPDX-License-Identifier: GPL-2.0+ +# Copyright 2019 Google LLC +#
+""" +Script to remove boards
+Usage:
- rmboard.py <board_name>...
+A single commit is created for each board removed.
+Some boards may depend on files provided by another and this will cause +problems, generally the removal of files which should not be removed.
+This script works by:
- Looking through the MAINTAINERS files which mention a board to find out
what files the board uses
- Looking through the Kconfig files which mention a board to find one that
needs to have material removed+Search for ## to update the commit message manually. +"""
+from __future__ import print_function
+import glob +import os +import re +import sys
+# Bring in the patman libraries +our_path = os.path.dirname(os.path.realpath(__file__)) +sys.path.append(os.path.join(our_path, '../tools/patman'))
+import command
+def rm_kconfig_include(path):
- """Remove a path from Kconfig files
- This function finds the given path in a 'source' statement in a Kconfig
- file and removes that line from the file. This is needed because the path
- is going to be removed, so any reference to it will cause a problem with
- Kconfig parsing.
- The changes are made locally and then added to the git staging area.
- Args:
path: Path to search for and remove- """
- cmd = ['git', 'grep', path]
- stdout = command.RunPipe([cmd], capture=True, raise_on_error=False).stdout
Is there any specific advantage of using patman/command.py instead of python's subprocess module directly?
- if not stdout:
return- fname = stdout.split(':')[0]
- print("Fixing up '%s' to remove reference to '%s'" % (fname, path))
- cmd = ['sed', '-i', '|%s|d' % path, fname]
- stdout = command.RunPipe([cmd], capture=True).stdout
- cmd = ['git', 'add', fname]
- stdout = command.RunPipe([cmd], capture=True).stdout
+def rm_board(board):
- """Create a commit which removes a single board
- This looks up the MAINTAINERS file to file files that need to be removed,
- then removes pieces from the Kconfig files that mention the board.
- Args:
board: Board name to remove- """
- # Find all MAINTAINERS and Kconfig files which mention the board
- cmd = ['git', 'grep', '-l', board]
- stdout = command.RunPipe([cmd], capture=True).stdout
- maintain = []
- kconfig = []
- for line in stdout.splitlines():
line = line.strip()if 'MAINTAINERS' in line:if line not in maintain:maintain.append(line)elif 'Kconfig' in line:kconfig.append(line)- paths = []
- cc = []
- # Look through the MAINTAINERS file to find things to remove
- for fname in maintain:
with open(fname) as fd:for line in fd:line = line.strip()fields = re.split('[ \t]', line, 1)if len(fields) == 2:if fields[0] == 'M:':cc.append(fields[1])elif fields[0] == 'F:':paths.append(fields[1].strip())- # Expannd any wildcards in the MAINTAINRERS file
Couple of typos s/Expannd/Expand/ and s/MAINTAINRERS/MAINTAINERS/
- real = []
- for path in paths:
if path[-1] == '/':path = path[:-1]if '*' in path:globbed = glob.glob(path)print("Expanded '%s' to '%s'" % (path, globbed))real += globbedelse:real.append(path)- # Search for Kconfig files in the resulting list. Remove any 'source' lines
- # which reference Kconfig files we want to remove
- for path in real:
cmd = ['find', path]stdout = (command.RunPipe([cmd], capture=True, raise_on_error=False).stdout)for fname in stdout.splitlines():if fname.endswith('Kconfig'):rm_kconfig_include(fname)- # Remove unwanted files
- cmd = ['git', 'rm', '-r'] + real
- stdout = command.RunPipe([cmd], capture=True).stdout
- ## Change the messages as needed
- msg = '''arm: Remove %s board
+This board has not been converted to CONFIG_DM_MMC by the deadline. +Remove it.
+''' % board
- for name in cc:
msg += 'Patch-cc: %s\n' % name- # Create the commit
- cmd = ['git', 'commit', '-s', '-m', msg]
- stdout = command.RunPipe([cmd], capture=True).stdout
- # Check if the board is mentioned anywhere else. The user will need to deal
- # with this
- cmd = ['git', 'grep', '-il', board]
- print(command.RunPipe([cmd], capture=True, raise_on_error=False).stdout)
- print(' '.join(cmd))
+for board in sys.argv[1:]:
- rm_board(board)
-- 2.21.0.1020.gf2820cf01a-goog
Hi Chris,
On Wed, 15 May 2019 at 01:54, Chris Packham judge.packham@gmail.com wrote:
On Wed, May 15, 2019 at 9:54 AM Simon Glass sjg@chromium.org wrote:
This script attempts to create a git commit which removes a single board. It is quite fallible and everything it does needs checking. But it can help speed up the process.
Signed-off-by: Simon Glass sjg@chromium.org
Did you mean to include this file in the series. It seems unrelated to the other patman/binman changes.
Well I originally had it as part of the series to drop/mangle boards that have not been migrated. But I decided it might fit better here since it was Python.
I'll send it again as a separate tool.
tools/rmboard.py | 150 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 150 insertions(+) create mode 100755 tools/rmboard.py
diff --git a/tools/rmboard.py b/tools/rmboard.py new file mode 100755 index 0000000000..17952f795d --- /dev/null +++ b/tools/rmboard.py @@ -0,0 +1,150 @@ +#! /usr/bin/python +# SPDX-License-Identifier: GPL-2.0+ +# Copyright 2019 Google LLC +#
+""" +Script to remove boards
+Usage:
- rmboard.py <board_name>...
+A single commit is created for each board removed.
+Some boards may depend on files provided by another and this will cause +problems, generally the removal of files which should not be removed.
+This script works by:
- Looking through the MAINTAINERS files which mention a board to find out
what files the board uses
- Looking through the Kconfig files which mention a board to find one that
needs to have material removed+Search for ## to update the commit message manually. +"""
+from __future__ import print_function
+import glob +import os +import re +import sys
+# Bring in the patman libraries +our_path = os.path.dirname(os.path.realpath(__file__)) +sys.path.append(os.path.join(our_path, '../tools/patman'))
+import command
+def rm_kconfig_include(path):
- """Remove a path from Kconfig files
- This function finds the given path in a 'source' statement in a Kconfig
- file and removes that line from the file. This is needed because the path
- is going to be removed, so any reference to it will cause a problem with
- Kconfig parsing.
- The changes are made locally and then added to the git staging area.
- Args:
path: Path to search for and remove- """
- cmd = ['git', 'grep', path]
- stdout = command.RunPipe([cmd], capture=True, raise_on_error=False).stdout
Is there any specific advantage of using patman/command.py instead of python's subprocess module directly?
Just for consistency with the rest of the code.
The only difference really is that 'command' allows output to be collected while the command is running rather than hanging the process until the command exits. In this case the difference doesn't matter though. since the command is pretty quick and we just wait for all the output to come.
So, mostly just for consistency.
- if not stdout:
return- fname = stdout.split(':')[0]
- print("Fixing up '%s' to remove reference to '%s'" % (fname, path))
- cmd = ['sed', '-i', '|%s|d' % path, fname]
- stdout = command.RunPipe([cmd], capture=True).stdout
- cmd = ['git', 'add', fname]
- stdout = command.RunPipe([cmd], capture=True).stdout
+def rm_board(board):
- """Create a commit which removes a single board
- This looks up the MAINTAINERS file to file files that need to be removed,
- then removes pieces from the Kconfig files that mention the board.
- Args:
board: Board name to remove- """
- # Find all MAINTAINERS and Kconfig files which mention the board
- cmd = ['git', 'grep', '-l', board]
- stdout = command.RunPipe([cmd], capture=True).stdout
- maintain = []
- kconfig = []
- for line in stdout.splitlines():
line = line.strip()if 'MAINTAINERS' in line:if line not in maintain:maintain.append(line)elif 'Kconfig' in line:kconfig.append(line)- paths = []
- cc = []
- # Look through the MAINTAINERS file to find things to remove
- for fname in maintain:
with open(fname) as fd:for line in fd:line = line.strip()fields = re.split('[ \t]', line, 1)if len(fields) == 2:if fields[0] == 'M:':cc.append(fields[1])elif fields[0] == 'F:':paths.append(fields[1].strip())- # Expannd any wildcards in the MAINTAINRERS file
Couple of typos s/Expannd/Expand/ and s/MAINTAINRERS/MAINTAINERS/
OK will fix, thanks.
Regards, Simon
With Python 3 this class has moved. Update the code to handle both cases.
Signed-off-by: Simon Glass sjg@chromium.org ---
tools/patman/func_test.py | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/tools/patman/func_test.py b/tools/patman/func_test.py index 31481157fc..2c4392dbc1 100644 --- a/tools/patman/func_test.py +++ b/tools/patman/func_test.py @@ -12,15 +12,20 @@ import sys import tempfile import unittest
+try: + from StringIO import StringIO +except ImportError: + from io import StringIO + import gitutil import patchstream import settings +import tools
@contextlib.contextmanager def capture(): import sys - from cStringIO import StringIO oldout,olderr = sys.stdout, sys.stderr try: out=[StringIO(), StringIO()]
Create helper functions in the tools module to deal with the differences between unicode in Python 2 (where we use the 'unicode' type) and Python 3 (where we use the 'str' type).
Signed-off-by: Simon Glass sjg@chromium.org ---
tools/patman/gitutil.py | 4 ++-- tools/patman/series.py | 3 ++- tools/patman/settings.py | 16 +++------------- tools/patman/tools.py | 32 ++++++++++++++++++++++++++++++++ 4 files changed, 39 insertions(+), 16 deletions(-)
diff --git a/tools/patman/gitutil.py b/tools/patman/gitutil.py index 11aeb73b74..fb9e67c0f0 100644 --- a/tools/patman/gitutil.py +++ b/tools/patman/gitutil.py @@ -12,6 +12,7 @@ import terminal
import checkpatch import settings +import tools
# True to use --no-decorate - we check this in Setup() use_no_decorate = True @@ -325,9 +326,8 @@ def BuildEmailList(in_list, tag=None, alias=None, raise_on_error=True): raw += LookupEmail(item, alias, raise_on_error=raise_on_error) result = [] for item in raw: + item = tools.FromUnicode(item) if not item in result: - if type(item) == unicode: - item = item.encode('utf-8') result.append(item) if tag: return ['%s %s%s%s' % (tag, quote, email, quote) for email in result] diff --git a/tools/patman/series.py b/tools/patman/series.py index 2735afaf88..0b71a896cb 100644 --- a/tools/patman/series.py +++ b/tools/patman/series.py @@ -11,6 +11,7 @@ import get_maintainer import gitutil import settings import terminal +import tools
# Series-xxx tags that we understand valid_series = ['to', 'cc', 'version', 'changes', 'prefix', 'notes', 'name', @@ -249,7 +250,7 @@ class Series(dict): cover_cc = gitutil.BuildEmailList(self.get('cover_cc', '')) cover_cc = [m.encode('utf-8') if type(m) != str else m for m in cover_cc] - cc_list = ', '.join([x.decode('utf-8') + cc_list = ', '.join([tools.ToUnicode(x) for x in set(cover_cc + all_ccs)]) print(cover_fname, cc_list.encode('utf-8'), file=fd)
diff --git a/tools/patman/settings.py b/tools/patman/settings.py index 07bf6a6ea4..b080136d88 100644 --- a/tools/patman/settings.py +++ b/tools/patman/settings.py @@ -14,6 +14,7 @@ import re
import command import gitutil +import tools
"""Default settings per-project.
@@ -99,17 +100,6 @@ class _ProjectConfigParser(ConfigParser.SafeConfigParser): for setting_name, setting_value in project_defaults.items(): self.set(project_settings, setting_name, setting_value)
- def _to_unicode(self, val): - """Make sure a value is of type 'unicode' - - Args: - val: string or unicode object - - Returns: - unicode version of val - """ - return val if isinstance(val, unicode) else val.decode('utf-8') - def get(self, section, option, *args, **kwargs): """Extend SafeConfigParser to try project_section before section.
@@ -127,7 +117,7 @@ class _ProjectConfigParser(ConfigParser.SafeConfigParser): val = ConfigParser.SafeConfigParser.get( self, section, option, *args, **kwargs ) - return self._to_unicode(val) + return tools.ToUnicode(val)
def items(self, section, *args, **kwargs): """Extend SafeConfigParser to add project_section to section. @@ -162,7 +152,7 @@ class _ProjectConfigParser(ConfigParser.SafeConfigParser):
item_dict = dict(top_items) item_dict.update(project_items) - return {(self._to_unicode(item), self._to_unicode(val)) + return {(tools.ToUnicode(item), tools.ToUnicode(val)) for item, val in item_dict.items()}
def ReadGitAliases(fname): diff --git a/tools/patman/tools.py b/tools/patman/tools.py index 0ad0fb9705..7e6a45a3b0 100644 --- a/tools/patman/tools.py +++ b/tools/patman/tools.py @@ -258,3 +258,35 @@ def GetBytes(byte, size): else: data = chr(byte) * size return data + +def ToUnicode(val): + """Make sure a value is a unicode string + + This allows some amount of compatibility between Python 2 and Python3. For + the former, it returns a unicode object. + + Args: + val: string or unicode object + + Returns: + unicode version of val + """ + if sys.version_info[0] >= 3: + return val + return val if isinstance(val, unicode) else val.decode('utf-8') + +def FromUnicode(val): + """Make sure a value is a non-unicode string + + This allows some amount of compatibility between Python 2 and Python3. For + the former, it converts a unicode object to a string. + + Args: + val: string or unicode object + + Returns: + non-unicode version of val + """ + if sys.version_info[0] >= 3: + return val + return val if isinstance(val, str) else val.encode('utf-8')
We use sets to produce the list of To and Cc lines for a series. This does not result in stable ordering of the recipients. Sort each list to ensure that the output is repeatable. This is necessary for tests.
Signed-off-by: Simon Glass sjg@chromium.org ---
tools/patman/func_test.py | 12 ++++++------ tools/patman/series.py | 12 ++++++------ 2 files changed, 12 insertions(+), 12 deletions(-)
diff --git a/tools/patman/func_test.py b/tools/patman/func_test.py index 2c4392dbc1..f7d5ad68f6 100644 --- a/tools/patman/func_test.py +++ b/tools/patman/func_test.py @@ -183,10 +183,10 @@ class TestFunctional(unittest.TestCase): self.assertEqual('Prefix:\t RFC', lines[line + 3]) self.assertEqual('Cover: 4 lines', lines[line + 4]) line += 5 - self.assertEqual(' Cc: %s' % mel.encode('utf-8'), lines[line + 0]) - self.assertEqual(' Cc: %s' % rick, lines[line + 1]) - self.assertEqual(' Cc: %s' % fred, lines[line + 2]) - self.assertEqual(' Cc: %s' % ed.encode('utf-8'), lines[line + 3]) + self.assertEqual(' Cc: %s' % fred, lines[line + 0]) + self.assertEqual(' Cc: %s' % ed.encode('utf-8'), lines[line + 1]) + self.assertEqual(' Cc: %s' % mel.encode('utf-8'), lines[line + 2]) + self.assertEqual(' Cc: %s' % rick, lines[line + 3]) expected = ('Git command: git send-email --annotate ' '--in-reply-to="%s" --to "u-boot@lists.denx.de" ' '--cc "%s" --cc-cmd "%s --cc-cmd %s" %s %s' @@ -197,8 +197,8 @@ class TestFunctional(unittest.TestCase):
self.assertEqual(('%s %s, %s' % (args[0], rick, stefan)) .encode('utf-8'), cc_lines[0]) - self.assertEqual(('%s %s, %s, %s, %s' % (args[1], fred, rick, stefan, - ed)).encode('utf-8'), cc_lines[1]) + self.assertEqual(('%s %s, %s, %s, %s' % (args[1], fred, ed, rick, + stefan)).encode('utf-8'), cc_lines[1])
expected = ''' This is a test of how the cover diff --git a/tools/patman/series.py b/tools/patman/series.py index 0b71a896cb..bbb30d849e 100644 --- a/tools/patman/series.py +++ b/tools/patman/series.py @@ -115,16 +115,16 @@ class Series(dict): commit = self.commits[upto] print(col.Color(col.GREEN, ' %s' % args[upto])) cc_list = list(self._generated_cc[commit.patch]) - for email in set(cc_list) - to_set - cc_set: + for email in sorted(set(cc_list) - to_set - cc_set): if email == None: email = col.Color(col.YELLOW, "<alias '%s' not found>" % tag) if email: print(' Cc: ', email) print - for item in to_set: + for item in sorted(to_set): print('To:\t ', item) - for item in cc_set - to_set: + for item in sorted(cc_set - to_set): print('Cc:\t ', item) print('Version: ', self.get('version')) print('Prefix:\t ', self.get('prefix')) @@ -132,7 +132,7 @@ class Series(dict): print('Cover: %d lines' % len(self.cover)) cover_cc = gitutil.BuildEmailList(self.get('cover_cc', '')) all_ccs = itertools.chain(cover_cc, *self._generated_cc.values()) - for email in set(all_ccs) - to_set - cc_set: + for email in sorted(set(all_ccs) - to_set - cc_set): print(' Cc: ', email) if cmd: print('Git command: %s' % cmd) @@ -243,7 +243,7 @@ class Series(dict): if limit is not None: cc = cc[:limit] all_ccs += cc - print(commit.patch, ', '.join(set(cc)), file=fd) + print(commit.patch, ', '.join(sorted(set(cc))), file=fd) self._generated_cc[commit.patch] = cc
if cover_fname: @@ -251,7 +251,7 @@ class Series(dict): cover_cc = [m.encode('utf-8') if type(m) != str else m for m in cover_cc] cc_list = ', '.join([tools.ToUnicode(x) - for x in set(cover_cc + all_ccs)]) + for x in sorted(set(cover_cc + all_ccs))]) print(cover_fname, cc_list.encode('utf-8'), file=fd)
fd.close()
The unicode type does not exist in Python 3 and when displaying strings they do not have the 'u' prefix. Adjusts the settings unit tests to deal with this difference, by converting the comparison value to a string, thus dropping the 'u'.
Signed-off-by: Simon Glass sjg@chromium.org ---
tools/patman/settings.py | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)
diff --git a/tools/patman/settings.py b/tools/patman/settings.py index b080136d88..c98911d522 100644 --- a/tools/patman/settings.py +++ b/tools/patman/settings.py @@ -58,26 +58,26 @@ class _ProjectConfigParser(ConfigParser.SafeConfigParser): # Check to make sure that bogus project gets general alias. >>> config = _ProjectConfigParser("zzz") >>> config.readfp(StringIO(sample_config)) - >>> config.get("alias", "enemies") - u'Evil evil@example.com' + >>> str(config.get("alias", "enemies")) + 'Evil evil@example.com'
# Check to make sure that alias gets overridden by project. >>> config = _ProjectConfigParser("sm") >>> config.readfp(StringIO(sample_config)) - >>> config.get("alias", "enemies") - u'Green G. ugly@example.com' + >>> str(config.get("alias", "enemies")) + 'Green G. ugly@example.com'
# Check to make sure that settings get merged with project. >>> config = _ProjectConfigParser("linux") >>> config.readfp(StringIO(sample_config)) - >>> sorted(config.items("settings")) - [(u'am_hero', u'True'), (u'process_tags', u'False')] + >>> sorted((str(a), str(b)) for (a, b) in config.items("settings")) + [('am_hero', 'True'), ('process_tags', 'False')]
# Check to make sure that settings works with unknown project. >>> config = _ProjectConfigParser("unknown") >>> config.readfp(StringIO(sample_config)) - >>> sorted(config.items("settings")) - [(u'am_hero', u'True')] + >>> sorted((str(a), str(b)) for (a, b) in config.items("settings")) + [('am_hero', 'True')] """ def __init__(self, project_name): """Construct _ProjectConfigParser.
Change the code so that it works on both Python 2 and Python 3. This works by using unicode instead of latin1 for the test input, and ensuring that the output is converted to a string rather than a unicode object on Python 2.
Signed-off-by: Simon Glass sjg@chromium.org ---
tools/patman/func_test.py | 25 ++++++++++++++----------- 1 file changed, 14 insertions(+), 11 deletions(-)
diff --git a/tools/patman/func_test.py b/tools/patman/func_test.py index f7d5ad68f6..50a2741439 100644 --- a/tools/patman/func_test.py +++ b/tools/patman/func_test.py @@ -129,10 +129,10 @@ class TestFunctional(unittest.TestCase): """ process_tags = True ignore_bad_tags = True - stefan = u'Stefan Brüns stefan.bruens@rwth-aachen.de' + stefan = b'Stefan Br\xc3\xbcns stefan.bruens@rwth-aachen.de'.decode('utf-8') rick = 'Richard III richard@palace.gov' - mel = u'Lord Mëlchett clergy@palace.gov' - ed = u'Lond Edmund Blackaddër <weasel@blackadder.org' + mel = b'Lord M\xc3\xablchett clergy@palace.gov'.decode('utf-8') + ed = b'Lond Edmund Blackadd\xc3\xabr <weasel@blackadder.org'.decode('utf-8') fred = 'Fred Bloggs f.bloggs@napier.net' add_maintainers = [stefan, rick] dry_run = True @@ -178,27 +178,30 @@ class TestFunctional(unittest.TestCase): while 'Cc:' in lines[line]: line += 1 self.assertEqual('To: u-boot@lists.denx.de', lines[line]) - self.assertEqual('Cc: %s' % stefan.encode('utf-8'), lines[line + 1]) + self.assertEqual('Cc: %s' % tools.FromUnicode(stefan), + lines[line + 1]) self.assertEqual('Version: 3', lines[line + 2]) self.assertEqual('Prefix:\t RFC', lines[line + 3]) self.assertEqual('Cover: 4 lines', lines[line + 4]) line += 5 self.assertEqual(' Cc: %s' % fred, lines[line + 0]) - self.assertEqual(' Cc: %s' % ed.encode('utf-8'), lines[line + 1]) - self.assertEqual(' Cc: %s' % mel.encode('utf-8'), lines[line + 2]) + self.assertEqual(' Cc: %s' % tools.FromUnicode(ed), + lines[line + 1]) + self.assertEqual(' Cc: %s' % tools.FromUnicode(mel), + lines[line + 2]) self.assertEqual(' Cc: %s' % rick, lines[line + 3]) expected = ('Git command: git send-email --annotate ' '--in-reply-to="%s" --to "u-boot@lists.denx.de" ' '--cc "%s" --cc-cmd "%s --cc-cmd %s" %s %s' % (in_reply_to, stefan, sys.argv[0], cc_file, cover_fname, - ' '.join(args))).encode('utf-8') + ' '.join(args))) line += 4 - self.assertEqual(expected, lines[line]) + self.assertEqual(expected, tools.ToUnicode(lines[line]))
- self.assertEqual(('%s %s, %s' % (args[0], rick, stefan)) - .encode('utf-8'), cc_lines[0]) + self.assertEqual(('%s %s, %s' % (args[0], rick, stefan)), + tools.ToUnicode(cc_lines[0])) self.assertEqual(('%s %s, %s, %s, %s' % (args[1], fred, ed, rick, - stefan)).encode('utf-8'), cc_lines[1]) + stefan)), tools.ToUnicode(cc_lines[1]))
expected = ''' This is a test of how the cover
Use the new functions in the tools module to handle conversion.
Signed-off-by: Simon Glass sjg@chromium.org ---
tools/patman/gitutil.py | 4 +--- tools/patman/series.py | 5 ++--- 2 files changed, 3 insertions(+), 6 deletions(-)
diff --git a/tools/patman/gitutil.py b/tools/patman/gitutil.py index fb9e67c0f0..dce7fa25b6 100644 --- a/tools/patman/gitutil.py +++ b/tools/patman/gitutil.py @@ -412,9 +412,7 @@ def EmailPatches(series, cover_fname, args, dry_run, raise_on_error, cc_fname, if smtp_server: cmd.append('--smtp-server=%s' % smtp_server) if in_reply_to: - if type(in_reply_to) != str: - in_reply_to = in_reply_to.encode('utf-8') - cmd.append('--in-reply-to="%s"' % in_reply_to) + cmd.append('--in-reply-to="%s"' % tools.FromUnicode(in_reply_to)) if thread: cmd.append('--thread')
diff --git a/tools/patman/series.py b/tools/patman/series.py index bbb30d849e..67103f03e6 100644 --- a/tools/patman/series.py +++ b/tools/patman/series.py @@ -239,7 +239,7 @@ class Series(dict): for x in set(cc) & set(settings.bounces): print(col.Color(col.YELLOW, 'Skipping "%s"' % x)) cc = set(cc) - set(settings.bounces) - cc = [m.encode('utf-8') if type(m) != str else m for m in cc] + cc = [tools.FromUnicode(m) for m in cc] if limit is not None: cc = cc[:limit] all_ccs += cc @@ -248,8 +248,7 @@ class Series(dict):
if cover_fname: cover_cc = gitutil.BuildEmailList(self.get('cover_cc', '')) - cover_cc = [m.encode('utf-8') if type(m) != str else m - for m in cover_cc] + cover_cc = [tools.FromUnicode(m) for m in cover_cc] cc_list = ', '.join([tools.ToUnicode(x) for x in sorted(set(cover_cc + all_ccs))]) print(cover_fname, cc_list.encode('utf-8'), file=fd)
Update the shebang to allow either Python 2 or Python 3.
Signed-off-by: Simon Glass sjg@chromium.org ---
tools/patman/patman.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tools/patman/patman.py b/tools/patman/patman.py index 27a2febf70..9605a36eff 100755 --- a/tools/patman/patman.py +++ b/tools/patman/patman.py @@ -1,4 +1,4 @@ -#!/usr/bin/env python2 +#!/usr/bin/env python # SPDX-License-Identifier: GPL-2.0+ # # Copyright (c) 2011 The Chromium OS Authors.
On Wed, May 15, 2019 at 9:54 AM Simon Glass sjg@chromium.org wrote:
This series updates patman to support Python 3:
- Avoid using 'unicode' type directly
- Use items() instead of iteritems()
- Make sure file I/O uses binary mode where necessary
- Change print statements to functions
- Use the built-in set() class
- Fix up generation of repeated bytes
A few patches for binman are included, but this still requires Python 2.
Couple of comments on 14/21 but the rest of the series
Reviewed-by: Chris Packham judge.packham@gmail.com
Simon Glass (21): patman: Update cros_subprocess to use bytearray patman: Convert print statements to Python 3 binman: Convert print statements to Python 3 binman: Don't show errors for failed tests binman: Remove use of Set() patman: Use items() instead of iteritems() binman: Use items() instead of iteritems() tools: binman: Open all binary files in binary mode tools: dtoc: Open all binary files in binary mode patman: Provide a way to get program output in binary mode binman: Use binary mode when compressing data binman: Drop an unused input file binman: Handle repeated bytes for Python 3 Add a simple script to remove boards patman: Support use of stringIO in Python 3 patman: Move unicode helpers to tools patman: Sort series output for repeatabily patman: Avoid unicode type in settings unit tests patman: Adjust functional tests for Python 3 patman: Tidy up a few more unicode conversions patman: Don't require Python 2
tools/binman/binman.py | 26 +++- tools/binman/bsection.py | 7 +- tools/binman/control.py | 8 +- tools/binman/elf.py | 4 +- tools/binman/elf_test.py | 2 +- tools/binman/entry.py | 5 +- tools/binman/etype/blob.py | 2 +- tools/binman/etype/fill.py | 4 +- tools/binman/etype/gbb.py | 2 +- tools/binman/etype/u_boot_spl_bss_pad.py | 2 +- tools/binman/ftest.py | 81 ++++++------ tools/binman/state.py | 7 +- tools/dtoc/fdt.py | 2 +- tools/patman/cros_subprocess.py | 53 +++++--- tools/patman/func_test.py | 41 ++++--- tools/patman/gitutil.py | 16 +-- tools/patman/patman.py | 2 +- tools/patman/series.py | 20 +-- tools/patman/settings.py | 34 ++--- tools/patman/test_util.py | 16 +-- tools/patman/tools.py | 55 ++++++++- tools/rmboard.py | 150 +++++++++++++++++++++++ 22 files changed, 385 insertions(+), 154 deletions(-) create mode 100755 tools/rmboard.py
-- 2.21.0.1020.gf2820cf01a-goog
On Wed, 15 May 2019 at 02:02, Chris Packham judge.packham@gmail.com wrote:
On Wed, May 15, 2019 at 9:54 AM Simon Glass sjg@chromium.org wrote:
This series updates patman to support Python 3:
- Avoid using 'unicode' type directly
- Use items() instead of iteritems()
- Make sure file I/O uses binary mode where necessary
- Change print statements to functions
- Use the built-in set() class
- Fix up generation of repeated bytes
A few patches for binman are included, but this still requires Python 2.
Couple of comments on 14/21 but the rest of the series
Reviewed-by: Chris Packham judge.packham@gmail.com
Simon Glass (21): patman: Update cros_subprocess to use bytearray patman: Convert print statements to Python 3 binman: Convert print statements to Python 3 binman: Don't show errors for failed tests binman: Remove use of Set() patman: Use items() instead of iteritems() binman: Use items() instead of iteritems() tools: binman: Open all binary files in binary mode tools: dtoc: Open all binary files in binary mode patman: Provide a way to get program output in binary mode binman: Use binary mode when compressing data binman: Drop an unused input file binman: Handle repeated bytes for Python 3 Add a simple script to remove boards patman: Support use of stringIO in Python 3 patman: Move unicode helpers to tools patman: Sort series output for repeatabily patman: Avoid unicode type in settings unit tests patman: Adjust functional tests for Python 3 patman: Tidy up a few more unicode conversions patman: Don't require Python 2
tools/binman/binman.py | 26 +++- tools/binman/bsection.py | 7 +- tools/binman/control.py | 8 +- tools/binman/elf.py | 4 +- tools/binman/elf_test.py | 2 +- tools/binman/entry.py | 5 +- tools/binman/etype/blob.py | 2 +- tools/binman/etype/fill.py | 4 +- tools/binman/etype/gbb.py | 2 +- tools/binman/etype/u_boot_spl_bss_pad.py | 2 +- tools/binman/ftest.py | 81 ++++++------ tools/binman/state.py | 7 +- tools/dtoc/fdt.py | 2 +- tools/patman/cros_subprocess.py | 53 +++++--- tools/patman/func_test.py | 41 ++++--- tools/patman/gitutil.py | 16 +-- tools/patman/patman.py | 2 +- tools/patman/series.py | 20 +-- tools/patman/settings.py | 34 ++--- tools/patman/test_util.py | 16 +-- tools/patman/tools.py | 55 ++++++++- tools/rmboard.py | 150 +++++++++++++++++++++++ 22 files changed, 385 insertions(+), 154 deletions(-) create mode 100755 tools/rmboard.py
-- 2.21.0.1020.gf2820cf01a-goog
Applied series to u-boot-dm/next
Due to a patchwork bug, some patches show up with missing spaces in the UI, e.g. here http://patchwork.ozlabs.org/bundle/sjg/dm/
So my script which emails on each patch doesn't work, sorry.
- Simon
participants (2)
-
Chris Packham -
Simon Glass