[U-Boot] [PATCH 0/9] Python 3.x support for patman & dtoc

This series makes patman & dtoc run on python 3.x, which allows them to work without hacks on distributions where the python binary is python 3.x. All changes are made in a way that continue to work on python 2.x, so both tools should work on either major version of python.
Paul Burton (9): patman: Replace tabs with spaces patman: Make print statements python 3.x safe patman: Make exception handling python 3.x safe patman: Import 'configparser' lower case to be python 3.x safe patman: Decode stdout/stderr as utf8, be python 3.x safe patman: Use items() to iterate over dictionaries in python 3.x dtoc: Use items() to iterate over dictionaries in python 3.x dtoc: Decode strings for struct.unpack on python 3.x dtoc: Make integer division python 3.x safe
tools/dtoc/dtoc.py | 26 ++++++++++++++++++++----- tools/dtoc/fdt_fallback.py | 8 +++++++- tools/dtoc/fdt_util.py | 3 +++ tools/patman/checkpatch.py | 16 ++++++++-------- tools/patman/command.py | 2 +- tools/patman/cros_subprocess.py | 6 +++--- tools/patman/get_maintainer.py | 2 +- tools/patman/gitutil.py | 18 +++++++++--------- tools/patman/patchstream.py | 6 +++--- tools/patman/patman.py | 12 ++++++------ tools/patman/series.py | 42 +++++++++++++++++++++-------------------- tools/patman/settings.py | 28 ++++++++++++++++++--------- tools/patman/terminal.py | 12 +++++++----- tools/patman/test.py | 2 +- 14 files changed, 111 insertions(+), 72 deletions(-)

In preparation for running on python 3.x, which will refuse to run scripts which mix tabs & spaces for indentation, replace 2 tab characters present in series.py with spaces.
Signed-off-by: Paul Burton paul.burton@imgtec.com ---
tools/patman/series.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/tools/patman/series.py b/tools/patman/series.py index cc6f80b..1ce30c6 100644 --- a/tools/patman/series.py +++ b/tools/patman/series.py @@ -225,7 +225,7 @@ class Series(dict): raise_on_error=raise_on_error) list += gitutil.BuildEmailList(commit.cc_list, raise_on_error=raise_on_error) - if add_maintainers: + if add_maintainers: list += get_maintainer.GetMaintainer(commit.patch) all_ccs += list print >>fd, commit.patch, ', '.join(set(list)) @@ -259,7 +259,7 @@ class Series(dict): """ git_prefix = gitutil.GetDefaultSubjectPrefix() if git_prefix: - git_prefix = '%s][' % git_prefix + git_prefix = '%s][' % git_prefix else: git_prefix = ''

On 26 September 2016 at 08:30, Paul Burton paul.burton@imgtec.com wrote:
In preparation for running on python 3.x, which will refuse to run scripts which mix tabs & spaces for indentation, replace 2 tab characters present in series.py with spaces.
Signed-off-by: Paul Burton paul.burton@imgtec.com
tools/patman/series.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
Acked-by: Simon Glass sjg@chromium.org

In python 3.x, print must be used as a function call. Convert all print statements to the function call style, importing from __future__ where we print with no trailing newline or print to a file object.
Signed-off-by: Paul Burton paul.burton@imgtec.com ---
tools/patman/checkpatch.py | 16 ++++++++-------- tools/patman/get_maintainer.py | 2 +- tools/patman/gitutil.py | 6 +++--- tools/patman/patchstream.py | 6 +++--- tools/patman/patman.py | 12 ++++++------ tools/patman/series.py | 38 ++++++++++++++++++++------------------ tools/patman/settings.py | 16 +++++++++------- tools/patman/terminal.py | 12 +++++++----- tools/patman/test.py | 2 +- 9 files changed, 58 insertions(+), 52 deletions(-)
diff --git a/tools/patman/checkpatch.py b/tools/patman/checkpatch.py index 3eef6de..be78fc5 100644 --- a/tools/patman/checkpatch.py +++ b/tools/patman/checkpatch.py @@ -83,7 +83,7 @@ def CheckPatch(fname, verbose=False):
for line in result.stdout.splitlines(): if verbose: - print line + print(line)
# A blank line indicates the end of a message if not line and item: @@ -151,17 +151,17 @@ def CheckPatches(verbose, args): error_count += result.errors warning_count += result.warnings check_count += result.checks - print '%d errors, %d warnings, %d checks for %s:' % (result.errors, - result.warnings, result.checks, col.Color(col.BLUE, fname)) + print('%d errors, %d warnings, %d checks for %s:' % (result.errors, + result.warnings, result.checks, col.Color(col.BLUE, fname))) if (len(result.problems) != result.errors + result.warnings + result.checks): - print "Internal error: some problems lost" + print("Internal error: some problems lost") for item in result.problems: - print GetWarningMsg(col, item.get('type', '<unknown>'), + print(GetWarningMsg(col, item.get('type', '<unknown>'), item.get('file', '<unknown>'), - item.get('line', 0), item.get('msg', 'message')) + item.get('line', 0), item.get('msg', 'message'))) print - #print stdout + #print(stdout) if error_count or warning_count or check_count: str = 'checkpatch.pl found %d error(s), %d warning(s), %d checks(s)' color = col.GREEN @@ -169,6 +169,6 @@ def CheckPatches(verbose, args): color = col.YELLOW if error_count: color = col.RED - print col.Color(color, str % (error_count, warning_count, check_count)) + print(col.Color(color, str % (error_count, warning_count, check_count))) return False return True diff --git a/tools/patman/get_maintainer.py b/tools/patman/get_maintainer.py index 00b4939..2deb5db 100644 --- a/tools/patman/get_maintainer.py +++ b/tools/patman/get_maintainer.py @@ -40,7 +40,7 @@ def GetMaintainer(fname, verbose=False): get_maintainer = FindGetMaintainer() if not get_maintainer: if verbose: - print "WARNING: Couldn't find get_maintainer.pl" + print("WARNING: Couldn't find get_maintainer.pl") return []
stdout = command.Output(get_maintainer, '--norolestats', fname) diff --git a/tools/patman/gitutil.py b/tools/patman/gitutil.py index bb7c9e0..c0fe093 100644 --- a/tools/patman/gitutil.py +++ b/tools/patman/gitutil.py @@ -491,7 +491,7 @@ def LookupEmail(lookup_name, alias=None, raise_on_error=True, level=0): if raise_on_error: raise OSError, msg else: - print col.Color(col.RED, msg) + print(col.Color(col.RED, msg)) return out_list
if lookup_name: @@ -500,7 +500,7 @@ def LookupEmail(lookup_name, alias=None, raise_on_error=True, level=0): if raise_on_error: raise ValueError, msg else: - print col.Color(col.RED, msg) + print(col.Color(col.RED, msg)) return out_list for item in alias[lookup_name]: todo = LookupEmail(item, alias, raise_on_error, level + 1) @@ -508,7 +508,7 @@ def LookupEmail(lookup_name, alias=None, raise_on_error=True, level=0): if not new_item in out_list: out_list.append(new_item)
- #print "No match for alias '%s'" % lookup_name + #print("No match for alias '%s'" % lookup_name) return out_list
def GetTopLevel(): diff --git a/tools/patman/patchstream.py b/tools/patman/patchstream.py index 69d5cfb..cd4667f 100644 --- a/tools/patman/patchstream.py +++ b/tools/patman/patchstream.py @@ -480,12 +480,12 @@ def FixPatches(series, fnames): commit.patch = fname result = FixPatch(backup_dir, fname, series, commit) if result: - print '%d warnings for %s:' % (len(result), fname) + print('%d warnings for %s:' % (len(result), fname)) for warn in result: - print '\t', warn + print('\t', warn) print count += 1 - print 'Cleaned %d patches' % count + print('Cleaned %d patches' % count) return series
def InsertCoverLetter(fname, series, count): diff --git a/tools/patman/patman.py b/tools/patman/patman.py index fe50eb4..fdbee67 100755 --- a/tools/patman/patman.py +++ b/tools/patman/patman.py @@ -93,11 +93,11 @@ elif options.test: suite.run(result)
# TODO: Surely we can just 'print' result? - print result + print(result) for test, err in result.errors: - print err + print(err) for test, err in result.failures: - print err + print(err)
# Called from git with a patch filename as argument # Printout a list of additional CC recipients for this patch @@ -110,7 +110,7 @@ elif options.cc_cmd: for cc in match.group(2).split(', '): cc = cc.strip() if cc: - print cc + print(cc) fd.close()
elif options.full_help: @@ -166,12 +166,12 @@ else: options.dry_run, not options.ignore_bad_tags, cc_file, in_reply_to=options.in_reply_to, thread=options.thread) else: - print col.Color(col.RED, "Not sending emails due to errors/warnings") + print(col.Color(col.RED, "Not sending emails due to errors/warnings"))
# For a dry run, just show our actions as a sanity check if options.dry_run: series.ShowActions(args, cmd, options.process_tags) if not its_a_go: - print col.Color(col.RED, "Email would not be sent") + print(col.Color(col.RED, "Email would not be sent"))
os.remove(cc_file) diff --git a/tools/patman/series.py b/tools/patman/series.py index 1ce30c6..38a452e 100644 --- a/tools/patman/series.py +++ b/tools/patman/series.py @@ -3,6 +3,8 @@ # SPDX-License-Identifier: GPL-2.0+ #
+from __future__ import print_function + import itertools import os
@@ -101,38 +103,38 @@ class Series(dict): cc_set = set(gitutil.BuildEmailList(self.cc));
col = terminal.Color() - print 'Dry run, so not doing much. But I would do this:' - print - print 'Send a total of %d patch%s with %scover letter.' % ( + print('Dry run, so not doing much. But I would do this:') + print() + print('Send a total of %d patch%s with %scover letter.' % ( len(args), '' if len(args) == 1 else 'es', - self.get('cover') and 'a ' or 'no ') + self.get('cover') and 'a ' or 'no '))
# TODO: Colour the patches according to whether they passed checks for upto in range(len(args)): commit = self.commits[upto] - print col.Color(col.GREEN, ' %s' % args[upto]) + print(col.Color(col.GREEN, ' %s' % args[upto])) cc_list = list(self._generated_cc[commit.patch]) for email in set(cc_list) - to_set - cc_set: if email == None: email = col.Color(col.YELLOW, "<alias '%s' not found>" % tag) if email: - print ' Cc: ',email + print(' Cc: ', email) print for item in to_set: - print 'To:\t ', item + print('To:\t ', item) for item in cc_set - to_set: - print 'Cc:\t ', item - print 'Version: ', self.get('version') - print 'Prefix:\t ', self.get('prefix') + print('Cc:\t ', item) + print('Version: ', self.get('version')) + print('Prefix:\t ', self.get('prefix')) if self.cover: - print 'Cover: %d lines' % len(self.cover) + print('Cover: %d lines' % len(self.cover)) cover_cc = gitutil.BuildEmailList(self.get('cover_cc', '')) all_ccs = itertools.chain(cover_cc, *self._generated_cc.values()) for email in set(all_ccs) - to_set - cc_set: - print ' Cc: ',email + print(' Cc: ', email) if cmd: - print 'Git command: %s' % cmd + print('Git command: %s' % cmd)
def MakeChangeLog(self, commit): """Create a list of changes for each version. @@ -191,13 +193,13 @@ class Series(dict): else: if version > 1: str = 'Change log missing for v%d' % version - print col.Color(col.RED, str) + print(col.Color(col.RED, str)) for version in changes_copy: str = 'Change log for unknown version v%d' % version - print col.Color(col.RED, str) + print(col.Color(col.RED, str)) elif self.changes: str = 'Change log exists, but no version is set' - print col.Color(col.RED, str) + print(col.Color(col.RED, str))
def MakeCcFile(self, process_tags, cover_fname, raise_on_error, add_maintainers): @@ -228,12 +230,12 @@ class Series(dict): if add_maintainers: list += get_maintainer.GetMaintainer(commit.patch) all_ccs += list - print >>fd, commit.patch, ', '.join(set(list)) + print(commit.patch, ', '.join(set(list)), file=fd) self._generated_cc[commit.patch] = list
if cover_fname: cover_cc = gitutil.BuildEmailList(self.get('cover_cc', '')) - print >>fd, cover_fname, ', '.join(set(cover_cc + all_ccs)) + print(cover_fname, ', '.join(set(cover_cc + all_ccs)), file=fd)
fd.close() return fname diff --git a/tools/patman/settings.py b/tools/patman/settings.py index ba2a68f..01f6c38 100644 --- a/tools/patman/settings.py +++ b/tools/patman/settings.py @@ -3,6 +3,8 @@ # SPDX-License-Identifier: GPL-2.0+ #
+from __future__ import print_function + import ConfigParser import os import re @@ -156,7 +158,7 @@ def ReadGitAliases(fname): try: fd = open(fname, 'r') except IOError: - print "Warning: Cannot find alias file '%s'" % fname + print("Warning: Cannot find alias file '%s'" % fname) return
re_line = re.compile('alias\s+(\S+)\s+(.*)') @@ -167,7 +169,7 @@ def ReadGitAliases(fname):
m = re_line.match(line) if not m: - print "Warning: Alias file line '%s' not understood" % line + print("Warning: Alias file line '%s' not understood" % line) continue
list = alias.get(m.group(1), []) @@ -200,10 +202,10 @@ def CreatePatmanConfigFile(config_fname): try: f = open(config_fname, 'w') except IOError: - print "Couldn't create patman config file\n" + print("Couldn't create patman config file\n") raise
- print >>f, "[alias]\nme: %s <%s>" % (name, email) + print("[alias]\nme: %s <%s>" % (name, email), file=f) f.close();
def _UpdateDefaults(parser, config): @@ -233,7 +235,7 @@ def _UpdateDefaults(parser, config): val = config.getint('settings', name) parser.set_default(name, val) else: - print "WARNING: Unknown setting %s" % name + print("WARNING: Unknown setting %s" % name)
def _ReadAliasFile(fname): """Read in the U-Boot git alias file if it exists. @@ -258,7 +260,7 @@ def _ReadAliasFile(fname): continue alias[words[1]] = [s.strip() for s in words[2].split(',')] if bad_line: - print bad_line + print(bad_line)
def Setup(parser, project_name, config_fname=''): """Set up the settings module by reading config files. @@ -276,7 +278,7 @@ def Setup(parser, project_name, config_fname=''): config_fname = '%s/.patman' % os.getenv('HOME')
if not os.path.exists(config_fname): - print "No config file found ~/.patman\nCreating one...\n" + print("No config file found ~/.patman\nCreating one...\n") CreatePatmanConfigFile(config_fname)
config.read(config_fname) diff --git a/tools/patman/terminal.py b/tools/patman/terminal.py index e78a7c1..d8a9e4c 100644 --- a/tools/patman/terminal.py +++ b/tools/patman/terminal.py @@ -8,6 +8,8 @@ This module handles terminal interaction including ANSI color codes. """
+from __future__ import print_function + import os import sys
@@ -52,9 +54,9 @@ def Print(text='', newline=True, colour=None): if colour: col = Color() text = col.Color(colour, text) - print text, + print(text, end='') if newline: - print + print()
def SetPrintTestMode(): """Go into test mode, where all printing is recorded""" @@ -79,11 +81,11 @@ def EchoPrintTestLines(): for line in print_test_list: if line.colour: col = Color() - print col.Color(line.colour, line.text), + print(col.Color(line.colour, line.text), end='') else: - print line.text, + print(line.text, end='') if line.newline: - print + print()
class Color(object): diff --git a/tools/patman/test.py b/tools/patman/test.py index e8f7472..8c39f66 100644 --- a/tools/patman/test.py +++ b/tools/patman/test.py @@ -181,7 +181,7 @@ index 0000000..2234c87 elif data_type == 'indent': indent = tab else: - print 'not implemented' + print('not implemented') return data % (signoff, tab, indent, tab)
def SetupData(self, data_type):

On 26 September 2016 at 08:30, Paul Burton paul.burton@imgtec.com wrote:
In python 3.x, print must be used as a function call. Convert all print statements to the function call style, importing from __future__ where we print with no trailing newline or print to a file object.
Signed-off-by: Paul Burton paul.burton@imgtec.com
tools/patman/checkpatch.py | 16 ++++++++-------- tools/patman/get_maintainer.py | 2 +- tools/patman/gitutil.py | 6 +++--- tools/patman/patchstream.py | 6 +++--- tools/patman/patman.py | 12 ++++++------ tools/patman/series.py | 38 ++++++++++++++++++++------------------ tools/patman/settings.py | 16 +++++++++------- tools/patman/terminal.py | 12 +++++++----- tools/patman/test.py | 2 +- 9 files changed, 58 insertions(+), 52 deletions(-)
Acked-by: Simon Glass sjg@chromium.org

Syntax for exception handling is a little more strict in python 3.x. Convert all uses to a form accepted by both python 2.x & python 3.x.
Signed-off-by: Paul Burton paul.burton@imgtec.com ---
tools/patman/command.py | 2 +- tools/patman/cros_subprocess.py | 2 +- tools/patman/gitutil.py | 12 ++++++------ 3 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/tools/patman/command.py b/tools/patman/command.py index d1f0ca5..bebc495 100644 --- a/tools/patman/command.py +++ b/tools/patman/command.py @@ -85,7 +85,7 @@ def RunPipe(pipe_list, infile=None, outfile=None,
try: last_pipe = cros_subprocess.Popen(cmd, cwd=cwd, **kwargs) - except Exception, err: + except Exception as err: result.exception = err if raise_on_error: raise Exception("Error running '%s': %s" % (user_pipestr, str)) diff --git a/tools/patman/cros_subprocess.py b/tools/patman/cros_subprocess.py index 0fc4a06..ebd4300 100644 --- a/tools/patman/cros_subprocess.py +++ b/tools/patman/cros_subprocess.py @@ -166,7 +166,7 @@ class Popen(subprocess.Popen): while read_set or write_set: try: rlist, wlist, _ = select.select(read_set, write_set, [], 0.2) - except select.error, e: + except select.error as e: if e.args[0] == errno.EINTR: continue raise diff --git a/tools/patman/gitutil.py b/tools/patman/gitutil.py index c0fe093..0d23079 100644 --- a/tools/patman/gitutil.py +++ b/tools/patman/gitutil.py @@ -139,7 +139,7 @@ def GetUpstream(git_dir, branch): leaf = merge.split('/')[-1] return '%s/%s' % (remote, leaf), None else: - raise ValueError, ("Cannot determine upstream branch for branch " + raise ValueError("Cannot determine upstream branch for branch " "'%s' remote='%s', merge='%s'" % (branch, remote, merge))
@@ -224,7 +224,7 @@ def Checkout(commit_hash, git_dir=None, work_tree=None, force=False): result = command.RunPipe([pipe], capture=True, raise_on_error=False, capture_stderr=True) if result.return_code != 0: - raise OSError, 'git checkout (%s): %s' % (pipe, result.stderr) + raise OSError('git checkout (%s): %s' % (pipe, result.stderr))
def Clone(git_dir, output_dir): """Checkout the selected commit for this build @@ -236,7 +236,7 @@ def Clone(git_dir, output_dir): result = command.RunPipe([pipe], capture=True, cwd=output_dir, capture_stderr=True) if result.return_code != 0: - raise OSError, 'git clone: %s' % result.stderr + raise OSError('git clone: %s' % result.stderr)
def Fetch(git_dir=None, work_tree=None): """Fetch from the origin repo @@ -252,7 +252,7 @@ def Fetch(git_dir=None, work_tree=None): pipe.append('fetch') result = command.RunPipe([pipe], capture=True, capture_stderr=True) if result.return_code != 0: - raise OSError, 'git fetch: %s' % result.stderr + raise OSError('git fetch: %s' % result.stderr)
def CreatePatches(start, count, series): """Create a series of patches from the top of the current branch. @@ -489,7 +489,7 @@ def LookupEmail(lookup_name, alias=None, raise_on_error=True, level=0): if level > 10: msg = "Recursive email alias at '%s'" % lookup_name if raise_on_error: - raise OSError, msg + raise OSError(msg) else: print(col.Color(col.RED, msg)) return out_list @@ -498,7 +498,7 @@ def LookupEmail(lookup_name, alias=None, raise_on_error=True, level=0): if not lookup_name in alias: msg = "Alias '%s' not found" % lookup_name if raise_on_error: - raise ValueError, msg + raise ValueError(msg) else: print(col.Color(col.RED, msg)) return out_list

On 26 September 2016 at 08:30, Paul Burton paul.burton@imgtec.com wrote:
Syntax for exception handling is a little more strict in python 3.x. Convert all uses to a form accepted by both python 2.x & python 3.x.
Signed-off-by: Paul Burton paul.burton@imgtec.com
tools/patman/command.py | 2 +- tools/patman/cros_subprocess.py | 2 +- tools/patman/gitutil.py | 12 ++++++------ 3 files changed, 8 insertions(+), 8 deletions(-)
Acked-by: Simon Glass sjg@chromium.org

In python 3.x module names used in import statements are case sensitive, and the configparser module is named in all lower-case. Import it as such in order to avoid errors when running with python 3.x.
Signed-off-by: Paul Burton paul.burton@imgtec.com ---
tools/patman/settings.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/tools/patman/settings.py b/tools/patman/settings.py index 01f6c38..3caf379 100644 --- a/tools/patman/settings.py +++ b/tools/patman/settings.py @@ -5,7 +5,11 @@
from __future__ import print_function
-import ConfigParser +try: + import configparser as ConfigParser +except: + import ConfigParser + import os import re

On 26 September 2016 at 08:30, Paul Burton paul.burton@imgtec.com wrote:
In python 3.x module names used in import statements are case sensitive, and the configparser module is named in all lower-case. Import it as such in order to avoid errors when running with python 3.x.
Signed-off-by: Paul Burton paul.burton@imgtec.com
tools/patman/settings.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
Acked-by: Simon Glass sjg@chromium.org

In python 3.x reading stdout or stdin will produce a bytestring rather than a string. Decode it in CommunicateFilter such that the rest of the code can continue to deal with strings. This works fine with python 2.x too.
Signed-off-by: Paul Burton paul.burton@imgtec.com ---
tools/patman/cros_subprocess.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/tools/patman/cros_subprocess.py b/tools/patman/cros_subprocess.py index ebd4300..922e560 100644 --- a/tools/patman/cros_subprocess.py +++ b/tools/patman/cros_subprocess.py @@ -189,7 +189,7 @@ class Popen(subprocess.Popen): data = "" # We will get an error on read if the pty is closed try: - data = os.read(self.stdout.fileno(), 1024) + data = os.read(self.stdout.fileno(), 1024).decode('utf8') except OSError: pass if data == "": @@ -204,7 +204,7 @@ class Popen(subprocess.Popen): data = "" # We will get an error on read if the pty is closed try: - data = os.read(self.stderr.fileno(), 1024) + data = os.read(self.stderr.fileno(), 1024).decode('utf8') except OSError: pass if data == "":

On 26 September 2016 at 08:30, Paul Burton paul.burton@imgtec.com wrote:
In python 3.x reading stdout or stdin will produce a bytestring rather than a string. Decode it in CommunicateFilter such that the rest of the code can continue to deal with strings. This works fine with python 2.x too.
Signed-off-by: Paul Burton paul.burton@imgtec.com
tools/patman/cros_subprocess.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
Acked-by: Simon Glass sjg@chromium.org

In python 3.x the iteritems() method has been removed from dictionaries, and the items() method does effectively the same thing. Convert the code to attempt to use iteritems() to be efficient on python 2.x, but use items() when that fails on python 3.x.
Signed-off-by: Paul Burton paul.burton@imgtec.com ---
tools/patman/settings.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/tools/patman/settings.py b/tools/patman/settings.py index 3caf379..8b10630 100644 --- a/tools/patman/settings.py +++ b/tools/patman/settings.py @@ -94,7 +94,11 @@ class _ProjectConfigParser(ConfigParser.SafeConfigParser): if not self.has_section(project_settings): self.add_section(project_settings) project_defaults = _default_settings.get(project_name, {}) - for setting_name, setting_value in project_defaults.iteritems(): + try: + iterator = project_defaults.iteritems() + except: + iterator = project_defaults.items() + for setting_name, setting_value in iterator: self.set(project_settings, setting_name, setting_value)
def get(self, section, option, *args, **kwargs):

Hi Paul,
On 26 September 2016 at 08:30, Paul Burton paul.burton@imgtec.com wrote:
In python 3.x the iteritems() method has been removed from dictionaries, and the items() method does effectively the same thing. Convert the code to attempt to use iteritems() to be efficient on python 2.x, but use items() when that fails on python 3.x.
Signed-off-by: Paul Burton paul.burton@imgtec.com
tools/patman/settings.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
I'd be happy with just using items(), since efficiency is not a concern here, and I'm not keen on the try..except approach for handling Python 2/3 differences.
diff --git a/tools/patman/settings.py b/tools/patman/settings.py index 3caf379..8b10630 100644 --- a/tools/patman/settings.py +++ b/tools/patman/settings.py @@ -94,7 +94,11 @@ class _ProjectConfigParser(ConfigParser.SafeConfigParser): if not self.has_section(project_settings): self.add_section(project_settings) project_defaults = _default_settings.get(project_name, {})
for setting_name, setting_value in project_defaults.iteritems():
try:
iterator = project_defaults.iteritems()
except:
iterator = project_defaults.items()
for setting_name, setting_value in iterator: self.set(project_settings, setting_name, setting_value)
def get(self, section, option, *args, **kwargs):
-- 2.10.0
Regards, Simon

In python 3.x the iteritems() method has been removed from dictionaries, and the items() method does effectively the same thing. Convert the code to attempt to use iteritems() to be efficient on python 2.x, but use items() when that fails on python 3.x.
Signed-off-by: Paul Burton paul.burton@imgtec.com ---
tools/dtoc/dtoc.py | 24 ++++++++++++++++++++---- tools/dtoc/fdt_fallback.py | 8 +++++++- 2 files changed, 27 insertions(+), 5 deletions(-)
diff --git a/tools/dtoc/dtoc.py b/tools/dtoc/dtoc.py index 518aa51..4ba8604 100755 --- a/tools/dtoc/dtoc.py +++ b/tools/dtoc/dtoc.py @@ -224,14 +224,22 @@ class DtbPlatdata: fields = {}
# Get a list of all the valid properties in this node. - for name, prop in node.props.iteritems(): + try: + iterator = node.props.iteritems() + except: + iterator = node.props.items() + for name, prop in iterator: if name not in PROP_IGNORE_LIST and name[0] != '#': fields[name] = copy.deepcopy(prop)
# If we've seen this node_name before, update the existing struct. if node_name in structs: struct = structs[node_name] - for name, prop in fields.iteritems(): + try: + iterator = fields.iteritems() + except: + iterator = fields.items() + for name, prop in iterator: oldprop = struct.get(name) if oldprop: oldprop.Widen(prop) @@ -246,7 +254,11 @@ class DtbPlatdata: for node in self._valid_nodes: node_name = self.GetCompatName(node) struct = structs[node_name] - for name, prop in node.props.iteritems(): + try: + iterator = node.props.iteritems() + except: + iterator = node.props.items() + for name, prop in iterator: if name not in PROP_IGNORE_LIST and name[0] != '#': prop.Widen(struct[name]) upto += 1 @@ -298,7 +310,11 @@ class DtbPlatdata: var_name = Conv_name_to_c(node.name) self.Buf('static struct %s%s %s%s = {\n' % (STRUCT_PREFIX, struct_name, VAL_PREFIX, var_name)) - for pname, prop in node.props.iteritems(): + try: + iterator = node.props.iteritems() + except: + iterator = node.props.items() + for pname, prop in iterator: if pname in PROP_IGNORE_LIST or pname[0] == '#': continue ptype = TYPE_NAMES[prop.type] diff --git a/tools/dtoc/fdt_fallback.py b/tools/dtoc/fdt_fallback.py index 0c0ebbc..498cb66 100644 --- a/tools/dtoc/fdt_fallback.py +++ b/tools/dtoc/fdt_fallback.py @@ -58,7 +58,13 @@ class Node(NodeBase): This fills in the props and subnodes properties, recursively searching into subnodes so that the entire tree is built. """ - for name, byte_list_str in self._fdt.GetProps(self.path).iteritems(): + props = self._fdt.GetProps(self.path) + try: + iterator = props.iteritems() + except: + iterator = props.items() + + for name, byte_list_str in iterator: prop = Prop(self, name, byte_list_str) self.props[name] = prop

On 26 September 2016 at 08:30, Paul Burton paul.burton@imgtec.com wrote:
In python 3.x the iteritems() method has been removed from dictionaries, and the items() method does effectively the same thing. Convert the code to attempt to use iteritems() to be efficient on python 2.x, but use items() when that fails on python 3.x.
Signed-off-by: Paul Burton paul.burton@imgtec.com
tools/dtoc/dtoc.py | 24 ++++++++++++++++++++---- tools/dtoc/fdt_fallback.py | 8 +++++++- 2 files changed, 27 insertions(+), 5 deletions(-)
Similar comment to previous patch...

On python 3.x struct.unpack will complain if we provide it with a string since it expects to operate on a bytes object. In order to satisfy this requirement, encode the string to a bytes object when running on python 3.x.
Signed-off-by: Paul Burton paul.burton@imgtec.com ---
tools/dtoc/fdt_util.py | 3 +++ 1 file changed, 3 insertions(+)
diff --git a/tools/dtoc/fdt_util.py b/tools/dtoc/fdt_util.py index 3a10838..e6d523b 100644 --- a/tools/dtoc/fdt_util.py +++ b/tools/dtoc/fdt_util.py @@ -8,6 +8,7 @@
import os import struct +import sys import tempfile
import command @@ -22,6 +23,8 @@ def fdt32_to_cpu(val): Return: A native-endian integer value """ + if sys.version_info > (3, 0): + val = val.encode('raw_unicode_escape') return struct.unpack('>I', val)[0]
def EnsureCompiled(fname):

On 26 September 2016 at 08:30, Paul Burton paul.burton@imgtec.com wrote:
On python 3.x struct.unpack will complain if we provide it with a string since it expects to operate on a bytes object. In order to satisfy this requirement, encode the string to a bytes object when running on python 3.x.
Signed-off-by: Paul Burton paul.burton@imgtec.com
tools/dtoc/fdt_util.py | 3 +++ 1 file changed, 3 insertions(+)
Acked-by: Simon Glass sjg@chromium.org

If we use the '/' operator then python 3.x will produce a float, and refuse to multiply the string sequence in Conv_name_to_c by it with:
TypeError: can't multiply sequence by non-int of type 'float'
Use the '//' operator instead to enforce that we want integer rather than floating point division.
Signed-off-by: Paul Burton paul.burton@imgtec.com
---
tools/dtoc/dtoc.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tools/dtoc/dtoc.py b/tools/dtoc/dtoc.py index 4ba8604..ac7ba6c 100755 --- a/tools/dtoc/dtoc.py +++ b/tools/dtoc/dtoc.py @@ -60,7 +60,7 @@ def Conv_name_to_c(name): def TabTo(num_tabs, str): if len(str) >= num_tabs * 8: return str + ' ' - return str + '\t' * (num_tabs - len(str) / 8) + return str + '\t' * (num_tabs - len(str) // 8)
class DtbPlatdata: """Provide a means to convert device tree binary data to platform data

On 26 September 2016 at 08:30, Paul Burton paul.burton@imgtec.com wrote:
If we use the '/' operator then python 3.x will produce a float, and refuse to multiply the string sequence in Conv_name_to_c by it with:
TypeError: can't multiply sequence by non-int of type 'float'
Use the '//' operator instead to enforce that we want integer rather than floating point division.
Signed-off-by: Paul Burton paul.burton@imgtec.com
tools/dtoc/dtoc.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Acked-by: Simon Glass sjg@chromium.org

Hi Paul,
On 26 September 2016 at 08:30, Paul Burton paul.burton@imgtec.com wrote:
This series makes patman & dtoc run on python 3.x, which allows them to work without hacks on distributions where the python binary is python 3.x. All changes are made in a way that continue to work on python 2.x, so both tools should work on either major version of python.
Thanks for doing this! Does it still install correctly with setup.py?
Paul Burton (9): patman: Replace tabs with spaces patman: Make print statements python 3.x safe patman: Make exception handling python 3.x safe patman: Import 'configparser' lower case to be python 3.x safe patman: Decode stdout/stderr as utf8, be python 3.x safe patman: Use items() to iterate over dictionaries in python 3.x dtoc: Use items() to iterate over dictionaries in python 3.x dtoc: Decode strings for struct.unpack on python 3.x dtoc: Make integer division python 3.x safe
tools/dtoc/dtoc.py | 26 ++++++++++++++++++++----- tools/dtoc/fdt_fallback.py | 8 +++++++- tools/dtoc/fdt_util.py | 3 +++ tools/patman/checkpatch.py | 16 ++++++++-------- tools/patman/command.py | 2 +- tools/patman/cros_subprocess.py | 6 +++--- tools/patman/get_maintainer.py | 2 +- tools/patman/gitutil.py | 18 +++++++++--------- tools/patman/patchstream.py | 6 +++--- tools/patman/patman.py | 12 ++++++------ tools/patman/series.py | 42 +++++++++++++++++++++-------------------- tools/patman/settings.py | 28 ++++++++++++++++++--------- tools/patman/terminal.py | 12 +++++++----- tools/patman/test.py | 2 +- 14 files changed, 111 insertions(+), 72 deletions(-)
-- 2.10.0
Regards, Simon

On Monday, 26 September 2016 18:33:19 BST Simon Glass wrote:
Hi Paul,
On 26 September 2016 at 08:30, Paul Burton paul.burton@imgtec.com wrote:
This series makes patman & dtoc run on python 3.x, which allows them to work without hacks on distributions where the python binary is python 3.x. All changes are made in a way that continue to work on python 2.x, so both tools should work on either major version of python.
Thanks for doing this! Does it still install correctly with setup.py?
Hi Simon,
No problem - I've been meaning to for a while!
It seems to install fine when using python2 (& neat, I hadn't noticed there was an installer before). When using python3 it installs fine but then when running it imports fail with messages like:
from patman import settings
Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/pburton/.local/lib/python3.5/site-packages/patman/settings.py", line 16, in <module> import command ImportError: No module named 'command'
I'm not sure what the best way to fix that is. We could import in all files the way patman.py does, wrapped in a try except block (though should that go the other way round to what it is now? ie. won't it currently always import the installed version even if you run patman from the source tree?) Alternatively we could run patman using a wrapper script in tools/ or move all the patman source a level down & have a wrapper script in tools/patman/ then use "from patman import x" in all cases. Any preferences?
Thanks, Paul
Paul Burton (9): patman: Replace tabs with spaces patman: Make print statements python 3.x safe patman: Make exception handling python 3.x safe patman: Import 'configparser' lower case to be python 3.x safe patman: Decode stdout/stderr as utf8, be python 3.x safe patman: Use items() to iterate over dictionaries in python 3.x dtoc: Use items() to iterate over dictionaries in python 3.x dtoc: Decode strings for struct.unpack on python 3.x dtoc: Make integer division python 3.x safe
tools/dtoc/dtoc.py | 26 ++++++++++++++++++++----- tools/dtoc/fdt_fallback.py | 8 +++++++- tools/dtoc/fdt_util.py | 3 +++ tools/patman/checkpatch.py | 16 ++++++++-------- tools/patman/command.py | 2 +- tools/patman/cros_subprocess.py | 6 +++--- tools/patman/get_maintainer.py | 2 +- tools/patman/gitutil.py | 18 +++++++++--------- tools/patman/patchstream.py | 6 +++--- tools/patman/patman.py | 12 ++++++------ tools/patman/series.py | 42 +++++++++++++++++++++-------------------- tools/patman/settings.py | 28 ++++++++++++++++++--------- tools/patman/terminal.py | 12 +++++++----- tools/patman/test.py | 2 +- 14 files changed, 111 insertions(+), 72 deletions(-)
-- 2.10.0
Regards, Simon

Hi Paul,
On 27 September 2016 at 09:07, Paul Burton paul.burton@imgtec.com wrote:
On Monday, 26 September 2016 18:33:19 BST Simon Glass wrote:
Hi Paul,
On 26 September 2016 at 08:30, Paul Burton paul.burton@imgtec.com wrote:
This series makes patman & dtoc run on python 3.x, which allows them to work without hacks on distributions where the python binary is python 3.x. All changes are made in a way that continue to work on python 2.x, so both tools should work on either major version of python.
Thanks for doing this! Does it still install correctly with setup.py?
Hi Simon,
No problem - I've been meaning to for a while!
It seems to install fine when using python2 (& neat, I hadn't noticed there was an installer before). When using python3 it installs fine but then when running it imports fail with messages like:
from patman import settings
Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/pburton/.local/lib/python3.5/site-packages/patman/settings.py", line 16, in <module> import command ImportError: No module named 'command'
I'm not sure what the best way to fix that is. We could import in all files the way patman.py does, wrapped in a try except block (though should that go the other way round to what it is now? ie. won't it currently always import the installed version even if you run patman from the source tree?) Alternatively we could run patman using a wrapper script in tools/ or move all the patman source a level down & have a wrapper script in tools/patman/ then use "from patman import x" in all cases. Any preferences?
Is it possible to update sys.path instead?
[...]
Regards, Simon
participants (2)
-
Paul Burton
-
Simon Glass