What is this? =============

This tool is a Python script which: - Creates patch directly from your branch - Cleans them up by removing unwanted tags - Inserts a cover letter with change lists - Runs the patches through checkpatch.pl and its own checks - Optionally emails them out to selected people

It is intended to automate patch creation and make it a less error-prone process. It is useful for U-Boot and Linux work so far, since it uses the checkpatch.pl script.

It is configured almost entirely by tags it finds in your commits. This means that you can work on a number of different branches at once, and keep the settings with each branch rather than having to git format-patch, git send-email, etc. with the correct parameters each time. So for example if you put:

in one of your commits, the series will be sent there.

See the README file for full details. END

Signed-off-by: Simon Glass sjg@chromium.org --- Changes in v2: - Fix up use of Color in series.py (avoid runtime error)

Changes in v3: - Add a few more tests (but still very incomplete) - Add option to display the README - Change cc-list tmpfile name to start with patman - Expand the README to include info about suggested workflow - Find checkpatch.pl in U-Boot tree - Fix generation of cc list to match patch names correctly - Support git's alias file if found in config sendemail.aliasesfile - Support recursive aliases - Tidy up gitutil and settings to separate code better - Use -M with git format-patch to so that file moves are tracked - Use @{upstream} instead of the previous mess (thanks Anton Staff)

Changes in v4: - Add --no-tags options to not Cc maintainers on tags - Add per-commit change log (automatically puts them all in cover letter) - Allow multiple Signed-off-by tags - Append valid tags to the patch - Make aliases additive so you can add more to an alias later - Sort change lists

tools/patman/.gitignore | 1 + tools/patman/README | 408 +++++++++++++++++++++++++++++++++++++++ tools/patman/checkpatch.py | 161 ++++++++++++++++ tools/patman/command.py | 72 +++++++ tools/patman/commit.py | 87 +++++++++ tools/patman/gitutil.py | 372 ++++++++++++++++++++++++++++++++++++ tools/patman/patchstream.py | 444 +++++++++++++++++++++++++++++++++++++++++++ tools/patman/patman | 1 + tools/patman/patman.py | 153 +++++++++++++++ tools/patman/series.py | 238 +++++++++++++++++++++++ tools/patman/settings.py | 81 ++++++++ tools/patman/terminal.py | 86 +++++++++ tools/patman/test.py | 250 ++++++++++++++++++++++++ 13 files changed, 2354 insertions(+), 0 deletions(-) create mode 100644 tools/patman/.gitignore create mode 100644 tools/patman/README create mode 100644 tools/patman/checkpatch.py create mode 100644 tools/patman/command.py create mode 100644 tools/patman/commit.py create mode 100644 tools/patman/gitutil.py create mode 100644 tools/patman/patchstream.py create mode 120000 tools/patman/patman create mode 100755 tools/patman/patman.py create mode 100644 tools/patman/series.py create mode 100644 tools/patman/settings.py create mode 100644 tools/patman/terminal.py create mode 100644 tools/patman/test.py

diff --git a/tools/patman/.gitignore b/tools/patman/.gitignore new file mode 100644 index 0000000..0d20b64 --- /dev/null +++ b/tools/patman/.gitignore @@ -0,0 +1 @@ +*.pyc diff --git a/tools/patman/README b/tools/patman/README new file mode 100644 index 0000000..ee38afc --- /dev/null +++ b/tools/patman/README @@ -0,0 +1,408 @@ +# Copyright (c) 2011 The Chromium OS Authors. +# +# See file CREDITS for list of people who contributed to this +# project. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License as +# published by the Free Software Foundation; either version 2 of +# the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, +# MA 02111-1307 USA +# + +What is this? +============= + +This tool is a Python script which: +- Creates patch directly from your branch +- Cleans them up by removing unwanted tags +- Inserts a cover letter with change lists +- Runs the patches through checkpatch.pl and its own checks +- Optionally emails them out to selected people + +It is intended to automate patch creation and make it a less +error-prone process. It is useful for U-Boot and Linux work so far, +since it uses the checkpatch.pl script. + +It is configured almost entirely by tags it finds in your commits. +This means that you can work on a number of different branches at +once, and keep the settings with each branch rather than having to +git format-patch, git send-email, etc. with the correct parameters +each time. So for example if you put: + +Series-to: fred.blogs@napier.co.nz + +in one of your commits, the series will be sent there. + + +How to use this tool +==================== + +This tool requires a certain way of working: + +- Maintain a number of branches, one for each patch series you are +working on +- Add tags into the commits within each branch to indicate where the +series should be sent, cover letter, version, etc. Most of these are +normally in the top commit so it is easy to change them with 'git +commit --amend' +- Each branch tracks the upstream branch, so that this script can +automatically determine the number of commits in it (optional) +- Check out a branch, and run this script to create and send out your +patches. Weeks later, change the patches and repeat, knowing that you +will get a consistent result each time. + + +How to configure it +=================== + +For most cases patman will locate and use the file 'doc/git-mailrc' in +your U-Boot directory. This contains most of the aliases you will need. + +To add your own, create a file ~/.config/patman directory like this: + +>>>> +# patman alias file + +[alias] +me: Simon Glass sjg@chromium.org + +u-boot: U-Boot Mailing List u-boot@lists.denx.de +wolfgang: Wolfgang Denk wd@denx.de +others: Mike Frysinger vapier@gentoo.org, Fred Bloggs f.bloggs@napier.net + +<<<< + +Aliases are recursive. + +The checkpatch.pl in the U-Boot tools/ subdirectory will be located and +used. Failing that you can put it into your path or ~/bin/checkpatch.pl + + +How to run it +============= + +First do a dry run: + +$ ./tools/scripts/patman/patman -n + +If it can't detect the upstream branch, try telling it how many patches +there are in your series: + +$ ./tools/scripts/patman/patman -n -c5 + +This will create patch files in your current directory and tell you who +it is thinking of sending them to. Take a look at the patch files. + +$ ./tools/scripts/patman/patman -n -c5 -s1 + +Similar to the above, but skip the first commit and take the next 5. This +is useful if your top commit is for setting up testing. + + +How to add tags +=============== + +To make this script useful you must add tags like the following into any +commit. Most can only appear once in the whole series. + +Series-to: email / alias + Email address / alias to send patch series to (you can add this + multiple times) + +Series-cc: email / alias, ... + Email address / alias to Cc patch series to (you can add this + multiple times) + +Series-version: n + Sets the version number of this patch series + +Series-prefix: prefix + Sets the subject prefix. Normally empty but it can be RFC for + RFC patches, or RESEND if you are being ignored. + +Cover-letter: +This is the patch set title +blah blah +more blah blah +END + Sets the cover letter contents for the series. The first line + will become the subject of the cover letter + +Series-notes: +blah blah +blah blah +more blah blah +END + Sets some notes for the patch series, which you don't want in + the commit messages, but do want to send, The notes are joined + together and put after the cover letter. Can appear multiple + times. + + Signed-off-by: Their Name <email> + A sign-off is added automatically to your patches (this is + probably a bug). If you put this tag in your patches, it will + override the default signoff that patman automatically adds. + + Tested-by: Their Name <email> + Acked-by: Their Name <email> + These indicate that someone has acked or tested your patch. + When you get this reply on the mailing list, you can add this + tag to the relevant commit and the script will include it when + you send out the next version. If 'Tested-by:' is set to + yourself, it will be removed. No one will believe you. + +Series-changes: n +- Guinea pig moved into its cage +- Other changes ending with a blank line +<blank line> + This can appear in any commit. It lists the changes for a + particular version n of that commit. The change list is + created based on this information. Each commit gets its own + change list and also the whole thing is repeated in the cover + letter (where duplicate change lines are merged). + + By adding your change lists into your commits it is easier to + keep track of what happened. When you amend a commit, remember + to update the log there and then, knowing that the script will + do the rest. + +Cc: Their Name <email> + This copies a single patch to another email address. + +Various other tags are silently removed, like these Chrome OS and +Gerrit tags: + +BUG=... +TEST=... +Change-Id: +Review URL: +Reviewed-on: +Reviewed-by: + + +Exercise for the reader: Try adding some tags to one of your current +patch series and see how the patches turn out. + + +Where Patches Are Sent +====================== + +Once the patches are created, patman sends them using gti send-email. The +whole series is sent to the recipients in Series-to: and Series-cc. +You can Cc individual patches to other people with the Cc: tag. Tags in the +subject are also picked up to Cc patches. For example, a commit like this: + +>>>> +commit 10212537b85ff9b6e09c82045127522c0f0db981 +Author: Mike Frysinger vapier@gentoo.org +Date: Mon Nov 7 23:18:44 2011 -0500 + + x86: arm: add a git mailrc file for maintainers + + This should make sending out e-mails to the right people easier. + + Cc: sandbox, mikef, ag + Cc: afleming +<<<< + +will create a patch which is copied to x86, arm, sandbox, mikef, ag and +afleming. + + +Example Work Flow +================= + +The basic workflow is to create your commits, add some tags to the top +commit, and type 'patman' to check and send them. + +Here is an example workflow for a series of 4 patches. Let's say you have +these rather contrived patches in the following order in branch us-cmd in +your tree where 'us' means your upstreaming activity (newest to oldest as +output by git log --oneline): + + 7c7909c wip + 89234f5 Don't include standard parser if hush is used + 8d640a7 mmc: sparc: Stop using builtin_run_command() + 0c859a9 Rename run_command2() to run_command() + a74443f sandbox: Rename run_command() to builtin_run_command() + +The first patch is some test things that enable your code to be compiled, +but that you don't want to submit because there is an existing patch for it +on the list. So you can tell patman to create and check some patches +(skipping the first patch) with: + + patman -s1 -n + +If you want to do all of them including the work-in-progress one, then +(if you are tracking an upstream branch): + + patman -n + +Let's say that patman reports an error in the second patch. Then: + + git rebase -i HEAD~6 + <change 'pick' to 'edit' in 89234f5> + <use editor to make code changes> + git add -u + git rebase --continue + +Now you have an updated patch series. To check it: + + patman -s1 -n + +Let's say it is now clean and you want to send it. Now you need to set up +the destination. So amend the top commit with: + + git commit --amend + +Use your editor to add some tags, so that the whole commit message is: + + The current run_command() is really only one of the options, with + hush providing the other. It really shouldn't be called directly + in case the hush parser is bring used, so rename this function to + better explain its purpose. + + Series-to: u-boot + Series-cc: bfin, marex + Series-prefix: RFC + Cover-letter: + Unified command execution in one place + + At present two parsers have similar code to execute commands. Also + cmd_usage() is called all over the place. This series adds a single + function which processes commands called cmd_process(). + END + + Change-Id: Ica71a14c1f0ecb5650f771a32fecb8d2eb9d8a17 + + +You want this to be an RFC and Cc the whole series to the bfin alias and +to Marek. Two of the patches have tags (those are the bits at the front of +the subject that say mmc: sparc: and sandbox:), so 8d640a7 will be Cc'd to +mmc and sparc, and the last one to sandbox. + +Now to send the patches, take off the -n flag: + + patman -s1 + +The patches will be created, shown in your editor, and then sent along with +the cover letter. Note that patman's tags are automatically removed so that +people on the list don't see your secret info. + +Of course patches often attract comments and you need to make some updates. +Let's say one person sent comments and you get an Acked-by: on one patch. +Also, the patch on the list that you were waiting for has been merged, +so you can drop your wip commit. So you resync with upstream: + + git fetch origin (or whatever upstream is called) + git rebase origin/master + +and use git rebase -i to edit the commits, dropping the wip one. You add +the ack tag to one commit: + + Acked-by: Heiko Schocher hs@denx.de + +update the Series-cc: in the top commit: + + Series-cc: bfin, marex, Heiko Schocher hs@denx.de + +and remove the Series-prefix: tag since it it isn't an RFC any more. The +series is now version two, so the series info in the top commit looks like +this: + + Series-to: u-boot + Series-cc: bfin, marex, Heiko Schocher hs@denx.de + Series-version: 2 + Cover-letter: + ... + +Finally, you need to add a change log to the two commits you changed. You +add change logs to each individual commit where the changes happened, like +this: + + Series-changes: 2 + - Updated the command decoder to reduce code size + - Wound the torque propounder up a little more + +(note the blank line at the end of the list) + +When you run patman it will collect all the change logs from the different +commits and combine them into the cover letter, if you have one. So finally +you have a new series of commits: + + faeb973 Don't include standard parser if hush is used + 1b2f2fe mmc: sparc: Stop using builtin_run_command() + cfbe330 Rename run_command2() to run_command() + 0682677 sandbox: Rename run_command() to builtin_run_command() + +so to send them: + + patman + +and it will create and send the version 2 series. + +General points: + +1. When you change back to the us-cmd branch days or weeks later all your +information is still there, safely stored in the commits. You don't need +to remember what version you are up to, who you sent the last lot of patches +to, or anything about the change logs. + +2. If you put tags in the subject, patman will Cc the maintainers +automatically in many cases. + +3. If you want to keep the commits from each series you sent so that you can +compare change and see what you did, you can either create a new branch for +each version, or just tag the branch before you start changing it: + + git tag sent/us-cmd-rfc + ...later... + git tag sent/us-cmd-v2 + +4. If you want to modify the patches a little before sending, you can do +this in your editor, but be careful! + +5. If you want to run git send-email yourself, use the -n flag which will +print out the command line patman would have used. + +6. It is a good idea to add the change log info as you change the commit, +not later when you can't remember which patch you changed. You can always +go back and change or remove logs from commits. + + +Other thoughts +============== + +This script has been split into sensible files but still needs work. +Most of these are indicated by a TODO in the code. + +It would be nice if this could handle the In-reply-to side of things. + +The tests are incomplete, as is customary. Use the -t flag to run them, +and make sure you are in the tools/scripts/patman directory first: + + $ cd /path/to/u-boot + $ cd tools/scripts/patman + $ patman -t + +Error handling doesn't always produce friendly error messages - e.g. +putting an incorrect tag in a commit may provide a confusing message. + +There might be a few other features not mentioned in this README. They +might be bugs. In particular, tags are case sensitive which is probably +a bad thing. + + +Simon Glass sjg@chromium.org +v1, v2, 19-Oct-11 +revised v3 24-Nov-11 diff --git a/tools/patman/checkpatch.py b/tools/patman/checkpatch.py new file mode 100644 index 0000000..a234277 --- /dev/null +++ b/tools/patman/checkpatch.py @@ -0,0 +1,161 @@ +# Copyright (c) 2011 The Chromium OS Authors. +# +# See file CREDITS for list of people who contributed to this +# project. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License as +# published by the Free Software Foundation; either version 2 of +# the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, +# MA 02111-1307 USA +# + +import command +import gitutil +import os +import re +import terminal + +def FindCheckPatch(): + try_list = [ + os.getcwd(), + os.path.join(os.getcwd(), '..', '..'), + os.path.join(gitutil.GetTopLevel(), 'tools'), + '%s/bin' % os.getenv('HOME'), + ] + # Look in current dir + for path in try_list: + fname = os.path.join(path, 'checkpatch.pl') + if os.path.isfile(fname): + return fname + + # Look upwwards for a Chrome OS tree + while not os.path.ismount(path): + fname = os.path.join(path, 'src', 'third_party', 'kernel', 'files', + 'scripts', 'checkpatch.pl') + if os.path.isfile(fname): + return fname + path = os.path.dirname(path) + print 'Could not find checkpatch.pl' + return None + +def CheckPatch(fname, verbose=False): + """Run checkpatch.pl on a file. + + Returns: + 4-tuple containing: + result: False=failure, True=ok + problems: List of problems, each a dict: + 'type'; error or warning + 'msg': text message + 'file' : filename + 'line': line number + lines: Number of lines + """ + result = False + error_count, warning_count, lines = 0, 0, 0 + problems = [] + chk = FindCheckPatch() + if not chk: + raise OSError, ('Cannot find checkpatch.pl - please put it in your ' + + '~/bin directory') + item = {} + stdout = command.Output(chk, '--no-tree', fname) + #pipe = subprocess.Popen(cmd, stdout=subprocess.PIPE) + #stdout, stderr = pipe.communicate() + + # total: 0 errors, 0 warnings, 159 lines checked + re_stats = re.compile('total: (\d+) errors, (\d+) warnings, (\d+)') + re_ok = re.compile('.*has no obvious style problems') + re_bad = re.compile('.*has style problems, please review') + re_error = re.compile('ERROR: (.*)') + re_warning = re.compile('WARNING: (.*)') + re_file = re.compile('#\d+: FILE: ([^:]*):(\d+):') + + for line in stdout.splitlines(): + if verbose: + print line + + # A blank line indicates the end of a message + if not line and item: + problems.append(item) + item = {} + match = re_stats.match(line) + if match: + error_count = int(match.group(1)) + warning_count = int(match.group(2)) + lines = int(match.group(3)) + elif re_ok.match(line): + result = True + elif re_bad.match(line): + result = False + match = re_error.match(line) + if match: + item['msg'] = match.group(1) + item['type'] = 'error' + match = re_warning.match(line) + if match: + item['msg'] = match.group(1) + item['type'] = 'warning' + match = re_file.match(line) + if match: + item['file'] = match.group(1) + item['line'] = int(match.group(2)) + + return result, problems, error_count, warning_count, lines, stdout + +def GetWarningMsg(col, msg_type, fname, line, msg): + '''Create a message for a given file/line + + Args: + msg_type: Message type ('error' or 'warning') + fname: Filename which reports the problem + line: Line number where it was noticed + msg: Message to report + ''' + if msg_type == 'warning': + msg_type = col.Color(col.YELLOW, msg_type) + elif msg_type == 'error': + msg_type = col.Color(col.RED, msg_type) + return '%s: %s,%d: %s' % (msg_type, fname, line, msg) + +def CheckPatches(verbose, args): + '''Run the checkpatch.pl script on each patch''' + error_count = 0 + warning_count = 0 + col = terminal.Color() + + for fname in args: + ok, problems, errors, warnings, lines, stdout = CheckPatch(fname, + verbose) + if not ok: + error_count += errors + warning_count += warnings + print '%d errors, %d warnings for %s:' % (errors, + warnings, fname) + if len(problems) != error_count + warning_count: + print "Internal error: some problems lost" + for item in problems: + print GetWarningMsg(col, item['type'], item['file'], + item['line'], item['msg']) + #print stdout + if error_count != 0 or warning_count != 0: + str = 'checkpatch.pl found %d error(s), %d warning(s)' % ( + error_count, warning_count) + color = col.GREEN + if warning_count: + color = col.YELLOW + if error_count: + color = col.RED + print col.Color(color, str) + return False + return True diff --git a/tools/patman/command.py b/tools/patman/command.py new file mode 100644 index 0000000..4b00250 --- /dev/null +++ b/tools/patman/command.py @@ -0,0 +1,72 @@ +# Copyright (c) 2011 The Chromium OS Authors. +# +# See file CREDITS for list of people who contributed to this +# project. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License as +# published by the Free Software Foundation; either version 2 of +# the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, +# MA 02111-1307 USA +# + +import os +import subprocess + +"""Shell command ease-ups for Python.""" + +def RunPipe(pipeline, infile=None, outfile=None, + capture=False, oneline=False, hide_stderr=False): + """ + Perform a command pipeline, with optional input/output filenames. + + hide_stderr Don't allow output of stderr (default False) + """ + last_pipe = None + while pipeline: + cmd = pipeline.pop(0) + kwargs = {} + if last_pipe is not None: + kwargs['stdin'] = last_pipe.stdout + elif infile: + kwargs['stdin'] = open(infile, 'rb') + if pipeline or capture: + kwargs['stdout'] = subprocess.PIPE + elif outfile: + kwargs['stdout'] = open(outfile, 'wb') + if hide_stderr: + kwargs['stderr'] = open('/dev/null', 'wb') + + last_pipe = subprocess.Popen(cmd, **kwargs) + + if capture: + ret = last_pipe.communicate()[0] + if not ret: + return None + elif oneline: + return ret.rstrip('\r\n') + else: + return ret + else: + return os.waitpid(last_pipe.pid, 0)[1] == 0 + +def Output(*cmd): + return RunPipe([cmd], capture=True) + +def OutputOneLine(*cmd): + return RunPipe([cmd], capture=True, oneline=True) + +def Run(*cmd, **kwargs): + return RunPipe([cmd], **kwargs) + +def RunList(cmd): + return RunPipe([cmd], capture=True) diff --git a/tools/patman/commit.py b/tools/patman/commit.py new file mode 100644 index 0000000..7144e54 --- /dev/null +++ b/tools/patman/commit.py @@ -0,0 +1,87 @@ +# Copyright (c) 2011 The Chromium OS Authors. +# +# See file CREDITS for list of people who contributed to this +# project. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License as +# published by the Free Software Foundation; either version 2 of +# the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, +# MA 02111-1307 USA +# + +import re + +# Separates a tag: at the beginning of the subject from the rest of it +re_subject_tag = re.compile('([^:]*):\s*(.*)') + +class Commit: + """Holds information about a single commit/patch in the series. + + Args: + hash: Commit hash (as a string) + + Variables: + hash: Commit hash + subject: Subject line + tags: List of maintainer tag strings + changes: Dict containing a list of changes (single line strings). + The dict is indexed by change version (an integer) + cc_list: List of people to aliases/emails to cc on this commit + """ + def __init__(self, hash): + self.hash = hash + self.subject = None + self.tags = [] + self.changes = {} + self.cc_list = [] + + def AddChange(self, version, info): + """Add a new change line to the change list for a version. + + Args: + version: Patch set version (integer: 1, 2, 3) + info: Description of change in this version + """ + if not self.changes.get(version): + self.changes[version] = [] + self.changes[version].append(info) + + def CheckTags(self): + """Create a list of subject tags in the commit + + Subject tags look like this: + + propounder: Change the widget to propound correctly + + Multiple tags are supported. The list is updated in self.tag + + Returns: + None if ok, else the name of a tag with no email alias + """ + str = self.subject + m = True + while m: + m = re_subject_tag.match(str) + if m: + tag = m.group(1) + self.tags.append(tag) + str = m.group(2) + return None + + def AddCc(self, cc_list): + """Add a list of people to Cc when we send this patch. + + Args: + cc_list: List of aliases or email addresses + """ + self.cc_list += cc_list diff --git a/tools/patman/gitutil.py b/tools/patman/gitutil.py new file mode 100644 index 0000000..48ca998 --- /dev/null +++ b/tools/patman/gitutil.py @@ -0,0 +1,372 @@ +# Copyright (c) 2011 The Chromium OS Authors. +# +# See file CREDITS for list of people who contributed to this +# project. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License as +# published by the Free Software Foundation; either version 2 of +# the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, +# MA 02111-1307 USA +# + +import command +import re +import os +import series +import settings +import subprocess +import sys +import terminal + + +def CountCommitsToBranch(): + """Returns number of commits between HEAD and the tracking branch. + + This looks back to the tracking branch and works out the number of commits + since then. + + Return: + Number of patches that exist on top of the branch + """ + pipe = [['git', 'log', '--oneline', '@{upstream}..'], + ['wc', '-l']] + stdout = command.RunPipe(pipe, capture=True, oneline=True) + patch_count = int(stdout) + return patch_count + +def CreatePatches(start, count, series): + """Create a series of patches from the top of the current branch. + + The patch files are written to the current directory using + git format-patch. + + Args: + start: Commit to start from: 0=HEAD, 1=next one, etc. + count: number of commits to include + Return: + Filename of cover letter + List of filenames of patch files + """ + if series.get('version'): + version = '%s ' % series['version'] + cmd = ['git', 'format-patch', '-M', '--signoff'] + if series.get('cover'): + cmd.append('--cover-letter') + prefix = series.GetPatchPrefix() + if prefix: + cmd += ['--subject-prefix=%s' % prefix] + cmd += ['HEAD~%d..HEAD~%d' % (start + count, start)] + + stdout = command.RunList(cmd) + files = stdout.splitlines() + + # We have an extra file if there is a cover letter + if series.get('cover'): + return files[0], files[1:] + else: + return None, files + +def ApplyPatch(verbose, fname): + """Apply a patch with git am to test it + + TODO: Convert these to use command, with stderr option + + Args: + fname: filename of patch file to apply + """ + cmd = ['git', 'am', fname] + pipe = subprocess.Popen(cmd, stdout=subprocess.PIPE, + stderr=subprocess.PIPE) + stdout, stderr = pipe.communicate() + re_error = re.compile('^error: patch failed: (.+):(\d+)') + for line in stderr.splitlines(): + if verbose: + print line + match = re_error.match(line) + if match: + print GetWarningMsg('warning', match.group(1), int(match.group(2)), + 'Patch failed') + return pipe.returncode == 0, stdout + +def ApplyPatches(verbose, args, start_point): + """Apply the patches with git am to make sure all is well + + Args: + verbose: Print out 'git am' output verbatim + args: List of patch files to apply + start_point: Number of commits back from HEAD to start applying. + Normally this is len(args), but it can be larger if a start + offset was given. + """ + error_count = 0 + col = terminal.Color() + + # Figure out our current position + cmd = ['git', 'name-rev', 'HEAD', '--name-only'] + pipe = subprocess.Popen(cmd, stdout=subprocess.PIPE) + stdout, stderr = pipe.communicate() + if pipe.returncode: + str = 'Could not find current commit name' + print col.Color(col.RED, str) + print stdout + return False + old_head = stdout.splitlines()[0] + + # Checkout the required start point + cmd = ['git', 'checkout', 'HEAD~%d' % start_point] + pipe = subprocess.Popen(cmd, stdout=subprocess.PIPE, + stderr=subprocess.PIPE) + stdout, stderr = pipe.communicate() + if pipe.returncode: + str = 'Could not move to commit before patch series' + print col.Color(col.RED, str) + print stdout, stderr + return False + + # Apply all the patches + for fname in args: + ok, stdout = ApplyPatch(verbose, fname) + if not ok: + print col.Color(col.RED, 'git am returned errors for %s: will ' + 'skip this patch' % fname) + if verbose: + print stdout + error_count += 1 + cmd = ['git', 'am', '--skip'] + pipe = subprocess.Popen(cmd, stdout=subprocess.PIPE) + stdout, stderr = pipe.communicate() + if pipe.returncode != 0: + print col.Color(col.RED, 'Unable to skip patch! Aborting...') + print stdout + break + + # Return to our previous position + cmd = ['git', 'checkout', old_head] + pipe = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + stdout, stderr = pipe.communicate() + if pipe.returncode: + print col.Color(col.RED, 'Could not move back to head commit') + print stdout, stderr + return error_count == 0 + +def BuildEmailList(in_list, tag=None, alias=None): + """Build a list of email addresses based on an input list. + + Takes a list of email addresses and aliases, and turns this into a list + of only email address, by resolving any aliases that are present. + + If the tag is given, then each email address is prepended with this + tag and a space. If the tag starts with a minus sign (indicating a + command line parameter) then the email address is quoted. + + Args: + in_list: List of aliases/email addresses + tag: Text to put before each address + + Returns: + List of email addresses + + >>> alias = {} + >>> alias['fred'] = ['f.bloggs@napier.co.nz'] + >>> alias['john'] = ['j.bloggs@napier.co.nz'] + >>> alias['mary'] = ['Mary Poppins m.poppins@cloud.net'] + >>> alias['boys'] = ['fred', ' john'] + >>> alias['all'] = ['fred ', 'john', ' mary '] + >>> BuildEmailList(['john', 'mary'], None, alias) + ['j.bloggs@napier.co.nz', 'Mary Poppins m.poppins@cloud.net'] + >>> BuildEmailList(['john', 'mary'], '--to', alias) + ['--to "j.bloggs@napier.co.nz"', \ +'--to "Mary Poppins m.poppins@cloud.net"'] + >>> BuildEmailList(['john', 'mary'], 'Cc', alias) + ['Cc j.bloggs@napier.co.nz', 'Cc Mary Poppins m.poppins@cloud.net'] + """ + quote = '"' if tag and tag[0] == '-' else '' + raw = [] + for item in in_list: + raw += LookupEmail(item, alias) + result = [] + for item in raw: + if not item in result: + result.append(item) + if tag: + return ['%s %s%s%s' % (tag, quote, email, quote) for email in result] + return result + +def EmailPatches(series, cover_fname, args, dry_run, cc_fname, + self_only=False, alias=None): + """Email a patch series. + + Args: + series: Series object containing destination info + cover_fname: filename of cover letter + args: list of filenames of patch files + dry_run: Just return the command that would be run + cc_fname: Filename of Cc file for per-commit Cc + self_only: True to just email to yourself as a test + + Returns: + Git command that was/would be run + + >>> alias = {} + >>> alias['fred'] = ['f.bloggs@napier.co.nz'] + >>> alias['john'] = ['j.bloggs@napier.co.nz'] + >>> alias['mary'] = ['m.poppins@cloud.net'] + >>> alias['boys'] = ['fred', ' john'] + >>> alias['all'] = ['fred ', 'john', ' mary '] + >>> alias[os.getenv('USER')] = ['this-is-me@me.com'] + >>> series = series.Series() + >>> series.to = ['fred'] + >>> series.cc = ['mary'] + >>> EmailPatches(series, 'cover', ['p1', 'p2'], True, 'cc-fname', False, \ + alias) + 'git send-email --annotate --to "f.bloggs@napier.co.nz" --cc \ +"m.poppins@cloud.net" --cc-cmd "./patman --cc-cmd cc-fname" cover p1 p2' + >>> EmailPatches(series, None, ['p1'], True, 'cc-fname', False, alias) + 'git send-email --annotate --to "f.bloggs@napier.co.nz" --cc \ +"m.poppins@cloud.net" --cc-cmd "./patman --cc-cmd cc-fname" p1' + >>> series.cc = ['all'] + >>> EmailPatches(series, 'cover', ['p1', 'p2'], True, 'cc-fname', True, \ + alias) + 'git send-email --annotate --to "this-is-me@me.com" --cc-cmd "./patman \ +--cc-cmd cc-fname" cover p1 p2' + >>> EmailPatches(series, 'cover', ['p1', 'p2'], True, 'cc-fname', False, \ + alias) + 'git send-email --annotate --to "f.bloggs@napier.co.nz" --cc \ +"f.bloggs@napier.co.nz" --cc "j.bloggs@napier.co.nz" --cc \ +"m.poppins@cloud.net" --cc-cmd "./patman --cc-cmd cc-fname" cover p1 p2' + """ + to = BuildEmailList(series.get('to'), '--to', alias) + if not to: + print ("No recipient, please add something like this to a commit\n" + "Series-to: Fred Bloggs f.blogs@napier.co.nz") + return + cc = BuildEmailList(series.get('cc'), '--cc', alias) + if self_only: + to = BuildEmailList([os.getenv('USER')], '--to', alias) + cc = [] + cmd = ['git', 'send-email', '--annotate'] + cmd += to + cmd += cc + cmd += ['--cc-cmd', '"%s --cc-cmd %s"' % (sys.argv[0], cc_fname)] + if cover_fname: + cmd.append(cover_fname) + cmd += args + str = ' '.join(cmd) + if not dry_run: + os.system(str) + return str + + +def LookupEmail(lookup_name, alias=None, level=0): + """If an email address is an alias, look it up and return the full name + + TODO: Why not just use git's own alias feature? + + Args: + lookup_name: Alias or email address to look up + + Returns: + tuple: + list containing a list of email addresses + + Raises: + OSError if a recursive alias reference was found + ValueError if an alias was not found + + >>> alias = {} + >>> alias['fred'] = ['f.bloggs@napier.co.nz'] + >>> alias['john'] = ['j.bloggs@napier.co.nz'] + >>> alias['mary'] = ['m.poppins@cloud.net'] + >>> alias['boys'] = ['fred', ' john', 'f.bloggs@napier.co.nz'] + >>> alias['all'] = ['fred ', 'john', ' mary '] + >>> alias['loop'] = ['other', 'john', ' mary '] + >>> alias['other'] = ['loop', 'john', ' mary '] + >>> LookupEmail('mary', alias) + ['m.poppins@cloud.net'] + >>> LookupEmail('arthur.wellesley@howe.ro.uk', alias) + ['arthur.wellesley@howe.ro.uk'] + >>> LookupEmail('boys', alias) + ['f.bloggs@napier.co.nz', 'j.bloggs@napier.co.nz'] + >>> LookupEmail('all', alias) + ['f.bloggs@napier.co.nz', 'j.bloggs@napier.co.nz', 'm.poppins@cloud.net'] + >>> LookupEmail('odd', alias) + Traceback (most recent call last): + ... + ValueError: Alias 'odd' not found + >>> LookupEmail('loop', alias) + Traceback (most recent call last): + ... + OSError: Recursive email alias at 'other' + """ + if not alias: + alias = settings.alias + lookup_name = lookup_name.strip() + if '@' in lookup_name: # Perhaps a real email address + return [lookup_name] + + lookup_name = lookup_name.lower() + + if level > 10: + raise OSError, "Recursive email alias at '%s'" % lookup_name + + out_list = [] + if lookup_name: + if not lookup_name in alias: + raise ValueError, "Alias '%s' not found" % lookup_name + for item in alias[lookup_name]: + todo = LookupEmail(item, alias, level + 1) + for new_item in todo: + if not new_item in out_list: + out_list.append(new_item) + + #print "No match for alias '%s'" % lookup_name + return out_list + +def GetTopLevel(): + """Return name of top-level directory for this git repo. + + Returns: + Full path to git top-level directory + + This test makes sure that we are running tests in the right subdir + + >>> os.path.realpath(os.getcwd()) == \ + os.path.join(GetTopLevel(), 'tools', 'scripts', 'patman') + True + """ + return command.OutputOneLine('git', 'rev-parse', '--show-toplevel') + +def GetAliasFile(): + """Gets the name of the git alias file. + + Returns: + Filename of git alias file, or None if none + """ + fname = command.OutputOneLine('git', 'config', 'sendemail.aliasesfile') + if fname: + fname = os.path.join(GetTopLevel(), fname.strip()) + return fname + +def Setup(): + """Set up git utils, by reading the alias files.""" + settings.Setup('') + + # Check for a git alias file also + alias_fname = GetAliasFile() + if alias_fname: + settings.ReadGitAliases(alias_fname) + +if __name__ == "__main__": + import doctest + + doctest.testmod() diff --git a/tools/patman/patchstream.py b/tools/patman/patchstream.py new file mode 100644 index 0000000..be40af3 --- /dev/null +++ b/tools/patman/patchstream.py @@ -0,0 +1,444 @@ +# Copyright (c) 2011 The Chromium OS Authors. +# +# See file CREDITS for list of people who contributed to this +# project. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License as +# published by the Free Software Foundation; either version 2 of +# the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, +# MA 02111-1307 USA +# + +import os +import re +import shutil +import tempfile + +import command +import commit +import gitutil +from series import Series + +# Tags that we detect and remove +re_remove = re.compile('^BUG=|^TEST=|^Change-Id:|^Review URL:' + '|Reviewed-on:|Reviewed-by:') + +# Lines which are allowed after a TEST= line +re_allowed_after_test = re.compile('^Signed-off-by:') + +# The start of the cover letter +re_cover = re.compile('^Cover-letter:') + +# Patch series tag +re_series = re.compile('^Series-(\w*): *(.*)') + +# Commit tags that we want to collect and keep +re_tag = re.compile('^(Tested-by|Acked-by|Signed-off-by|Cc): (.*)') + +# The start of a new commit in the git log +re_commit = re.compile('^commit (.*)') + +# We detect these since checkpatch doesn't always do it +re_space_before_tab = re.compile('^[+].* \t') + +# States we can be in - can we use range() and still have comments? +STATE_MSG_HEADER = 0 # Still in the message header +STATE_PATCH_SUBJECT = 1 # In patch subject (first line of log for a commit) +STATE_PATCH_HEADER = 2 # In patch header (after the subject) +STATE_DIFFS = 3 # In the diff part (past --- line) + +class PatchStream: + """Class for detecting/injecting tags in a patch or series of patches + + We support processing the output of 'git log' to read out the tags we + are interested in. We can also process a patch file in order to remove + unwanted tags or inject additional ones. These correspond to the two + phases of processing. + """ + def __init__(self, series, name=None, is_log=False): + self.skip_blank = False # True to skip a single blank line + self.found_test = False # Found a TEST= line + self.lines_after_test = 0 # MNumber of lines found after TEST= + self.warn = [] # List of warnings we have collected + self.linenum = 1 # Output line number we are up to + self.in_section = None # Name of start...END section we are in + self.notes = [] # Series notes + self.section = [] # The current section...END section + self.series = series # Info about the patch series + self.is_log = is_log # True if indent like git log + self.in_change = 0 # Non-zero if we are in a change list + self.blank_count = 0 # Number of blank lines stored up + self.state = STATE_MSG_HEADER # What state are we in? + self.tags = [] # Tags collected, like Tested-by... + self.signoff = [] # Contents of signoff line + self.commit = None # Current commit + + def AddToSeries(self, line, name, value): + """Add a new Series-xxx tag. + + When a Series-xxx tag is detected, we come here to record it, if we + are scanning a 'git log'. + + Args: + line: Source line containing tag (useful for debug/error messages) + name: Tag name (part after 'Series-') + value: Tag value (part after 'Series-xxx: ') + """ + if name == 'notes': + self.in_section = name + self.skip_blank = False + if self.is_log: + self.series.AddTag(self.commit, line, name, value) + + def CloseCommit(self): + """Save the current commit into our commit list, and reset our state""" + if self.commit and self.is_log: + self.series.AddCommit(self.commit) + self.commit = None + + def FormatTags(self, tags): + out_list = [] + for tag in sorted(tags): + if tag.startswith('Cc:'): + tag_list = tag[4:].split(',') + out_list += gitutil.BuildEmailList(tag_list, 'Cc:') + else: + out_list.append(tag) + return out_list + + def ProcessLine(self, line): + """Process a single line of a patch file or commit log + + This process a line and returns a list of lines to output. The list + may be empty or may contain multiple output lines. + + This is where all the complicated logic is located. The class's + state is used to move between different states and detect things + properly. + + We can be in one of two modes: + self.is_log == True: This is 'git log' mode, where most output is + indented by 4 characters and we are scanning for tags + + self.is_log == False: This is 'patch' mode, where we already have + all the tags, and are processing patches to remove junk we + don't want, and add things we think are required. + + Args: + line: text line to process + + Returns: + list of output lines, or [] if nothing should be output + """ + # Initially we have no output. Prepare the input line string + out = [] + line = line.rstrip('\n') + if self.is_log: + if line[:4] == ' ': + line = line[4:] + + # Handle state transition and skipping blank lines + series_match = re_series.match(line) + commit_match = re_commit.match(line) if self.is_log else None + tag_match = None + if self.state == STATE_PATCH_HEADER: + tag_match = re_tag.match(line) + is_blank = not line.strip() + if is_blank: + if (self.state == STATE_MSG_HEADER + or self.state == STATE_PATCH_SUBJECT): + self.state += 1 + + # We don't have a subject in the text stream of patch files + # It has its own line with a Subject: tag + if not self.is_log and self.state == STATE_PATCH_SUBJECT: + self.state += 1 + elif commit_match: + self.state = STATE_MSG_HEADER + + # If we are in a section, keep collecting lines until we see END + if self.in_section: + if line == 'END': + if self.in_section == 'cover': + self.series.cover = self.section + elif self.in_section == 'notes': + if self.is_log: + self.series.notes += self.section + else: + self.warn.append("Unknown section '%s'" % self.in_section) + self.in_section = None + self.skip_blank = True + self.section = [] + else: + self.section.append(line) + + # Detect the commit subject + elif not is_blank and self.state == STATE_PATCH_SUBJECT: + self.commit.subject = line + + # Detect the tags we want to remove, and skip blank lines + elif re_remove.match(line): + self.skip_blank = True + + # TEST= should be the last thing in the commit, so remove + # everything after it + if line.startswith('TEST='): + self.found_test = True + elif self.skip_blank and is_blank: + self.skip_blank = False + + # Detect the start of a cover letter section + elif re_cover.match(line): + self.in_section = 'cover' + self.skip_blank = False + + # If we are in a change list, key collected lines until a blank one + elif self.in_change: + if is_blank: + # Blank line ends this change list + self.in_change = 0 + else: + self.series.AddChange(self.in_change, self.commit, line) + self.skip_blank = False + + # Detect Series-xxx tags + elif series_match: + name = series_match.group(1) + value = series_match.group(2) + if name == 'changes': + # value is the version number: e.g. 1, or 2 + try: + value = int(value) + except ValueError as str: + raise ValueError("%s: Cannot decode version info '%s'" % + (self.commit.hash, line)) + self.in_change = int(value) + else: + self.AddToSeries(line, name, value) + self.skip_blank = True + + # Detect the start of a new commit + elif commit_match: + self.CloseCommit() + self.commit = commit.Commit(commit_match.group(1)[:7]) + + # Detect tags in the commit message + elif tag_match: + # Onlly allow a single signoff tag + if tag_match.group(1) == 'Signed-off-by': + if self.signoff: + self.warn.append('Patch has more than one Signed-off-by ' + 'tag') + self.signoff += [line] + + # Remove Tested-by self, since few will take much notice + elif (tag_match.group(1) == 'Tested-by' and + tag_match.group(2).find(os.getenv('USER') + '@') != -1): + self.warn.append("Ignoring %s" % line) + elif tag_match.group(1) == 'Cc': + self.commit.AddCc(tag_match.group(2).split(',')) + else: + self.tags.append(line); + + # Well that means this is an ordinary line + else: + pos = 1 + # Look for ugly ASCII characters + for ch in line: + # TODO: Would be nicer to report source filename and line + if ord(ch) > 0x80: + self.warn.append("Line %d/%d ('%s') has funny ascii char" % + (self.linenum, pos, line)) + pos += 1 + + # Look for space before tab + m = re_space_before_tab.match(line) + if m: + self.warn.append('Line %d/%d has space before tab' % + (self.linenum, m.start())) + + # OK, we have a valid non-blank line + out = [line] + self.linenum += 1 + self.skip_blank = False + if self.state == STATE_DIFFS: + pass + + # If this is the start of the diffs section, emit our tags and + # change log + elif line == '---': + self.state = STATE_DIFFS + + # Output the tags (signeoff first), then change list + out = [] + if self.signoff: + out += self.signoff + log = self.series.MakeChangeLog(self.commit) + out += self.FormatTags(self.tags) + out += [line] + log + elif self.found_test: + if not re_allowed_after_test.match(line): + self.lines_after_test += 1 + + return out + + def Finalize(self): + """Close out processing of this patch stream""" + self.CloseCommit() + if self.lines_after_test: + self.warn.append('Found %d lines after TEST=' % + self.lines_after_test) + + def ProcessStream(self, infd, outfd): + """Copy a stream from infd to outfd, filtering out unwanting things. + + This is used to process patch files one at a time. + + Args: + infd: Input stream file object + outfd: Output stream file object + """ + # Extract the filename from each diff, for nice warnings + fname = None + last_fname = None + re_fname = re.compile('diff --git a/(.*) b/.*') + while True: + line = infd.readline() + if not line: + break + out = self.ProcessLine(line) + + # Try to detect blank lines at EOF + for line in out: + match = re_fname.match(line) + if match: + last_fname = fname + fname = match.group(1) + if line == '+': + self.blank_count += 1 + else: + if self.blank_count and (line == '-- ' or match): + self.warn.append("Found possible blank line(s) at " + "end of file '%s'" % last_fname) + outfd.write('+\n' * self.blank_count) + outfd.write(line + '\n') + self.blank_count = 0 + self.Finalize() + + +def GetMetaData(start, count): + """Reads out patch series metadata from the commits + + This does a 'git log' on the relevant commits and pulls out the tags we + are interested in. + + Args: + start: Commit to start from: 0=HEAD, 1=next one, etc. + count: Number of commits to list + """ + pipe = [['git', 'log', '--reverse', 'HEAD~%d' % start, '-n%d' % count]] + stdout = command.RunPipe(pipe, capture=True) + series = Series() + ps = PatchStream(series, is_log=True) + for line in stdout.splitlines(): + ps.ProcessLine(line) + ps.Finalize() + return series + +def FixPatch(backup_dir, fname, series, commit): + """Fix up a patch file, by adding/removing as required. + + We remove our tags from the patch file, insert changes lists, etc. + The patch file is processed in place, and overwritten. + + A backup file is put into backup_dir (if not None). + + Args: + fname: Filename to patch file to process + series: Series information about this patch set + commit: Commit object for this patch file + Return: + A list of errors, or [] if all ok. + """ + handle, tmpname = tempfile.mkstemp() + outfd = os.fdopen(handle, 'w') + infd = open(fname, 'r') + ps = PatchStream(series) + ps.commit = commit + ps.ProcessStream(infd, outfd) + infd.close() + outfd.close() + + # Create a backup file if required + if backup_dir: + shutil.copy(fname, os.path.join(backup_dir, os.path.basename(fname))) + shutil.move(tmpname, fname) + return ps.warn + +def FixPatches(series, fnames): + """Fix up a list of patches identified by filenames + + The patch files are processed in place, and overwritten. + + Args: + series: The series object + fnames: List of patch files to process + """ + # Current workflow creates patches, so we shouldn't need a backup + backup_dir = None #tempfile.mkdtemp('clean-patch') + count = 0 + for fname in fnames: + commit = series.commits[count] + commit.patch = fname + result = FixPatch(backup_dir, fname, series, commit) + if result: + print '%d warnings for %s:' % (len(result), fname) + for warn in result: + print '\t', warn + print + count += 1 + print 'Cleaned %d patches' % count + return series + +def InsertCoverLetter(fname, series, count): + """Inserts a cover letter with the required info into patch 0 + + Args: + fname: Input / output filename of the cover letter file + series: Series object + count: Number of patches in the series + """ + fd = open(fname, 'r') + lines = fd.readlines() + fd.close() + + fd = open(fname, 'w') + text = series.cover + prefix = series.GetPatchPrefix() + for line in lines: + if line.startswith('Subject:'): + # TODO: if more than 10 patches this should save 00/xx, not 0/xx + line = 'Subject: [%s 0/%d] %s\n' % (prefix, count, text[0]) + + # Insert our cover letter + elif line.startswith('*** BLURB HERE ***'): + # First the blurb test + line = '\n'.join(text[1:]) + '\n' + if series.get('notes'): + line += '\n'.join(series.notes) + '\n' + + # Now the change list + out = series.MakeChangeLog(None) + line += '\n' + '\n'.join(out) + fd.write(line) + fd.close() diff --git a/tools/patman/patman b/tools/patman/patman new file mode 120000 index 0000000..6cc3d7a --- /dev/null +++ b/tools/patman/patman @@ -0,0 +1 @@ +patman.py \ No newline at end of file diff --git a/tools/patman/patman.py b/tools/patman/patman.py new file mode 100755 index 0000000..cfe06d0 --- /dev/null +++ b/tools/patman/patman.py @@ -0,0 +1,153 @@ +#!/usr/bin/python +# +# Copyright (c) 2011 The Chromium OS Authors. +# +# See file CREDITS for list of people who contributed to this +# project. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License as +# published by the Free Software Foundation; either version 2 of +# the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, +# MA 02111-1307 USA +# + +"""See README for more information""" + +from optparse import OptionParser +import os +import re +import sys +import unittest + +# Our modules +import checkpatch +import command +import gitutil +import patchstream +import terminal +import test + + +parser = OptionParser() +parser.add_option('-H', '--full-help', action='store_true', dest='full_help', + default=False, help='Display the README file') +parser.add_option('-c', '--count', dest='count', type='int', + default=-1, help='Automatically create patches from top n commits') +parser.add_option('-i', '--ignore-errors', action='store_true', + dest='ignore_errors', default=False, + help='Send patches email even if patch errors are found') +parser.add_option('-n', '--dry-run', action='store_true', dest='dry_run', + default=False, help="Do a try run (create but don't email patches)") +parser.add_option('-s', '--start', dest='start', type='int', + default=0, help='Commit to start creating patches from (0 = HEAD)') +parser.add_option('-t', '--test', action='store_true', dest='test', + default=False, help='run tests') +parser.add_option('-v', '--verbose', action='store_true', dest='verbose', + default=False, help='Verbose output of errors and warnings') +parser.add_option('--cc-cmd', dest='cc_cmd', type='string', action='store', + default=None, help='Output cc list for patch file (used by git)') +parser.add_option('--no-tags', action='store_false', dest='process_tags', + default=True, help="Don't process subject tags as aliaes") + +parser.usage = """patman [options] + +Create patches from commits in a branch, check them and email them as +specified by tags you place in the commits. Use -n to """ + +(options, args) = parser.parse_args() + +# Run our meagre tests +if options.test: + import doctest + + sys.argv = [sys.argv[0]] + suite = unittest.TestLoader().loadTestsFromTestCase(test.TestPatch) + result = unittest.TestResult() + suite.run(result) + + suite = doctest.DocTestSuite('gitutil') + suite.run(result) + + # TODO: Surely we can just 'print' result? + print result + for test, err in result.errors: + print err + for test, err in result.failures: + print err + +# Called from git with a patch filename as argument +# Printout a list of additional CC recipients for this patch +elif options.cc_cmd: + fd = open(options.cc_cmd, 'r') + re_line = re.compile('(\S*) (.*)') + for line in fd.readlines(): + match = re_line.match(line) + if match and match.group(1) == args[0]: + for cc in match.group(2).split(', '): + cc = cc.strip() + if cc: + print cc + fd.close() + +elif options.full_help: + pager = os.getenv('PAGER') + if not pager: + pager = 'more' + fname = os.path.join(os.path.dirname(sys.argv[0]), 'README') + command.Run(pager, fname) + +# Process commits, produce patches files, check them, email them +else: + gitutil.Setup() + + if options.count == -1: + # Work out how many patches to send if we can + options.count = gitutil.CountCommitsToBranch() - options.start + + col = terminal.Color() + if not options.count: + str = 'No commits found to process - please use -c flag' + print col.Color(col.RED, str) + sys.exit(1) + + # Read the metadata from the commits + if options.count: + series = patchstream.GetMetaData(options.start, options.count) + cover_fname, args = gitutil.CreatePatches(options.start, options.count, + series) + + # Fix up the patch files to our liking, and insert the cover letter + series = patchstream.FixPatches(series, args) + if series and cover_fname and series.get('cover'): + patchstream.InsertCoverLetter(cover_fname, series, options.count) + + # Do a few checks on the series + series.DoChecks() + + # Check the patches, and run them through 'git am' just to be sure + ok = checkpatch.CheckPatches(options.verbose, args) + if not gitutil.ApplyPatches(options.verbose, args, + options.count + options.start): + ok = False + + # Email the patches out (giving the user time to check / cancel) + cmd = '' + if ok or options.ignore_errors: + cc_file = series.MakeCcFile(options.process_tags) + cmd = gitutil.EmailPatches(series, cover_fname, args, + options.dry_run, cc_file) + os.remove(cc_file) + + # For a dry run, just show our actions as a sanity check + if options.dry_run: + series.ShowActions(args, cmd, options.process_tags) diff --git a/tools/patman/series.py b/tools/patman/series.py new file mode 100644 index 0000000..05d9e73 --- /dev/null +++ b/tools/patman/series.py @@ -0,0 +1,238 @@ +# Copyright (c) 2011 The Chromium OS Authors. +# +# See file CREDITS for list of people who contributed to this +# project. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License as +# published by the Free Software Foundation; either version 2 of +# the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, +# MA 02111-1307 USA +# + +import os + +import gitutil +import terminal + +# Series-xxx tags that we understand +valid_series = ['to', 'cc', 'version', 'changes', 'prefix', 'notes']; + +class Series(dict): + """Holds information about a patch series, including all tags. + + Vars: + cc: List of aliases/emails to Cc all patches to + commits: List of Commit objects, one for each patch + cover: List of lines in the cover letter + notes: List of lines in the notes + changes: (dict) List of changes for each version, The key is + the integer version number + """ + def __init__(self): + self.cc = [] + self.to = [] + self.commits = [] + self.cover = None + self.notes = [] + self.changes = {} + + # These make us more like a dictionary + def __setattr__(self, name, value): + self[name] = value + + def __getattr__(self, name): + return self[name] + + def AddTag(self, commit, line, name, value): + """Add a new Series-xxx tag along with its value. + + Args: + line: Source line containing tag (useful for debug/error messages) + name: Tag name (part after 'Series-') + value: Tag value (part after 'Series-xxx: ') + """ + # If we already have it, then add to our list + if name in self: + values = value.split(',') + values = [str.strip() for str in values] + if type(self[name]) != type([]): + raise ValueError("In %s: line '%s': Cannot add another value " + "'%s' to series '%s'" % + (commit.hash, line, values, self[name])) + self[name] += values + + # Otherwise just set the value + elif name in valid_series: + self[name] = value + else: + raise ValueError("In %s: line '%s': Unknown 'Series-%s': valid " + "options are %s" % (self.commit.hash, line, name, + ', '.join(valid_series))) + + def AddCommit(self, commit): + """Add a commit into our list of commits + + We create a list of tags in the commit subject also. + + Args: + commit: Commit object to add + """ + commit.CheckTags() + self.commits.append(commit) + + def ShowActions(self, args, cmd, process_tags): + """Show what actions we will/would perform + + Args: + args: List of patch files we created + cmd: The git command we would have run + process_tags: Process tags as if they were aliases + """ + col = terminal.Color() + print 'Dry run, so not doing much. But I would do this:' + print + print 'Send a total of %d patch%s with %scover letter.' % ( + len(args), '' if len(args) == 1 else 'es', + self.get('cover') and 'a ' or 'no ') + + # TODO: Colour the patches according to whether they passed checks + for upto in range(len(args)): + commit = self.commits[upto] + print col.Color(col.GREEN, ' %s' % args[upto]) + cc_list = [] + if process_tags: + cc_list += gitutil.BuildEmailList(commit.tags) + cc_list += gitutil.BuildEmailList(commit.cc_list) + + for email in cc_list: + if email == None: + email = col.Color(col.YELLOW, "<alias '%s' not found>" + % tag) + if email: + print ' Cc: ',email + print + for item in gitutil.BuildEmailList(self.get('to', '<none>')): + print 'To:\t ', item + for item in gitutil.BuildEmailList(self.cc): + print 'Cc:\t ', item + print 'Version: ', self.get('version') + print 'Prefix:\t ', self.get('prefix') + if self.cover: + print 'Cover: %d lines' % len(self.cover) + if cmd: + print 'Git command: %s' % cmd + + def MakeChangeLog(self, commit): + """Create a list of changes for each version. + + Return: + The change log as a list of strings, one per line + + Changes in v1: + - Fix the widget + - Jog the dial + + Changes in v2: + - Jog the dial back closer to the widget + + etc. + """ + final = [] + need_blank = False + for change in sorted(self.changes): + out = [] + for this_commit, text in self.changes[change]: + if commit and this_commit != commit: + continue + if text not in out: + out.append(text) + if out: + out = ['Changes in v%d:' % change] + sorted(out) + if need_blank: + out = [''] + out + final += out + need_blank = True + if self.changes: + final.append('') + return final + + def DoChecks(self): + """Check that each version has a change log + + Print an error if something is wrong. + """ + col = terminal.Color() + if self.get('version'): + changes_copy = dict(self.changes) + for version in range(2, int(self.version) + 1): + if self.changes.get(version): + del changes_copy[version] + else: + str = 'Change log missing for v%d' % version + print col.Color(col.RED, str) + for version in changes_copy: + str = 'Change log for unknown version v%d' % version + print col.Color(col.RED, str) + elif self.changes: + str = 'Change log exists, but no version is set' + print col.Color(col.RED, str) + + def MakeCcFile(self, process_tags): + """Make a cc file for us to use for per-commit Cc automation + + Args: + process_tags: Process tags as if they were aliases + Return: + Filename of temp file created + """ + # Look for commit tags (of the form 'xxx:' at the start of the subject) + fname = '/tmp/patman.%d' % os.getpid() + fd = open(fname, 'w') + for commit in self.commits: + list = [] + if process_tags: + list += gitutil.BuildEmailList(commit.tags) + list += gitutil.BuildEmailList(commit.cc_list) + print >>fd, commit.patch, ', '.join(list) + + fd.close() + return fname + + def AddChange(self, version, commit, info): + """Add a new change line to a version. + + This will later appear in the change log. + + Args: + version: version number to add change list to + info: change line for this version + """ + if not self.changes.get(version): + self.changes[version] = [] + self.changes[version].append([commit, info]) + + def GetPatchPrefix(self): + """Get the patch version string + + Return: + Patch string, like 'RFC PATCH v5' or just 'PATCH' + """ + version = '' + if self.get('version'): + version = ' v%s' % self['version'] + + # Get patch name prefix + prefix = '' + if self.get('prefix'): + prefix = '%s ' % self['prefix'] + return '%sPATCH%s' % (prefix, version) diff --git a/tools/patman/settings.py b/tools/patman/settings.py new file mode 100644 index 0000000..049c709 --- /dev/null +++ b/tools/patman/settings.py @@ -0,0 +1,81 @@ +# Copyright (c) 2011 The Chromium OS Authors. +# +# See file CREDITS for list of people who contributed to this +# project. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License as +# published by the Free Software Foundation; either version 2 of +# the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, +# MA 02111-1307 USA +# + +import ConfigParser +import os +import re + +import command + + +def ReadGitAliases(fname): + """Read a git alias file. This is in the form used by git: + + alias uboot u-boot@lists.denx.de + alias wd Wolfgang Denk wd@denx.de + + Args: + fname: Filename to read + """ + try: + fd = open(fname, 'r') + except IOError: + print "Warning: Cannot find alias file '%s'" % fname + return + + re_line = re.compile('alias\s+(\S+)\s+(.*)') + for line in fd.readlines(): + line = line.strip() + if not line or line[0] == '#': + continue + + m = re_line.match(line) + if not m: + print "Warning: Alias file line '%s' not understood" % line + continue + + list = alias.get(m.group(1), []) + for item in m.group(2).split(','): + item = item.strip() + if item: + list.append(item) + alias[m.group(1)] = list + + fd.close() + +def Setup(config_fname=''): + """Set up the settings module by reading config files. + + Args: + config_fname: Config filename to read ('' for default) + """ + settings = ConfigParser.SafeConfigParser() + if config_fname == '': + config_fname = '%s/.config/patman' % os.getenv('HOME') + if config_fname: + settings.read(config_fname) + + for name, value in settings.items('alias'): + alias[name] = value.split(',') + + +# These are the aliases we understand, indexed by alias. Each member is a list. +alias = {} diff --git a/tools/patman/terminal.py b/tools/patman/terminal.py new file mode 100644 index 0000000..838c828 --- /dev/null +++ b/tools/patman/terminal.py @@ -0,0 +1,86 @@ +# Copyright (c) 2011 The Chromium OS Authors. +# +# See file CREDITS for list of people who contributed to this +# project. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License as +# published by the Free Software Foundation; either version 2 of +# the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, +# MA 02111-1307 USA +# + +"""Terminal utilities + +This module handles terminal interaction including ANSI color codes. +""" + +class Color(object): + """Conditionally wraps text in ANSI color escape sequences.""" + BLACK, RED, GREEN, YELLOW, BLUE, MAGENTA, CYAN, WHITE = range(8) + BOLD = -1 + COLOR_START = '\033[1;%dm' + BOLD_START = '\033[1m' + RESET = '\033[0m' + + def __init__(self, enabled=True): + """Create a new Color object, optionally disabling color output. + + Args: + enabled: True if color output should be enabled. If False then this + class will not add color codes at all. + """ + self._enabled = enabled + + def Start(self, color): + """Returns a start color code. + + Args: + color: Color to use, .e.g BLACK, RED, etc. + + Returns: + If color is enabled, returns an ANSI sequence to start the given color, + otherwise returns empty string + """ + if self._enabled: + return self.COLOR_START % (color + 30) + return '' + + def Stop(self): + """Retruns a stop color code. + + Returns: + If color is enabled, returns an ANSI color reset sequence, otherwise + returns empty string + """ + if self._enabled: + return self.RESET + return '' + + def Color(self, color, text): + """Returns text with conditionally added color escape sequences. + + Keyword arguments: + color: Text color -- one of the color constants defined in this class. + text: The text to color. + + Returns: + If self._enabled is False, returns the original text. If it's True, + returns text with color escape sequences based on the value of color. + """ + if not self._enabled: + return text + if color == self.BOLD: + start = self.BOLD_START + else: + start = self.COLOR_START % (color + 30) + return start + text + self.RESET diff --git a/tools/patman/test.py b/tools/patman/test.py new file mode 100644 index 0000000..f801ced --- /dev/null +++ b/tools/patman/test.py @@ -0,0 +1,250 @@ +# +# Copyright (c) 2011 The Chromium OS Authors. +# +# See file CREDITS for list of people who contributed to this +# project. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License as +# published by the Free Software Foundation; either version 2 of +# the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, +# MA 02111-1307 USA +# + +import os +import tempfile +import unittest + +import checkpatch +import gitutil +import patchstream +import series + + +class TestPatch(unittest.TestCase): + """Test this program + + TODO: Write tests for the rest of the functionality + """ + + def testBasic(self): + """Test basic filter operation""" + data=''' + +From 656c9a8c31fa65859d924cd21da920d6ba537fad Mon Sep 17 00:00:00 2001 +From: Simon Glass sjg@chromium.org +Date: Thu, 28 Apr 2011 09:58:51 -0700 +Subject: [PATCH (resend) 3/7] Tegra2: Add more clock support + +This adds functions to enable/disable clocks and reset to on-chip peripherals. + +BUG=chromium-os:13875 +TEST=build U-Boot for Seaboard, boot + +Change-Id: I80fe1d0c0b7dd10aa58ce5bb1d9290b6664d5413 + +Review URL: http://codereview.chromium.org/6900006 + +Signed-off-by: Simon Glass sjg@chromium.org +--- + arch/arm/cpu/armv7/tegra2/Makefile | 2 +- + arch/arm/cpu/armv7/tegra2/ap20.c | 57 ++---- + arch/arm/cpu/armv7/tegra2/clock.c | 163 +++++++++++++++++ +''' + expected=''' + +From 656c9a8c31fa65859d924cd21da920d6ba537fad Mon Sep 17 00:00:00 2001 +From: Simon Glass sjg@chromium.org +Date: Thu, 28 Apr 2011 09:58:51 -0700 +Subject: [PATCH (resend) 3/7] Tegra2: Add more clock support + +This adds functions to enable/disable clocks and reset to on-chip peripherals. + +Signed-off-by: Simon Glass sjg@chromium.org +--- + arch/arm/cpu/armv7/tegra2/Makefile | 2 +- + arch/arm/cpu/armv7/tegra2/ap20.c | 57 ++---- + arch/arm/cpu/armv7/tegra2/clock.c | 163 +++++++++++++++++ +''' + out = '' + inhandle, inname = tempfile.mkstemp() + infd = os.fdopen(inhandle, 'w') + infd.write(data) + infd.close() + + exphandle, expname = tempfile.mkstemp() + expfd = os.fdopen(exphandle, 'w') + expfd.write(expected) + expfd.close() + + patchstream.FixPatch(None, inname, series.Series(), None) + rc = os.system('diff -u %s %s' % (inname, expname)) + self.assertEqual(rc, 0) + + os.remove(inname) + os.remove(expname) + + def GetData(self, data_type): + data=''' +From 4924887af52713cabea78420eff03badea8f0035 Mon Sep 17 00:00:00 2001 +From: Simon Glass sjg@chromium.org +Date: Thu, 7 Apr 2011 10:14:41 -0700 +Subject: [PATCH 1/4] Add microsecond boot time measurement + +This defines the basics of a new boot time measurement feature. This allows +logging of very accurate time measurements as the boot proceeds, by using +an available microsecond counter. + +%s +--- + README | 11 ++++++++ + common/bootstage.c | 50 ++++++++++++++++++++++++++++++++++++ + include/bootstage.h | 71 +++++++++++++++++++++++++++++++++++++++++++++++++++ + include/common.h | 8 ++++++ + 5 files changed, 141 insertions(+), 0 deletions(-) + create mode 100644 common/bootstage.c + create mode 100644 include/bootstage.h + +diff --git a/README b/README +index 6f3748d..f9e4e65 100644 +--- a/README ++++ b/README +@@ -2026,6 +2026,17 @@ The following options need to be configured: + example, some LED's) on your board. At the moment, + the following checkpoints are implemented: + ++- Time boot progress ++ CONFIG_BOOTSTAGE ++ ++ Define this option to enable microsecond boot stage timing ++ on supported platforms. For this to work your platform ++ needs to define a function timer_get_us() which returns the ++ number of microseconds since reset. This would normally ++ be done in your SOC or board timer.c file. ++ ++ You can add calls to bootstage_mark() to set time markers. ++ + - Standalone program support: + CONFIG_STANDALONE_LOAD_ADDR + +diff --git a/common/bootstage.c b/common/bootstage.c +new file mode 100644 +index 0000000..2234c87 +--- /dev/null ++++ b/common/bootstage.c +@@ -0,0 +1,50 @@ ++/* ++ * Copyright (c) 2011, Google Inc. All rights reserved. ++ * ++ * See file CREDITS for list of people who contributed to this ++ * project. ++ * ++ * This program is free software; you can redistribute it and/or ++ * modify it under the terms of the GNU General Public License as ++ * published by the Free Software Foundation; either version 2 of ++ * the License, or (at your option) any later version. ++ * ++ * This program is distributed in the hope that it will be useful, ++ * but WITHOUT ANY WARRANTY; without even the implied warranty of ++ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the ++ * GNU General Public License for more details. ++ * ++ * You should have received a copy of the GNU General Public License ++ * along with this program; if not, write to the Free Software ++ * Foundation, Inc., 59 Temple Place, Suite 330, Boston, ++ * MA 02111-1307 USA ++ */ ++ ++ ++/* ++ * This module records the progress of boot and arbitrary commands, and ++ * permits accurate timestamping of each. The records can optionally be ++ * passed to kernel in the ATAGs ++ */ ++ ++#include <common.h> ++ ++ ++struct bootstage_record { ++ uint32_t time_us; ++ const char *name; ++}; ++ ++static struct bootstage_record record[BOOTSTAGE_COUNT]; ++ ++uint32_t bootstage_mark(enum bootstage_id id, const char *name) ++{ ++ struct bootstage_record *rec = &record[id]; ++ ++ /* Only record the first event for each */ ++%sif (!rec->name) { ++ rec->time_us = (uint32_t)timer_get_us(); ++ rec->name = name; ++ } ++%sreturn rec->time_us; ++} +-- +1.7.3.1 +''' + signoff = 'Signed-off-by: Simon Glass sjg@chromium.org\n' + tab = ' ' + if data_type == 'good': + pass + elif data_type == 'no-signoff': + signoff = '' + elif data_type == 'spaces': + tab = ' ' + else: + print 'not implemented' + return data % (signoff, tab, tab) + + def SetupData(self, data_type): + inhandle, inname = tempfile.mkstemp() + infd = os.fdopen(inhandle, 'w') + data = self.GetData(data_type) + infd.write(data) + infd.close() + return inname + + def testCheckpatch(self): + """Test checkpatch operation""" + inf = self.SetupData('good') + result, problems, err, warn, lines, stdout = checkpatch.CheckPatch(inf) + self.assertEqual(result, True) + self.assertEqual(problems, []) + self.assertEqual(err, 0) + self.assertEqual(warn, 0) + self.assertEqual(lines, 67) + os.remove(inf) + + inf = self.SetupData('no-signoff') + result, problems, err, warn, lines, stdout = checkpatch.CheckPatch(inf) + self.assertEqual(result, False) + self.assertEqual(len(problems), 1) + self.assertEqual(err, 1) + self.assertEqual(warn, 0) + self.assertEqual(lines, 67) + os.remove(inf) + + inf = self.SetupData('spaces') + result, problems, err, warn, lines, stdout = checkpatch.CheckPatch(inf) + self.assertEqual(result, False) + self.assertEqual(len(problems), 2) + self.assertEqual(err, 0) + self.assertEqual(warn, 2) + self.assertEqual(lines, 67) + os.remove(inf) + + +if __name__ == "__main__": + unittest.main() + gitutil.RunTests()