[PATCH v2 1/2] binman: Do not pollute source tree when build with `make O=...`

Importing libraries in Python caches the bytecode by default. Since we run scripts in source tree it ignores the current directory settings, which is $(srctree), and creates cache just in the middle of the source tree. Move cache to the current directory.
Signed-off-by: Andy Shevchenko andriy.shevchenko@linux.intel.com --- v2: reused our_path tools/binman/main.py | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
diff --git a/tools/binman/main.py b/tools/binman/main.py index 8c1e478d54ce..4d8b124c7468 100755 --- a/tools/binman/main.py +++ b/tools/binman/main.py @@ -16,9 +16,18 @@ import sys import traceback import unittest
+# Get the absolute path to this file at run-time +our_path = os.path.dirname(sys.argv[0]) + +# +# Do not pollute source tree with cache files: +# https://stackoverflow.com/a/60024195/2511795 +# https://bugs.python.org/issue33499 +# +sys.pycache_prefix = os.path.relpath(our_path, os.environ['srctree']) + # Bring in the patman and dtoc libraries (but don't override the first path # in PYTHONPATH) -our_path = os.path.dirname(os.path.realpath(__file__)) sys.path.insert(2, os.path.join(our_path, '..'))
from patman import test_util

Instead of joining hard coded '..' to the run-time path of the executable, take just a dirname out of it. Besides that, use $(srctree) where it makes sense.
Signed-off-by: Andy Shevchenko andriy.shevchenko@linux.intel.com --- v2: new patch tools/binman/main.py | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-)
diff --git a/tools/binman/main.py b/tools/binman/main.py index 4d8b124c7468..2b3e58d1aabd 100755 --- a/tools/binman/main.py +++ b/tools/binman/main.py @@ -19,24 +19,26 @@ import unittest # Get the absolute path to this file at run-time our_path = os.path.dirname(sys.argv[0])
+# Extract $(srctree) from Kbuild environment +srctree = os.environ['srctree'] + # # Do not pollute source tree with cache files: # https://stackoverflow.com/a/60024195/2511795 # https://bugs.python.org/issue33499 # -sys.pycache_prefix = os.path.relpath(our_path, os.environ['srctree']) +sys.pycache_prefix = os.path.relpath(our_path, srctree)
# Bring in the patman and dtoc libraries (but don't override the first path # in PYTHONPATH) -sys.path.insert(2, os.path.join(our_path, '..')) +sys.path.insert(2, os.path.dirname(our_path))
from patman import test_util
# Bring in the libfdt module sys.path.insert(2, 'scripts/dtc/pylibfdt') -sys.path.insert(2, os.path.join(our_path, '../../scripts/dtc/pylibfdt')) -sys.path.insert(2, os.path.join(our_path, - '../../build-sandbox_spl/scripts/dtc/pylibfdt')) +sys.path.insert(2, os.path.join(srctree, 'scripts/dtc/pylibfdt')) +sys.path.insert(2, os.path.join(srctree, 'build-sandbox_spl/scripts/dtc/pylibfdt'))
# When running under python-coverage on Ubuntu 16.04, the dist-packages # directories are dropped from the python path. Add them in so that we can find

Hi Andy,
On Tue, 30 Nov 2021 at 12:04, Andy Shevchenko andriy.shevchenko@linux.intel.com wrote:
Instead of joining hard coded '..' to the run-time path of the executable, take just a dirname out of it. Besides that, use $(srctree) where it makes sense.
Signed-off-by: Andy Shevchenko andriy.shevchenko@linux.intel.com
v2: new patch tools/binman/main.py | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-)
Looks good, apart from my comment on the previous patch, which applies here too.
Regards, Simon

Hi Andy,
On Tue, 30 Nov 2021 at 12:04, Andy Shevchenko andriy.shevchenko@linux.intel.com wrote:
Importing libraries in Python caches the bytecode by default. Since we run scripts in source tree it ignores the current directory settings, which is $(srctree), and creates cache just in the middle of the source tree. Move cache to the current directory.
Signed-off-by: Andy Shevchenko andriy.shevchenko@linux.intel.com
v2: reused our_path tools/binman/main.py | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
This look useful, but we cannot rely on 'srcdir' being in the environment. For example, most binman development is done just by running 'binman test' in the source tre. So perhaps default to the current directory is 'srcdir' is not set?
Regards, Simon
diff --git a/tools/binman/main.py b/tools/binman/main.py index 8c1e478d54ce..4d8b124c7468 100755 --- a/tools/binman/main.py +++ b/tools/binman/main.py @@ -16,9 +16,18 @@ import sys import traceback import unittest
+# Get the absolute path to this file at run-time +our_path = os.path.dirname(sys.argv[0])
+# +# Do not pollute source tree with cache files: +# https://stackoverflow.com/a/60024195/2511795 +# https://bugs.python.org/issue33499 +# +sys.pycache_prefix = os.path.relpath(our_path, os.environ['srctree'])
# Bring in the patman and dtoc libraries (but don't override the first path # in PYTHONPATH) -our_path = os.path.dirname(os.path.realpath(__file__)) sys.path.insert(2, os.path.join(our_path, '..'))
from patman import test_util
2.33.0

On Friday, December 3, 2021, Simon Glass sjg@chromium.org wrote:
Hi Andy,
On Tue, 30 Nov 2021 at 12:04, Andy Shevchenko andriy.shevchenko@linux.intel.com wrote:
Importing libraries in Python caches the bytecode by default. Since we run scripts in source tree it ignores the current directory settings, which is $(srctree), and creates cache just in the middle of the source tree. Move cache to the current directory.
Signed-off-by: Andy Shevchenko andriy.shevchenko@linux.intel.com
v2: reused our_path tools/binman/main.py | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
This look useful, but we cannot rely on 'srcdir' being in the environment.
True and code is aware of that. Nothing needs to be fixed.
For example, most binman development is done just by running 'binman test' in the source tre. So perhaps default to the current directory is 'srcdir' is not set?
Regards, Simon
diff --git a/tools/binman/main.py b/tools/binman/main.py index 8c1e478d54ce..4d8b124c7468 100755 --- a/tools/binman/main.py +++ b/tools/binman/main.py @@ -16,9 +16,18 @@ import sys import traceback import unittest
+# Get the absolute path to this file at run-time +our_path = os.path.dirname(sys.argv[0])
+# +# Do not pollute source tree with cache files: +# https://stackoverflow.com/a/60024195/2511795 +# https://bugs.python.org/issue33499 +# +sys.pycache_prefix = os.path.relpath(our_path, os.environ['srctree'])
# Bring in the patman and dtoc libraries (but don't override the first
path
# in PYTHONPATH) -our_path = os.path.dirname(os.path.realpath(__file__)) sys.path.insert(2, os.path.join(our_path, '..'))
from patman import test_util
2.33.0

Hi Andy,
On Fri, 3 Dec 2021 at 00:55, Andy Shevchenko andy.shevchenko@gmail.com wrote:
On Friday, December 3, 2021, Simon Glass sjg@chromium.org wrote:
Hi Andy,
On Tue, 30 Nov 2021 at 12:04, Andy Shevchenko andriy.shevchenko@linux.intel.com wrote:
Importing libraries in Python caches the bytecode by default. Since we run scripts in source tree it ignores the current directory settings, which is $(srctree), and creates cache just in the middle of the source tree. Move cache to the current directory.
Signed-off-by: Andy Shevchenko andriy.shevchenko@linux.intel.com
v2: reused our_path tools/binman/main.py | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
This look useful, but we cannot rely on 'srcdir' being in the environment.
True and code is aware of that. Nothing needs to be fixed.
What am I missing?
$ binman test Traceback (most recent call last): File "/home/sglass/bin/binman", line 23, in <module> srctree = os.environ['srctree'] File "/usr/lib/python3.8/os.py", line 675, in __getitem__ raise KeyError(key) from None KeyError: 'srctree'
For example, most binman development is done just by running 'binman test' in the source tre. So perhaps default to the current directory is 'srcdir' is not set?
[..]
Regards, Simon

On Fri, Dec 03, 2021 at 01:13:12PM -0700, Simon Glass wrote:
On Fri, 3 Dec 2021 at 00:55, Andy Shevchenko andy.shevchenko@gmail.com wrote:
On Friday, December 3, 2021, Simon Glass sjg@chromium.org wrote:
On Tue, 30 Nov 2021 at 12:04, Andy Shevchenko andriy.shevchenko@linux.intel.com wrote:
...
This look useful, but we cannot rely on 'srcdir' being in the environment.
True and code is aware of that. Nothing needs to be fixed.
What am I missing?
$ binman test Traceback (most recent call last): File "/home/sglass/bin/binman", line 23, in <module> srctree = os.environ['srctree'] File "/usr/lib/python3.8/os.py", line 675, in __getitem__ raise KeyError(key) from None KeyError: 'srctree'
I see, you mean that you run it when it's not in build tree?
For example, most binman development is done just by running 'binman test' in the source tre. So perhaps default to the current directory is 'srcdir' is not set?
Ah, you mean you run it manually and not via `make`.
os.environ.get('srctree', '')
should help I suppose.
P.S. What is the 'srcdir' you are referring to all the time?

On Fri, Dec 03, 2021 at 10:29:10PM +0200, Andy Shevchenko wrote:
On Fri, Dec 03, 2021 at 01:13:12PM -0700, Simon Glass wrote:
...
Ah, you mean you run it manually and not via `make`.
os.environ.get('srctree', '')
should help I suppose.
I haven't heard back from you, I suppose that the above solves the issue, I'm about to send v3.
P.S. What is the 'srcdir' you are referring to all the time?
And please clarify, maybe I missed something, what is the 'srcdir'? Should I use it instead of 'srctree' or what?
participants (3)
-
Andy Shevchenko
-
Andy Shevchenko
-
Simon Glass