[PATCH v3 1/2] binman: Do not pollute source tree when build with `make O=...`
Importing libraries in Python caches the bytecode by default. Since we run scripts in source tree it ignores the current directory settings, which is $(srctree), and creates cache just in the middle of the source tree. Move cache to the current directory.
Signed-off-by: Andy Shevchenko andriy.shevchenko@linux.intel.com --- v3: avoided crash (Simon), preserved tree hierarchy tools/binman/main.py | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)
diff --git a/tools/binman/main.py b/tools/binman/main.py index 8c1e478d54ce..d19ded491af7 100755 --- a/tools/binman/main.py +++ b/tools/binman/main.py @@ -16,9 +16,20 @@ import sys import traceback import unittest
+# Get the absolute path to this file at run-time +our_path = os.path.dirname(os.path.realpath(__file__)) +our1_path = os.path.dirname(our_path) +our2_path = os.path.dirname(our1_path) + +# +# Do not pollute source tree with cache files: +# https://stackoverflow.com/a/60024195/2511795 +# https://bugs.python.org/issue33499 +# +sys.pycache_prefix = os.path.relpath(our_path, os.environ.get('srctree', our2_path)) + # Bring in the patman and dtoc libraries (but don't override the first path # in PYTHONPATH) -our_path = os.path.dirname(os.path.realpath(__file__)) sys.path.insert(2, os.path.join(our_path, '..'))
from patman import test_util
Instead of joining hard coded '..' to the run-time path of the executable, take just a dirname out of it. Besides that, use $(srctree) where it makes sense.
Signed-off-by: Andy Shevchenko andriy.shevchenko@linux.intel.com --- v3: avoided crash (Simon), reused one level up folder variable tools/binman/main.py | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-)
diff --git a/tools/binman/main.py b/tools/binman/main.py index d19ded491af7..6f5ee4e21db7 100755 --- a/tools/binman/main.py +++ b/tools/binman/main.py @@ -21,24 +21,26 @@ our_path = os.path.dirname(os.path.realpath(__file__)) our1_path = os.path.dirname(our_path) our2_path = os.path.dirname(our1_path)
+# Extract $(srctree) from Kbuild environment, or use relative paths below +srctree = os.environ.get('srctree', our2_path) + # # Do not pollute source tree with cache files: # https://stackoverflow.com/a/60024195/2511795 # https://bugs.python.org/issue33499 # -sys.pycache_prefix = os.path.relpath(our_path, os.environ.get('srctree', our2_path)) +sys.pycache_prefix = os.path.relpath(our_path, srctree)
# Bring in the patman and dtoc libraries (but don't override the first path # in PYTHONPATH) -sys.path.insert(2, os.path.join(our_path, '..')) +sys.path.insert(2, our1_path)
from patman import test_util
# Bring in the libfdt module sys.path.insert(2, 'scripts/dtc/pylibfdt') -sys.path.insert(2, os.path.join(our_path, '../../scripts/dtc/pylibfdt')) -sys.path.insert(2, os.path.join(our_path, - '../../build-sandbox_spl/scripts/dtc/pylibfdt')) +sys.path.insert(2, os.path.join(srctree, 'scripts/dtc/pylibfdt')) +sys.path.insert(2, os.path.join(srctree, 'build-sandbox_spl/scripts/dtc/pylibfdt'))
# When running under python-coverage on Ubuntu 16.04, the dist-packages # directories are dropped from the python path. Add them in so that we can find
Instead of joining hard coded '..' to the run-time path of the executable, take just a dirname out of it. Besides that, use $(srctree) where it makes sense.
Signed-off-by: Andy Shevchenko andriy.shevchenko@linux.intel.com --- v3: avoided crash (Simon), reused one level up folder variable tools/binman/main.py | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-)
Applied to u-boot-dm/next, thanks!
Importing libraries in Python caches the bytecode by default. Since we run scripts in source tree it ignores the current directory settings, which is $(srctree), and creates cache just in the middle of the source tree. Move cache to the current directory.
Signed-off-by: Andy Shevchenko andriy.shevchenko@linux.intel.com --- v3: avoided crash (Simon), preserved tree hierarchy tools/binman/main.py | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)
Applied to u-boot-dm/next, thanks!
Hi Andy,
On Tue, 14 Dec 2021 at 17:33, Simon Glass sjg@chromium.org wrote:
Importing libraries in Python caches the bytecode by default. Since we run scripts in source tree it ignores the current directory settings, which is $(srctree), and creates cache just in the middle of the source tree. Move cache to the current directory.
Signed-off-by: Andy Shevchenko andriy.shevchenko@linux.intel.com
v3: avoided crash (Simon), preserved tree hierarchy tools/binman/main.py | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)
Applied to u-boot-dm/next, thanks!
I didn't notice this before, but this seems to create files like this:
./tools/binman/usr/lib/python3/dist-packages/elftools/common/construct_utils.cpython-39.pyc
We don't really want to 'recache' the common Python files. Do you think we should revert this patch, or find another fix?
Regards, Simon
On Thu, Dec 23, 2021 at 7:08 AM Simon Glass sjg@chromium.org wrote:
On Tue, 14 Dec 2021 at 17:33, Simon Glass sjg@chromium.org wrote:
Importing libraries in Python caches the bytecode by default. Since we run scripts in source tree it ignores the current directory settings, which is $(srctree), and creates cache just in the middle of the source tree. Move cache to the current directory.
Signed-off-by: Andy Shevchenko andriy.shevchenko@linux.intel.com
v3: avoided crash (Simon), preserved tree hierarchy tools/binman/main.py | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)
Applied to u-boot-dm/next, thanks!
I didn't notice this before, but this seems to create files like this:
./tools/binman/usr/lib/python3/dist-packages/elftools/common/construct_utils.cpython-39.pyc
We don't really want to 'recache' the common Python files. Do you think we should revert this patch, or find another fix?
I'm not sure I understand. efitools is not common, it's a separate (non-standard) module and it's cached.
Hi Andy,
On Thu, 23 Dec 2021 at 02:07, Andy Shevchenko andy.shevchenko@gmail.com wrote:
On Thu, Dec 23, 2021 at 7:08 AM Simon Glass sjg@chromium.org wrote:
On Tue, 14 Dec 2021 at 17:33, Simon Glass sjg@chromium.org wrote:
Importing libraries in Python caches the bytecode by default. Since we run scripts in source tree it ignores the current directory settings, which is $(srctree), and creates cache just in the middle of the source tree. Move cache to the current directory.
Signed-off-by: Andy Shevchenko andriy.shevchenko@linux.intel.com
v3: avoided crash (Simon), preserved tree hierarchy tools/binman/main.py | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)
Applied to u-boot-dm/next, thanks!
I didn't notice this before, but this seems to create files like this:
./tools/binman/usr/lib/python3/dist-packages/elftools/common/construct_utils.cpython-39.pyc
We don't really want to 'recache' the common Python files. Do you think we should revert this patch, or find another fix?
I'm not sure I understand. efitools is not common, it's a separate (non-standard) module and it's cached.
The problem is that we don't want to create this file in the source directory.
For me this file is in:
/usr/lib/python3/dist-packages/elftools/
but it could be in ~/.local I suppose.
In any case, the cache file should go there, shouldn't it?
Regards, Simon
participants (3)
-
Andy Shevchenko -
Andy Shevchenko -
Simon Glass