[PATCH] binman: Avoid requiring a home directory on startup
This is needed to download tools, but we may not need to do this. At present binman fails to start if HOME is not set.
Use the current directory as a default to avoid this.
Signed-off-by: Simon Glass sjg@chromium.org ---
tools/binman/bintool.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tools/binman/bintool.py b/tools/binman/bintool.py index 8fda13ff012..f460243e796 100644 --- a/tools/binman/bintool.py +++ b/tools/binman/bintool.py @@ -43,7 +43,7 @@ FETCH_NAMES = { # Status of tool fetching FETCHED, FAIL, PRESENT, STATUS_COUNT = range(4)
-DOWNLOAD_DESTDIR = os.path.join(os.getenv('HOME'), 'bin') +DOWNLOAD_DESTDIR = os.path.expanduser('~/bin')
class Bintool: """Tool which operates on binaries to help produce entry contents
From: Simon Glass sjg@chromium.org Date: Fri, 10 Feb 2023 13:59:46 -0700
This is needed to download tools, but we may not need to do this. At present binman fails to start if HOME is not set.
Use the current directory as a default to avoid this.
Signed-off-by: Simon Glass sjg@chromium.org
Not new, but $HOME/bin seems like an *extremely* poor choice for a directory to download random binaries from the internet into as this directory is often part of a user's $PATH.
tools/binman/bintool.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tools/binman/bintool.py b/tools/binman/bintool.py index 8fda13ff012..f460243e796 100644 --- a/tools/binman/bintool.py +++ b/tools/binman/bintool.py @@ -43,7 +43,7 @@ FETCH_NAMES = { # Status of tool fetching FETCHED, FAIL, PRESENT, STATUS_COUNT = range(4)
-DOWNLOAD_DESTDIR = os.path.join(os.getenv('HOME'), 'bin') +DOWNLOAD_DESTDIR = os.path.expanduser('~/bin')
class Bintool: """Tool which operates on binaries to help produce entry contents -- 2.39.1.581.gbfd45094c4-goog
Hi Mark,
On Fri, 10 Feb 2023 at 14:23, Mark Kettenis mark.kettenis@xs4all.nl wrote:
From: Simon Glass sjg@chromium.org Date: Fri, 10 Feb 2023 13:59:46 -0700
This is needed to download tools, but we may not need to do this. At present binman fails to start if HOME is not set.
Use the current directory as a default to avoid this.
Signed-off-by: Simon Glass sjg@chromium.org
Not new, but $HOME/bin seems like an *extremely* poor choice for a directory to download random binaries from the internet into as this directory is often part of a user's $PATH.
But isn't that the point? It is designed to bring in tools for use by the user.
Regards, Simon
considering, iiuc, the user has to execute the command to opt-in to installing the programs, putting it into a dir that's in $PATH is reasonable
i would quibble that ~/bin is archaic and everyone should be using ~/.local/bin nowadays. but that can be a followup. https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html -mike
On Tue, Feb 14, 2023 at 12:48:56PM -0700, Simon Glass wrote:
Hi Mark,
On Fri, 10 Feb 2023 at 14:23, Mark Kettenis mark.kettenis@xs4all.nl wrote:
From: Simon Glass sjg@chromium.org Date: Fri, 10 Feb 2023 13:59:46 -0700
This is needed to download tools, but we may not need to do this. At present binman fails to start if HOME is not set.
Use the current directory as a default to avoid this.
Signed-off-by: Simon Glass sjg@chromium.org
Not new, but $HOME/bin seems like an *extremely* poor choice for a directory to download random binaries from the internet into as this directory is often part of a user's $PATH.
But isn't that the point? It is designed to bring in tools for use by the user.
Downloading things from the internet and putting them in to the default PATH always and forever is also kinda not great? buildman --fetch-arch places things elsewhere and some sub directory of ~/.buildman-toolchains is less risky.
On Tue, Feb 14, 2023 at 3:08 PM Tom Rini trini@konsulko.com wrote:
Downloading things from the internet and putting them in to the default PATH always and forever is also kinda not great?
you just described a standard distribution. this is like literally how all of them work. not to mention every other language-specific distro tool out there (e.g. Python pip, Perl cpan, Go, etc...).
maybe you'd like more guarantees on top (e.g. signature verification) which is reasonable.
but to be clear, this script is already merged & in the tree, so your feedback doesn't block this patch. -mike
On Tue, Feb 14, 2023 at 03:12:46PM -0500, Mike Frysinger wrote:
On Tue, Feb 14, 2023 at 3:08 PM Tom Rini trini@konsulko.com wrote:
Downloading things from the internet and putting them in to the default PATH always and forever is also kinda not great?
you just described a standard distribution. this is like literally how all of them work. not to mention every other language-specific distro tool out there (e.g. Python pip, Perl cpan, Go, etc...).
maybe you'd like more guarantees on top (e.g. signature verification) which is reasonable.
but to be clear, this script is already merged & in the tree, so your feedback doesn't block this patch.
Yes, exactly. This is a fix on top of what we do today, so it should go in. But modern distributions only install signed packages, and language-specific tools tend to be a hive of bad examples. Looking over binman right now, I see that we're either using apt (and oh, there's "aot" typo in one spot) or downloading from a known Google drive, for only a few less common tools.
So yes, I would like to see some ideas on how to improve things in the future so we aren't putting the binaries somewhere that's not a default (or frequently common) PATH location.
Hi Tom,
On Tue, 14 Feb 2023 at 13:27, Tom Rini trini@konsulko.com wrote:
On Tue, Feb 14, 2023 at 03:12:46PM -0500, Mike Frysinger wrote:
On Tue, Feb 14, 2023 at 3:08 PM Tom Rini trini@konsulko.com wrote:
Downloading things from the internet and putting them in to the default PATH always and forever is also kinda not great?
you just described a standard distribution. this is like literally how all of them work. not to mention every other language-specific distro tool out there (e.g. Python pip, Perl cpan, Go, etc...).
maybe you'd like more guarantees on top (e.g. signature verification) which is reasonable.
but to be clear, this script is already merged & in the tree, so your feedback doesn't block this patch.
Yes, exactly. This is a fix on top of what we do today, so it should go in. But modern distributions only install signed packages, and language-specific tools tend to be a hive of bad examples. Looking over binman right now, I see that we're either using apt (and oh, there's "aot" typo in one spot) or downloading from a known Google drive, for only a few less common tools.
So yes, I would like to see some ideas on how to improve things in the future so we aren't putting the binaries somewhere that's not a default (or frequently common) PATH location.
Are you thinking they should go in ~/.binman-tools or something like that? Then we would need to tell people to add it to their path. But we could make binman look there automatically.
Regards, Simon
On Thu, Feb 16, 2023 at 05:12:33PM -0700, Simon Glass wrote:
Hi Tom,
On Tue, 14 Feb 2023 at 13:27, Tom Rini trini@konsulko.com wrote:
On Tue, Feb 14, 2023 at 03:12:46PM -0500, Mike Frysinger wrote:
On Tue, Feb 14, 2023 at 3:08 PM Tom Rini trini@konsulko.com wrote:
Downloading things from the internet and putting them in to the default PATH always and forever is also kinda not great?
you just described a standard distribution. this is like literally how all of them work. not to mention every other language-specific distro tool out there (e.g. Python pip, Perl cpan, Go, etc...).
maybe you'd like more guarantees on top (e.g. signature verification) which is reasonable.
but to be clear, this script is already merged & in the tree, so your feedback doesn't block this patch.
Yes, exactly. This is a fix on top of what we do today, so it should go in. But modern distributions only install signed packages, and language-specific tools tend to be a hive of bad examples. Looking over binman right now, I see that we're either using apt (and oh, there's "aot" typo in one spot) or downloading from a known Google drive, for only a few less common tools.
So yes, I would like to see some ideas on how to improve things in the future so we aren't putting the binaries somewhere that's not a default (or frequently common) PATH location.
Are you thinking they should go in ~/.binman-tools or something like that? Then we would need to tell people to add it to their path. But we could make binman look there automatically.
We should document that it's where we're putting stuff, not so much "tell" them, unless you mean as a note when downloading. But yes, ~/.binman-tools sounds reasonable. Maybe a flag to point elsewhere?
Hi Tom,
On Thu, 16 Feb 2023 at 17:19, Tom Rini trini@konsulko.com wrote:
On Thu, Feb 16, 2023 at 05:12:33PM -0700, Simon Glass wrote:
Hi Tom,
On Tue, 14 Feb 2023 at 13:27, Tom Rini trini@konsulko.com wrote:
On Tue, Feb 14, 2023 at 03:12:46PM -0500, Mike Frysinger wrote:
On Tue, Feb 14, 2023 at 3:08 PM Tom Rini trini@konsulko.com wrote:
Downloading things from the internet and putting them in to the default PATH always and forever is also kinda not great?
you just described a standard distribution. this is like literally how all of them work. not to mention every other language-specific distro tool out there (e.g. Python pip, Perl cpan, Go, etc...).
maybe you'd like more guarantees on top (e.g. signature verification) which is reasonable.
but to be clear, this script is already merged & in the tree, so your feedback doesn't block this patch.
Yes, exactly. This is a fix on top of what we do today, so it should go in. But modern distributions only install signed packages, and language-specific tools tend to be a hive of bad examples. Looking over binman right now, I see that we're either using apt (and oh, there's "aot" typo in one spot) or downloading from a known Google drive, for only a few less common tools.
So yes, I would like to see some ideas on how to improve things in the future so we aren't putting the binaries somewhere that's not a default (or frequently common) PATH location.
Are you thinking they should go in ~/.binman-tools or something like that? Then we would need to tell people to add it to their path. But we could make binman look there automatically.
We should document that it's where we're putting stuff, not so much "tell" them, unless you mean as a note when downloading. But yes, ~/.binman-tools sounds reasonable. Maybe a flag to point elsewhere?
OK I will take a look.
Regards, Simon
Hi all,
On 2/17/23 03:55, Simon Glass wrote:
Hi Tom,
On Thu, 16 Feb 2023 at 17:19, Tom Rini trini@konsulko.com wrote:
On Thu, Feb 16, 2023 at 05:12:33PM -0700, Simon Glass wrote:
Hi Tom,
On Tue, 14 Feb 2023 at 13:27, Tom Rini trini@konsulko.com wrote:
On Tue, Feb 14, 2023 at 03:12:46PM -0500, Mike Frysinger wrote:
On Tue, Feb 14, 2023 at 3:08 PM Tom Rini trini@konsulko.com wrote:
Downloading things from the internet and putting them in to the default PATH always and forever is also kinda not great?
you just described a standard distribution. this is like literally how all of them work. not to mention every other language-specific distro tool out there (e.g. Python pip, Perl cpan, Go, etc...).
maybe you'd like more guarantees on top (e.g. signature verification) which is reasonable.
but to be clear, this script is already merged & in the tree, so your feedback doesn't block this patch.
Yes, exactly. This is a fix on top of what we do today, so it should go in. But modern distributions only install signed packages, and language-specific tools tend to be a hive of bad examples. Looking over binman right now, I see that we're either using apt (and oh, there's "aot" typo in one spot) or downloading from a known Google drive, for only a few less common tools.
So yes, I would like to see some ideas on how to improve things in the future so we aren't putting the binaries somewhere that's not a default (or frequently common) PATH location.
Are you thinking they should go in ~/.binman-tools or something like that? Then we would need to tell people to add it to their path. But we could make binman look there automatically.
We should document that it's where we're putting stuff, not so much "tell" them, unless you mean as a note when downloading. But yes, ~/.binman-tools sounds reasonable. Maybe a flag to point elsewhere?
OK I will take a look.
I think this should be directly put into the output/build directory used by U-Boot, because what happens when you have two U-Boot git repos with different version requirements for those host tools? Then you need to make sure you're not building both at the same time, that you update them properly before each build, etc.
We also have an issue with buildman creating (image) temporary files in the current git repo which makes people unhappy because they then get a dirty tree because we tend to forget to update the .gitignore. What about building out-of-tree by default like Buildroot does? e.g. with make O=$PWD/output/ ?
Cheers, Quentin
Hi Quentin,
On Fri, 17 Feb 2023 at 05:21, Quentin Schulz quentin.schulz@theobroma-systems.com wrote:
Hi all,
On 2/17/23 03:55, Simon Glass wrote:
Hi Tom,
On Thu, 16 Feb 2023 at 17:19, Tom Rini trini@konsulko.com wrote:
On Thu, Feb 16, 2023 at 05:12:33PM -0700, Simon Glass wrote:
Hi Tom,
On Tue, 14 Feb 2023 at 13:27, Tom Rini trini@konsulko.com wrote:
On Tue, Feb 14, 2023 at 03:12:46PM -0500, Mike Frysinger wrote:
On Tue, Feb 14, 2023 at 3:08 PM Tom Rini trini@konsulko.com wrote: > Downloading things from the internet and putting them in to the default > PATH always and forever is also kinda not great?
you just described a standard distribution. this is like literally how all of them work. not to mention every other language-specific distro tool out there (e.g. Python pip, Perl cpan, Go, etc...).
maybe you'd like more guarantees on top (e.g. signature verification) which is reasonable.
but to be clear, this script is already merged & in the tree, so your feedback doesn't block this patch.
Yes, exactly. This is a fix on top of what we do today, so it should go in. But modern distributions only install signed packages, and language-specific tools tend to be a hive of bad examples. Looking over binman right now, I see that we're either using apt (and oh, there's "aot" typo in one spot) or downloading from a known Google drive, for only a few less common tools.
So yes, I would like to see some ideas on how to improve things in the future so we aren't putting the binaries somewhere that's not a default (or frequently common) PATH location.
Are you thinking they should go in ~/.binman-tools or something like that? Then we would need to tell people to add it to their path. But we could make binman look there automatically.
We should document that it's where we're putting stuff, not so much "tell" them, unless you mean as a note when downloading. But yes, ~/.binman-tools sounds reasonable. Maybe a flag to point elsewhere?
OK I will take a look.
I think this should be directly put into the output/build directory used by U-Boot, because what happens when you have two U-Boot git repos with different version requirements for those host tools? Then you need to make sure you're not building both at the same time, that you update them properly before each build, etc.
My advice: *Don't do that*
So far as binman is concerned, a tool is a tool. Tools should be backwards compatible so updating to the new one should fix all the problems.
The problem with using the output dir is we then have to download them for each build, or cache them somewhere. To my mind, the 'binman tool' feature is a convenience to reduce the pain involved in obtaining tools needed to build. It is a not a panacea for strange situations.
We also have an issue with buildman creating (image) temporary files in the current git repo which makes people unhappy because they then get a dirty tree because we tend to forget to update the .gitignore. What about building out-of-tree by default like Buildroot does? e.g. with make O=$PWD/output/ ?
I don't tend to use in-tree builds, but, yes, this is a problem.
I believe the best solution is, I think, to make binman put its intermediate files in a subdir. At present there is no distinction between output files and files produced along the way for debugging purposes. This has been on my mind for a while and has been discussed at least once before, too.
We do want the output files to be in the build directory. It would be annoying to put u-boot.itb in a subdir.
Binman should all track all the files it produced (both final output and intermediate) so that info is available.
For now, the solution is to use 'make O=xxx' as you say. With that, binman should not write anything to the current dir. If it does, that is a bug.
Regards, Simon
Hi Simon,
On 2/18/23 00:49, Simon Glass wrote:
Hi Quentin,
On Fri, 17 Feb 2023 at 05:21, Quentin Schulz quentin.schulz@theobroma-systems.com wrote:
Hi all,
On 2/17/23 03:55, Simon Glass wrote:
Hi Tom,
On Thu, 16 Feb 2023 at 17:19, Tom Rini trini@konsulko.com wrote:
On Thu, Feb 16, 2023 at 05:12:33PM -0700, Simon Glass wrote:
Hi Tom,
On Tue, 14 Feb 2023 at 13:27, Tom Rini trini@konsulko.com wrote:
On Tue, Feb 14, 2023 at 03:12:46PM -0500, Mike Frysinger wrote: > On Tue, Feb 14, 2023 at 3:08 PM Tom Rini trini@konsulko.com wrote: >> Downloading things from the internet and putting them in to the default >> PATH always and forever is also kinda not great? > > you just described a standard distribution. this is like literally > how all of them work. not to mention every other language-specific > distro tool out there (e.g. Python pip, Perl cpan, Go, etc...). > > maybe you'd like more guarantees on top (e.g. signature verification) > which is reasonable. > > but to be clear, this script is already merged & in the tree, so your > feedback doesn't block this patch.
Yes, exactly. This is a fix on top of what we do today, so it should go in. But modern distributions only install signed packages, and language-specific tools tend to be a hive of bad examples. Looking over binman right now, I see that we're either using apt (and oh, there's "aot" typo in one spot) or downloading from a known Google drive, for only a few less common tools.
So yes, I would like to see some ideas on how to improve things in the future so we aren't putting the binaries somewhere that's not a default (or frequently common) PATH location.
Are you thinking they should go in ~/.binman-tools or something like that? Then we would need to tell people to add it to their path. But we could make binman look there automatically.
We should document that it's where we're putting stuff, not so much "tell" them, unless you mean as a note when downloading. But yes, ~/.binman-tools sounds reasonable. Maybe a flag to point elsewhere?
OK I will take a look.
I think this should be directly put into the output/build directory used by U-Boot, because what happens when you have two U-Boot git repos with different version requirements for those host tools? Then you need to make sure you're not building both at the same time, that you update them properly before each build, etc.
My advice: *Don't do that*
So far as binman is concerned, a tool is a tool. Tools should be backwards compatible so updating to the new one should fix all the problems.
That's a very bold claim :)
The problem with using the output dir is we then have to download them for each build, or cache them somewhere. To my mind, the 'binman tool' feature is a convenience to reduce the pain involved in obtaining tools needed to build. It is a not a panacea for strange situations.
Have the default in the build directory and allow the user to define an out-of-tree directory if they want to cache them somewhere? Similar to Yocto with SSTATE_DIR/DL_DIR, Buildroot with BR2_DL_DIR for example.
Cheers, Quentin
Hi Quentin,
On Mon, 20 Feb 2023 at 04:15, Quentin Schulz quentin.schulz@theobroma-systems.com wrote:
Hi Simon,
On 2/18/23 00:49, Simon Glass wrote:
Hi Quentin,
On Fri, 17 Feb 2023 at 05:21, Quentin Schulz quentin.schulz@theobroma-systems.com wrote:
Hi all,
On 2/17/23 03:55, Simon Glass wrote:
Hi Tom,
On Thu, 16 Feb 2023 at 17:19, Tom Rini trini@konsulko.com wrote:
On Thu, Feb 16, 2023 at 05:12:33PM -0700, Simon Glass wrote:
Hi Tom,
On Tue, 14 Feb 2023 at 13:27, Tom Rini trini@konsulko.com wrote: > > On Tue, Feb 14, 2023 at 03:12:46PM -0500, Mike Frysinger wrote: >> On Tue, Feb 14, 2023 at 3:08 PM Tom Rini trini@konsulko.com wrote: >>> Downloading things from the internet and putting them in to the default >>> PATH always and forever is also kinda not great? >> >> you just described a standard distribution. this is like literally >> how all of them work. not to mention every other language-specific >> distro tool out there (e.g. Python pip, Perl cpan, Go, etc...). >> >> maybe you'd like more guarantees on top (e.g. signature verification) >> which is reasonable. >> >> but to be clear, this script is already merged & in the tree, so your >> feedback doesn't block this patch. > > Yes, exactly. This is a fix on top of what we do today, so it should go > in. But modern distributions only install signed packages, and > language-specific tools tend to be a hive of bad examples. Looking over > binman right now, I see that we're either using apt (and oh, there's > "aot" typo in one spot) or downloading from a known Google drive, for > only a few less common tools. > > So yes, I would like to see some ideas on how to improve things in the > future so we aren't putting the binaries somewhere that's not a default > (or frequently common) PATH location.
Are you thinking they should go in ~/.binman-tools or something like that? Then we would need to tell people to add it to their path. But we could make binman look there automatically.
We should document that it's where we're putting stuff, not so much "tell" them, unless you mean as a note when downloading. But yes, ~/.binman-tools sounds reasonable. Maybe a flag to point elsewhere?
OK I will take a look.
I think this should be directly put into the output/build directory used by U-Boot, because what happens when you have two U-Boot git repos with different version requirements for those host tools? Then you need to make sure you're not building both at the same time, that you update them properly before each build, etc.
My advice: *Don't do that*
So far as binman is concerned, a tool is a tool. Tools should be backwards compatible so updating to the new one should fix all the problems.
That's a very bold claim :)
The problem with using the output dir is we then have to download them for each build, or cache them somewhere. To my mind, the 'binman tool' feature is a convenience to reduce the pain involved in obtaining tools needed to build. It is a not a panacea for strange situations.
Have the default in the build directory and allow the user to define an out-of-tree directory if they want to cache them somewhere? Similar to Yocto with SSTATE_DIR/DL_DIR, Buildroot with BR2_DL_DIR for example.
OK, but why do you want to use the build directory at all? It seems like a hassle to set up. With the series I sent it is automatic and all that is needed is to add ~/.binman-tools to your path.
Regards, Simon
Hi Quentin,
On Mon, 20 Feb 2023 at 04:15, Quentin Schulz quentin.schulz@theobroma-systems.com wrote:
Hi Simon,
On 2/18/23 00:49, Simon Glass wrote:
Hi Quentin,
On Fri, 17 Feb 2023 at 05:21, Quentin Schulz quentin.schulz@theobroma-systems.com wrote:
Hi all,
On 2/17/23 03:55, Simon Glass wrote:
Hi Tom,
On Thu, 16 Feb 2023 at 17:19, Tom Rini trini@konsulko.com wrote:
On Thu, Feb 16, 2023 at 05:12:33PM -0700, Simon Glass wrote:
Hi Tom,
On Tue, 14 Feb 2023 at 13:27, Tom Rini trini@konsulko.com wrote: > > On Tue, Feb 14, 2023 at 03:12:46PM -0500, Mike Frysinger wrote: >> On Tue, Feb 14, 2023 at 3:08 PM Tom Rini trini@konsulko.com wrote: >>> Downloading things from the internet and putting them in to the default >>> PATH always and forever is also kinda not great? >> >> you just described a standard distribution. this is like literally >> how all of them work. not to mention every other language-specific >> distro tool out there (e.g. Python pip, Perl cpan, Go, etc...). >> >> maybe you'd like more guarantees on top (e.g. signature verification) >> which is reasonable. >> >> but to be clear, this script is already merged & in the tree, so your >> feedback doesn't block this patch. > > Yes, exactly. This is a fix on top of what we do today, so it should go > in. But modern distributions only install signed packages, and > language-specific tools tend to be a hive of bad examples. Looking over > binman right now, I see that we're either using apt (and oh, there's > "aot" typo in one spot) or downloading from a known Google drive, for > only a few less common tools. > > So yes, I would like to see some ideas on how to improve things in the > future so we aren't putting the binaries somewhere that's not a default > (or frequently common) PATH location.
Are you thinking they should go in ~/.binman-tools or something like that? Then we would need to tell people to add it to their path. But we could make binman look there automatically.
We should document that it's where we're putting stuff, not so much "tell" them, unless you mean as a note when downloading. But yes, ~/.binman-tools sounds reasonable. Maybe a flag to point elsewhere?
OK I will take a look.
I think this should be directly put into the output/build directory used by U-Boot, because what happens when you have two U-Boot git repos with different version requirements for those host tools? Then you need to make sure you're not building both at the same time, that you update them properly before each build, etc.
My advice: *Don't do that*
So far as binman is concerned, a tool is a tool. Tools should be backwards compatible so updating to the new one should fix all the problems.
That's a very bold claim :)
The problem with using the output dir is we then have to download them for each build, or cache them somewhere. To my mind, the 'binman tool' feature is a convenience to reduce the pain involved in obtaining tools needed to build. It is a not a panacea for strange situations.
Have the default in the build directory and allow the user to define an out-of-tree directory if they want to cache them somewhere? Similar to Yocto with SSTATE_DIR/DL_DIR, Buildroot with BR2_DL_DIR for example.
OK, but why do you want to use the build directory at all? It seems like a hassle to set up. With the series I sent it is automatic and all that is needed is to add ~/.binman-tools to your path.
Regards, Simon
Applied to u-boot-dm/next, thanks!
participants (5)
-
Mark Kettenis -
Mike Frysinger -
Quentin Schulz -
Simon Glass -
Tom Rini