[U-Boot] [PATCH v2] Reproducible U-Boot build support, using SOURCE_DATE_EPOCH
In order to achieve reproducible builds in U-Boot, timestamps that are defined at build-time have to be somewhat eliminated. The SOURCE_DATE_EPOCH environment variable allows setting a fixed value for those timestamps.
Simply by setting SOURCE_DATE_EPOCH to a fixed value, a number of targets can be built reproducibly. This is the case for e.g. sunxi devices.
However, some other devices might need some more tweaks, especially regarding the image generation tools.
Signed-off-by: Paul Kocialkowski contact@paulk.fr --- Makefile | 7 ++++--- README | 12 ++++++++++++ tools/default_image.c | 21 ++++++++++++++++++++- 3 files changed, 36 insertions(+), 4 deletions(-)
diff --git a/Makefile b/Makefile index 54ef2cd..596974c 100644 --- a/Makefile +++ b/Makefile @@ -1232,9 +1232,10 @@ define filechk_version.h endef
define filechk_timestamp.h - (LC_ALL=C date +'#define U_BOOT_DATE "%b %d %C%y"'; \ - LC_ALL=C date +'#define U_BOOT_TIME "%T"'; \ - LC_ALL=C date +'#define U_BOOT_TZ "%z"') + (SOURCE_DATE="$${SOURCE_DATE_EPOCH:+@$$SOURCE_DATE_EPOCH}"; \ + LC_ALL=C date -u -d "$${SOURCE_DATE:-now}" +'#define U_BOOT_DATE "%b %d %C%y"'; \ + LC_ALL=C date -u -d "$${SOURCE_DATE:-now}" +'#define U_BOOT_TIME "%T"'; \ + LC_ALL=C date -u -d "$${SOURCE_DATE:-now}" +'#define U_BOOT_TZ "%z"' ) endef
$(version_h): include/config/uboot.release FORCE diff --git a/README b/README index 4e0ff9f..1bcb63c 100644 --- a/README +++ b/README @@ -5081,6 +5081,18 @@ This firmware often needs to be loaded during U-Boot booting. - CONFIG_SYS_MEM_TOP_HIDE_MIN Define minimum DDR size to be hided from top of the DDR memory
+Reproducible builds +------------------- + +In order to achieve reproducible builds, timestamps used in the U-Boot build +process have to be set to a fixed value. + +This is done using the SOURCE_DATE_EPOCH environment variable. +SOURCE_DATE_EPOCH is to be set on the build host's shell, not as a configuration +option for U-Boot or an environment variable in U-Boot. + +SOURCE_DATE_EPOCH should be set to a number of seconds since the epoch, in UTC. + Building the Software: ======================
diff --git a/tools/default_image.c b/tools/default_image.c index cf5c0d4..18940af 100644 --- a/tools/default_image.c +++ b/tools/default_image.c @@ -88,6 +88,9 @@ static void image_set_header(void *ptr, struct stat *sbuf, int ifd, struct image_tool_params *params) { uint32_t checksum; + char *source_date_epoch; + struct tm *time_universal; + time_t time;
image_header_t * hdr = (image_header_t *)ptr;
@@ -96,9 +99,25 @@ static void image_set_header(void *ptr, struct stat *sbuf, int ifd, sizeof(image_header_t)), sbuf->st_size - sizeof(image_header_t));
+ source_date_epoch = getenv("SOURCE_DATE_EPOCH"); + if (source_date_epoch != NULL) { + time = (time_t) strtol(source_date_epoch, NULL, 10); + + time_universal = gmtime(&time); + if (time_universal == NULL) { + fprintf(stderr, "%s: SOURCE_DATE_EPOCH is not valid\n", + __func__); + time = 0; + } else { + time = mktime(time_universal); + } + } else { + time = sbuf->st_mtime; + } + /* Build new header */ image_set_magic(hdr, IH_MAGIC); - image_set_time(hdr, sbuf->st_mtime); + image_set_time(hdr, time); image_set_size(hdr, sbuf->st_size - sizeof(image_header_t)); image_set_load(hdr, params->addr); image_set_ep(hdr, params->ep);
On Sun, Jul 26, 2015 at 06:48:15PM +0200, Paul Kocialkowski wrote:
In order to achieve reproducible builds in U-Boot, timestamps that are defined at build-time have to be somewhat eliminated. The SOURCE_DATE_EPOCH environment variable allows setting a fixed value for those timestamps.
Simply by setting SOURCE_DATE_EPOCH to a fixed value, a number of targets can be built reproducibly. This is the case for e.g. sunxi devices.
However, some other devices might need some more tweaks, especially regarding the image generation tools.
Signed-off-by: Paul Kocialkowski contact@paulk.fr
Applied to u-boot/master, thanks!
Hi,
On Tue, Jul 28, 2015 at 11:00 PM, Tom Rini trini@konsulko.com wrote:
On Sun, Jul 26, 2015 at 06:48:15PM +0200, Paul Kocialkowski wrote:
In order to achieve reproducible builds in U-Boot, timestamps that are defined at build-time have to be somewhat eliminated. The SOURCE_DATE_EPOCH environment variable allows setting a fixed value for those timestamps.
Simply by setting SOURCE_DATE_EPOCH to a fixed value, a number of targets can be built reproducibly. This is the case for e.g. sunxi devices.
However, some other devices might need some more tweaks, especially regarding the image generation tools.
Signed-off-by: Paul Kocialkowski contact@paulk.fr
Applied to u-boot/master, thanks!
--
This commit breaks the following commit:
commit f3f431a712729a1af94d01bd1bfde17a252ff02c Author: Chris Packham judge.packham@gmail.com Date: Sun May 10 21:02:09 2015 +1200
Makefile: Add U_BOOT_TZ and include in version
Define U_BOOT_TZ alongside U_BOOT_TIME and U_BOOT_DATE and use it to include the timezone in the version output.
Acked-by: Simon Glass sjg@chromium.org Signed-off-by: Chris Packham judge.packham@gmail.com
Before this commit I have: U-Boot 2015.07-00345-g9c57487 (Jul 31 2015 - 10:49:31 +0800)
After this commit I have: U-Boot 2015.07-00346-gf3f431a (Jul 31 2015 - 02:50:54 +0000)
As you see: the timezone information is missing, and U-Boot's timestamp is now GMT+0 (the correct one should be GMT+8)
Is this intended behavior? Or if not, please fix it.
Regards, Bin
On Fri, Jul 31, 2015 at 2:54 PM, Bin Meng bmeng.cn@gmail.com wrote:
Hi,
On Tue, Jul 28, 2015 at 11:00 PM, Tom Rini trini@konsulko.com wrote:
On Sun, Jul 26, 2015 at 06:48:15PM +0200, Paul Kocialkowski wrote:
In order to achieve reproducible builds in U-Boot, timestamps that are defined at build-time have to be somewhat eliminated. The SOURCE_DATE_EPOCH environment variable allows setting a fixed value for those timestamps.
Simply by setting SOURCE_DATE_EPOCH to a fixed value, a number of targets can be built reproducibly. This is the case for e.g. sunxi devices.
However, some other devices might need some more tweaks, especially regarding the image generation tools.
Signed-off-by: Paul Kocialkowski contact@paulk.fr
Applied to u-boot/master, thanks!
--
This commit breaks the following commit:
commit f3f431a712729a1af94d01bd1bfde17a252ff02c Author: Chris Packham judge.packham@gmail.com Date: Sun May 10 21:02:09 2015 +1200
Makefile: Add U_BOOT_TZ and include in version Define U_BOOT_TZ alongside U_BOOT_TIME and U_BOOT_DATE and use it to include the timezone in the version output. Acked-by: Simon Glass <sjg@chromium.org> Signed-off-by: Chris Packham <judge.packham@gmail.com>Before this commit I have: U-Boot 2015.07-00345-g9c57487 (Jul 31 2015 - 10:49:31 +0800)
After this commit I have: U-Boot 2015.07-00346-gf3f431a (Jul 31 2015 - 02:50:54 +0000)
As you see: the timezone information is missing, and U-Boot's timestamp is now GMT+0 (the correct one should be GMT+8)
Is this intended behavior? Or if not, please fix it.
Regards, Bin
The problem is that date -u implies UTC. So +0000 is right if you do want the time to be displayed in UTC (for me living at +12/+13 UTC makes my head hurt because all the dates are yesterday).
The intent of f3f431a7 was to reflect the timezone that the build machine was set to. Dropping the -u would probably be sufficient but perhaps it would be better to make whole lot conditional on SOURCE_DATE_EPOCH being set. Something like this (untested, apologies gmail web interface)
ifneq ($(SOURCE_DATE_EPOCH),) define filechk_timestamp.h (SOURCE_DATE="$(SOURCE_DATE_EPOCH)"; \ LC_ALL=C date -u -d "$${SOURCE_DATE}" +'#define U_BOOT_DATE "%b %d %C%y"'; \ LC_ALL=C date -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TIME "%T"'; \ LC_ALL=C date -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TZ "%z"' ) endef else define filechk_timestamp.h (LC_ALL=C date +'#define U_BOOT_DATE "%b %d %C%y"'; \ LC_ALL=C date +'#define U_BOOT_TIME "%T"'; \ LC_ALL=C date +'#define U_BOOT_TZ "%z"') endef endif
When building with SOURCE_DATE_EPOCH the timezone is in UTC. When building normally the timezone is taken from the build machine's locale setting.
Fixes: f3f431a71272 ("Reproducible U-Boot build support, using SOURCE_DATE_EPOCH") Signed-off-by: Chris Packham judge.packham@gmail.com --- Hopefully this will suit everyone. I've done a quick sanity test with sandbox and the date shown in the version output looks to be what I expect with/without SOURCE_DATE_EPOCH.
Makefile | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-)
diff --git a/Makefile b/Makefile index 394ed09..f75c730 100644 --- a/Makefile +++ b/Makefile @@ -1230,10 +1230,16 @@ define filechk_version.h endef
define filechk_timestamp.h - (SOURCE_DATE="$${SOURCE_DATE_EPOCH:+@$$SOURCE_DATE_EPOCH}"; \ - LC_ALL=C date -u -d "$${SOURCE_DATE:-now}" +'#define U_BOOT_DATE "%b %d %C%y"'; \ - LC_ALL=C date -u -d "$${SOURCE_DATE:-now}" +'#define U_BOOT_TIME "%T"'; \ - LC_ALL=C date -u -d "$${SOURCE_DATE:-now}" +'#define U_BOOT_TZ "%z"' ) + (if test -n "$${SOURCE_DATE_EPOCH}"; then \ + SOURCE_DATE="@$${SOURCE_DATE_EPOCH}"; \ + LC_ALL=C date -u -d "$${SOURCE_DATE}" +'#define U_BOOT_DATE "%b %d %C%y"'; \ + LC_ALL=C date -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TIME "%T"'; \ + LC_ALL=C date -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TZ "%z"'; \ + else \ + LC_ALL=C date +'#define U_BOOT_DATE "%b %d %C%y"'; \ + LC_ALL=C date +'#define U_BOOT_TIME "%T"'; \ + LC_ALL=C date +'#define U_BOOT_TZ "%z"'; \ + fi) endef
$(version_h): include/config/uboot.release FORCE
On Fri, Jul 31, 2015 at 6:04 PM, Chris Packham judge.packham@gmail.com wrote:
When building with SOURCE_DATE_EPOCH the timezone is in UTC. When building normally the timezone is taken from the build machine's locale setting.
Fixes: f3f431a71272 ("Reproducible U-Boot build support, using SOURCE_DATE_EPOCH") Signed-off-by: Chris Packham judge.packham@gmail.com
Hopefully this will suit everyone. I've done a quick sanity test with sandbox and the date shown in the version output looks to be what I expect with/without SOURCE_DATE_EPOCH.
Makefile | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-)
diff --git a/Makefile b/Makefile index 394ed09..f75c730 100644 --- a/Makefile +++ b/Makefile @@ -1230,10 +1230,16 @@ define filechk_version.h endef
define filechk_timestamp.h
(SOURCE_DATE="$${SOURCE_DATE_EPOCH:+@$$SOURCE_DATE_EPOCH}"; \LC_ALL=C date -u -d "$${SOURCE_DATE:-now}" +'#define U_BOOT_DATE "%b %d %C%y"'; \LC_ALL=C date -u -d "$${SOURCE_DATE:-now}" +'#define U_BOOT_TIME "%T"'; \LC_ALL=C date -u -d "$${SOURCE_DATE:-now}" +'#define U_BOOT_TZ "%z"' )
(if test -n "$${SOURCE_DATE_EPOCH}"; then \SOURCE_DATE="@$${SOURCE_DATE_EPOCH}"; \LC_ALL=C date -u -d "$${SOURCE_DATE}" +'#define U_BOOT_DATE "%b %d %C%y"'; \LC_ALL=C date -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TIME "%T"'; \LC_ALL=C date -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TZ "%z"'; \else \LC_ALL=C date +'#define U_BOOT_DATE "%b %d %C%y"'; \LC_ALL=C date +'#define U_BOOT_TIME "%T"'; \LC_ALL=C date +'#define U_BOOT_TZ "%z"'; \fi)endef
$(version_h): include/config/uboot.release FORCE
Tested-by: Bin Meng bmeng.cn@gmail.com
Le vendredi 31 juillet 2015 à 22:04 +1200, Chris Packham a écrit :
When building with SOURCE_DATE_EPOCH the timezone is in UTC. When building normally the timezone is taken from the build machine's locale setting.
Thanks a lot for introducing this, I thought it might disturb a few people.
Fixes: f3f431a71272 ("Reproducible U-Boot build support, using SOURCE_DATE_EPOCH")
It is incorrect to say that your patch fixes my commit, because my commit didn't introduce anything unexpected or faulty.
If anything, it was merged before people gave their opinion about whether UTC should be default. I was also going to introduce a patch like yours later. For the record, UTC is default for Coreboot's timestamps.
Thus, please remove that line.
Otherwise, this is: Tested-By: Paul Kocialkowski contact@paulk.fr
Signed-off-by: Chris Packham judge.packham@gmail.com
Hopefully this will suit everyone. I've done a quick sanity test with sandbox and the date shown in the version output looks to be what I expect with/without SOURCE_DATE_EPOCH.
Makefile | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-)
diff --git a/Makefile b/Makefile index 394ed09..f75c730 100644 --- a/Makefile +++ b/Makefile @@ -1230,10 +1230,16 @@ define filechk_version.h endef
define filechk_timestamp.h
- (SOURCE_DATE="$${SOURCE_DATE_EPOCH:+@$$SOURCE_DATE_EPOCH}"; \
- LC_ALL=C date -u -d "$${SOURCE_DATE:-now}" +'#define U_BOOT_DATE "%b %d %C%y"'; \
- LC_ALL=C date -u -d "$${SOURCE_DATE:-now}" +'#define U_BOOT_TIME "%T"'; \
- LC_ALL=C date -u -d "$${SOURCE_DATE:-now}" +'#define U_BOOT_TZ "%z"' )
- (if test -n "$${SOURCE_DATE_EPOCH}"; then \
SOURCE_DATE="@$${SOURCE_DATE_EPOCH}"; \LC_ALL=C date -u -d "$${SOURCE_DATE}" +'#define U_BOOT_DATE "%b %d %C%y"'; \LC_ALL=C date -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TIME "%T"'; \LC_ALL=C date -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TZ "%z"'; \- else \
LC_ALL=C date +'#define U_BOOT_DATE "%b %d %C%y"'; \LC_ALL=C date +'#define U_BOOT_TIME "%T"'; \LC_ALL=C date +'#define U_BOOT_TZ "%z"'; \- fi)
endef
$(version_h): include/config/uboot.release FORCE
(note I had some patman config that erroneously included extra people on the original Cc list, I've now dropped them)
On Sat, Aug 1, 2015 at 5:05 AM, Paul Kocialkowski contact@paulk.fr wrote:
Le vendredi 31 juillet 2015 à 22:04 +1200, Chris Packham a écrit :
When building with SOURCE_DATE_EPOCH the timezone is in UTC. When building normally the timezone is taken from the build machine's locale setting.
Thanks a lot for introducing this, I thought it might disturb a few people.
Fixes: f3f431a71272 ("Reproducible U-Boot build support, using SOURCE_DATE_EPOCH")
It is incorrect to say that your patch fixes my commit, because my commit didn't introduce anything unexpected or faulty.
If anything, it was merged before people gave their opinion about whether UTC should be default. I was also going to introduce a patch like yours later. For the record, UTC is default for Coreboot's timestamps.
Thus, please remove that line.
Done. v2 on it's way.
The other mailing list thread also talked about the possibility of having SOURCE_DATE_TZOFFSET I'll look at a proof of concept for that.
Otherwise, this is: Tested-By: Paul Kocialkowski contact@paulk.fr
Signed-off-by: Chris Packham judge.packham@gmail.com
Hopefully this will suit everyone. I've done a quick sanity test with sandbox and the date shown in the version output looks to be what I expect with/without SOURCE_DATE_EPOCH.
Makefile | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-)
diff --git a/Makefile b/Makefile index 394ed09..f75c730 100644 --- a/Makefile +++ b/Makefile @@ -1230,10 +1230,16 @@ define filechk_version.h endef
define filechk_timestamp.h
(SOURCE_DATE="$${SOURCE_DATE_EPOCH:+@$$SOURCE_DATE_EPOCH}"; \LC_ALL=C date -u -d "$${SOURCE_DATE:-now}" +'#define U_BOOT_DATE "%b %d %C%y"'; \LC_ALL=C date -u -d "$${SOURCE_DATE:-now}" +'#define U_BOOT_TIME "%T"'; \LC_ALL=C date -u -d "$${SOURCE_DATE:-now}" +'#define U_BOOT_TZ "%z"' )
(if test -n "$${SOURCE_DATE_EPOCH}"; then \SOURCE_DATE="@$${SOURCE_DATE_EPOCH}"; \LC_ALL=C date -u -d "$${SOURCE_DATE}" +'#define U_BOOT_DATE "%b %d %C%y"'; \LC_ALL=C date -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TIME "%T"'; \LC_ALL=C date -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TZ "%z"'; \else \LC_ALL=C date +'#define U_BOOT_DATE "%b %d %C%y"'; \LC_ALL=C date +'#define U_BOOT_TIME "%T"'; \LC_ALL=C date +'#define U_BOOT_TZ "%z"'; \fi)endef
$(version_h): include/config/uboot.release FORCE
When building with SOURCE_DATE_EPOCH the timezone is in UTC. When building normally the timezone is taken from the build machine's locale setting.
Signed-off-by: Chris Packham judge.packham@gmail.com Tested-by: Bin Meng bmeng.cn@gmail.com Tested-by: Paul Kocialkowski contact@paulk.fr ---
Changes in v2: - Collect some tested-by tags - Remove reference to f3f431a71272 in the commit message - Drop Ccs that were erroneously added when submitting v1, remaining Ccs are from the original mailing list thread
Makefile | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-)
diff --git a/Makefile b/Makefile index 394ed09..f75c730 100644 --- a/Makefile +++ b/Makefile @@ -1230,10 +1230,16 @@ define filechk_version.h endef
define filechk_timestamp.h - (SOURCE_DATE="$${SOURCE_DATE_EPOCH:+@$$SOURCE_DATE_EPOCH}"; \ - LC_ALL=C date -u -d "$${SOURCE_DATE:-now}" +'#define U_BOOT_DATE "%b %d %C%y"'; \ - LC_ALL=C date -u -d "$${SOURCE_DATE:-now}" +'#define U_BOOT_TIME "%T"'; \ - LC_ALL=C date -u -d "$${SOURCE_DATE:-now}" +'#define U_BOOT_TZ "%z"' ) + (if test -n "$${SOURCE_DATE_EPOCH}"; then \ + SOURCE_DATE="@$${SOURCE_DATE_EPOCH}"; \ + LC_ALL=C date -u -d "$${SOURCE_DATE}" +'#define U_BOOT_DATE "%b %d %C%y"'; \ + LC_ALL=C date -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TIME "%T"'; \ + LC_ALL=C date -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TZ "%z"'; \ + else \ + LC_ALL=C date +'#define U_BOOT_DATE "%b %d %C%y"'; \ + LC_ALL=C date +'#define U_BOOT_TIME "%T"'; \ + LC_ALL=C date +'#define U_BOOT_TZ "%z"'; \ + fi) endef
$(version_h): include/config/uboot.release FORCE
Hi Tom,
On Sat, Aug 1, 2015 at 9:43 PM, Chris Packham judge.packham@gmail.com wrote:
When building with SOURCE_DATE_EPOCH the timezone is in UTC. When building normally the timezone is taken from the build machine's locale setting.
Signed-off-by: Chris Packham judge.packham@gmail.com Tested-by: Bin Meng bmeng.cn@gmail.com Tested-by: Paul Kocialkowski contact@paulk.fr
Changes in v2:
- Collect some tested-by tags
- Remove reference to f3f431a71272 in the commit message
- Drop Ccs that were erroneously added when submitting v1, remaining Ccs are from the original mailing list thread
Makefile | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-)
diff --git a/Makefile b/Makefile index 394ed09..f75c730 100644 --- a/Makefile +++ b/Makefile @@ -1230,10 +1230,16 @@ define filechk_version.h endef
define filechk_timestamp.h
(SOURCE_DATE="$${SOURCE_DATE_EPOCH:+@$$SOURCE_DATE_EPOCH}"; \LC_ALL=C date -u -d "$${SOURCE_DATE:-now}" +'#define U_BOOT_DATE "%b %d %C%y"'; \LC_ALL=C date -u -d "$${SOURCE_DATE:-now}" +'#define U_BOOT_TIME "%T"'; \LC_ALL=C date -u -d "$${SOURCE_DATE:-now}" +'#define U_BOOT_TZ "%z"' )
(if test -n "$${SOURCE_DATE_EPOCH}"; then \SOURCE_DATE="@$${SOURCE_DATE_EPOCH}"; \LC_ALL=C date -u -d "$${SOURCE_DATE}" +'#define U_BOOT_DATE "%b %d %C%y"'; \LC_ALL=C date -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TIME "%T"'; \LC_ALL=C date -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TZ "%z"'; \else \LC_ALL=C date +'#define U_BOOT_DATE "%b %d %C%y"'; \LC_ALL=C date +'#define U_BOOT_TIME "%T"'; \LC_ALL=C date +'#define U_BOOT_TZ "%z"'; \fi)endef
$(version_h): include/config/uboot.release FORCE
2.5.0.rc0
Ping? I think this may have been missed due to an excessive Cc list (which I've now culled).
On 31/07/15 07:25, Chris Packham wrote:
The problem is that date -u implies UTC. So +0000 is right if you do want the time to be displayed in UTC (for me living at +12/+13 UTC makes my head hurt because all the dates are yesterday).
One of the main problems with human-readable date/time is that it implies a calendar, with all its exceptions like leap years, changing DST, etc. Many RFCs, and the POSIX date functions, are highly flawed and conflate calendars with absolute time, e.g. don't specify they mean the Gregorian calendar, and don't specify what to do with dates before when the Gregorian calendar was introduced.
If you want to be precise about these things (so as to not cause problems in the future), then you need to specify all of that when dealing with human-readable dates. This is why for now, the Reproducible Builds timestamps proposal only defines SOURCE_DATE_EPOCH. We might extend it later, though.
The intent of f3f431a7 was to reflect the timezone that the build machine was set to. Dropping the -u would probably be sufficient but perhaps it would be better to make whole lot conditional on SOURCE_DATE_EPOCH being set. Something like this (untested, apologies gmail web interface)
ifneq ($(SOURCE_DATE_EPOCH),) define filechk_timestamp.h (SOURCE_DATE="$(SOURCE_DATE_EPOCH)"; \ LC_ALL=C date -u -d "$${SOURCE_DATE}" +'#define U_BOOT_DATE "%b %d %C%y"'; \ LC_ALL=C date -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TIME "%T"'; \ LC_ALL=C date -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TZ "%z"' ) endef else define filechk_timestamp.h (LC_ALL=C date +'#define U_BOOT_DATE "%b %d %C%y"'; \ LC_ALL=C date +'#define U_BOOT_TIME "%T"'; \ LC_ALL=C date +'#define U_BOOT_TZ "%z"') endef endif
Yes, it is best to force the TZ as UTC for now, if SOURCE_DATE_EPOCH is set.
At some point later, we might add SOURCE_DATE_TZOFFSET, or other formats that include a timezone offset, or something. (That would match Git's internal timestamp format, which is "$epochts $tzoffset", and can be translated unambiguously into ISO8601 and RFC2822 formats). For now though, we decided to keep things simple. But please say so, if you think that is something you (or others who use timestamps in build systems) would really want.
BTW, you need to do "@$${SOURCE_DATE_EPOCH}" when giving timestamps to date(1):
$ date -d @1438330000 +%s 1438330000 $ date -d 1438330000 +%s date: invalid date ‘1438330000’ 1
X
Along with SOURCE_DATE_EPOCH SOURCE_DATE_TZ can be used to recreate a build with a specific date timestamp. This allows the verification of source supplied with a pre-compiled binary.
If SOURCE_DATE_EPOCH is supplied SOURCE_DATE_TZ can be used to specify what will appear in the output of the version command. If SOURCE_DATE_TZ is not specified UTC will be used. SOURCE_DATE_TZ on it's own will not have an affect.
Signed-off-by: Chris Packham judge.packham@gmail.com --- This is a quick proof of concept that allows some control of how the timezone is displayed.
Makefile | 7 ++++--- README | 9 ++++++--- 2 files changed, 10 insertions(+), 6 deletions(-)
diff --git a/Makefile b/Makefile index f75c730..5102a9c 100644 --- a/Makefile +++ b/Makefile @@ -1232,9 +1232,10 @@ endef define filechk_timestamp.h (if test -n "$${SOURCE_DATE_EPOCH}"; then \ SOURCE_DATE="@$${SOURCE_DATE_EPOCH}"; \ - LC_ALL=C date -u -d "$${SOURCE_DATE}" +'#define U_BOOT_DATE "%b %d %C%y"'; \ - LC_ALL=C date -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TIME "%T"'; \ - LC_ALL=C date -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TZ "%z"'; \ + SOURCE_TZ="$${SOURCE_DATE_TZ:-UTC}"; \ + TZ="$${SOURCE_TZ}" LC_ALL=C date -d "$${SOURCE_DATE}" +'#define U_BOOT_DATE "%b %d %C%y"'; \ + TZ="$${SOURCE_TZ}" LC_ALL=C date -d "$${SOURCE_DATE}" +'#define U_BOOT_TIME "%T"'; \ + TZ="$${SOURCE_TZ}" LC_ALL=C date -d "$${SOURCE_DATE}" +'#define U_BOOT_TZ "%z"'; \ else \ LC_ALL=C date +'#define U_BOOT_DATE "%b %d %C%y"'; \ LC_ALL=C date +'#define U_BOOT_TIME "%T"'; \ diff --git a/README b/README index 1bcb63c..d2e3e94 100644 --- a/README +++ b/README @@ -5087,11 +5087,14 @@ Reproducible builds In order to achieve reproducible builds, timestamps used in the U-Boot build process have to be set to a fixed value.
-This is done using the SOURCE_DATE_EPOCH environment variable. -SOURCE_DATE_EPOCH is to be set on the build host's shell, not as a configuration -option for U-Boot or an environment variable in U-Boot. +This is done using the SOURCE_DATE_EPOCH and SOURCE_DATE_TZ environment +variables. These are to be set on the build host's shell, not as a +configuration option for U-Boot or an environment variable in U-Boot.
SOURCE_DATE_EPOCH should be set to a number of seconds since the epoch, in UTC. +SOURCE_DATE_TZ will default to UTC but can be set to another timezone and is +used to determine how the date is displayed (tzselect(1) can be used to +determine an appropriate value).
Building the Software: ======================
Le samedi 01 août 2015 à 22:32 +1200, Chris Packham a écrit :
Along with SOURCE_DATE_EPOCH SOURCE_DATE_TZ can be used to recreate a build with a specific date timestamp. This allows the verification of source supplied with a pre-compiled binary.
If SOURCE_DATE_EPOCH is supplied SOURCE_DATE_TZ can be used to specify what will appear in the output of the version command. If SOURCE_DATE_TZ is not specified UTC will be used. SOURCE_DATE_TZ on it's own will not have an affect.
Well, I worked with the assumption that SOURCE_DATE_EPOCH would always be provided in UTC, but I see no harm in providing SOURCE_DATE_TZ as well, provided that it falls back to UTC when not set, as is the current behaviour of tour patch.
Looks good to me!
Signed-off-by: Chris Packham judge.packham@gmail.com
This is a quick proof of concept that allows some control of how the timezone is displayed.
Makefile | 7 ++++--- README | 9 ++++++--- 2 files changed, 10 insertions(+), 6 deletions(-)
diff --git a/Makefile b/Makefile index f75c730..5102a9c 100644 --- a/Makefile +++ b/Makefile @@ -1232,9 +1232,10 @@ endef define filechk_timestamp.h (if test -n "$${SOURCE_DATE_EPOCH}"; then \ SOURCE_DATE="@$${SOURCE_DATE_EPOCH}"; \
LC_ALL=C date -u -d "$${SOURCE_DATE}" +'#define U_BOOT_DATE "%b %d %C%y"'; \LC_ALL=C date -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TIME "%T"'; \LC_ALL=C date -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TZ "%z"'; \
SOURCE_TZ="$${SOURCE_DATE_TZ:-UTC}"; \TZ="$${SOURCE_TZ}" LC_ALL=C date -d "$${SOURCE_DATE}" +'#define U_BOOT_DATE "%b %d %C%y"'; \TZ="$${SOURCE_TZ}" LC_ALL=C date -d "$${SOURCE_DATE}" +'#define U_BOOT_TIME "%T"'; \TZ="$${SOURCE_TZ}" LC_ALL=C date -d "$${SOURCE_DATE}" +'#define U_BOOT_TZ "%z"'; \diff --git a/README b/README index 1bcb63c..d2e3e94 100644 --- a/README +++ b/README @@ -5087,11 +5087,14 @@ Reproducible builds In order to achieve reproducible builds, timestamps used in the U-Boot build process have to be set to a fixed value.
-This is done using the SOURCE_DATE_EPOCH environment variable. -SOURCE_DATE_EPOCH is to be set on the build host's shell, not as a configuration -option for U-Boot or an environment variable in U-Boot. +This is done using the SOURCE_DATE_EPOCH and SOURCE_DATE_TZ environment +variables. These are to be set on the build host's shell, not as a +configuration option for U-Boot or an environment variable in U-Boot.
SOURCE_DATE_EPOCH should be set to a number of seconds since the epoch, in UTC. +SOURCE_DATE_TZ will default to UTC but can be set to another timezone and is +used to determine how the date is displayed (tzselect(1) can be used to +determine an appropriate value).
Building the Software:
On 01/08/15 20:47, Paul Kocialkowski wrote:
Le samedi 01 août 2015 à 22:32 +1200, Chris Packham a écrit :
Along with SOURCE_DATE_EPOCH SOURCE_DATE_TZ can be used to recreate a build with a specific date timestamp. This allows the verification of source supplied with a pre-compiled binary.
If SOURCE_DATE_EPOCH is supplied SOURCE_DATE_TZ can be used to specify what will appear in the output of the version command. If SOURCE_DATE_TZ is not specified UTC will be used. SOURCE_DATE_TZ on it's own will not have an affect.
Well, I worked with the assumption that SOURCE_DATE_EPOCH would always be provided in UTC, but I see no harm in providing SOURCE_DATE_TZ as well, provided that it falls back to UTC when not set, as is the current behaviour of tour patch.
To clarify, SOURCE_DATE_EPOCH is a unix timestamp which is defined as the number of seconds (excluding leap seconds) since Jan 1 1970 UTC. There is no way to specify this "in another timezone"; there is no ambiguity here.
However, I am not sure that us at Reproducible Builds will actually adopt this variable. We haven't talked about it beyond my previous email [1], it was just me quickly skimming off my thoughts and firing off quick ideas. My personal concern about SOURCE_DATE_TZ is that it implies that it could take actual named time zones, like "EST" or "Europe/London"; it is **extremely unlikely** that we will do anything like this soon, because this would involve timezone databases and complex things like that. This is why in my previous email I suggested the SOURCE_DATE_TZOFFSET variable. Also, the TZ variable as specified by POSIX specifies the offset in the *opposite* direction to what ISO8601 and RFC2822 displays.
The git internal timestamp format only supports timezone offsets in the common direction, opposite to TZ, i.e. positive for east of Greenwich and negative for west.
I'd suggest to leave out SOURCE_DATE_TZ for now, until we come up with a more precisely defined *meaning* for this variable. It is meant to be a standard across all tools, so some time to think through the issues is necessary. Otherwise we risk leaving a clusterfuck of a legacy, like the POSIX time functions.
X
[1] https://lists.alioth.debian.org/pipermail/reproducible-builds/Week-of-Mon-20... or http://lists.denx.de/pipermail/u-boot/2015-July/221301.html
On 02/08/15 00:02, Ximin Luo wrote:
However, I am not sure that us at Reproducible Builds will actually adopt this variable. We haven't talked about it beyond my previous email [1], it was just me quickly skimming off my thoughts and firing off quick ideas.
Whoops, I mean the _TZ variable. The _EPOCH variable has been decided and we're trying to get tools to support this.
On Sat, Aug 01, 2015 at 10:32:49PM +1200, Chris Packham wrote:
Along with SOURCE_DATE_EPOCH SOURCE_DATE_TZ can be used to recreate a build with a specific date timestamp. This allows the verification of source supplied with a pre-compiled binary.
If SOURCE_DATE_EPOCH is supplied SOURCE_DATE_TZ can be used to specify what will appear in the output of the version command. If SOURCE_DATE_TZ is not specified UTC will be used. SOURCE_DATE_TZ on it's own will not have an affect.
Signed-off-by: Chris Packham judge.packham@gmail.com
This is a quick proof of concept that allows some control of how the timezone is displayed.
Can you please re-submit this, and the related EPOC patches as a series together? Thanks!
On Thu, Aug 13, 2015 at 4:40 AM, Tom Rini trini@konsulko.com wrote:
On Sat, Aug 01, 2015 at 10:32:49PM +1200, Chris Packham wrote:
Along with SOURCE_DATE_EPOCH SOURCE_DATE_TZ can be used to recreate a build with a specific date timestamp. This allows the verification of source supplied with a pre-compiled binary.
If SOURCE_DATE_EPOCH is supplied SOURCE_DATE_TZ can be used to specify what will appear in the output of the version command. If SOURCE_DATE_TZ is not specified UTC will be used. SOURCE_DATE_TZ on it's own will not have an affect.
Signed-off-by: Chris Packham judge.packham@gmail.com
This is a quick proof of concept that allows some control of how the timezone is displayed.
Can you please re-submit this, and the related EPOC patches as a series together? Thanks!
Hi Tom,
Will do although Ximin Luo had some concerns[1]. The code works but there may (will?) be some churn in the naming/concept for the TZ part.
On Thu, Jul 30, 2015 at 11:54 PM, Bin Meng bmeng.cn@gmail.com wrote:
This commit breaks the following commit:
commit f3f431a712729a1af94d01bd1bfde17a252ff02c Author: Chris Packham judge.packham@gmail.com Date: Sun May 10 21:02:09 2015 +1200
Makefile: Add U_BOOT_TZ and include in version Define U_BOOT_TZ alongside U_BOOT_TIME and U_BOOT_DATE and use it to include the timezone in the version output. Acked-by: Simon Glass <sjg@chromium.org> Signed-off-by: Chris Packham <judge.packham@gmail.com>Before this commit I have: U-Boot 2015.07-00345-g9c57487 (Jul 31 2015 - 10:49:31 +0800)
After this commit I have: U-Boot 2015.07-00346-gf3f431a (Jul 31 2015 - 02:50:54 +0000)
As you see: the timezone information is missing, and U-Boot's timestamp is now GMT+0 (the correct one should be GMT+8)
Is this intended behavior? Or if not, please fix it.
I notice the same behavior here and I agree it would be nice to fix this.
Thanks
On 07/28/2015 05:00 PM, Tom Rini wrote:
On Sun, Jul 26, 2015 at 06:48:15PM +0200, Paul Kocialkowski wrote:
In order to achieve reproducible builds in U-Boot, timestamps that are defined at build-time have to be somewhat eliminated. The SOURCE_DATE_EPOCH environment variable allows setting a fixed value for those timestamps.
Simply by setting SOURCE_DATE_EPOCH to a fixed value, a number of targets can be built reproducibly. This is the case for e.g. sunxi devices.
However, some other devices might need some more tweaks, especially regarding the image generation tools.
Signed-off-by: Paul Kocialkowski contact@paulk.fr
Applied to u-boot/master, thanks!
This commit breaks build on non GNU hosts (like OS X and persumably other *BSD hosts). Before, those hosts where supported, so for me this has to be fixed for 2015.10
We need a) some mechanism to search for the GNU date variant or b) some wrapper to provide the correct output on those host machines.
I vote for a), it is acceptable to have the GNU date available but we should error on 'no GNU date available'. Furthermore we need to have the date command exchangeable by e.g. gdate, gnudate, ... maybe with full path.
Andreas
On 2015-08-25, Andreas Bießmann wrote:
On 07/28/2015 05:00 PM, Tom Rini wrote:
On Sun, Jul 26, 2015 at 06:48:15PM +0200, Paul Kocialkowski wrote:
In order to achieve reproducible builds in U-Boot, timestamps that are defined at build-time have to be somewhat eliminated. The SOURCE_DATE_EPOCH environment variable allows setting a fixed value for those timestamps.
Simply by setting SOURCE_DATE_EPOCH to a fixed value, a number of targets can be built reproducibly. This is the case for e.g. sunxi devices.
However, some other devices might need some more tweaks, especially regarding the image generation tools.
Signed-off-by: Paul Kocialkowski contact@paulk.fr
Applied to u-boot/master, thanks!
This commit breaks build on non GNU hosts (like OS X and persumably other *BSD hosts). Before, those hosts where supported, so for me this has to be fixed for 2015.10
We need a) some mechanism to search for the GNU date variant or b) some wrapper to provide the correct output on those host machines.
I vote for a), it is acceptable to have the GNU date available but we should error on 'no GNU date available'. Furthermore we need to have the date command exchangeable by e.g. gdate, gnudate, ... maybe with full path.
There was a proposed patch which only uses the GNU date extensions if SOURCE_DATE_EPOCH environment variable is set, would this sufficiently address your concerns, at least for the short term?
Message-Id: 1438337042-30762-1-git-send-email-judge.packham@gmail.com http://lists.denx.de/pipermail/u-boot/2015-August/221429.html
live well, vagrant
On 08/25/2015 11:55 AM, Vagrant Cascadian wrote:
On 2015-08-25, Andreas Bießmann wrote:
On 07/28/2015 05:00 PM, Tom Rini wrote:
On Sun, Jul 26, 2015 at 06:48:15PM +0200, Paul Kocialkowski wrote:
In order to achieve reproducible builds in U-Boot, timestamps that are defined at build-time have to be somewhat eliminated. The SOURCE_DATE_EPOCH environment variable allows setting a fixed value for those timestamps.
Simply by setting SOURCE_DATE_EPOCH to a fixed value, a number of targets can be built reproducibly. This is the case for e.g. sunxi devices.
However, some other devices might need some more tweaks, especially regarding the image generation tools.
Signed-off-by: Paul Kocialkowski contact@paulk.fr
Applied to u-boot/master, thanks!
This commit breaks build on non GNU hosts (like OS X and persumably other *BSD hosts). Before, those hosts where supported, so for me this has to be fixed for 2015.10
We need a) some mechanism to search for the GNU date variant or b) some wrapper to provide the correct output on those host machines.
I vote for a), it is acceptable to have the GNU date available but we should error on 'no GNU date available'. Furthermore we need to have the date command exchangeable by e.g. gdate, gnudate, ... maybe with full path.
There was a proposed patch which only uses the GNU date extensions if SOURCE_DATE_EPOCH environment variable is set, would this sufficiently address your concerns, at least for the short term?
Message-Id: 1438337042-30762-1-git-send-email-judge.packham@gmail.com http://lists.denx.de/pipermail/u-boot/2015-August/221429.html
thanks for the pointer, normal builds work with that change.
Andreas
Le mardi 25 août 2015 à 12:20 +0200, Andreas Bießmann a écrit :
On 08/25/2015 11:55 AM, Vagrant Cascadian wrote:
On 2015-08-25, Andreas Bießmann wrote:
On 07/28/2015 05:00 PM, Tom Rini wrote:
On Sun, Jul 26, 2015 at 06:48:15PM +0200, Paul Kocialkowski wrote:
In order to achieve reproducible builds in U-Boot, timestamps that are defined at build-time have to be somewhat eliminated. The SOURCE_DATE_EPOCH environment variable allows setting a fixed value for those timestamps.
Simply by setting SOURCE_DATE_EPOCH to a fixed value, a number of targets can be built reproducibly. This is the case for e.g. sunxi devices.
However, some other devices might need some more tweaks, especially regarding the image generation tools.
Signed-off-by: Paul Kocialkowski contact@paulk.fr
Applied to u-boot/master, thanks!
This commit breaks build on non GNU hosts (like OS X and persumably other *BSD hosts). Before, those hosts where supported, so for me this has to be fixed for 2015.10
We need a) some mechanism to search for the GNU date variant or b) some wrapper to provide the correct output on those host machines.
I vote for a), it is acceptable to have the GNU date available but we should error on 'no GNU date available'. Furthermore we need to have the date command exchangeable by e.g. gdate, gnudate, ... maybe with full path.
There was a proposed patch which only uses the GNU date extensions if SOURCE_DATE_EPOCH environment variable is set, would this sufficiently address your concerns, at least for the short term?
Message-Id: 1438337042-30762-1-git-send-email-judge.packham@gmail.com http://lists.denx.de/pipermail/u-boot/2015-August/221429.html
thanks for the pointer, normal builds work with that change.
I think we should get that patch merged so that it doesn't affect regular builds on non-GNU hosts. It is also relevant for the purpose it serves initially, too.
Hi,
(just stating the (IMHO) obvious here…)
On Dienstag, 25. August 2015, Vagrant Cascadian wrote:
There was a proposed patch which only uses the GNU date extensions if SOURCE_DATE_EPOCH environment variable is set, would this sufficiently address your concerns, at least for the short term?
there also should be a solution which works on BSD in the not so long term if we want the SOURCE_DATE_EPOCH specification to take off..!
(and this probably also affects other software where we want to (or did) introduce SOURCE_DATE_EPOCH!)
cheers, Holger
The SOURCE_DATE_EPOCH mechanism for reproducible builds requires the GNU variant of date. Respect this and search it, error on missing GNU date.
Signed-off-by: Andreas Bießmann andreas.devel@googlemail.com --- This commit tries to figure out if we have a GNU date variant available. It errors on missing GNU date when it is required (for SOURCE_DATE_EPOCH set). The result is:
---8<--- abiessmann@punisher % PATH=$ARMv5_PATH:$PATH CROSS_COMPILE=arm-v5te-linux-gnueabi- make SOURCE_DATE_EPOCH="`date -R`" O=/tmp/picosam ARCH=arm include/generated/timestamp_autogenerated.h make[1]: Entering directory '/tmp/picosam' /home/abiessmann/src/u-boot/Makefile:1303: *** "No GNU date found". Stop. make[1]: Leaving directory '/tmp/picosam' Makefile:146: recipe for target 'sub-make' failed make: *** [sub-make] Error 2 abiessmann@punisher % PATH=$ARMv5_PATH:$PATH CROSS_COMPILE=arm-v5te-linux-gnueabi- make O=/tmp/picosam ARCH=arm include/generated/timestamp_autogenerated.h make[1]: Entering directory '/tmp/picosam' CHK include/generated/timestamp_autogenerated.h UPD include/generated/timestamp_autogenerated.h make[1]: Leaving directory '/tmp/picosam' --->8---
It applies on top of http://patchwork.ozlabs.org/patch/506856/ (Makefile: Use correct timezone for U_BOOT_TZ).
Makefile | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-)
diff --git a/Makefile b/Makefile index b9b2375..95eae64 100644 --- a/Makefile +++ b/Makefile @@ -346,6 +346,9 @@ PERL = perl PYTHON = python DTC = dtc CHECK = sparse +GNUDATE := $(foreach date,gdate date.gnu date, \ + $(shell _date=`which $(date)`; \ + $${_date} --version 2> /dev/null | $(AWK) "/GNU coreutils/ { print "$${_date}"; }"))
CHECKFLAGS := -D__linux__ -Dlinux -D__STDC__ -Dunix -D__unix__ \ -Wbitwise -Wno-return-void -D__CHECK_ENDIAN__ $(CF) @@ -1281,9 +1284,9 @@ endef define filechk_timestamp.h (if test -n "$${SOURCE_DATE_EPOCH}"; then \ SOURCE_DATE="@$${SOURCE_DATE_EPOCH}"; \ - LC_ALL=C date -u -d "$${SOURCE_DATE}" +'#define U_BOOT_DATE "%b %d %C%y"'; \ - LC_ALL=C date -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TIME "%T"'; \ - LC_ALL=C date -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TZ "%z"'; \ + LC_ALL=C $(GNUDATE) -u -d "$${SOURCE_DATE}" +'#define U_BOOT_DATE "%b %d %C%y"'; \ + LC_ALL=C $(GNUDATE) -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TIME "%T"'; \ + LC_ALL=C $(GNUDATE) -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TZ "%z"'; \ else \ LC_ALL=C date +'#define U_BOOT_DATE "%b %d %C%y"'; \ LC_ALL=C date +'#define U_BOOT_TIME "%T"'; \ @@ -1295,6 +1298,11 @@ $(version_h): include/config/uboot.release FORCE $(call filechk,version.h)
$(timestamp_h): $(srctree)/Makefile FORCE +ifneq ($(strip $(SOURCE_DATE_EPOCH)),) +ifeq ($(strip $(GNUDATE)),) + $(error "No GNU date found") +endif +endif $(call filechk,timestamp.h)
# ---------------------------------------------------------------------------
On Thursday, August 27, 2015 at 10:13:49 AM, Andreas Bießmann wrote:
The SOURCE_DATE_EPOCH mechanism for reproducible builds requires the GNU variant of date.
Why does it require the GNU date ?
Respect this and search it, error on missing GNU date.
Wouldn't it make more sense to fix the code to NOT depend on GNU extensions ?
Signed-off-by: Andreas Bießmann andreas.devel@googlemail.com
[...]
Best regards, Marek Vasut
Hi Marek,
On 08/27/2015 10:28 AM, Marek Vasut wrote:
On Thursday, August 27, 2015 at 10:13:49 AM, Andreas Bießmann wrote:
The SOURCE_DATE_EPOCH mechanism for reproducible builds requires the GNU variant of date.
Why does it require the GNU date ?
Cause of the -d switch. It is used in GNU date variants to set the time to print the string from. On the other hand *BSD variants use it to set the DST flag.
Respect this and search it, error on missing GNU date.
Wouldn't it make more sense to fix the code to NOT depend on GNU extensions ?
That would be great, but so we need to check for the tool and use the right switches depending on that. I think forcing the user to provide the right tool here is Ok.
Best regards
Andreas
On Thursday, August 27, 2015 at 11:01:19 AM, Andreas Bießmann wrote:
Hi Marek,
Hi!
On 08/27/2015 10:28 AM, Marek Vasut wrote:
On Thursday, August 27, 2015 at 10:13:49 AM, Andreas Bießmann wrote:
The SOURCE_DATE_EPOCH mechanism for reproducible builds requires the GNU variant of date.
Why does it require the GNU date ?
Cause of the -d switch. It is used in GNU date variants to set the time to print the string from. On the other hand *BSD variants use it to set the DST flag.
This sort of stuff should be in the commit message :)
Respect this and search it, error on missing GNU date.
Wouldn't it make more sense to fix the code to NOT depend on GNU extensions ?
That would be great, but so we need to check for the tool and use the right switches depending on that. I think forcing the user to provide the right tool here is Ok.
I think we don't have a better option here anyway, right ? Maybe try checking uname ? But you can have linux with bsd userspace ... and probably BSD with GNU userspace, meh.
Best regards, Marek Vasut
Le jeudi 27 août 2015 à 10:13 +0200, Andreas Bießmann a écrit :
The SOURCE_DATE_EPOCH mechanism for reproducible builds requires the GNU variant of date. Respect this and search it, error on missing GNU date.
Well, IMHO we shouldn't check for GNU date but for the extensions it implements. Those could be included in other date implementations. For instance, it looks like busybox's date implements the -u and -d options too.
Signed-off-by: Andreas Bießmann andreas.devel@googlemail.com
This commit tries to figure out if we have a GNU date variant available. It errors on missing GNU date when it is required (for SOURCE_DATE_EPOCH set). The result is:
Also, I don't think it should be up to the Makefile to figure out which date binary to use. It should be up to the user to put the right one as "date", e.g. with an alias. That is, unless the name for GNU date is something really standardized on non-GNU systems.
---8<--- abiessmann@punisher % PATH=$ARMv5_PATH:$PATH CROSS_COMPILE=arm-v5te-linux-gnueabi- make SOURCE_DATE_EPOCH="`date -R`" O=/tmp/picosam ARCH=arm include/generated/timestamp_autogenerated.h make[1]: Entering directory '/tmp/picosam' /home/abiessmann/src/u-boot/Makefile:1303: *** "No GNU date found". Stop. make[1]: Leaving directory '/tmp/picosam' Makefile:146: recipe for target 'sub-make' failed make: *** [sub-make] Error 2 abiessmann@punisher % PATH=$ARMv5_PATH:$PATH CROSS_COMPILE=arm-v5te-linux-gnueabi- make O=/tmp/picosam ARCH=arm include/generated/timestamp_autogenerated.h make[1]: Entering directory '/tmp/picosam' CHK include/generated/timestamp_autogenerated.h UPD include/generated/timestamp_autogenerated.h make[1]: Leaving directory '/tmp/picosam' --->8---
It applies on top of http://patchwork.ozlabs.org/patch/506856/ (Makefile: Use correct timezone for U_BOOT_TZ).
Makefile | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-)
diff --git a/Makefile b/Makefile index b9b2375..95eae64 100644 --- a/Makefile +++ b/Makefile @@ -346,6 +346,9 @@ PERL = perl PYTHON = python DTC = dtc CHECK = sparse +GNUDATE := $(foreach date,gdate date.gnu date, \
$(shell _date=`which $(date)`; \$${_date} --version 2> /dev/null | $(AWK) "/GNU coreutils/ { print \"$${_date}\"; }"))
I advise running date with the needed options (-u -d) and checked the error code instead of this. And of course, the variable names shouldn't mention GNU date.
CHECKFLAGS := -D__linux__ -Dlinux -D__STDC__ -Dunix -D__unix__ \ -Wbitwise -Wno-return-void -D__CHECK_ENDIAN__ $(CF) @@ -1281,9 +1284,9 @@ endef define filechk_timestamp.h (if test -n "$${SOURCE_DATE_EPOCH}"; then \ SOURCE_DATE="@$${SOURCE_DATE_EPOCH}"; \
LC_ALL=C date -u -d "$${SOURCE_DATE}" +'#define U_BOOT_DATE "%b %d %C%y"'; \LC_ALL=C date -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TIME "%T"'; \LC_ALL=C date -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TZ "%z"'; \
LC_ALL=C $(GNUDATE) -u -d "$${SOURCE_DATE}" +'#define U_BOOT_DATE "%b %d %C%y"'; \LC_ALL=C $(GNUDATE) -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TIME "%T"'; \LC_ALL=C $(GNUDATE) -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TZ "%z"'; \@@ -1295,6 +1298,11 @@ $(version_h): include/config/uboot.release FORCE $(call filechk,version.h)
$(timestamp_h): $(srctree)/Makefile FORCE +ifneq ($(strip $(SOURCE_DATE_EPOCH)),) +ifeq ($(strip $(GNUDATE)),)
- $(error "No GNU date found")
+endif +endif $(call filechk,timestamp.h)
# ---------------------------------------------------------------------------
On 08/27/2015 10:32 AM, Paul Kocialkowski wrote:
Le jeudi 27 août 2015 à 10:13 +0200, Andreas Bießmann a écrit :
The SOURCE_DATE_EPOCH mechanism for reproducible builds requires the GNU variant of date. Respect this and search it, error on missing GNU date.
Well, IMHO we shouldn't check for GNU date but for the extensions it implements. Those could be included in other date implementations. For instance, it looks like busybox's date implements the -u and -d options too.
But BSD date used the -d switch to set the timezone [1] ...
---8<--- -d dst Set the kernel's value for daylight saving time. If dst is non-zero, future calls to gettimeofday(2) will return a non-zero for tz_dsttime. --->8---
And it is the same for OS X [2].
Signed-off-by: Andreas Bießmann andreas.devel@googlemail.com
This commit tries to figure out if we have a GNU date variant available. It errors on missing GNU date when it is required (for SOURCE_DATE_EPOCH set). The result is:
Also, I don't think it should be up to the Makefile to figure out which date binary to use. It should be up to the user to put the right one as "date", e.g. with an alias. That is, unless the name for GNU date is something really standardized on non-GNU systems.
Most *BSD systems use gdate or date.gnu for the GNU variant of date, if available. The date tool available is the one of the *BSD distributiuon. HAving a shell alias for date covering gdate is for some users not an option. So it has to be some make variable. For me it would be neat to have the make find some possible solution (e.g. search for the GNU variant on systems which have it).
diff --git a/Makefile b/Makefile index b9b2375..95eae64 100644 --- a/Makefile +++ b/Makefile @@ -346,6 +346,9 @@ PERL = perl PYTHON = python DTC = dtc CHECK = sparse +GNUDATE := $(foreach date,gdate date.gnu date, \
$(shell _date=`which $(date)`; \$${_date} --version 2> /dev/null | $(AWK) "/GNU coreutils/ { print \"$${_date}\"; }"))I advise running date with the needed options (-u -d) and checked the error code instead of this. And of course, the variable names shouldn't mention GNU date.
Good point, let's see how my OS X date behaves with a set '-d' switch.
Andreas
[1] http://www.freebsd.org/cgi/man.cgi?query=date [2] https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPa...
The SOURCE_DATE_EPOCH mechanism for reproducible builds require some date with -u and -d switch to print the date string of another time. In other words it requires some date that behaves like the GNU date.
Respect this and search a working date, error on no working version.
Signed-off-by: Andreas Bießmann andreas.devel@googlemail.com --- This commit tries to figure out if we have a date variant available that supports the '-u' and '-d "@0"' switches. It errors on non-working variants of date.
To respect *BSD host systems search for gdate and date.gnu. Those pre- and suffixes are widespread used for the GNU variant of a tool also avialable on *BSD systems.
The result is:
---8<--- abiessmann@punisher % PATH=$ARMv5_PATH:$PATH CROSS_COMPILE=arm-v5te-linux-gnueabi- make O=/tmp/picosam ARCH=arm include/generated/timestamp_autogenerated.h make[1]: Entering directory '/tmp/picosam' CHK include/generated/timestamp_autogenerated.h UPD include/generated/timestamp_autogenerated.h make[1]: Leaving directory '/tmp/picosam' abiessmann@punisher % PATH=$ARMv5_PATH:$PATH CROSS_COMPILE=arm-v5te-linux-gnueabi- make O=/tmp/picosam SOURCE_DATE_EPOCH="0" ARCH=arm include/generated/timestamp_autogenerated.h make[1]: Entering directory '/tmp/picosam' /home/abiessmann/src/u-boot/Makefile:1304: *** "Your gdate/date.gnu/date does not support the '-u' and '-d' switches like GNU date does!". Stop. make[1]: Leaving directory '/tmp/picosam' Makefile:146: recipe for target 'sub-make' failed make: *** [sub-make] Error 2 --->8---
It applies on top of http://patchwork.ozlabs.org/patch/506856/ (Makefile: Use correct timezone for U_BOOT_TZ).
Changes in v2: * check for '-u' and '-d "@0"' switch rather than for the GNU variant
Makefile | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-)
diff --git a/Makefile b/Makefile index b9b2375..b797e38 100644 --- a/Makefile +++ b/Makefile @@ -346,6 +346,10 @@ PERL = perl PYTHON = python DTC = dtc CHECK = sparse +DATE := $(foreach date,gdate date.gnu date, \ + $(shell _date=`which $(date)`; \ + $${_date} -u -d "@0" >/dev/null 2>&1; \ + test $$? -eq 0 && echo $${_date}))
CHECKFLAGS := -D__linux__ -Dlinux -D__STDC__ -Dunix -D__unix__ \ -Wbitwise -Wno-return-void -D__CHECK_ENDIAN__ $(CF) @@ -1281,9 +1285,9 @@ endef define filechk_timestamp.h (if test -n "$${SOURCE_DATE_EPOCH}"; then \ SOURCE_DATE="@$${SOURCE_DATE_EPOCH}"; \ - LC_ALL=C date -u -d "$${SOURCE_DATE}" +'#define U_BOOT_DATE "%b %d %C%y"'; \ - LC_ALL=C date -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TIME "%T"'; \ - LC_ALL=C date -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TZ "%z"'; \ + LC_ALL=C $(DATE) -u -d "$${SOURCE_DATE}" +'#define U_BOOT_DATE "%b %d %C%y"'; \ + LC_ALL=C $(DATE) -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TIME "%T"'; \ + LC_ALL=C $(DATE) -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TZ "%z"'; \ else \ LC_ALL=C date +'#define U_BOOT_DATE "%b %d %C%y"'; \ LC_ALL=C date +'#define U_BOOT_TIME "%T"'; \ @@ -1295,6 +1299,11 @@ $(version_h): include/config/uboot.release FORCE $(call filechk,version.h)
$(timestamp_h): $(srctree)/Makefile FORCE +ifneq ($(strip $(SOURCE_DATE_EPOCH)),) +ifeq ($(strip $(DATE)),) + $(error "Your gdate/date.gnu/date does not support the '-u' and '-d' switches like GNU date does!") +endif +endif $(call filechk,timestamp.h)
# ---------------------------------------------------------------------------
Le jeudi 27 août 2015 à 11:30 +0200, Andreas Bießmann a écrit :
The SOURCE_DATE_EPOCH mechanism for reproducible builds require some date with -u and -d switch to print the date string of another time. In other words it requires some date that behaves like the GNU date.
Respect this and search a working date, error on no working version.
Looks good to me, except one nitpick, see below.
Signed-off-by: Andreas Bießmann andreas.devel@googlemail.com
This commit tries to figure out if we have a date variant available that supports the '-u' and '-d "@0"' switches. It errors on non-working variants of date.
To respect *BSD host systems search for gdate and date.gnu. Those pre- and suffixes are widespread used for the GNU variant of a tool also avialable on *BSD systems.
Fair enough then, that's fine with me.
The result is:
---8<--- abiessmann@punisher % PATH=$ARMv5_PATH:$PATH CROSS_COMPILE=arm-v5te-linux-gnueabi- make O=/tmp/picosam ARCH=arm include/generated/timestamp_autogenerated.h make[1]: Entering directory '/tmp/picosam' CHK include/generated/timestamp_autogenerated.h UPD include/generated/timestamp_autogenerated.h make[1]: Leaving directory '/tmp/picosam' abiessmann@punisher % PATH=$ARMv5_PATH:$PATH CROSS_COMPILE=arm-v5te-linux-gnueabi- make O=/tmp/picosam SOURCE_DATE_EPOCH="0" ARCH=arm include/generated/timestamp_autogenerated.h make[1]: Entering directory '/tmp/picosam' /home/abiessmann/src/u-boot/Makefile:1304: *** "Your gdate/date.gnu/date does not support the '-u' and '-d' switches like GNU date does!". Stop. make[1]: Leaving directory '/tmp/picosam' Makefile:146: recipe for target 'sub-make' failed make: *** [sub-make] Error 2 --->8---
It applies on top of http://patchwork.ozlabs.org/patch/506856/ (Makefile: Use correct timezone for U_BOOT_TZ).
Changes in v2:
- check for '-u' and '-d "@0"' switch rather than for the GNU variant
Makefile | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-)
diff --git a/Makefile b/Makefile index b9b2375..b797e38 100644 --- a/Makefile +++ b/Makefile @@ -346,6 +346,10 @@ PERL = perl PYTHON = python DTC = dtc CHECK = sparse +DATE := $(foreach date,gdate date.gnu date, \
$(shell _date=`which $(date)`; \$${_date} -u -d "@0" >/dev/null 2>&1; \test $$? -eq 0 && echo $${_date}))
First, I don't understand why you need to call date with the full path: if which can find it, then calling the binary without its full path should do just as well, right?
Then, correct me if I'm wrong, but calling test and using && is overkill, you could simply do: $${_date} -u -d "@0" >/dev/null 2>&1 && echo $${_date}
So in the end, the whole line would look like:
DATE := $(foreach date,gdate date.gnu date, \ $($${date} -u -d "@0" >/dev/null 2>&1 \ && echo $${date}))
Let me know what you think (and please test it as well).
CHECKFLAGS := -D__linux__ -Dlinux -D__STDC__ -Dunix -D__unix__ \ -Wbitwise -Wno-return-void -D__CHECK_ENDIAN__ $(CF) @@ -1281,9 +1285,9 @@ endef define filechk_timestamp.h (if test -n "$${SOURCE_DATE_EPOCH}"; then \ SOURCE_DATE="@$${SOURCE_DATE_EPOCH}"; \
LC_ALL=C date -u -d "$${SOURCE_DATE}" +'#define U_BOOT_DATE "%b %d %C%y"'; \LC_ALL=C date -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TIME "%T"'; \LC_ALL=C date -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TZ "%z"'; \
LC_ALL=C $(DATE) -u -d "$${SOURCE_DATE}" +'#define U_BOOT_DATE "%b %d %C%y"'; \LC_ALL=C $(DATE) -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TIME "%T"'; \LC_ALL=C $(DATE) -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TZ "%z"'; \@@ -1295,6 +1299,11 @@ $(version_h): include/config/uboot.release FORCE $(call filechk,version.h)
$(timestamp_h): $(srctree)/Makefile FORCE +ifneq ($(strip $(SOURCE_DATE_EPOCH)),) +ifeq ($(strip $(DATE)),)
- $(error "Your gdate/date.gnu/date does not support the '-u' and '-d' switches like GNU date does!")
+endif +endif $(call filechk,timestamp.h)
# ---------------------------------------------------------------------------
On 08/27/2015 03:03 PM, Paul Kocialkowski wrote:
Le jeudi 27 août 2015 à 11:30 +0200, Andreas Bießmann a écrit :
Changes in v2:
- check for '-u' and '-d "@0"' switch rather than for the GNU variant
Makefile | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-)
diff --git a/Makefile b/Makefile index b9b2375..b797e38 100644 --- a/Makefile +++ b/Makefile @@ -346,6 +346,10 @@ PERL = perl PYTHON = python DTC = dtc CHECK = sparse +DATE := $(foreach date,gdate date.gnu date, \
$(shell _date=`which $(date)`; \$${_date} -u -d "@0" >/dev/null 2>&1; \test $$? -eq 0 && echo $${_date}))First, I don't understand why you need to call date with the full path: if which can find it, then calling the binary without its full path should do just as well, right?
You'r right.
Then, correct me if I'm wrong, but calling test and using && is overkill, you could simply do: $${_date} -u -d "@0" >/dev/null 2>&1 && echo $${_date}
Also true.
So in the end, the whole line would look like:
DATE := $(foreach date,gdate date.gnu date, \ $($${date} -u -d "@0" >/dev/null 2>&1 \ && echo $${date}))
Let me know what you think (and please test it as well).
That should work. I wonder however why we don't include this snippet in the filechk_timestamp.h script directly. In fact the $(DATE) (a date that support -d '@0' switch) is just used there, so why clobber the Makefile with it?
---8<--- define filechk_timestamp.h (if test -n "$${SOURCE_DATE_EPOCH}"; then \ date=""; \ for _date in gdate date.gnu date; do \ $${_date} -u -d "$${SOURCE_DATE}" >/dev/null 2>&1 && \ date="$${_date}"; \ done; \ if test -n "$${date}"; then \ SOURCE_DATE="@$${SOURCE_DATE_EPOCH}"; \ LC_ALL=C $${date} -u -d "$${SOURCE_DATE}" +'#define U_BOOT_DATE "%b %d %C%y"'; \ LC_ALL=C $${date} -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TIME "%T"'; \ LC_ALL=C $${date} -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TZ "%z"'; \ else \ return 42; \ fi; \ else \ LC_ALL=C date +'#define U_BOOT_DATE "%b %d %C%y"'; \ LC_ALL=C date +'#define U_BOOT_TIME "%T"'; \ LC_ALL=C date +'#define U_BOOT_TZ "%z"'; \ fi) endef --->8---
CHECKFLAGS := -D__linux__ -Dlinux -D__STDC__ -Dunix -D__unix__ \ -Wbitwise -Wno-return-void -D__CHECK_ENDIAN__ $(CF) @@ -1281,9 +1285,9 @@ endef define filechk_timestamp.h (if test -n "$${SOURCE_DATE_EPOCH}"; then \ SOURCE_DATE="@$${SOURCE_DATE_EPOCH}"; \
LC_ALL=C date -u -d "$${SOURCE_DATE}" +'#define U_BOOT_DATE "%b %d %C%y"'; \LC_ALL=C date -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TIME "%T"'; \LC_ALL=C date -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TZ "%z"'; \
LC_ALL=C $(DATE) -u -d "$${SOURCE_DATE}" +'#define U_BOOT_DATE "%b %d %C%y"'; \LC_ALL=C $(DATE) -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TIME "%T"'; \LC_ALL=C $(DATE) -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TZ "%z"'; \@@ -1295,6 +1299,11 @@ $(version_h): include/config/uboot.release FORCE $(call filechk,version.h)
$(timestamp_h): $(srctree)/Makefile FORCE +ifneq ($(strip $(SOURCE_DATE_EPOCH)),) +ifeq ($(strip $(DATE)),)
- $(error "Your gdate/date.gnu/date does not support the '-u' and '-d' switches like GNU date does!")
+endif +endif $(call filechk,timestamp.h)
# ---------------------------------------------------------------------------
Le jeudi 27 août 2015 à 15:52 +0200, Andreas Bießmann a écrit :
On 08/27/2015 03:03 PM, Paul Kocialkowski wrote:
Le jeudi 27 août 2015 à 11:30 +0200, Andreas Bießmann a écrit :
Changes in v2:
- check for '-u' and '-d "@0"' switch rather than for the GNU variant
Makefile | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-)
diff --git a/Makefile b/Makefile index b9b2375..b797e38 100644 --- a/Makefile +++ b/Makefile @@ -346,6 +346,10 @@ PERL = perl PYTHON = python DTC = dtc CHECK = sparse +DATE := $(foreach date,gdate date.gnu date, \
$(shell _date=`which $(date)`; \$${_date} -u -d "@0" >/dev/null 2>&1; \test $$? -eq 0 && echo $${_date}))First, I don't understand why you need to call date with the full path: if which can find it, then calling the binary without its full path should do just as well, right?
You'r right.
Then, correct me if I'm wrong, but calling test and using && is overkill, you could simply do: $${_date} -u -d "@0" >/dev/null 2>&1 && echo $${_date}
Also true.
So in the end, the whole line would look like:
DATE := $(foreach date,gdate date.gnu date, \ $($${date} -u -d "@0" >/dev/null 2>&1 \ && echo $${date}))
Let me know what you think (and please test it as well).
That should work. I wonder however why we don't include this snippet in the filechk_timestamp.h script directly. In fact the $(DATE) (a date that support -d '@0' switch) is just used there, so why clobber the Makefile with it?
I'm fine with both options, but your suggestion also makes it more obvious why we're doing that check at all.
---8<--- define filechk_timestamp.h (if test -n "$${SOURCE_DATE_EPOCH}"; then \ date=""; \ for _date in gdate date.gnu date; do \ $${_date} -u -d "$${SOURCE_DATE}" >/dev/null 2>&1 && \ date="$${_date}"; \ done; \ if test -n "$${date}"; then \ SOURCE_DATE="@$${SOURCE_DATE_EPOCH}"; \ LC_ALL=C $${date} -u -d "$${SOURCE_DATE}" +'#define U_BOOT_DATE "%b %d %C%y"'; \ LC_ALL=C $${date} -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TIME "%T"'; \ LC_ALL=C $${date} -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TZ "%z"'; \ else \ return 42; \ fi; \ else \ LC_ALL=C date +'#define U_BOOT_DATE "%b %d %C%y"'; \ LC_ALL=C date +'#define U_BOOT_TIME "%T"'; \ LC_ALL=C date +'#define U_BOOT_TZ "%z"'; \ fi) endef --->8---
CHECKFLAGS := -D__linux__ -Dlinux -D__STDC__ -Dunix -D__unix__ \ -Wbitwise -Wno-return-void -D__CHECK_ENDIAN__ $(CF) @@ -1281,9 +1285,9 @@ endef define filechk_timestamp.h (if test -n "$${SOURCE_DATE_EPOCH}"; then \ SOURCE_DATE="@$${SOURCE_DATE_EPOCH}"; \
LC_ALL=C date -u -d "$${SOURCE_DATE}" +'#define U_BOOT_DATE "%b %d %C%y"'; \LC_ALL=C date -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TIME "%T"'; \LC_ALL=C date -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TZ "%z"'; \
LC_ALL=C $(DATE) -u -d "$${SOURCE_DATE}" +'#define U_BOOT_DATE "%b %d %C%y"'; \LC_ALL=C $(DATE) -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TIME "%T"'; \LC_ALL=C $(DATE) -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TZ "%z"'; \@@ -1295,6 +1299,11 @@ $(version_h): include/config/uboot.release FORCE $(call filechk,version.h)
$(timestamp_h): $(srctree)/Makefile FORCE +ifneq ($(strip $(SOURCE_DATE_EPOCH)),) +ifeq ($(strip $(DATE)),)
- $(error "Your gdate/date.gnu/date does not support the '-u' and '-d' switches like GNU date does!")
+endif +endif $(call filechk,timestamp.h)
# ---------------------------------------------------------------------------
The SOURCE_DATE_EPOCH mechanism for reproducible builds require some date(1) with -d switch to print the relevant date and time strings of another point of time.
In other words it requires some date(1) that behaves like the GNU date(1) [1]. The BSD date(1) [2] on the other hand has the same switch but with a different meaning.
Respect this and check the date(1) abilities before usage, error on non working version. Use the well known pre- and suffixes for the GNU variant of a tool on *BSD hosts to search for a working date(1) version.
[1] http://man7.org/linux/man-pages/man1/date.1.html [2] http://www.freebsd.org/cgi/man.cgi?query=date
Signed-off-by: Andreas Bießmann andreas.devel@googlemail.com --- This commit tries to figure out if we have a date variant available that supports the '-u' and '-d "@0"' switches. It errors on non-working variants of date.
To respect *BSD host systems search for gdate and date.gnu. Those pre- and suffixes are widespread used for the GNU variant of a tool also avialable on *BSD systems.
The result is:
---8<--- abiessmann@punisher % PATH=$ARMv5_PATH:$PATH CROSS_COMPILE=arm-v5te-linux-gnueabi- make O=/tmp/picosam ARCH=arm include/generated/timestamp_autogenerated.h make[1]: Entering directory '/tmp/picosam' CHK include/generated/timestamp_autogenerated.h UPD include/generated/timestamp_autogenerated.h make[1]: Leaving directory '/tmp/picosam' abiessmann@punisher % PATH=$ARMv5_PATH:$PATH CROSS_COMPILE=arm-v5te-linux-gnueabi- make O=/tmp/picosam SOURCE_DATE_EPOCH="$(date +'%s')" ARCH=arm include/generated/timestamp_autogenerated.h make[1]: Entering directory '/tmp/picosam' CHK include/generated/timestamp_autogenerated.h /home/abiessmann/src/u-boot/Makefile:1311: recipe for target 'include/generated/timestamp_autogenerated.h' failed make[1]: *** [include/generated/timestamp_autogenerated.h] Error 42 make[1]: Leaving directory '/tmp/picosam' Makefile:146: recipe for target 'sub-make' failed make: *** [sub-make] Error 2 --->8---
It applies on top of http://patchwork.ozlabs.org/patch/506856/ (Makefile: Use correct timezone for U_BOOT_TZ).
Changes in v3: * move the check in the timestamp.h generation script * remove RFC * reword commit message
Changes in v2: * check for '-u' and '-d "@0"' switch rather than for the GNU variant
Makefile | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-)
diff --git a/Makefile b/Makefile index e8ea71c..81f12b8 100644 --- a/Makefile +++ b/Makefile @@ -1279,12 +1279,24 @@ define filechk_version.h echo #define LD_VERSION_STRING "$$($(LD) --version | head -n 1)"; ) endef
+# The SOURCE_DATE_EPOCH mechanism requires a date that behaves like GNU date. +# The BSD date on the other hand behaves different and would produce errors +# with the misused '-d' switch. Respect that and search a working date with +# well known pre- and suffixes for the GNU variant of date. define filechk_timestamp.h (if test -n "$${SOURCE_DATE_EPOCH}"; then \ SOURCE_DATE="@$${SOURCE_DATE_EPOCH}"; \ - LC_ALL=C date -u -d "$${SOURCE_DATE}" +'#define U_BOOT_DATE "%b %d %C%y"'; \ - LC_ALL=C date -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TIME "%T"'; \ - LC_ALL=C date -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TZ "%z"'; \ + DATE=""; \ + for date in gdate date.gnu date; do \ + $${date} -u -d "$${SOURCE_DATE}" >/dev/null 2>&1 && DATE="$${date}"; \ + done; \ + if test -n "$${DATE}"; then \ + LC_ALL=C $${DATE} -u -d "$${SOURCE_DATE}" +'#define U_BOOT_DATE "%b %d %C%y"'; \ + LC_ALL=C $${DATE} -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TIME "%T"'; \ + LC_ALL=C $${DATE} -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TZ "%z"'; \ + else \ + return 42; \ + fi; \ else \ LC_ALL=C date +'#define U_BOOT_DATE "%b %d %C%y"'; \ LC_ALL=C date +'#define U_BOOT_TIME "%T"'; \
On Fri, Aug 28, 2015 at 10:29:55AM +0200, Andreas Bießmann wrote:
The SOURCE_DATE_EPOCH mechanism for reproducible builds require some date(1) with -d switch to print the relevant date and time strings of another point of time.
In other words it requires some date(1) that behaves like the GNU date(1) [1]. The BSD date(1) [2] on the other hand has the same switch but with a different meaning.
Respect this and check the date(1) abilities before usage, error on non working version. Use the well known pre- and suffixes for the GNU variant of a tool on *BSD hosts to search for a working date(1) version.
[1] http://man7.org/linux/man-pages/man1/date.1.html [2] http://www.freebsd.org/cgi/man.cgi?query=date
Signed-off-by: Andreas Bießmann andreas.devel@googlemail.com
Applied to u-boot/master, thanks!
Hi,
On Freitag, 28. August 2015, Tom Rini wrote:
Applied to u-boot/master, thanks!
yay, glad to see this thread come to a happy ending, both nicely and quickly! ;-) Thanks to everyone involved!
cheers, Holger
Le vendredi 28 août 2015 à 10:29 +0200, Andreas Bießmann a écrit :
The SOURCE_DATE_EPOCH mechanism for reproducible builds require some date(1) with -d switch to print the relevant date and time strings of another point of time.
In other words it requires some date(1) that behaves like the GNU date(1) [1]. The BSD date(1) [2] on the other hand has the same switch but with a different meaning.
Respect this and check the date(1) abilities before usage, error on non working version. Use the well known pre- and suffixes for the GNU variant of a tool on *BSD hosts to search for a working date(1) version.
Looks good to me! Have you actually tried setting SOURCE_DATE_EPOCH to a fixed value, building U-Boot twice and checking that the produced file is the very same?
[1] http://man7.org/linux/man-pages/man1/date.1.html [2] http://www.freebsd.org/cgi/man.cgi?query=date
Signed-off-by: Andreas Bießmann andreas.devel@googlemail.com
This commit tries to figure out if we have a date variant available that supports the '-u' and '-d "@0"' switches. It errors on non-working variants of date.
To respect *BSD host systems search for gdate and date.gnu. Those pre- and suffixes are widespread used for the GNU variant of a tool also avialable on *BSD systems.
The result is:
---8<--- abiessmann@punisher % PATH=$ARMv5_PATH:$PATH CROSS_COMPILE=arm-v5te-linux-gnueabi- make O=/tmp/picosam ARCH=arm include/generated/timestamp_autogenerated.h make[1]: Entering directory '/tmp/picosam' CHK include/generated/timestamp_autogenerated.h UPD include/generated/timestamp_autogenerated.h make[1]: Leaving directory '/tmp/picosam' abiessmann@punisher % PATH=$ARMv5_PATH:$PATH CROSS_COMPILE=arm-v5te-linux-gnueabi- make O=/tmp/picosam SOURCE_DATE_EPOCH="$(date +'%s')" ARCH=arm include/generated/timestamp_autogenerated.h make[1]: Entering directory '/tmp/picosam' CHK include/generated/timestamp_autogenerated.h /home/abiessmann/src/u-boot/Makefile:1311: recipe for target 'include/generated/timestamp_autogenerated.h' failed make[1]: *** [include/generated/timestamp_autogenerated.h] Error 42 make[1]: Leaving directory '/tmp/picosam' Makefile:146: recipe for target 'sub-make' failed make: *** [sub-make] Error 2 --->8---
It applies on top of http://patchwork.ozlabs.org/patch/506856/ (Makefile: Use correct timezone for U_BOOT_TZ).
Changes in v3:
- move the check in the timestamp.h generation script
- remove RFC
- reword commit message
Changes in v2:
- check for '-u' and '-d "@0"' switch rather than for the GNU variant
Makefile | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-)
diff --git a/Makefile b/Makefile index e8ea71c..81f12b8 100644 --- a/Makefile +++ b/Makefile @@ -1279,12 +1279,24 @@ define filechk_version.h echo #define LD_VERSION_STRING "$$($(LD) --version | head -n 1)"; ) endef
+# The SOURCE_DATE_EPOCH mechanism requires a date that behaves like GNU date. +# The BSD date on the other hand behaves different and would produce errors +# with the misused '-d' switch. Respect that and search a working date with +# well known pre- and suffixes for the GNU variant of date. define filechk_timestamp.h (if test -n "$${SOURCE_DATE_EPOCH}"; then \ SOURCE_DATE="@$${SOURCE_DATE_EPOCH}"; \
LC_ALL=C date -u -d "$${SOURCE_DATE}" +'#define U_BOOT_DATE "%b %d %C%y"'; \LC_ALL=C date -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TIME "%T"'; \LC_ALL=C date -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TZ "%z"'; \
DATE=""; \for date in gdate date.gnu date; do \$${date} -u -d "$${SOURCE_DATE}" >/dev/null 2>&1 && DATE="$${date}"; \done; \if test -n "$${DATE}"; then \LC_ALL=C $${DATE} -u -d "$${SOURCE_DATE}" +'#define U_BOOT_DATE "%b %d %C%y"'; \LC_ALL=C $${DATE} -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TIME "%T"'; \LC_ALL=C $${DATE} -u -d "$${SOURCE_DATE}" +'#define U_BOOT_TZ "%z"'; \else \return 42; \fi; \
On 09/01/2015 07:03 PM, Paul Kocialkowski wrote:
Le vendredi 28 août 2015 à 10:29 +0200, Andreas Bießmann a écrit :
The SOURCE_DATE_EPOCH mechanism for reproducible builds require some date(1) with -d switch to print the relevant date and time strings of another point of time.
In other words it requires some date(1) that behaves like the GNU date(1) [1]. The BSD date(1) [2] on the other hand has the same switch but with a different meaning.
Respect this and check the date(1) abilities before usage, error on non working version. Use the well known pre- and suffixes for the GNU variant of a tool on *BSD hosts to search for a working date(1) version.
Looks good to me! Have you actually tried setting SOURCE_DATE_EPOCH to a fixed value, building U-Boot twice and checking that the produced file is the very same?
It works:
---8<--- andreas@imac % shasum /tmp/u-boot.img.1441178617 u-boot.img /tmp/u-boot.bin.1441178617 u-boot.bin /tmp/boot.bin.1441178617 boot.bin 1a779ba79efa1874a5b307650392737d861005bb /tmp/u-boot.img.1441178617 1a779ba79efa1874a5b307650392737d861005bb u-boot.img dbe56f02e510ee251a2e5f9f8b6ce430884557bb /tmp/u-boot.bin.1441178617 dbe56f02e510ee251a2e5f9f8b6ce430884557bb u-boot.bin 4aa9b3ba9641febae0bfdbb374c54c287b463ac0 /tmp/boot.bin.1441178617 4aa9b3ba9641febae0bfdbb374c54c287b463ac0 boot.bin andreas@imac % uname -a Darwin imac 14.5.0 Darwin Kernel Version 14.5.0: Wed Jul 29 02:26:53 PDT 2015; root:xnu-2782.40.9~1/RELEASE_X86_64 x86_64 andreas@imac % git describe v2015.10-rc2-304-gb7e84c9 --->8---
Both are fresh builds with the same SOURCE_DATE_EPOCH set. However using the time stamp from timestamp_autogenerated.h does _not_ work due to the wrong TZ (+0200 for me, UTC with SOURCE_DATE_EPOCH). But I guess this is a known problem.
Andreas
On 2015-07-26, Paul Kocialkowski wrote:
In order to achieve reproducible builds in U-Boot, timestamps that are defined at build-time have to be somewhat eliminated. The SOURCE_DATE_EPOCH environment variable allows setting a fixed value for those timestamps.
...
However, some other devices might need some more tweaks, especially regarding the image generation tools.
With this patch, there is still variation based on timezone in any of the u-boot.img and u-boot-sunxi-with-spl.bin produced in the Debian packages:
https://reproducible.debian.net/rb-pkg/unstable/armhf/u-boot.html
The good news is that all the u-boot.bin targets are produced reproducibly, so here's to progress!
I think the use of "time = mktime(time_universal);" is where the problem lies:
diff --git a/tools/default_image.c b/tools/default_image.c index cf5c0d4..18940af 100644 --- a/tools/default_image.c +++ b/tools/default_image.c @@ -96,9 +99,25 @@ static void image_set_header(void *ptr, struct stat *sbuf, int ifd, sizeof(image_header_t)), sbuf->st_size - sizeof(image_header_t));
- source_date_epoch = getenv("SOURCE_DATE_EPOCH");
- if (source_date_epoch != NULL) {
time = (time_t) strtol(source_date_epoch, NULL, 10);time_universal = gmtime(&time);if (time_universal == NULL) {fprintf(stderr, "%s: SOURCE_DATE_EPOCH is not valid\n",__func__);time = 0;} else {time = mktime(time_universal);}- } else {
time = sbuf->st_mtime;- }
- /* Build new header */ image_set_magic(hdr, IH_MAGIC);
- image_set_time(hdr, sbuf->st_mtime);
- image_set_time(hdr, time); image_set_size(hdr, sbuf->st_size - sizeof(image_header_t)); image_set_load(hdr, params->addr); image_set_ep(hdr, params->ep);
-- 1.9.1
According to the mktime manpage:
The mktime() function converts a broken-down time structure, expressed as local time, to calendar time representation.
So my interpetation is that it's taking the UTC time and converts it into local time using the configured timezone... not sure what would be a viable alternative to mktime.
Running with the TZ=UTC environment variable exported works around the problem; not sure if it would be appropriate to always run with TZ=UTC when SOURCE_DATE_EPOCH is set...
live well, vagrant
Le jeudi 24 septembre 2015 à 09:05 -0700, Vagrant Cascadian a écrit :
On 2015-07-26, Paul Kocialkowski wrote:
In order to achieve reproducible builds in U-Boot, timestamps that are defined at build-time have to be somewhat eliminated. The SOURCE_DATE_EPOCH environment variable allows setting a fixed value for those timestamps.
...
However, some other devices might need some more tweaks, especially regarding the image generation tools.
With this patch, there is still variation based on timezone in any of the u-boot.img and u-boot-sunxi-with-spl.bin produced in the Debian packages:
https://reproducible.debian.net/rb-pkg/unstable/armhf/u-boot.html
Thanks for reporting this!
The good news is that all the u-boot.bin targets are produced reproducibly, so here's to progress!
Good, that's a nice first step forward.
I think the use of "time = mktime(time_universal);" is where the problem lies:
[…]
According to the mktime manpage:
The mktime() function converts a broken-down time structure, expressed as local time, to calendar time representation.So my interpetation is that it's taking the UTC time and converts it into local time using the configured timezone... not sure what would be a viable alternative to mktime.
That seems to make sense. Come to think of it, it probably was not necessary to call gmtime in the first place: if SOURCE_DATE_EPOCH is always in UTC, we should be able to stick that as-is in the time variable. At best, gmtime + mktime (assuming mktime working in UTC) would give us back the same timestamp.
What do you think? Please let me know if I'm wrong.
Running with the TZ=UTC environment variable exported works around the problem; not sure if it would be appropriate to always run with TZ=UTC when SOURCE_DATE_EPOCH is set...
Well that's too much of a workaround to be a reliable solution for the long term, IMHO.
On 2015-09-28, Paul Kocialkowski wrote:
Le jeudi 24 septembre 2015 à 09:05 -0700, Vagrant Cascadian a écrit :
I think the use of "time = mktime(time_universal);" is where the problem lies:
[…]
According to the mktime manpage:
The mktime() function converts a broken-down time structure, expressed as local time, to calendar time representation.So my interpetation is that it's taking the UTC time and converts it into local time using the configured timezone... not sure what would be a viable alternative to mktime.
That seems to make sense. Come to think of it, it probably was not necessary to call gmtime in the first place: if SOURCE_DATE_EPOCH is always in UTC, we should be able to stick that as-is in the time variable. At best, gmtime + mktime (assuming mktime working in UTC) would give us back the same timestamp.
What do you think? Please let me know if I'm wrong.
This patch on top of 2015.10-rc4 seems to resolve the issue for me:
Index: u-boot/tools/default_image.c =================================================================== --- u-boot.orig/tools/default_image.c +++ u-boot/tools/default_image.c @@ -108,8 +108,6 @@ static void image_set_header(void *ptr, fprintf(stderr, "%s: SOURCE_DATE_EPOCH is not valid\n", __func__); time = 0; - } else { - time = mktime(time_universal); } } else { time = sbuf->st_mtime;
It still checks for the validity of SOURCE_DATE_EPOCH using gmtime, but doesn't call mktime at all, just re-uses the value set from SOURCE_DATE_EPOCH.
live well, vagrant
Le mercredi 30 septembre 2015 à 08:50 -0700, Vagrant Cascadian a écrit :
On 2015-09-28, Paul Kocialkowski wrote:
What do you think? Please let me know if I'm wrong.
This patch on top of 2015.10-rc4 seems to resolve the issue for me:
Index: u-boot/tools/default_image.c
--- u-boot.orig/tools/default_image.c +++ u-boot/tools/default_image.c @@ -108,8 +108,6 @@ static void image_set_header(void *ptr, fprintf(stderr, "%s: SOURCE_DATE_EPOCH is not valid\n", __func__); time = 0;
} else {time = mktime(time_universal);It still checks for the validity of SOURCE_DATE_EPOCH using gmtime, but doesn't call mktime at all, just re-uses the value set from SOURCE_DATE_EPOCH.
That's a good plan! I guess we should also fully get rid of the time_universal variable and make the check inline:
if (gmtime(&time) == NULL)
and of course drop the else statement.
Would you like to craft that patch for upstream U-Boot? If not, I'd be happy to do it.
Thanks for your work!
On Thu, 24 Sep 2015 09:05:10 -0700 Vagrant Cascadian vagrant@debian.org wrote:
On 2015-07-26, Paul Kocialkowski wrote:
In order to achieve reproducible builds in U-Boot, timestamps that are defined at build-time have to be somewhat eliminated. The SOURCE_DATE_EPOCH environment variable allows setting a fixed value for those timestamps.
...
However, some other devices might need some more tweaks, especially regarding the image generation tools.
With this patch, there is still variation based on timezone in any of the u-boot.img and u-boot-sunxi-with-spl.bin produced in the Debian packages:
https://reproducible.debian.net/rb-pkg/unstable/armhf/u-boot.html
The good news is that all the u-boot.bin targets are produced reproducibly, so here's to progress!
Hello,
I see that Debian is building U-Boot 2015.07 according to the information from that link. It may be a good idea to ensure that the following simple patch is also applied: http://git.denx.de/?p=u-boot.git;a=commitdiff;h=bfb05d0187d70274c77d02dc0de5...
It had been submitted at the day when somebody (probably you) was troubleshooting reproducible build issues and asking questions in the U-Boot IRC channel.
participants (11)
-
Andreas Bießmann -
Bin Meng -
Chris Packham -
Fabio Estevam -
Holger Levsen -
Marek Vasut -
Paul Kocialkowski -
Siarhei Siamashka -
Tom Rini -
Vagrant Cascadian -
Ximin Luo