[U-Boot] [PATCH v3 0/2] fit: Image node compression
This patch series adds compression support for non-kernel FIT image nodes (e.g. FDTs). The first patch adds the compression support itself, the second adds a new feature to compatible string matching that allows it to be useful with compressed FDTs.
Sandbox-tested with FIT images with and without compressed FDTs, with and without overlays, in both compatible string matching and direct config selection modes. Also expanded the test_fit pytest to include a case with compressed kernel, FDT and ramdisk.
Julius Werner (2): fit: Support compression for non-kernel components (e.g. FDT) - Changes for v2: - Changed from only supporting compressed FDTs to supporting all non-kernel image node types. fit: Support compat string property in configuration node
common/image-fit.c | 140 +++++++++++++++++++++++++++------------------ 1 file changed, 83 insertions(+), 57 deletions(-)
This patch adds support for compressing non-kernel image nodes in a FIT image (kernel nodes could already be compressed previously). This can reduce the size of FIT images and therefore improve boot times (especially when an image bundles many different kernel FDTs). The images will automatically be decompressed on load.
This patch does not support extracting compatible strings from compressed FDTs, so it's not very helpful in conjunction with CONFIG_FIT_BEST_MATCH yet, but it can already be used in environments that select the configuration to load explicitly.
Signed-off-by: Julius Werner jwerner@chromium.org --- common/image-fit.c | 86 +++++++++++++++++++++++---------------- test/py/tests/test_fit.py | 29 +++++++++++-- 2 files changed, 77 insertions(+), 38 deletions(-)
diff --git a/common/image-fit.c b/common/image-fit.c index ac901e131c..469c5c8f49 100644 --- a/common/image-fit.c +++ b/common/image-fit.c @@ -22,6 +22,7 @@ DECLARE_GLOBAL_DATA_PTR; #endif /* !USE_HOSTCC*/
+#include <bootm.h> #include <image.h> #include <bootstage.h> #include <u-boot/crc.h> @@ -1576,6 +1577,13 @@ int fit_conf_find_compat(const void *fit, const void *fdt) kfdt_name); continue; } + + if (!fit_image_check_comp(fit, kfdt_noffset, IH_COMP_NONE)) { + debug("Can't extract compat from "%s" (compressed)\n", + kfdt_name); + continue; + } + /* * Get a pointer to this configuration's fdt. */ @@ -1795,11 +1803,12 @@ int fit_image_load(bootm_headers_t *images, ulong addr, const char *fit_uname_config; const char *fit_base_uname_config; const void *fit; - const void *buf; + void *buf; + void *loadbuf; size_t size; int type_ok, os_ok; - ulong load, data, len; - uint8_t os; + ulong load, load_end, data, len; + uint8_t os, comp; #ifndef USE_HOSTCC uint8_t os_arch; #endif @@ -1895,12 +1904,6 @@ int fit_image_load(bootm_headers_t *images, ulong addr, images->os.arch = os_arch; #endif
- if (image_type == IH_TYPE_FLATDT && - !fit_image_check_comp(fit, noffset, IH_COMP_NONE)) { - puts("FDT image is compressed"); - return -EPROTONOSUPPORT; - } - bootstage_mark(bootstage_id + BOOTSTAGE_SUB_CHECK_ALL); type_ok = fit_image_check_type(fit, noffset, image_type) || fit_image_check_type(fit, noffset, IH_TYPE_FIRMWARE) || @@ -1931,7 +1934,8 @@ int fit_image_load(bootm_headers_t *images, ulong addr, bootstage_mark(bootstage_id + BOOTSTAGE_SUB_CHECK_ALL_OK);
/* get image data address and length */ - if (fit_image_get_data_and_size(fit, noffset, &buf, &size)) { + if (fit_image_get_data_and_size(fit, noffset, + (const void **)&buf, &size)) { printf("Could not find %s subimage data!\n", prop_name); bootstage_error(bootstage_id + BOOTSTAGE_SUB_GET_DATA); return -ENOENT; @@ -1939,30 +1943,15 @@ int fit_image_load(bootm_headers_t *images, ulong addr,
#if !defined(USE_HOSTCC) && defined(CONFIG_FIT_IMAGE_POST_PROCESS) /* perform any post-processing on the image data */ - board_fit_image_post_process((void **)&buf, &size); + board_fit_image_post_process(&buf, &size); #endif
len = (ulong)size;
- /* verify that image data is a proper FDT blob */ - if (image_type == IH_TYPE_FLATDT && fdt_check_header(buf)) { - puts("Subimage data is not a FDT"); - return -ENOEXEC; - } - bootstage_mark(bootstage_id + BOOTSTAGE_SUB_GET_DATA_OK);
- /* - * Work-around for eldk-4.2 which gives this warning if we try to - * cast in the unmap_sysmem() call: - * warning: initialization discards qualifiers from pointer target type - */ - { - void *vbuf = (void *)buf; - - data = map_to_sysmem(vbuf); - } - + data = map_to_sysmem(buf); + load = data; if (load_op == FIT_LOAD_IGNORED) { /* Don't load */ } else if (fit_image_get_load(fit, noffset, &load)) { @@ -1974,8 +1963,6 @@ int fit_image_load(bootm_headers_t *images, ulong addr, } } else if (load_op != FIT_LOAD_OPTIONAL_NON_ZERO || load) { ulong image_start, image_end; - ulong load_end; - void *dst;
/* * move image data to the load address, @@ -1993,14 +1980,45 @@ int fit_image_load(bootm_headers_t *images, ulong addr,
printf(" Loading %s from 0x%08lx to 0x%08lx\n", prop_name, data, load); + } else { + load = data; /* No load address specified */ + } + + comp = IH_COMP_NONE; + loadbuf = buf; + /* Kernel images get decompressed later in bootm_load_os(). */ + if (!(image_type == IH_TYPE_KERNEL || + image_type == IH_TYPE_KERNEL_NOLOAD) && + !fit_image_get_comp(fit, noffset, &comp) && + comp != IH_COMP_NONE) { + ulong max_decomp_len = len * 20; + if (load == data) { + loadbuf = malloc(max_decomp_len); + load = map_to_sysmem(loadbuf); + } else { + loadbuf = map_sysmem(load, max_decomp_len); + } + if (bootm_decomp_image(comp, load, data, image_type, + loadbuf, buf, len, max_decomp_len, &load_end)) { + printf("Error decompressing %s\n", prop_name);
- dst = map_sysmem(load, len); - memmove(dst, buf, len); - data = load; + return -ENOEXEC; + } + len = load_end - load; + } else if (load != data) { + loadbuf = map_sysmem(load, len); + memcpy(loadbuf, buf, len); } + + /* verify that image data is a proper FDT blob */ + if (image_type == IH_TYPE_FLATDT && fdt_check_header(loadbuf)) { + puts("Subimage data is not a FDT"); + return -ENOEXEC; + } + bootstage_mark(bootstage_id + BOOTSTAGE_SUB_LOAD);
- *datap = data; + *datap = load; *lenp = len; if (fit_unamep) *fit_unamep = (char *)fit_uname; diff --git a/test/py/tests/test_fit.py b/test/py/tests/test_fit.py index 49d6fea571..8009d2907b 100755 --- a/test/py/tests/test_fit.py +++ b/test/py/tests/test_fit.py @@ -24,7 +24,7 @@ base_its = ''' type = "kernel"; arch = "sandbox"; os = "linux"; - compression = "none"; + compression = "%(compression)s"; load = <0x40000>; entry = <0x8>; }; @@ -39,11 +39,11 @@ base_its = ''' }; fdt@1 { description = "snow"; - data = /incbin/("u-boot.dtb"); + data = /incbin/("%(fdt)s"); type = "flat_dt"; arch = "sandbox"; %(fdt_load)s - compression = "none"; + compression = "%(compression)s"; signature@1 { algo = "sha1,rsa2048"; key-name-hint = "dev"; @@ -56,7 +56,7 @@ base_its = ''' arch = "sandbox"; os = "linux"; %(ramdisk_load)s - compression = "none"; + compression = "%(compression)s"; }; ramdisk@2 { description = "snow"; @@ -221,6 +221,10 @@ def test_fit(u_boot_console): print(data, file=fd) return fname
+ def make_compressed(filename): + util.run_and_log(cons, ['gzip', '-f', '-k', filename]) + return filename + '.gz' + def find_matching(text, match): """Find a match in a line of text, and return the unmatched line portion
@@ -312,6 +316,7 @@ def test_fit(u_boot_console): loadables1 = make_kernel('test-loadables1.bin', 'lenrek') loadables2 = make_ramdisk('test-loadables2.bin', 'ksidmar') kernel_out = make_fname('kernel-out.bin') + fdt = make_fname('u-boot.dtb') fdt_out = make_fname('fdt-out.dtb') ramdisk_out = make_fname('ramdisk-out.bin') loadables1_out = make_fname('loadables1-out.bin') @@ -326,6 +331,7 @@ def test_fit(u_boot_console): 'kernel_addr' : 0x40000, 'kernel_size' : filesize(kernel),
+ 'fdt' : fdt, 'fdt_out' : fdt_out, 'fdt_addr' : 0x80000, 'fdt_size' : filesize(control_dtb), @@ -351,6 +357,7 @@ def test_fit(u_boot_console): 'loadables2_load' : '',
'loadables_config' : '', + 'compression' : 'none', }
# Make a basic FIT and a script to load it @@ -417,6 +424,20 @@ def test_fit(u_boot_console): check_equal(loadables2, loadables2_out, 'Loadables2 (ramdisk) not loaded')
+ # Kernel, FDT and Ramdisk all compressed + with cons.log.section('(Kernel + FDT + Ramdisk) compressed'): + params['compression'] = 'gzip' + params['kernel'] = make_compressed(kernel) + params['fdt'] = make_compressed(fdt) + params['ramdisk'] = make_compressed(ramdisk) + fit = make_fit(mkimage, params) + cons.restart_uboot() + output = cons.run_command_list(cmd.splitlines()) + check_equal(kernel, kernel_out, 'Kernel not loaded') + check_equal(control_dtb, fdt_out, 'FDT not loaded') + check_equal(ramdisk, ramdisk_out, 'Ramdisk not loaded') + + cons = u_boot_console try: # We need to use our own device tree file. Remember to restore it
Hi Julius,
This patch adds support for compressing non-kernel image nodes in a FIT image (kernel nodes could already be compressed previously). This can reduce the size of FIT images and therefore improve boot times (especially when an image bundles many different kernel FDTs). The images will automatically be decompressed on load.
This patch does not support extracting compatible strings from compressed FDTs, so it's not very helpful in conjunction with CONFIG_FIT_BEST_MATCH yet, but it can already be used in environments that select the configuration to load explicitly.
Signed-off-by: Julius Werner jwerner@chromium.org
common/image-fit.c | 86 +++++++++++++++++++++++---------------- test/py/tests/test_fit.py | 29 +++++++++++-- 2 files changed, 77 insertions(+), 38 deletions(-)
Is there a change log for this patch?I think we discussed having a test.
Regards, Simon
Is there a change log for this patch?I think we discussed having a test.
Sorry, forgot that. Is it okay if I just put it here or do you need me to resend a v4?
--- Changes for v2: - made decompression work for all non-kernel images, not just FDTs Changes for v3: - fixed up debug output that was still written for v1 - fixed a mistake with handling FIT_LOAD_OPTIONAL_NON_ZERO when 'load' was 0 (i.e. unset) - added compression test case to test_fit
Hi Julius,
On Mon, 13 May 2019 at 19:13, Julius Werner jwerner@chromium.org wrote:
Is there a change log for this patch?I think we discussed having a test.
Sorry, forgot that. Is it okay if I just put it here or do you need me to resend a v4?
I think it should be a v4 so it looks right in patchwork.
Also for this patch, can you please add documentation - see source_file_format.txt
Changes for v2:
- made decompression work for all non-kernel images, not just FDTs
Changes for v3:
- fixed up debug output that was still written for v1
- fixed a mistake with handling FIT_LOAD_OPTIONAL_NON_ZERO when
'load' was 0 (i.e. unset)
- added compression test case to test_fit
- Simon
On Tue, May 14, 2019 at 12:55 PM Simon Glass sjg@chromium.org wrote:
Hi Julius,
On Mon, 13 May 2019 at 19:13, Julius Werner jwerner@chromium.org wrote:
Is there a change log for this patch?I think we discussed having a test.
Sorry, forgot that. Is it okay if I just put it here or do you need me to resend a v4?
I think it should be a v4 so it looks right in patchwork.
Also for this patch, can you please add documentation - see source_file_format.txt
I had submitted a patch for that file already last year but then got stuck in implementing just this, see commit fd15a9e2565f here: https://github.com/u-boot/u-boot/commit/fd15a9e2565f831bf95c2152d1966d068a64...
I don't know if this compression support needs anything more specific in that doc file. If anything, we could explicitly state that compression is now supported for all nodes, but then again, the file didn't tell that it was only supported for Kernels...
Regards, Simon
Changes for v2:
- made decompression work for all non-kernel images, not just FDTs
Changes for v3:
- fixed up debug output that was still written for v1
- fixed a mistake with handling FIT_LOAD_OPTIONAL_NON_ZERO when
'load' was 0 (i.e. unset)
- added compression test case to test_fit
- Simon
Hi Simon,
On Tue, 14 May 2019 at 05:16, Simon Goldschmidt simon.k.r.goldschmidt@gmail.com wrote:
On Tue, May 14, 2019 at 12:55 PM Simon Glass sjg@chromium.org wrote:
Hi Julius,
On Mon, 13 May 2019 at 19:13, Julius Werner jwerner@chromium.org wrote:
Is there a change log for this patch?I think we discussed having a test.
Sorry, forgot that. Is it okay if I just put it here or do you need me to resend a v4?
I think it should be a v4 so it looks right in patchwork.
Also for this patch, can you please add documentation - see source_file_format.txt
I had submitted a patch for that file already last year but then got stuck in implementing just this, see commit fd15a9e2565f here: https://github.com/u-boot/u-boot/commit/fd15a9e2565f831bf95c2152d1966d068a64...
I don't know if this compression support needs anything more specific in that doc file. If anything, we could explicitly state that compression is now supported for all nodes, but then again, the file didn't tell that it was only supported for Kernels...
Now that gmail is fixed I can reply more easily...
Fair enough.
Compression for a DT node is a pain, since it must be decompressed before being accessed and even reading the compatible string is hard. This is the purpose of one of Julius' patches.
The DT format is fairly well documented and I would like to keep it that way, as well as extend the tests as we add new functions.
Regards, Simon
Like Simon (Goldschmidt) said, the current documentation for the "compression" property in image nodes already matches this, so I don't think I need to add anything there. I'll update the other patch to add documentation for compatible strings in config nodes, and then upload an .its example for compressed FDTs there (because the example makes more sense when using both features together).
On Tue, May 14, 2019 at 8:11 PM Simon Glass sjg@chromium.org wrote:
Hi Simon,
On Tue, 14 May 2019 at 05:16, Simon Goldschmidt simon.k.r.goldschmidt@gmail.com wrote:
On Tue, May 14, 2019 at 12:55 PM Simon Glass sjg@chromium.org wrote:
Hi Julius,
On Mon, 13 May 2019 at 19:13, Julius Werner jwerner@chromium.org wrote:
Is there a change log for this patch?I think we discussed having a test.
Sorry, forgot that. Is it okay if I just put it here or do you need me to resend a v4?
I think it should be a v4 so it looks right in patchwork.
Also for this patch, can you please add documentation - see source_file_format.txt
I had submitted a patch for that file already last year but then got stuck in implementing just this, see commit fd15a9e2565f here: https://github.com/u-boot/u-boot/commit/fd15a9e2565f831bf95c2152d1966d068a64...
I don't know if this compression support needs anything more specific in that doc file. If anything, we could explicitly state that compression is now supported for all nodes, but then again, the file didn't tell that it was only supported for Kernels...
Now that gmail is fixed I can reply more easily...
Fair enough.
Compression for a DT node is a pain, since it must be decompressed before being accessed and even reading the compatible string is hard. This is the purpose of one of Julius' patches.
The DT format is fairly well documented and I would like to keep it that way, as well as extend the tests as we add new functions.
Regards, Simon
This patch adds support for an optional optimization to compatible string matching where the compatible string property from the root node of the kernel FDT can be copied into the configuration node of the FIT image. This is most useful when using compressed FDTs or when using FDT overlays, where the traditional extraction of the compatible string from the kernel FDT itself is not easily possible.
Signed-off-by: Julius Werner jwerner@chromium.org --- common/image-fit.c | 67 +++++++++++++++++++++++++++------------------- 1 file changed, 39 insertions(+), 28 deletions(-)
diff --git a/common/image-fit.c b/common/image-fit.c index 469c5c8f49..d32add6419 100644 --- a/common/image-fit.c +++ b/common/image-fit.c @@ -1522,6 +1522,10 @@ int fit_check_format(const void *fit) * compatible list, "foo,bar", matches a compatible string in the root of fdt1. * "bim,bam" in fdt2 matches the second string which isn't as good as fdt1. * + * As an optimization, the compatible property from the FDT's root node can be + * copied into the configuration node in the FIT image. This is required to + * match configurations with compressed FDTs. + * * returns: * offset to the configuration to use if one was found * -1 otherwise @@ -1554,55 +1558,62 @@ int fit_conf_find_compat(const void *fit, const void *fdt) for (noffset = fdt_next_node(fit, confs_noffset, &ndepth); (noffset >= 0) && (ndepth > 0); noffset = fdt_next_node(fit, noffset, &ndepth)) { - const void *kfdt; + const void *fdt; const char *kfdt_name; - int kfdt_noffset; + int kfdt_noffset, compat_noffset; const char *cur_fdt_compat; int len; - size_t size; + size_t sz; int i;
if (ndepth > 1) continue;
- kfdt_name = fdt_getprop(fit, noffset, "fdt", &len); - if (!kfdt_name) { - debug("No fdt property found.\n"); - continue; - } - kfdt_noffset = fdt_subnode_offset(fit, images_noffset, - kfdt_name); - if (kfdt_noffset < 0) { - debug("No image node named "%s" found.\n", - kfdt_name); - continue; - } + /* If there's a compat property in the config node, use that. */ + if (fdt_getprop(fit, noffset, "compatible", NULL)) { + fdt = fit; /* search in FIT image */ + compat_noffset = noffset; /* search under config node */ + } else { /* Otherwise extract it from the kernel FDT. */ + kfdt_name = fdt_getprop(fit, noffset, "fdt", &len); + if (!kfdt_name) { + debug("No fdt property found.\n"); + continue; + } + kfdt_noffset = fdt_subnode_offset(fit, images_noffset, + kfdt_name); + if (kfdt_noffset < 0) { + debug("No image node named "%s" found.\n", + kfdt_name); + continue; + }
- if (!fit_image_check_comp(fit, kfdt_noffset, IH_COMP_NONE)) { - debug("Can't extract compat from "%s" (compressed)\n", - kfdt_name); - continue; - } + if (!fit_image_check_comp(fit, kfdt_noffset, + IH_COMP_NONE)) { + debug("Can't extract compat from "%s" " + "(compressed)\n", kfdt_name); + continue; + }
- /* - * Get a pointer to this configuration's fdt. - */ - if (fit_image_get_data(fit, kfdt_noffset, &kfdt, &size)) { - debug("Failed to get fdt "%s".\n", kfdt_name); - continue; + /* search in this config's kernel FDT */ + if (fit_image_get_data(fit, kfdt_noffset, &fdt, &sz)) { + debug("Failed to get fdt "%s".\n", kfdt_name); + continue; + } + + compat_noffset = 0; /* search kFDT under root node */ }
len = fdt_compat_len; cur_fdt_compat = fdt_compat; /* * Look for a match for each U-Boot compatibility string in - * turn in this configuration's fdt. + * turn in the compat string property. */ for (i = 0; len > 0 && (!best_match_offset || best_match_pos > i); i++) { int cur_len = strlen(cur_fdt_compat) + 1;
- if (!fdt_node_check_compatible(kfdt, 0, + if (!fdt_node_check_compatible(fdt, compat_noffset, cur_fdt_compat)) { best_match_offset = noffset; best_match_pos = i;
Hi,
This patch adds support for an optional optimization to compatible string matching where the compatible string property from the root node of the kernel FDT can be copied into the configuration node of the FIT image. This is most useful when using compressed FDTs or when using FDT overlays, where the traditional extraction of the compatible string from the kernel FDT itself is not easily possible.
Signed-off-by: Julius Werner jwerner@chromium.org
common/image-fit.c | 67 +++++++++++++++++++++++++++------------------- 1 file changed, 39 insertions(+), 28 deletions(-)
Can you please add a change log, as this is a v3 patch.
Regards, Simon
Can you please add a change log, as this is a v3 patch.
I didn't change anything in this one. (Should I have not called it v3 then? Sorry, I'm not familiar with the submission process here. I thought letting the individual patches in the series have different version numbers for the same set would be more confusing.)
Hi Julius,
On Mon, 13 May 2019 at 19:15, Julius Werner jwerner@chromium.org wrote:
Can you please add a change log, as this is a v3 patch.
I didn't change anything in this one. (Should I have not called it v3 then? Sorry, I'm not familiar with the submission process here. I thought letting the individual patches in the series have different version numbers for the same set would be more confusing.)
If there are no changes for a version, the patch should say so.
Do you use patman for submission? It does all of this for you reliably each time.
For this one, there should be a documentation update too. Also, for the series I think you should add a new example to the docs, since you are adding two new features.
Regards, Simon
participants (3)
-
Julius Werner -
Simon Glass -
Simon Goldschmidt