[PATCH 1/2] schemas: Add firmware node schema
Add a motivation and purpose for this new proposed node.
Signed-off-by: Simon Glass sjg@chromium.org ---
dtschema/schemas/firmware.yaml | 83 ++++++++++++++++++++++++++++++++++ 1 file changed, 83 insertions(+) create mode 100644 dtschema/schemas/firmware.yaml
diff --git a/dtschema/schemas/firmware.yaml b/dtschema/schemas/firmware.yaml new file mode 100644 index 0000000..4439a70 --- /dev/null +++ b/dtschema/schemas/firmware.yaml @@ -0,0 +1,83 @@ +# SPDX-License-Identifier: GPL-2.0-only OR BSD-2-clause +# Copyright 2023 Google LLC +# + +%YAML 1.2 +--- +$id: http://devicetree.org/schemas/firmware.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml# + +title: /firmware Node + +maintainers: + - Simon Glass sjg@chromium.org + +description: | + The '/firmware' node does not represent a real device, but serves as a place + for recording information about the main firmware used on the device, such as + a map of its contents. This is used by the Operating System (OS), user space + programs and possibly other firmware components. Data in the '/firmware' node + does not itself represent the hardware. + + Properties in this node should be common to (and used by) at least two + firmware projects, such as U-Boot and TF-A. Project-specific subnodes can be + used for properties which are specific to a single project. + + Purpose of '/firmware' node + --------------------------- + + Firmware has traditionally been fairly opaque to the OS, with the OS taking + no interest in its contents, version, layout or how it might be updated. This + is less than ideal, since firmware is an important part of the system and + visibility into its operation is every bit as important as visbility into the + OS and user-space programs within the system. + + The traditional approach has been to let firmware deal with firmware, and the + OS deal with everything else. Updating firmware has been handled by firmware. + For example, the UEFI spec defines a way for the OS to post a 'capsule' which + is discovered next time the system boots, permitting firmware updates. But + firmware updates in firmware are highly problematic. They require a reboot + and a sometimes-lengthy wait with a strange-looking interface unfamiliar + to most users. It seems better to make the update as transparent as possible + to the user. As an example of that, ChromeOS has full knowledge of the + firmware version and layout, updates it in the background from user space and + instantly selects the new firmware when the user reboots or logs out. + + A common objection to considering the system holistically is that some parts + of the system are inaccessible to the OS, such as a secure enclave. However + this does not preclude providing visibility into what is present in that + enclave. Firmware-version information is still useful. Firmware updates are + still needed and can still be initiated from user space. + + Another objection is that firmware should provide an interface to the OS, + while keeping its structure private. This thinking is largely driven by + extrapolating from how firmware has been handled in the 'BIOS' days. + It should be considered a degenerate case rather than the norm. As complexity + increases, it creates an artificial boundary between two pieces of the whole. + Mechanisms then need to be invented to cross this unnecessary chasm. An + example of this is Intel's Dynamic Platform and Thermal Framework (DPTF), + which consists of user-space, OS and firmware components all working towards + a shared goal. We need a standard description of these cross-system pieces. + + In order to 'teach the OS about firmware', we need a place to put this + information. That is the purpose of this node. + + In an Open Source world the entire model of firmware needs to adjust to be + more open, more visible and managed just like any other part of the system. + The major goal is to standardise how firmware is presented to the OS and user + space, so that common utilities can be used to manage the entire system, + including the firmware. For example, fwupd can look in this node for + information on how to update the firmware, similar to how VBE works. [1] + It is likely that other purposes will come to light over time. + + [1] https://github.com/fwupd/fwupd/tree/main/plugins/vbe + +properties: + $nodename: + const: firmware + + "#address-cells": true + "#size-cells": true + +additionalProperties: + type: object
I am unsure whether to add this with a generic name, such as 'layout', but for now am using /firmware/binman to avoid conflicts with any other firmware-layout schema that others might be working on.
Signed-off-by: Simon Glass sjg@chromium.org ---
dtschema/schemas/firmware/binman.yaml | 51 ++++++++++++++++++ dtschema/schemas/firmware/binman/entry.yaml | 57 +++++++++++++++++++++ 2 files changed, 108 insertions(+) create mode 100644 dtschema/schemas/firmware/binman.yaml create mode 100644 dtschema/schemas/firmware/binman/entry.yaml
diff --git a/dtschema/schemas/firmware/binman.yaml b/dtschema/schemas/firmware/binman.yaml new file mode 100644 index 0000000..4b1ecf6 --- /dev/null +++ b/dtschema/schemas/firmware/binman.yaml @@ -0,0 +1,51 @@ +# SPDX-License-Identifier: (GPL-2.0 OR BSD-2-Clause) +# Copyright 2023 Google LLC + +%YAML 1.2 +--- +$id: http://devicetree.org/schemas/firmware/binman.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml# + +title: Binman firmware layout + +maintainers: + - Simon Glass sjg@chromium.org + +description: | + The binman node provides a layout for firmware, used when packaging firmware + from multiple projects. For now it just supports a very simple set of + features, as a starting point for discussion. + + Documentation for Binman is available at: + + https://u-boot.readthedocs.io/en/latest/develop/package/binman.html + + with the current image-description format at: + + https://u-boot.readthedocs.io/en/latest/develop/package/binman.html#image-de... + +properties: + $nodename: + const: binman + +required: + - compatible + +additionalProperties: false + +examples: + - | + - | + firmware { + binman { + compatible = "u-boot,binman"; + + u-boot { + size = <0xa0000>; + }; + + atf-bl31 { + offset = <0x100000>; + }; + }; + }; diff --git a/dtschema/schemas/firmware/binman/entry.yaml b/dtschema/schemas/firmware/binman/entry.yaml new file mode 100644 index 0000000..16b84c5 --- /dev/null +++ b/dtschema/schemas/firmware/binman/entry.yaml @@ -0,0 +1,57 @@ +# SPDX-License-Identifier: (GPL-2.0 OR BSD-2-Clause) +# Copyright 2023 Google LLC + +%YAML 1.2 +--- +$id: http://devicetree.org/schemas/firmware/binman/entry.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml# + +title: Binman entry + +maintainers: + - Simon Glass sjg@chromium.org + +description: | + The entry node specifies a single entry in the firmware. + + Entries have a specific type, such as "u-boot" or "atf-bl31". If the type + is missing, the name is used as the type. + + Note: This definition is intended to be hierarchical, so that entries can + appear in other entries. Schema for that is TBD. + +properties: + $nodename: + pattern: "^[-a-z]+(-[0-9]+)?$" + + type: + $ref: /schemas/types.yaml#/definitions/string + + offset: + $ref: /schemas/types.yaml#/definitions/uint32 + description: | + Provides the offset of this entry from the start of its parent section. + + size: + $ref: /schemas/types.yaml#/definitions/uint32 + description: | + Provides the size of this entry in bytes. + +additionalProperties: false + +examples: + - | + firmware { + binman { + compatible = "u-boot,binman"; + + u-boot { + size = <0xa0000>; + }; + + second-area { + type = "atf-bl31"; + offset = <0x100000>; + }; + }; + };
Hi Simon,
I know I haven't been able to look at binman stuff for a long time, but I've been occasionally thinking about it through the course of a year and think binman severely needs a design-wise review before things get entrenched, even in the most fundamental parts. I do see the cross-project collaboration intention here, but still...
There's also the issue of binman having multiple different device-trees: its input, the ones in fdtmaps per image, the ones injected to U-Boot dtb files per image. I'd say each has different needs, and those differences have to be figured out before specified upstream. I can only guess this is about the one that'll be in a u-boot.dtb.
I want to go through binman more thoroughly, but a lot of changes happened since I last looked at it and I'm a bit slow at writing things, so won't exactly be soon. That being said, here's some ideas off the top of my head, for inspiration on both this schema and binman itself.
On 2023-07-12 00:18 +03:00, Simon Glass wrote:
I am unsure whether to add this with a generic name, such as 'layout', but for now am using /firmware/binman to avoid conflicts with any other firmware-layout schema that others might be working on.
Signed-off-by: Simon Glass sjg@chromium.org
I've been thinking of compatible = "data,<type>" for entries, so proposing 'data' here. A big ask, but it might be the one schema to unify them all if we look at things as "description of data" instead of "firmware layout".
Also consider data layouts in-memory. For example it could be an alternative to /chosen linux,initrd-start to specify initramfs location. Or things like keys or logs received from previous stages / other parts? Weak examples, but maybe might connect better into firmware handoff things. (Sorry, I don't know much about those.)
I'm also thinking about things like JPEG/BMP files for ACPI BGRT-like boot splash, unique firmware/configuration/calibration data for drivers, but they don't exactly need to be in-memory I guess.
dtschema/schemas/firmware/binman.yaml | 51 ++++++++++++++++++ dtschema/schemas/firmware/binman/entry.yaml | 57 +++++++++++++++++++++ 2 files changed, 108 insertions(+) create mode 100644 dtschema/schemas/firmware/binman.yaml create mode 100644 dtschema/schemas/firmware/binman/entry.yaml
diff --git a/dtschema/schemas/firmware/binman.yaml b/dtschema/schemas/firmware/binman.yaml new file mode 100644 index 0000000..4b1ecf6 --- /dev/null +++ b/dtschema/schemas/firmware/binman.yaml @@ -0,0 +1,51 @@ +# SPDX-License-Identifier: (GPL-2.0 OR BSD-2-Clause) +# Copyright 2023 Google LLC
+%YAML 1.2 +--- +$id: http://devicetree.org/schemas/firmware/binman.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml#
+title: Binman firmware layout
+maintainers:
- Simon Glass sjg@chromium.org
+description: |
- The binman node provides a layout for firmware, used when packaging firmware
- from multiple projects. For now it just supports a very simple set of
- features, as a starting point for discussion.
- Documentation for Binman is available at:
- https://u-boot.readthedocs.io/en/latest/develop/package/binman.html
- with the current image-description format at:
- https://u-boot.readthedocs.io/en/latest/develop/package/binman.html#image-de...
+properties:
- $nodename:
- const: binman
I think having a single /firmware/binman node for a single image *and* having a "multiple-images" mode under it is quite bad because the single-image mode is multiple-images with one image. Multiple images should be the only mode of operation
Furthermore, each image should be able to function independently of others, so I think to help enforce that maybe there shouldn't be any formal binman node. Merely instances of images that may happen to be under one parent node that may happen to be called /firmware/binman. The images can be compatible = "data,image" or something and we could search by that.
Or, heh, a single image node that happens to be named /firmware/binman because we are embedding it into a u-boot.dtb in that image. But perhaps we shouldn't restrain things like that. A bit contrived, but why should we be unable to start from an image in SPI and make it load something from microSD?
Going even further, let me suggest putting the image nodes as children of the devices whose layout the image describes. Then I imagine we can have something like a "data driver" (both for U-Boot and Linux) that reads the entry data from the parent data/block device and makes it available for other drivers or in sysfs.
+required:
- compatible
+additionalProperties: false
Including the binman version or a version number for the spec might be useful later on.
+examples:
- |
- |
- firmware {
binman {compatible = "u-boot,binman";
Consider including "image" or "namespace" in the compatible string to disambiguate. Not sure of the conventions, "binman,*" could work just as well, though I'm partial to "data,*" as I've suggested.
u-boot {size = <0xa0000>;};atf-bl31 {offset = <0x100000>;};};- };
diff --git a/dtschema/schemas/firmware/binman/entry.yaml b/dtschema/schemas/firmware/binman/entry.yaml new file mode 100644 index 0000000..16b84c5 --- /dev/null +++ b/dtschema/schemas/firmware/binman/entry.yaml @@ -0,0 +1,57 @@ +# SPDX-License-Identifier: (GPL-2.0 OR BSD-2-Clause) +# Copyright 2023 Google LLC
+%YAML 1.2 +--- +$id: http://devicetree.org/schemas/firmware/binman/entry.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml#
+title: Binman entry
+maintainers:
- Simon Glass sjg@chromium.org
+description: |
- The entry node specifies a single entry in the firmware.
- Entries have a specific type, such as "u-boot" or "atf-bl31". If the type
- is missing, the name is used as the type.
I really dislike the name being a fallback for the type, it's guiding people to using types as the names (or even worse, to assuming names can only be same as the type), leading to ugly combinations of "blob-ext", "fdt-N", "fit", "section" and so on. Node names should be descriptive even when you don't have the device-tree source in front of you, think of "binman extract" and others. Ones like "u-boot" and "atf-bl31" are fine only because they are very specific entry types.
- Note: This definition is intended to be hierarchical, so that entries can
- appear in other entries. Schema for that is TBD.
It might be better to redesign things bottom-up -- first raw data, then a few data types, then sections and images that hold these. Oh, and I have a feeling section/image can be the same thing if we think hard enough (e.g. explicit node labels/references for fdtmap).
+properties:
- $nodename:
- pattern: "^[-a-z]+(-[0-9]+)?$"
- type:
- $ref: /schemas/types.yaml#/definitions/string
Consider compatible = "data,bytes" instead, making it required. Or maybe "data,blob" but that's not what's there on the binman side. (e.g. rename "blob" to "file", "blob-ext" could be "file" combined with a globally-available "optional" property.)
- offset:
- $ref: /schemas/types.yaml#/definitions/uint32
- description: |
Provides the offset of this entry from the start of its parent section.- size:
- $ref: /schemas/types.yaml#/definitions/uint32
- description: |
Provides the size of this entry in bytes.
Can't we use "reg" for these, and even entry@<addr> in the definitions? I know the latter conflicts with the fdt@SEQ stuff, but that should be a binman-input-only thing, and I dislike that anyway. Maybe could work if we put image nodes under block devices...
Or maybe even better, I think we should make it like FIT: allow a "data" property that has it inside the dtb, or a pair of "data-offset/position" and "data-size" properties if it's outside. Inlining data inside the dtb could help us do nice things in binman, like hashes/signatures as entry types instead of special-casing them.
In fact it could be possible to turn binman images into a FIT 2.0 if we do some more work on top of that, instead of nesting a FIT inside a binman image.
+additionalProperties: false
+examples:
- |
- firmware {
binman {compatible = "u-boot,binman";u-boot {size = <0xa0000>;};second-area {type = "atf-bl31";offset = <0x100000>;};};- };
Hi Alper,
On Thu, 20 Jul 2023 at 09:23, Alper Nebi Yasak alpernebiyasak@gmail.com wrote:
Hi Simon,
I know I haven't been able to look at binman stuff for a long time, but I've been occasionally thinking about it through the course of a year and think binman severely needs a design-wise review before things get entrenched, even in the most fundamental parts. I do see the cross-project collaboration intention here, but still...
Yes, I agree. I was expecting a few more comments but I suppose not that many people worry too much about open firmware. Many of the points you've mentioned here have been in my mind as we've moved from a prototype 6 years ago to what Binman is today, and I know we've discussed some before on the mailing list. I really appreciated you collecting your thoughts on this. The whole thing needs a new look and I hope that the discussion here can help us settle on an industry standard for this stuff.
So...
There's also the issue of binman having multiple different device-trees: its input, the ones in fdtmaps per image, the ones injected to U-Boot dtb files per image. I'd say each has different needs, and those differences have to be figured out before specified upstream. I can only guess this is about the one that'll be in a u-boot.dtb.
Well, there is really only one. The fdtmap is actually the same schema, except that it mentions only the image that it is embedded in, i.e. if the fdtmap is for the SPI image, then the fdtmap in SPI flash only contains the definition for the SPI image, not the MMC image which is in a different device. The input is the same schema, albeit that things like 'offset' may be filled in by Binman automatically when it packs things.
I want to go through binman more thoroughly, but a lot of changes happened since I last looked at it and I'm a bit slow at writing things, so won't exactly be soon. That being said, here's some ideas off the top of my head, for inspiration on both this schema and binman itself.
Do you mean the code? There are definitely some abstractions that are stretching a bit, but it is mostly holding together for now.
On 2023-07-12 00:18 +03:00, Simon Glass wrote:
I am unsure whether to add this with a generic name, such as 'layout', but for now am using /firmware/binman to avoid conflicts with any other firmware-layout schema that others might be working on.
Signed-off-by: Simon Glass sjg@chromium.org
I've been thinking of compatible = "data,<type>" for entries, so proposing 'data' here. A big ask, but it might be the one schema to unify them all if we look at things as "description of data" instead of "firmware layout".
As I mentioned in the commit message I was a bit unsure about just coming out with a generic name, but let's do it and see what people think.
Also consider data layouts in-memory. For example it could be an alternative to /chosen linux,initrd-start to specify initramfs location. Or things like keys or logs received from previous stages / other parts? Weak examples, but maybe might connect better into firmware handoff things. (Sorry, I don't know much about those.)
Yes I suspect at the high level those are better handled by firmware handoff, which is some other work going on. But I would like this to be a standard for firmware images, so BInman could perhaps plug into the handoff size of things too,
I'm also thinking about things like JPEG/BMP files for ACPI BGRT-like boot splash, unique firmware/configuration/calibration data for drivers, but they don't exactly need to be in-memory I guess.
Sure,
dtschema/schemas/firmware/binman.yaml | 51 ++++++++++++++++++ dtschema/schemas/firmware/binman/entry.yaml | 57 +++++++++++++++++++++ 2 files changed, 108 insertions(+) create mode 100644 dtschema/schemas/firmware/binman.yaml create mode 100644 dtschema/schemas/firmware/binman/entry.yaml
diff --git a/dtschema/schemas/firmware/binman.yaml b/dtschema/schemas/firmware/binman.yaml new file mode 100644 index 0000000..4b1ecf6 --- /dev/null +++ b/dtschema/schemas/firmware/binman.yaml @@ -0,0 +1,51 @@ +# SPDX-License-Identifier: (GPL-2.0 OR BSD-2-Clause) +# Copyright 2023 Google LLC
+%YAML 1.2 +--- +$id: http://devicetree.org/schemas/firmware/binman.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml#
+title: Binman firmware layout
+maintainers:
- Simon Glass sjg@chromium.org
+description: |
- The binman node provides a layout for firmware, used when packaging firmware
- from multiple projects. For now it just supports a very simple set of
- features, as a starting point for discussion.
- Documentation for Binman is available at:
- https://u-boot.readthedocs.io/en/latest/develop/package/binman.html
- with the current image-description format at:
- https://u-boot.readthedocs.io/en/latest/develop/package/binman.html#image-de...
+properties:
- $nodename:
- const: binman
I think having a single /firmware/binman node for a single image *and* having a "multiple-images" mode under it is quite bad because the single-image mode is multiple-images with one image. Multiple images should be the only mode of operation
Yes, I think that is pretty clear now. Most boards have ended up with more than a single image, which I had not envisaged at the start.
Furthermore, each image should be able to function independently of others, so I think to help enforce that maybe there shouldn't be any formal binman node. Merely instances of images that may happen to be under one parent node that may happen to be called /firmware/binman. The images can be compatible = "data,image" or something and we could search by that.
An image is a section with some extra properties. But yes that seems OK to me.
Or, heh, a single image node that happens to be named /firmware/binman because we are embedding it into a u-boot.dtb in that image. But perhaps we shouldn't restrain things like that. A bit contrived, but why should we be unable to start from an image in SPI and make it load something from microSD?
Well, you can do that, although U-Boot currently has no mechanism to connect the binman definition to the media. We need use cases for this stuff, IMO, since designing something before it is needed by a particular board is just a pain.
Going even further, let me suggest putting the image nodes as children of the devices whose layout the image describes. Then I imagine we can have something like a "data driver" (both for U-Boot and Linux) that reads the entry data from the parent data/block device and makes it available for other drivers or in sysfs.
Yes, we have that on x86 actually (grep for "fwstore_spi" in U-Boot). I'm sure people will weigh in on this, but I feel it is a natural thing to do. Of course, we may have the same image structure which can appear on SD or eMMC, so perhaps we should have a property in the device, like data,image-desc = <&binman_image_node> so that the image desc can be in the /firmware node.
+required:
- compatible
+additionalProperties: false
Including the binman version or a version number for the spec might be useful later on.
I worry a little about that. Devicetree schemas are supposed to be backwards-compatible. But you are right in that we may add new features later and that may involve new properties. As an example, the 'image-pos' property was added to Binman long after the 'offset' property. Perhaps we need a way for readers to see what level of support is present in the file?
But for now I am going to leave it out, since I'm not sure what to do here. If you have a proposal, please let me know.
+examples:
- |
- |
- firmware {
binman {compatible = "u-boot,binman";Consider including "image" or "namespace" in the compatible string to disambiguate. Not sure of the conventions, "binman,*" could work just as well, though I'm partial to "data,*" as I've suggested.
OK I can try 'data,image' and see what people think.
u-boot {size = <0xa0000>;};atf-bl31 {offset = <0x100000>;};};- };
diff --git a/dtschema/schemas/firmware/binman/entry.yaml b/dtschema/schemas/firmware/binman/entry.yaml new file mode 100644 index 0000000..16b84c5 --- /dev/null +++ b/dtschema/schemas/firmware/binman/entry.yaml @@ -0,0 +1,57 @@ +# SPDX-License-Identifier: (GPL-2.0 OR BSD-2-Clause) +# Copyright 2023 Google LLC
+%YAML 1.2 +--- +$id: http://devicetree.org/schemas/firmware/binman/entry.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml#
+title: Binman entry
+maintainers:
- Simon Glass sjg@chromium.org
+description: |
- The entry node specifies a single entry in the firmware.
- Entries have a specific type, such as "u-boot" or "atf-bl31". If the type
- is missing, the name is used as the type.
I really dislike the name being a fallback for the type, it's guiding people to using types as the names (or even worse, to assuming names can only be same as the type), leading to ugly combinations of "blob-ext", "fdt-N", "fit", "section" and so on. Node names should be descriptive even when you don't have the device-tree source in front of you, think of "binman extract" and others. Ones like "u-boot" and "atf-bl31" are fine only because they are very specific entry types.
Well this doesn't have to be in the schema itself, since Binman can state that it handles fixing up these things. I really like the name-as-type thing since it simplifies so much of the documentation, test dts files and makes it much easier for people to understand initially.
- Note: This definition is intended to be hierarchical, so that entries can
- appear in other entries. Schema for that is TBD.
It might be better to redesign things bottom-up -- first raw data, then a few data types, then sections and images that hold these. Oh, and I have a feeling section/image can be the same thing if we think hard enough (e.g. explicit node labels/references for fdtmap).
Yes section and image are essentially the same, as you can see in the code. But we do need top-level nodes which represent images.
+properties:
- $nodename:
- pattern: "^[-a-z]+(-[0-9]+)?$"
- type:
- $ref: /schemas/types.yaml#/definitions/string
Consider compatible = "data,bytes" instead, making it required. Or maybe "data,blob" but that's not what's there on the binman side. (e.g. rename "blob" to "file", "blob-ext" could be "file" combined with a globally-available "optional" property.)
I don't like 'file' at all, sorry. Where it comes from is not the point here; we are worried about what it is. Since we have no information on its format, it is a blob, IMO. Binman has an "optional" property which means that the entry itself is optional.
For now I just have a generic entry, with no definition of the available types. For a first series I very much want to avoid getting into this level of detail. It is easy to add stuff later.
- offset:
- $ref: /schemas/types.yaml#/definitions/uint32
- description: |
Provides the offset of this entry from the start of its parent section.- size:
- $ref: /schemas/types.yaml#/definitions/uint32
- description: |
Provides the size of this entry in bytes.Can't we use "reg" for these, and even entry@<addr> in the definitions? I know the latter conflicts with the fdt@SEQ stuff, but that should be a binman-input-only thing, and I dislike that anyway. Maybe could work if we put image nodes under block devices...
That was an early decision, based on the fact that generally you don't specify the offset or size, so they are optional; reg does not work well with an optional address or size. Also, we have image-pos as the absolute position (which again I have elected to worry about later once we have an agreed initial schema). Does reg mean an absolute address or a relative one ('ranges', anyone...nooooo!!!)? To me it is stretching the meaning.
That said, I can see some benefit in writing a 'reg' property when all the entries are packed:
reg = <offset size>
i.e. the offset is relative to its containing section. That would simplify some of the 'replace' logic, since offset and size can be left in there.
Or maybe even better, I think we should make it like FIT: allow a "data" property that has it inside the dtb, or a pair of "data-offset/position" and "data-size" properties if it's outside.
Eh? The point of the entry is to declare the position of actual data. If the data is elsewhere, then the entry will be too.
Inlining data inside the dtb could help us do nice things in binman, like hashes/signatures as entry types instead of special-casing them.
We already do that though, right? See testHash() for example. This is a powerful feature of a firmware-layout description / Binman, IMO, since all the metadata you need is right there.
In fact it could be possible to turn binman images into a FIT 2.0 if we do some more work on top of that, instead of nesting a FIT inside a binman image.
Well binman needs to produce things which are not FITs. Bear in mind that the output can be a binary file in any format. It doesn't necessarily have to have any metadata in it at all, certainly not a FIT header.
+additionalProperties: false
+examples:
- |
- firmware {
binman {compatible = "u-boot,binman";u-boot {size = <0xa0000>;};second-area {type = "atf-bl31";offset = <0x100000>;};};- };
Thanks again for your encouraging comments. I'll do a v2 with the above changes.
Regards, Simon
(I think I've strayed far away from schema/dtb side of things towards binman-the-program side, so I'm dropping Rob and devicetree@ from Cc).
On 2023-07-20 22:55 +03:00, Simon Glass wrote:
On Thu, 20 Jul 2023 at 09:23, Alper Nebi Yasak alpernebiyasak@gmail.com wrote:
There's also the issue of binman having multiple different device-trees: its input, the ones in fdtmaps per image, the ones injected to U-Boot dtb files per image. I'd say each has different needs, and those differences have to be figured out before specified upstream. I can only guess this is about the one that'll be in a u-boot.dtb.
Well, there is really only one. The fdtmap is actually the same schema, except that it mentions only the image that it is embedded in, i.e. if the fdtmap is for the SPI image, then the fdtmap in SPI flash only contains the definition for the SPI image, not the MMC image which is in a different device. The input is the same schema, albeit that things like 'offset' may be filled in by Binman automatically when it packs things.
I was thinking of things like generator nodes and templates and "filename"s in blob entries that (IMO) only make sense as binman inputs and never should be in a fdtmap. Trying to highlight a difference between "how to build this image" and "what this image contains".
But I guess it's OK to call them the same schema unless something has two conflicting meanings in input-domain and output-domain, and I can't think of any counter-examples now.
I want to go through binman more thoroughly, but a lot of changes happened since I last looked at it and I'm a bit slow at writing things, so won't exactly be soon. That being said, here's some ideas off the top of my head, for inspiration on both this schema and binman itself.
Do you mean the code? There are definitely some abstractions that are stretching a bit, but it is mostly holding together for now.
I mean both the implementation and the design. There's a lot more on my mind, some more examples:
- Precise structures for data instead of thinking of them as black boxes - Heuristically parsing arbitrary images/data for e.g. binman extract - Deconstructing input files to reuse their pieces for building images - Explicit dependency tracking and resolution for data and layout - Making things act more like native Python objects - In fact, making it entirely operable with just Python code - Decoupling internals from the control dtbs ("entry._node" etc.) - Ensuring reproducible builds and testing for it - Fuzzing as a replacement for most tests - ...
I think it has the beginnings of a very nice declarative-style tool, but has to embrace that style to get there. (And I guess I'll have to do the work to properly express myself on some points...)
[...]
Or maybe even better, I think we should make it like FIT: allow a "data" property that has it inside the dtb, or a pair of "data-offset/position" and "data-size" properties if it's outside.
Eh? The point of the entry is to declare the position of actual data. If the data is elsewhere, then the entry will be too.
Sorry, I'm jumping into a different contexts here without explaining it. I'm seeing a similarity between images that start with a fdtmap and the images built by "mkimage -T fit -E". Then I'm extrapolating to the non-"-E" case. Then extending to make it possible for a "fdtmap + data" binman description to result in something almost backwards-compatible with FIT, which could replace most uses of a fit etype.
(Of course, backwards compatible only if you add config nodes, and flatten or don't nest entries, and whatnot. And fit etype would still stay.)
Inlining data inside the dtb could help us do nice things in binman, like hashes/signatures as entry types instead of special-casing them.
We already do that though, right? See testHash() for example. This is a powerful feature of a firmware-layout description / Binman, IMO, since all the metadata you need is right there.
Having that functionality in state.py and having to work around it for mkimage/fit etypes (because we want to embed them into the data instead of the binman dtb) is what I mean by special-casing.
If we had them as etypes, and could embed arbitrary entries as "data" in binman dtb, I think we would have the best of both worlds -- avoid calling mkimage just for hash/signature embedding, have it still possible to put those in binman metadata, and enable a foundation for other sign-verify flows (leaning towards replacing custom signing tools/scripts with binman).
In fact it could be possible to turn binman images into a FIT 2.0 if we do some more work on top of that, instead of nesting a FIT inside a binman image.
Well binman needs to produce things which are not FITs. Bear in mind that the output can be a binary file in any format. It doesn't necessarily have to have any metadata in it at all, certainly not a FIT header.
I know, I don't mean it as the only mode of operation, I hope I have explained better this time.
Thanks again for your encouraging comments. I'll do a v2 with the above changes.
I'm glad to hear that you appreciate my comments, because trying to review your work always feels quite hubristic to me, haha.
Hi Alper,
On Thu, 3 Aug 2023 at 13:29, Alper Nebi Yasak alpernebiyasak@gmail.com wrote:
(I think I've strayed far away from schema/dtb side of things towards binman-the-program side, so I'm dropping Rob and devicetree@ from Cc).
On 2023-07-20 22:55 +03:00, Simon Glass wrote:
On Thu, 20 Jul 2023 at 09:23, Alper Nebi Yasak alpernebiyasak@gmail.com
wrote:
There's also the issue of binman having multiple different
device-trees:
its input, the ones in fdtmaps per image, the ones injected to U-Boot dtb files per image. I'd say each has different needs, and those differences have to be figured out before specified upstream. I can
only
guess this is about the one that'll be in a u-boot.dtb.
Well, there is really only one. The fdtmap is actually the same schema, except that it mentions only the image that it is embedded in, i.e. if the fdtmap is for the SPI image, then the fdtmap in SPI flash only contains the definition for the SPI image, not the MMC image which is in a different device. The input is the same schema, albeit that things like 'offset' may be filled in by Binman automatically when it packs things.
I was thinking of things like generator nodes and templates and "filename"s in blob entries that (IMO) only make sense as binman inputs and never should be in a fdtmap. Trying to highlight a difference between "how to build this image" and "what this image contains".
But I guess it's OK to call them the same schema unless something has two conflicting meanings in input-domain and output-domain, and I can't think of any counter-examples now.
Yes I see that distinction, but I certainly want to avoid anything that would conflict. Also, it is the 'final' format (fdtmap) for which I most want to have a binding.
I want to go through binman more thoroughly, but a lot of changes happened since I last looked at it and I'm a bit slow at writing
things,
so won't exactly be soon. That being said, here's some ideas off the
top
of my head, for inspiration on both this schema and binman itself.
Do you mean the code? There are definitely some abstractions that are stretching a bit, but it is mostly holding together for now.
I mean both the implementation and the design. There's a lot more on my mind, some more examples:
- Precise structures for data instead of thinking of them as black boxes
- Heuristically parsing arbitrary images/data for e.g. binman extract
- Deconstructing input files to reuse their pieces for building images
- Explicit dependency tracking and resolution for data and layout
- Making things act more like native Python objects
- In fact, making it entirely operable with just Python code
- Decoupling internals from the control dtbs ("entry._node" etc.)
- Ensuring reproducible builds and testing for it
- Fuzzing as a replacement for most tests
- ...
I think it has the beginnings of a very nice declarative-style tool, but has to embrace that style to get there. (And I guess I'll have to do the work to properly express myself on some points...)
Yes, patches are best, too. E.g. pick one of the above and come up with a proposal!
[...]
Or maybe even better, I think we should make it like FIT: allow a
"data"
property that has it inside the dtb, or a pair of
"data-offset/position"
and "data-size" properties if it's outside.
Eh? The point of the entry is to declare the position of actual data. If the data is elsewhere, then the entry will be too.
Sorry, I'm jumping into a different contexts here without explaining it. I'm seeing a similarity between images that start with a fdtmap and the images built by "mkimage -T fit -E". Then I'm extrapolating to the non-"-E" case. Then extending to make it possible for a "fdtmap + data" binman description to result in something almost backwards-compatible with FIT, which could replace most uses of a fit etype.
(Of course, backwards compatible only if you add config nodes, and flatten or don't nest entries, and whatnot. And fit etype would still
stay.)
OK I see. Yes, the -E FIT could use some of binman's descriptive power. I haven't really thought about it, but I agree.
Inlining data inside the dtb could help us do nice things in binman, like hashes/signatures as entry types instead of special-casing them.
We already do that though, right? See testHash() for example. This is a powerful feature of a firmware-layout description / Binman, IMO, since all the metadata you need is right there.
Having that functionality in state.py and having to work around it for mkimage/fit etypes (because we want to embed them into the data instead of the binman dtb) is what I mean by special-casing.
If we had them as etypes, and could embed arbitrary entries as "data" in binman dtb, I think we would have the best of both worlds -- avoid calling mkimage just for hash/signature embedding, have it still possible to put those in binman metadata, and enable a foundation for other sign-verify flows (leaning towards replacing custom signing tools/scripts with binman).
Sounds good to me.
In fact it could be possible to turn binman images into a FIT 2.0 if we do some more work on top of that, instead of nesting a FIT inside a binman image.
Well binman needs to produce things which are not FITs. Bear in mind that the output can be a binary file in any format. It doesn't necessarily have to have any metadata in it at all, certainly not a FIT header.
I know, I don't mean it as the only mode of operation, I hope I have explained better this time.
Thanks again for your encouraging comments. I'll do a v2 with the above
changes.
I'm glad to hear that you appreciate my comments, because trying to review your work always feels quite hubristic to me, haha.
Well Binman was started as a rewrite of a tool hacked together years back in Chrome OS. I wanted it to grow as the use cases developed and then see where it ended up. I think it is now at the pointer where there is enough functionality that we can seriously think about how best to describe images, which is why I am pushing this binding at present.
Regards, Simon
On Tue, Jul 11, 2023 at 3:18 PM Simon Glass sjg@chromium.org wrote:
Add a motivation and purpose for this new proposed node.
Signed-off-by: Simon Glass sjg@chromium.org
dtschema/schemas/firmware.yaml | 83 ++++++++++++++++++++++++++++++++++ 1 file changed, 83 insertions(+) create mode 100644 dtschema/schemas/firmware.yaml
diff --git a/dtschema/schemas/firmware.yaml b/dtschema/schemas/firmware.yaml new file mode 100644 index 0000000..4439a70 --- /dev/null +++ b/dtschema/schemas/firmware.yaml @@ -0,0 +1,83 @@ +# SPDX-License-Identifier: GPL-2.0-only OR BSD-2-clause +# Copyright 2023 Google LLC +#
+%YAML 1.2 +--- +$id: http://devicetree.org/schemas/firmware.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml#
+title: /firmware Node
+maintainers:
- Simon Glass sjg@chromium.org
+description: |
- The '/firmware' node does not represent a real device, but serves as a place
- for recording information about the main firmware used on the device, such as
- a map of its contents. This is used by the Operating System (OS), user space
- programs and possibly other firmware components. Data in the '/firmware' node
- does not itself represent the hardware.
- Properties in this node should be common to (and used by) at least two
- firmware projects, such as U-Boot and TF-A. Project-specific subnodes can be
- used for properties which are specific to a single project.
- Purpose of '/firmware' node
- Firmware has traditionally been fairly opaque to the OS, with the OS taking
- no interest in its contents, version, layout or how it might be updated. This
- is less than ideal, since firmware is an important part of the system and
- visibility into its operation is every bit as important as visbility into the
- OS and user-space programs within the system.
- The traditional approach has been to let firmware deal with firmware, and the
- OS deal with everything else. Updating firmware has been handled by firmware.
- For example, the UEFI spec defines a way for the OS to post a 'capsule' which
- is discovered next time the system boots, permitting firmware updates. But
- firmware updates in firmware are highly problematic. They require a reboot
- and a sometimes-lengthy wait with a strange-looking interface unfamiliar
- to most users. It seems better to make the update as transparent as possible
- to the user. As an example of that, ChromeOS has full knowledge of the
- firmware version and layout, updates it in the background from user space and
- instantly selects the new firmware when the user reboots or logs out.
Perhaps if OS based firmware updates are useful, then UEFI should gain that capability rather than inventing some way to do it with DT. Seems like a worthy goal, just needs wider review IMO.
- A common objection to considering the system holistically is that some parts
- of the system are inaccessible to the OS, such as a secure enclave. However
- this does not preclude providing visibility into what is present in that
- enclave. Firmware-version information is still useful. Firmware updates are
- still needed and can still be initiated from user space.
- Another objection is that firmware should provide an interface to the OS,
- while keeping its structure private. This thinking is largely driven by
- extrapolating from how firmware has been handled in the 'BIOS' days.
It's also the case that the OS may not have direct access to the h/w needed.
- It should be considered a degenerate case rather than the norm. As complexity
- increases, it creates an artificial boundary between two pieces of the whole.
- Mechanisms then need to be invented to cross this unnecessary chasm. An
- example of this is Intel's Dynamic Platform and Thermal Framework (DPTF),
- which consists of user-space, OS and firmware components all working towards
- a shared goal. We need a standard description of these cross-system pieces.
- In order to 'teach the OS about firmware', we need a place to put this
- information. That is the purpose of this node.
- In an Open Source world the entire model of firmware needs to adjust to be
- more open, more visible and managed just like any other part of the system.
- The major goal is to standardise how firmware is presented to the OS and user
- space, so that common utilities can be used to manage the entire system,
- including the firmware. For example, fwupd can look in this node for
- information on how to update the firmware, similar to how VBE works. [1]
- It is likely that other purposes will come to light over time.
It's good we're documenting /firmware, but your use seems different to what's already in place. Generally, /firmware has been for providers which are implemented by firmware and are not on any bus. SCMI for example. PSCI is another example, but it predated using /firmware so it's typically just put at the top level (and also PSCI should have been made discoverable like any SMCCC interface).
+properties:
- $nodename:
- const: firmware
- "#address-cells": true
- "#size-cells": true
What address space is this? It's not memory mapped because you don't have 'ranges'.
Rob
Hi Rob,
On Fri, 14 Jul 2023 at 10:58, Rob Herring robh@kernel.org wrote:
On Tue, Jul 11, 2023 at 3:18 PM Simon Glass sjg@chromium.org wrote:
Add a motivation and purpose for this new proposed node.
Signed-off-by: Simon Glass sjg@chromium.org
dtschema/schemas/firmware.yaml | 83 ++++++++++++++++++++++++++++++++++ 1 file changed, 83 insertions(+) create mode 100644 dtschema/schemas/firmware.yaml
diff --git a/dtschema/schemas/firmware.yaml b/dtschema/schemas/firmware.yaml new file mode 100644 index 0000000..4439a70 --- /dev/null +++ b/dtschema/schemas/firmware.yaml @@ -0,0 +1,83 @@ +# SPDX-License-Identifier: GPL-2.0-only OR BSD-2-clause +# Copyright 2023 Google LLC +#
+%YAML 1.2 +--- +$id: http://devicetree.org/schemas/firmware.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml#
+title: /firmware Node
+maintainers:
- Simon Glass sjg@chromium.org
+description: |
- The '/firmware' node does not represent a real device, but serves as a place
- for recording information about the main firmware used on the device, such as
- a map of its contents. This is used by the Operating System (OS), user space
- programs and possibly other firmware components. Data in the '/firmware' node
- does not itself represent the hardware.
- Properties in this node should be common to (and used by) at least two
- firmware projects, such as U-Boot and TF-A. Project-specific subnodes can be
- used for properties which are specific to a single project.
- Purpose of '/firmware' node
- Firmware has traditionally been fairly opaque to the OS, with the OS taking
- no interest in its contents, version, layout or how it might be updated. This
- is less than ideal, since firmware is an important part of the system and
- visibility into its operation is every bit as important as visbility into the
- OS and user-space programs within the system.
- The traditional approach has been to let firmware deal with firmware, and the
- OS deal with everything else. Updating firmware has been handled by firmware.
- For example, the UEFI spec defines a way for the OS to post a 'capsule' which
- is discovered next time the system boots, permitting firmware updates. But
- firmware updates in firmware are highly problematic. They require a reboot
- and a sometimes-lengthy wait with a strange-looking interface unfamiliar
- to most users. It seems better to make the update as transparent as possible
- to the user. As an example of that, ChromeOS has full knowledge of the
- firmware version and layout, updates it in the background from user space and
- instantly selects the new firmware when the user reboots or logs out.
Perhaps if OS based firmware updates are useful, then UEFI should gain that capability rather than inventing some way to do it with DT. Seems like a worthy goal, just needs wider review IMO.
Perhaps it should, although it would involve changing the spec, etc. In any case it would be a very strange world if we mandated UEFI everywhere.
Yes I am looking for wider review, partly since the work to document all the firmware-image complexity is happening mostly in U-Boot at present.
- A common objection to considering the system holistically is that some parts
- of the system are inaccessible to the OS, such as a secure enclave. However
- this does not preclude providing visibility into what is present in that
- enclave. Firmware-version information is still useful. Firmware updates are
- still needed and can still be initiated from user space.
- Another objection is that firmware should provide an interface to the OS,
- while keeping its structure private. This thinking is largely driven by
- extrapolating from how firmware has been handled in the 'BIOS' days.
It's also the case that the OS may not have direct access to the h/w needed.
I tried to cover that in the paragraph you quote immediately above...what is missing?
- It should be considered a degenerate case rather than the norm. As complexity
- increases, it creates an artificial boundary between two pieces of the whole.
- Mechanisms then need to be invented to cross this unnecessary chasm. An
- example of this is Intel's Dynamic Platform and Thermal Framework (DPTF),
- which consists of user-space, OS and firmware components all working towards
- a shared goal. We need a standard description of these cross-system pieces.
- In order to 'teach the OS about firmware', we need a place to put this
- information. That is the purpose of this node.
- In an Open Source world the entire model of firmware needs to adjust to be
- more open, more visible and managed just like any other part of the system.
- The major goal is to standardise how firmware is presented to the OS and user
- space, so that common utilities can be used to manage the entire system,
- including the firmware. For example, fwupd can look in this node for
- information on how to update the firmware, similar to how VBE works. [1]
- It is likely that other purposes will come to light over time.
It's good we're documenting /firmware, but your use seems different to what's already in place. Generally, /firmware has been for providers which are implemented by firmware and are not on any bus. SCMI for example. PSCI is another example, but it predated using /firmware so it's typically just put at the top level (and also PSCI should have been made discoverable like any SMCCC interface).
The intent here is that the node is not on any bus.
I'm happy to create a new node if that is better.
+properties:
- $nodename:
- const: firmware
- "#address-cells": true
- "#size-cells": true
What address space is this? It's not memory mapped because you don't have 'ranges'.
Probably this should be 'false' ? I suppose there may be an address space, for example in memory-mapped SPI flash, but that is not very common. The firmware could be stored in eMMC or UFS, which are not memory-mapped.
Regards, Simon
Hi Rob,
On Fri, 14 Jul 2023 at 11:57, Simon Glass sjg@chromium.org wrote:
Hi Rob,
On Fri, 14 Jul 2023 at 10:58, Rob Herring robh@kernel.org wrote:
On Tue, Jul 11, 2023 at 3:18 PM Simon Glass sjg@chromium.org wrote:
Add a motivation and purpose for this new proposed node.
Signed-off-by: Simon Glass sjg@chromium.org
dtschema/schemas/firmware.yaml | 83 ++++++++++++++++++++++++++++++++++ 1 file changed, 83 insertions(+) create mode 100644 dtschema/schemas/firmware.yaml
diff --git a/dtschema/schemas/firmware.yaml b/dtschema/schemas/firmware.yaml new file mode 100644 index 0000000..4439a70 --- /dev/null +++ b/dtschema/schemas/firmware.yaml @@ -0,0 +1,83 @@ +# SPDX-License-Identifier: GPL-2.0-only OR BSD-2-clause +# Copyright 2023 Google LLC +#
+%YAML 1.2 +--- +$id: http://devicetree.org/schemas/firmware.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml#
+title: /firmware Node
+maintainers:
- Simon Glass sjg@chromium.org
+description: |
- The '/firmware' node does not represent a real device, but serves as a place
- for recording information about the main firmware used on the device, such as
- a map of its contents. This is used by the Operating System (OS), user space
- programs and possibly other firmware components. Data in the '/firmware' node
- does not itself represent the hardware.
- Properties in this node should be common to (and used by) at least two
- firmware projects, such as U-Boot and TF-A. Project-specific subnodes can be
- used for properties which are specific to a single project.
- Purpose of '/firmware' node
- Firmware has traditionally been fairly opaque to the OS, with the OS taking
- no interest in its contents, version, layout or how it might be updated. This
- is less than ideal, since firmware is an important part of the system and
- visibility into its operation is every bit as important as visbility into the
- OS and user-space programs within the system.
- The traditional approach has been to let firmware deal with firmware, and the
- OS deal with everything else. Updating firmware has been handled by firmware.
- For example, the UEFI spec defines a way for the OS to post a 'capsule' which
- is discovered next time the system boots, permitting firmware updates. But
- firmware updates in firmware are highly problematic. They require a reboot
- and a sometimes-lengthy wait with a strange-looking interface unfamiliar
- to most users. It seems better to make the update as transparent as possible
- to the user. As an example of that, ChromeOS has full knowledge of the
- firmware version and layout, updates it in the background from user space and
- instantly selects the new firmware when the user reboots or logs out.
Perhaps if OS based firmware updates are useful, then UEFI should gain that capability rather than inventing some way to do it with DT. Seems like a worthy goal, just needs wider review IMO.
Perhaps it should, although it would involve changing the spec, etc. In any case it would be a very strange world if we mandated UEFI everywhere.
Yes I am looking for wider review, partly since the work to document all the firmware-image complexity is happening mostly in U-Boot at present.
- A common objection to considering the system holistically is that some parts
- of the system are inaccessible to the OS, such as a secure enclave. However
- this does not preclude providing visibility into what is present in that
- enclave. Firmware-version information is still useful. Firmware updates are
- still needed and can still be initiated from user space.
- Another objection is that firmware should provide an interface to the OS,
- while keeping its structure private. This thinking is largely driven by
- extrapolating from how firmware has been handled in the 'BIOS' days.
It's also the case that the OS may not have direct access to the h/w needed.
I tried to cover that in the paragraph you quote immediately above...what is missing?
- It should be considered a degenerate case rather than the norm. As complexity
- increases, it creates an artificial boundary between two pieces of the whole.
- Mechanisms then need to be invented to cross this unnecessary chasm. An
- example of this is Intel's Dynamic Platform and Thermal Framework (DPTF),
- which consists of user-space, OS and firmware components all working towards
- a shared goal. We need a standard description of these cross-system pieces.
- In order to 'teach the OS about firmware', we need a place to put this
- information. That is the purpose of this node.
- In an Open Source world the entire model of firmware needs to adjust to be
- more open, more visible and managed just like any other part of the system.
- The major goal is to standardise how firmware is presented to the OS and user
- space, so that common utilities can be used to manage the entire system,
- including the firmware. For example, fwupd can look in this node for
- information on how to update the firmware, similar to how VBE works. [1]
- It is likely that other purposes will come to light over time.
It's good we're documenting /firmware, but your use seems different to what's already in place. Generally, /firmware has been for providers which are implemented by firmware and are not on any bus. SCMI for example. PSCI is another example, but it predated using /firmware so it's typically just put at the top level (and also PSCI should have been made discoverable like any SMCCC interface).
The intent here is that the node is not on any bus.
I'm happy to create a new node if that is better.
+properties:
- $nodename:
- const: firmware
- "#address-cells": true
- "#size-cells": true
What address space is this? It's not memory mapped because you don't have 'ranges'.
Probably this should be 'false' ? I suppose there may be an address space, for example in memory-mapped SPI flash, but that is not very common. The firmware could be stored in eMMC or UFS, which are not memory-mapped.
Are there any other comments on this? Is there a way to tag it so that other firmware people might see it?
Regards, Simon
participants (3)
-
Alper Nebi Yasak -
Rob Herring -
Simon Glass