Hi Alper,
On Thu, 3 Aug 2023 at 13:29, Alper Nebi Yasak alpernebiyasak@gmail.com wrote:
(I think I've strayed far away from schema/dtb side of things towards binman-the-program side, so I'm dropping Rob and devicetree@ from Cc).
On 2023-07-20 22:55 +03:00, Simon Glass wrote:
On Thu, 20 Jul 2023 at 09:23, Alper Nebi Yasak alpernebiyasak@gmail.com
wrote:
There's also the issue of binman having multiple different
device-trees:
its input, the ones in fdtmaps per image, the ones injected to U-Boot dtb files per image. I'd say each has different needs, and those differences have to be figured out before specified upstream. I can
only
guess this is about the one that'll be in a u-boot.dtb.
Well, there is really only one. The fdtmap is actually the same schema, except that it mentions only the image that it is embedded in, i.e. if the fdtmap is for the SPI image, then the fdtmap in SPI flash only contains the definition for the SPI image, not the MMC image which is in a different device. The input is the same schema, albeit that things like 'offset' may be filled in by Binman automatically when it packs things.
I was thinking of things like generator nodes and templates and "filename"s in blob entries that (IMO) only make sense as binman inputs and never should be in a fdtmap. Trying to highlight a difference between "how to build this image" and "what this image contains".
But I guess it's OK to call them the same schema unless something has two conflicting meanings in input-domain and output-domain, and I can't think of any counter-examples now.
Yes I see that distinction, but I certainly want to avoid anything that would conflict. Also, it is the 'final' format (fdtmap) for which I most want to have a binding.
I want to go through binman more thoroughly, but a lot of changes happened since I last looked at it and I'm a bit slow at writing
things,
so won't exactly be soon. That being said, here's some ideas off the
top
of my head, for inspiration on both this schema and binman itself.
Do you mean the code? There are definitely some abstractions that are stretching a bit, but it is mostly holding together for now.
I mean both the implementation and the design. There's a lot more on my mind, some more examples:
- Precise structures for data instead of thinking of them as black boxes
- Heuristically parsing arbitrary images/data for e.g. binman extract
- Deconstructing input files to reuse their pieces for building images
- Explicit dependency tracking and resolution for data and layout
- Making things act more like native Python objects
- In fact, making it entirely operable with just Python code
- Decoupling internals from the control dtbs ("entry._node" etc.)
- Ensuring reproducible builds and testing for it
- Fuzzing as a replacement for most tests
- ...
I think it has the beginnings of a very nice declarative-style tool, but has to embrace that style to get there. (And I guess I'll have to do the work to properly express myself on some points...)
Yes, patches are best, too. E.g. pick one of the above and come up with a proposal!
[...]
Or maybe even better, I think we should make it like FIT: allow a
"data"
property that has it inside the dtb, or a pair of
"data-offset/position"
and "data-size" properties if it's outside.
Eh? The point of the entry is to declare the position of actual data. If the data is elsewhere, then the entry will be too.
Sorry, I'm jumping into a different contexts here without explaining it. I'm seeing a similarity between images that start with a fdtmap and the images built by "mkimage -T fit -E". Then I'm extrapolating to the non-"-E" case. Then extending to make it possible for a "fdtmap + data" binman description to result in something almost backwards-compatible with FIT, which could replace most uses of a fit etype.
(Of course, backwards compatible only if you add config nodes, and flatten or don't nest entries, and whatnot. And fit etype would still
stay.)
OK I see. Yes, the -E FIT could use some of binman's descriptive power. I haven't really thought about it, but I agree.
Inlining data inside the dtb could help us do nice things in binman, like hashes/signatures as entry types instead of special-casing them.
We already do that though, right? See testHash() for example. This is a powerful feature of a firmware-layout description / Binman, IMO, since all the metadata you need is right there.
Having that functionality in state.py and having to work around it for mkimage/fit etypes (because we want to embed them into the data instead of the binman dtb) is what I mean by special-casing.
If we had them as etypes, and could embed arbitrary entries as "data" in binman dtb, I think we would have the best of both worlds -- avoid calling mkimage just for hash/signature embedding, have it still possible to put those in binman metadata, and enable a foundation for other sign-verify flows (leaning towards replacing custom signing tools/scripts with binman).
Sounds good to me.
In fact it could be possible to turn binman images into a FIT 2.0 if we do some more work on top of that, instead of nesting a FIT inside a binman image.
Well binman needs to produce things which are not FITs. Bear in mind that the output can be a binary file in any format. It doesn't necessarily have to have any metadata in it at all, certainly not a FIT header.
I know, I don't mean it as the only mode of operation, I hope I have explained better this time.
Thanks again for your encouraging comments. I'll do a v2 with the above
changes.
I'm glad to hear that you appreciate my comments, because trying to review your work always feels quite hubristic to me, haha.
Well Binman was started as a rewrite of a tool hacked together years back in Chrome OS. I wanted it to grow as the use cases developed and then see where it ended up. I think it is now at the pointer where there is enough functionality that we can seriously think about how best to describe images, which is why I am pushing this binding at present.
Regards, Simon