[U-Boot] [PATCH v2 0/2] rockchip: make_fit_atf.py Eliminate pyelftools dependency

newer
[U-Boot] [PATCH v3] spl: kconfig:...

older
[U-Boot] [PATCH v4 0/4] SiFive SPI...

Chris Webb

14 Jul 2019 14 Jul '19

10:45 a.m.

[Sorry for a respin so soon, but this fixes the overzealous documentation update and also ensures the script's behaviour is identical to the existing implementation in the case of a completely empty bl31 ELF file.]

Building for Rockchip, make_fit_atf.py depends on pyelftools, a non-bundled python module that is not used elsewhere in building u-boot or the kernel.

We only use pyelftools to pull out PT_LOAD segments. ELF is very simple, so doing this manually is easy and spares users the extra dependency. In fact, a straightforward implementation shrinks make_fit_aft.py rather than adding complexity.

Andy Yan's patch

https://gitlab.denx.de/u-boot/custodians/u-boot-rockchip/commit/619f002db864...

is a prerequisite for this. The new code does not erroneously count GNU_STACK sections and therefore without the above patch, the loadables in the conf section will always be one short, even where GNU_STACK is present.

I have tested this script with python 2 and 3 against all bl31 elf files at

https://github.com/rockchip-linux/rkbin/tree/master/bin/rk33

as well as an rk3399 bl31.elf built locally from the master branch of

https://github.com/ARM-software/arm-trusted-firmware

to which my toolchain added the extra GNU_STACK segment.

In each case, identical *.bin files were produced compared to the pyelftools implementation, and a correct .its configuration (without off-by-one error) was emitted, with and without dtbs supplied to the script.

Changes in v2: - no exception on a valid but completely empty bl31 ELF file - restrict documentation updates to the Rockchip docs (!)

Show replies by date

Chris Webb

14 Jul 14 Jul

10:46 a.m.

New subject: [U-Boot] [PATCH v2 1/2] rockchip: make_fit_atf.py: Eliminate pyelftools dependency

make_fit_aft.py depends on the non-standard library pyelftools to pull out PT_LOAD segments from ELF files. However, this is as easy to do manually, without imposing the extra dependency on users.

Structures in the ELF file are unpacked into variables named to exactly match the ELF spec to ensure the destructuring code is reasonably self-documenting.

Signed-off-by: Chris Webb chris@arachsys.com --- arch/arm/mach-rockchip/make_fit_atf.py | 75 +++++++++++--------------- 1 file changed, 32 insertions(+), 43 deletions(-)

diff --git a/arch/arm/mach-rockchip/make_fit_atf.py b/arch/arm/mach-rockchip/make_fit_atf.py index db0ae96ca8..9acc1edfc6 100755 --- a/arch/arm/mach-rockchip/make_fit_atf.py +++ b/arch/arm/mach-rockchip/make_fit_atf.py @@ -13,16 +13,7 @@ import os import sys import getopt import logging - -# pip install pyelftools -from elftools.elf.elffile import ELFFile - -ELF_SEG_P_TYPE = 'p_type' -ELF_SEG_P_PADDR = 'p_paddr' -ELF_SEG_P_VADDR = 'p_vaddr' -ELF_SEG_P_OFFSET = 'p_offset' -ELF_SEG_P_FILESZ = 'p_filesz' -ELF_SEG_P_MEMSZ = 'p_memsz' +import struct

DT_HEADER = """ /* @@ -118,33 +109,19 @@ def append_conf_node(file, dtbs, segments): file.write('\n')

def generate_atf_fit_dts_uboot(fit_file, uboot_file_name): - num_load_seg = 0 - p_paddr = 0xFFFFFFFF - with open(uboot_file_name, 'rb') as uboot_file: - uboot = ELFFile(uboot_file) - for i in range(uboot.num_segments()): - seg = uboot.get_segment(i) - if seg.__getitem__(ELF_SEG_P_TYPE) == 'PT_LOAD': - p_paddr = seg.__getitem__(ELF_SEG_P_PADDR) - num_load_seg = num_load_seg + 1 - - assert (p_paddr != 0xFFFFFFFF and num_load_seg == 1) - + segments = unpack_elf(uboot_file_name) + if len(segments) != 1: + raise ValueError("Invalid u-boot ELF image '%s'" % uboot_file_name) + index, entry, p_paddr, data = segments[0] fit_file.write(DT_UBOOT % p_paddr)

def generate_atf_fit_dts_bl31(fit_file, bl31_file_name, dtbs_file_name): - with open(bl31_file_name, 'rb') as bl31_file: - bl31 = ELFFile(bl31_file) - elf_entry = bl31.header['e_entry'] - segments = bl31.num_segments() - for i in range(segments): - seg = bl31.get_segment(i) - if seg.__getitem__(ELF_SEG_P_TYPE) == 'PT_LOAD': - paddr = seg.__getitem__(ELF_SEG_P_PADDR) - append_bl31_node(fit_file, i + 1, paddr, elf_entry) + segments = unpack_elf(bl31_file_name) + for index, entry, paddr, data in segments: + append_bl31_node(fit_file, index + 1, paddr, entry) append_fdt_node(fit_file, dtbs_file_name) fit_file.write(DT_IMAGES_NODE_END) - append_conf_node(fit_file, dtbs_file_name, segments) + append_conf_node(fit_file, dtbs_file_name, len(segments))

def generate_atf_fit_dts(fit_file_name, bl31_file_name, uboot_file_name, dtbs_file_name): # Generate FIT script for ATF image. @@ -162,17 +139,29 @@ def generate_atf_fit_dts(fit_file_name, bl31_file_name, uboot_file_name, dtbs_fi fit_file.close()

def generate_atf_binary(bl31_file_name): - with open(bl31_file_name, 'rb') as bl31_file: - bl31 = ELFFile(bl31_file) - - num = bl31.num_segments() - for i in range(num): - seg = bl31.get_segment(i) - if seg.__getitem__(ELF_SEG_P_TYPE) == 'PT_LOAD': - paddr = seg.__getitem__(ELF_SEG_P_PADDR) - file_name = 'bl31_0x%08x.bin' % paddr - with open(file_name, "wb") as atf: - atf.write(seg.data()) + for index, entry, paddr, data in unpack_elf(bl31_file_name): + file_name = 'bl31_0x%08x.bin' % paddr + with open(file_name, "wb") as atf: + atf.write(data) + +def unpack_elf(filename): + with open(filename, 'rb') as file: + elf = file.read() + if elf[0:7] != b'\x7fELF\x02\x01\x01' or elf[18:20] != b'\xb7\x00': + raise ValueError("Invalid arm64 ELF file '%s'" % filename) + + e_entry, e_phoff = struct.unpack_from('<2Q', elf, 0x18) + e_phentsize, e_phnum = struct.unpack_from('<2H', elf, 0x36) + segments = [] + + for index in range(e_phnum): + offset = e_phoff + e_phentsize*index + p_type, p_flags, p_offset = struct.unpack_from('<LLQ', elf, offset) + if p_type == 1: # PT_LOAD + p_paddr, p_filesz = struct.unpack_from('<2Q', elf, offset + 0x18) + p_data = elf[p_offset:p_offset + p_filesz] + segments.append((index, e_entry, p_paddr, p_data)) + return segments

def main(): uboot_elf = "./u-boot"

Andy Yan

16 Jul 16 Jul

10:45 a.m.

New subject: [U-Boot] [PATCH v2 1/2] rockchip: make_fit_atf.py: Eliminate pyelftools dependency

Hi Chris:

On 7/14/19 4:46 PM, Chris Webb wrote:

...

make_fit_aft.py depends on the non-standard library pyelftools to pull out PT_LOAD segments from ELF files. However, this is as easy to do manually, without imposing the extra dependency on users.

Structures in the ELF file are unpacked into variables named to exactly match the ELF spec to ensure the destructuring code is reasonably self-documenting.

Signed-off-by: Chris Webb chris@arachsys.com

arch/arm/mach-rockchip/make_fit_atf.py | 75 +++++++++++--------------- 1 file changed, 32 insertions(+), 43 deletions(-)

diff --git a/arch/arm/mach-rockchip/make_fit_atf.py b/arch/arm/mach-rockchip/make_fit_atf.py index db0ae96ca8..9acc1edfc6 100755 --- a/arch/arm/mach-rockchip/make_fit_atf.py +++ b/arch/arm/mach-rockchip/make_fit_atf.py @@ -13,16 +13,7 @@ import os import sys import getopt import logging

-# pip install pyelftools -from elftools.elf.elffile import ELFFile

-ELF_SEG_P_TYPE = 'p_type' -ELF_SEG_P_PADDR = 'p_paddr' -ELF_SEG_P_VADDR = 'p_vaddr' -ELF_SEG_P_OFFSET = 'p_offset' -ELF_SEG_P_FILESZ = 'p_filesz' -ELF_SEG_P_MEMSZ = 'p_memsz' +import struct

DT_HEADER = """ /* @@ -118,33 +109,19 @@ def append_conf_node(file, dtbs, segments): file.write('\n')

def generate_atf_fit_dts_uboot(fit_file, uboot_file_name):
num_load_seg = 0

p_paddr = 0xFFFFFFFF

with open(uboot_file_name, 'rb') as uboot_file:
   uboot = ELFFile(uboot_file)
   for i in range(uboot.num_segments()):
       seg = uboot.get_segment(i)
       if seg.__getitem__(ELF_SEG_P_TYPE) == 'PT_LOAD':
           p_paddr = seg.__getitem__(ELF_SEG_P_PADDR)
           num_load_seg = num_load_seg + 1
assert (p_paddr != 0xFFFFFFFF and num_load_seg == 1)
segments = unpack_elf(uboot_file_name)

if len(segments) != 1:
   raise ValueError("Invalid u-boot ELF image '%s'" % uboot_file_name)
index, entry, p_paddr, data = segments[0] fit_file.write(DT_UBOOT % p_paddr)

def generate_atf_fit_dts_bl31(fit_file, bl31_file_name, dtbs_file_name):
with open(bl31_file_name, 'rb') as bl31_file:
   bl31 = ELFFile(bl31_file)
   elf_entry = bl31.header['e_entry']
   segments = bl31.num_segments()
   for i in range(segments):
       seg = bl31.get_segment(i)
       if seg.__getitem__(ELF_SEG_P_TYPE) == 'PT_LOAD':
           paddr = seg.__getitem__(ELF_SEG_P_PADDR)
           append_bl31_node(fit_file, i + 1, paddr, elf_entry)
segments = unpack_elf(bl31_file_name)

for index, entry, paddr, data in segments:
   append_bl31_node(fit_file, index + 1, paddr, entry)
append_fdt_node(fit_file, dtbs_file_name)
fit_file.write(DT_IMAGES_NODE_END)
append_conf_node(fit_file, dtbs_file_name, segments)

append_conf_node(fit_file, dtbs_file_name, len(segments))

def generate_atf_fit_dts(fit_file_name, bl31_file_name, uboot_file_name, dtbs_file_name): # Generate FIT script for ATF image.

@@ -162,17 +139,29 @@ def generate_atf_fit_dts(fit_file_name, bl31_file_name, uboot_file_name, dtbs_fi fit_file.close()

def generate_atf_binary(bl31_file_name):
with open(bl31_file_name, 'rb') as bl31_file:
   bl31 = ELFFile(bl31_file)
   num = bl31.num_segments()
   for i in range(num):
       seg = bl31.get_segment(i)
       if seg.__getitem__(ELF_SEG_P_TYPE) == 'PT_LOAD':
           paddr = seg.__getitem__(ELF_SEG_P_PADDR)
           file_name = 'bl31_0x%08x.bin' % paddr
           with open(file_name, "wb") as atf:
               atf.write(seg.data())
for index, entry, paddr, data in unpack_elf(bl31_file_name):
   file_name = 'bl31_0x%08x.bin' % paddr
   with open(file_name, "wb") as atf:
       atf.write(data)
+def unpack_elf(filename):
with open(filename, 'rb') as file:
   elf = file.read()
if elf[0:7] != b'\x7fELF\x02\x01\x01' or elf[18:20] != b'\xb7\x00':
   raise ValueError("Invalid arm64 ELF file '%s'" % filename)
e_entry, e_phoff = struct.unpack_from('<2Q', elf, 0x18)

e_phentsize, e_phnum = struct.unpack_from('<2H', elf, 0x36)

segments = []

for index in range(e_phnum):
   offset = e_phoff + e_phentsize*index

One small coding style issue:

offset = e_phoff + e_phentsize * index

...

   p_type, p_flags, p_offset = struct.unpack_from('<LLQ', elf, offset)

```
   if p_type == 1: # PT_LOAD
```

       p_paddr, p_filesz = struct.unpack_from('<2Q', elf, offset + 0x18)

       p_data = elf[p_offset:p_offset + p_filesz]

       segments.append((index, e_entry, p_paddr, p_data))

return segments

def main(): uboot_elf = "./u-boot"

Kever Yang

1:04 p.m.

New subject: [U-Boot] [PATCH v2 1/2] rockchip: make_fit_atf.py: Eliminate pyelftools dependency

Chris,

On 2019/7/14 下午4:46, Chris Webb wrote:

...

make_fit_aft.py depends on the non-standard library pyelftools to pull out PT_LOAD segments from ELF files. However, this is as easy to do manually, without imposing the extra dependency on users.

Structures in the ELF file are unpacked into variables named to exactly match the ELF spec to ensure the destructuring code is reasonably self-documenting.

Signed-off-by: Chris Webb chris@arachsys.com

Please fix the coding style comment by Andy, for other codes:

Reviewed-by: Kever Yang Kever.yang@rock-chips.com

Thanks, - Kever

...

arch/arm/mach-rockchip/make_fit_atf.py | 75 +++++++++++--------------- 1 file changed, 32 insertions(+), 43 deletions(-)

diff --git a/arch/arm/mach-rockchip/make_fit_atf.py b/arch/arm/mach-rockchip/make_fit_atf.py index db0ae96ca8..9acc1edfc6 100755 --- a/arch/arm/mach-rockchip/make_fit_atf.py +++ b/arch/arm/mach-rockchip/make_fit_atf.py @@ -13,16 +13,7 @@ import os import sys import getopt import logging

-# pip install pyelftools -from elftools.elf.elffile import ELFFile

-ELF_SEG_P_TYPE = 'p_type' -ELF_SEG_P_PADDR = 'p_paddr' -ELF_SEG_P_VADDR = 'p_vaddr' -ELF_SEG_P_OFFSET = 'p_offset' -ELF_SEG_P_FILESZ = 'p_filesz' -ELF_SEG_P_MEMSZ = 'p_memsz' +import struct

DT_HEADER = """ /* @@ -118,33 +109,19 @@ def append_conf_node(file, dtbs, segments): file.write('\n')

def generate_atf_fit_dts_uboot(fit_file, uboot_file_name):
num_load_seg = 0

p_paddr = 0xFFFFFFFF

with open(uboot_file_name, 'rb') as uboot_file:
   uboot = ELFFile(uboot_file)
   for i in range(uboot.num_segments()):
       seg = uboot.get_segment(i)
       if seg.__getitem__(ELF_SEG_P_TYPE) == 'PT_LOAD':
           p_paddr = seg.__getitem__(ELF_SEG_P_PADDR)
           num_load_seg = num_load_seg + 1
assert (p_paddr != 0xFFFFFFFF and num_load_seg == 1)
segments = unpack_elf(uboot_file_name)

if len(segments) != 1:
   raise ValueError("Invalid u-boot ELF image '%s'" % uboot_file_name)
index, entry, p_paddr, data = segments[0] fit_file.write(DT_UBOOT % p_paddr)

def generate_atf_fit_dts_bl31(fit_file, bl31_file_name, dtbs_file_name):
with open(bl31_file_name, 'rb') as bl31_file:
   bl31 = ELFFile(bl31_file)
   elf_entry = bl31.header['e_entry']
   segments = bl31.num_segments()
   for i in range(segments):
       seg = bl31.get_segment(i)
       if seg.__getitem__(ELF_SEG_P_TYPE) == 'PT_LOAD':
           paddr = seg.__getitem__(ELF_SEG_P_PADDR)
           append_bl31_node(fit_file, i + 1, paddr, elf_entry)
segments = unpack_elf(bl31_file_name)

for index, entry, paddr, data in segments:
   append_bl31_node(fit_file, index + 1, paddr, entry)
append_fdt_node(fit_file, dtbs_file_name)
fit_file.write(DT_IMAGES_NODE_END)
append_conf_node(fit_file, dtbs_file_name, segments)

append_conf_node(fit_file, dtbs_file_name, len(segments))

def generate_atf_fit_dts(fit_file_name, bl31_file_name, uboot_file_name, dtbs_file_name): # Generate FIT script for ATF image.

@@ -162,17 +139,29 @@ def generate_atf_fit_dts(fit_file_name, bl31_file_name, uboot_file_name, dtbs_fi fit_file.close()

def generate_atf_binary(bl31_file_name):
with open(bl31_file_name, 'rb') as bl31_file:
   bl31 = ELFFile(bl31_file)
   num = bl31.num_segments()
   for i in range(num):
       seg = bl31.get_segment(i)
       if seg.__getitem__(ELF_SEG_P_TYPE) == 'PT_LOAD':
           paddr = seg.__getitem__(ELF_SEG_P_PADDR)
           file_name = 'bl31_0x%08x.bin' % paddr
           with open(file_name, "wb") as atf:
               atf.write(seg.data())
for index, entry, paddr, data in unpack_elf(bl31_file_name):
   file_name = 'bl31_0x%08x.bin' % paddr
   with open(file_name, "wb") as atf:
       atf.write(data)
+def unpack_elf(filename):
with open(filename, 'rb') as file:
   elf = file.read()
if elf[0:7] != b'\x7fELF\x02\x01\x01' or elf[18:20] != b'\xb7\x00':
   raise ValueError("Invalid arm64 ELF file '%s'" % filename)
e_entry, e_phoff = struct.unpack_from('<2Q', elf, 0x18)

e_phentsize, e_phnum = struct.unpack_from('<2H', elf, 0x36)

segments = []

for index in range(e_phnum):
   offset = e_phoff + e_phentsize*index
   p_type, p_flags, p_offset = struct.unpack_from('<LLQ', elf, offset)
   if p_type == 1: # PT_LOAD
       p_paddr, p_filesz = struct.unpack_from('<2Q', elf, offset + 0x18)
       p_data = elf[p_offset:p_offset + p_filesz]
       segments.append((index, e_entry, p_paddr, p_data))
return segments

def main(): uboot_elf = "./u-boot"

Chris Webb

1:46 p.m.

New subject: [U-Boot] [PATCH v2 1/2] rockchip: make_fit_atf.py: Eliminate pyelftools dependency

Hi Andy and Kever

Andy Yan andy.yan@rock-chips.com writes:

...

One small coding style issue:

offset = e_phoff + e_phentsize * index

Thanks, will fix this.

Kever Yang kever.yang@rock-chips.com writes:

...

Please fix the coding style comment by Andy, for other codes:

Reviewed-by: Kever Yang Kever.yang@rock-chips.com

Newbie contributor so can I check: is it right to add this Reviewed-by: line to the end of the patch commit message after my Signed-off-by: when I resend the patches as v3 with the style correction?

Many thanks, Chris.

Chris Webb

14 Jul 14 Jul

10:47 a.m.

New subject: [U-Boot] [PATCH v2 2/2] rockchip: Remove obsolete references to pyelftools

make_fit_atf.py no longer requires pyelftools, and nothing else in the rockchip build requires it either, so remove references to installing it from the documentation.

Signed-off-by: Chris Webb chris@arachsys.com --- board/rockchip/evb_rk3399/README | 6 ------ doc/README.rockchip | 4 ---- 2 files changed, 10 deletions(-)

diff --git a/board/rockchip/evb_rk3399/README b/board/rockchip/evb_rk3399/README index 6469821987..ea3258cf37 100644 --- a/board/rockchip/evb_rk3399/README +++ b/board/rockchip/evb_rk3399/README @@ -35,12 +35,6 @@ Get the Source and prebuild binary

...

git clone https://github.com/rockchip-linux/rkbin.git git clone https://github.com/rockchip-linux/rkdeveloptool.git

-Get some prerequisites -====================== - -You need the Python elftools.elf.elffile library for make_fit_atf.py to work: - - > sudo apt-get install python-pyelftools

Compile ATF =========== diff --git a/doc/README.rockchip b/doc/README.rockchip index 02e2497b15..8ccbb87264 100644 --- a/doc/README.rockchip +++ b/doc/README.rockchip @@ -123,10 +123,6 @@ For example:

Option 2: Package the image with SPL:

- - We need the Python elftools.elf.elffile library for make_fit_atf.py to work - - => sudo apt-get install python-pyelftools - - Export cross compiler path for aarch64

- Compile ATF

Kever Yang

16 Jul 16 Jul

1:02 p.m.

New subject: [U-Boot] [PATCH v2 2/2] rockchip: Remove obsolete references to pyelftools

On 2019/7/14 下午4:47, Chris Webb wrote:

...

make_fit_atf.py no longer requires pyelftools, and nothing else in the rockchip build requires it either, so remove references to installing it from the documentation.

Signed-off-by: Chris Webb chris@arachsys.com

Reviewed-by: Kever Yang Kever.yang@rock-chips.com

Thanks, - Kever

...

board/rockchip/evb_rk3399/README | 6 ------ doc/README.rockchip | 4 ---- 2 files changed, 10 deletions(-)

diff --git a/board/rockchip/evb_rk3399/README b/board/rockchip/evb_rk3399/README index 6469821987..ea3258cf37 100644 --- a/board/rockchip/evb_rk3399/README +++ b/board/rockchip/evb_rk3399/README @@ -35,12 +35,6 @@ Get the Source and prebuild binary > git clone https://github.com/rockchip-linux/rkbin.git > git clone https://github.com/rockchip-linux/rkdeveloptool.git

-Get some prerequisites

-You need the Python elftools.elf.elffile library for make_fit_atf.py to work:

...
sudo apt-get install python-pyelftools

Compile ATF

diff --git a/doc/README.rockchip b/doc/README.rockchip index 02e2497b15..8ccbb87264 100644 --- a/doc/README.rockchip +++ b/doc/README.rockchip @@ -123,10 +123,6 @@ For example:
 Option 2: Package the image with SPL:
We need the Python elftools.elf.elffile library for make_fit_atf.py to work
=> sudo apt-get install python-pyelftools
Export cross compiler path for aarch64

Compile ATF

2125

Age (days ago)

2127

Last active (days ago)

List overview

Download

6 comments

3 participants

tags (0)

participants (3)

Andy Yan
Chris Webb
Kever Yang