[PATCH 00/18] New CI image and fixes

newer
Pull request efi-2024-10-rc1-3

older
[PATCH] riscv: dts: jh7110: Enable...

Jiaxun Yang

17 Jul 2024 17 Jul '24

4:29 p.m.

Hi all,

This series build a new CI image based on Ubuntu focal with LoongArch64 support, fixed various python scripts for python 3.12, fixed various problems popped up when testing againt latest software.

Last two commits are for demonstration purpose and not for commit into repo.

CI runs passed at azure [1].

Thanks [1]: https://flygoat.visualstudio.com/u-boot/_build/results?buildId=64&view=r...

Signed-off-by: Jiaxun Yang jiaxun.yang@flygoat.com --- Jiaxun Yang (18): binman: Replace pkg_resources with importlib.resources py: Replace distutils.core with setuptools doc/sphinx: Remove usage of six py: Bump pylint version and clear warnings binman: Workaround lz4 cli padding in test cases tests/test_event_dump: Relax match rule for output lib/charset & efi: Fix possible unaligned accesses CI: Ensure pip install is always performed in venv CI: Dockerfile: Set global git name & email config CI: Dockerfile: Bump base OS version CI: Dockerfile: Bump GRUB to 2.12 CI: Dockerfile: Bump QEMU to 9.0.1 CI: Dockerfile: Bump fiptool to 2.10.4 CI: Dockerfile: Bump genimage to 17 CI: Dockerfile: Use kernel.org upstream for trace-cmd CI: Dockerfile: Bump coreboot to 24.05 CI: Dockerfile: Add LoongArch64 support [NFC] Use Jiaxun's CI Image

Best regards,

-- Jiaxun Yang jiaxun.yang@flygoat.com

Show replies by date

Jiaxun Yang

17 Jul 17 Jul

4:29 p.m.

New subject: [PATCH 01/18] binman: Replace pkg_resources with importlib.resources

pkg_resources is deprecated long ago and being removed in python 3.12.

Reimplement functions with importlib.resources.

Link: https://docs.python.org/3/library/importlib.resources.html Signed-off-by: Jiaxun Yang jiaxun.yang@flygoat.com --- tools/binman/control.py | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-)

diff --git a/tools/binman/control.py b/tools/binman/control.py index 2f00279232b8..5549b0ad2185 100644 --- a/tools/binman/control.py +++ b/tools/binman/control.py @@ -8,12 +8,11 @@ from collections import OrderedDict import glob try: - import importlib.resources -except ImportError: # pragma: no cover + from importlib import resources +except ImportError: # for Python 3.6 - import importlib_resources + import importlib_resources as resources import os -import pkg_resources import re

import sys @@ -96,12 +95,12 @@ def _ReadMissingBlobHelp(): msg = '' return tag, msg

- my_data = pkg_resources.resource_string(__name__, 'missing-blob-help') + my_data = resources.files(__package__).joinpath('missing-blob-help').read_text() re_tag = re.compile('^([-a-z0-9]+):$') result = {} tag = None msg = '' - for line in my_data.decode('utf-8').splitlines(): + for line in my_data.splitlines(): if not line.startswith('#'): m_tag = re_tag.match(line) if m_tag: @@ -151,8 +150,9 @@ def GetEntryModules(include_testing=True): Returns: Set of paths to entry class filenames """ - glob_list = pkg_resources.resource_listdir(__name__, 'etype') - glob_list = [fname for fname in glob_list if fname.endswith('.py')] + directory = resources.files("binman.etype") + glob_list = [entry.name for entry in directory.iterdir() + if entry.name.endswith('.py')] return set([os.path.splitext(os.path.basename(item))[0] for item in glob_list if include_testing or '_testing' not in item]) @@ -735,7 +735,7 @@ def Binman(args): global state

if args.full_help: - with importlib.resources.path('binman', 'README.rst') as readme: + with resources.path('binman', 'README.rst') as readme: tools.print_full_help(str(readme)) return 0

-- 2.45.2

Simon Glass

19 Jul 19 Jul

5:05 p.m.

New subject: [PATCH 01/18] binman: Replace pkg_resources with importlib.resources

On Wed, 17 Jul 2024 at 15:29, Jiaxun Yang jiaxun.yang@flygoat.com wrote:

...

pkg_resources is deprecated long ago and being removed in python 3.12.

Reimplement functions with importlib.resources.

Link: https://docs.python.org/3/library/importlib.resources.html Signed-off-by: Jiaxun Yang jiaxun.yang@flygoat.com

tools/binman/control.py | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-)

Reviewed-by: Simon Glass sjg@chromium.org

Jiaxun Yang

17 Jul 17 Jul

4:29 p.m.

New subject: [PATCH 02/18] py: Replace distutils.core with setuptools

distutils is deprecated long ago and being removed in python 3.12.

Signed-off-by: Jiaxun Yang jiaxun.yang@flygoat.com --- tools/binman/setup.py | 2 +- tools/buildman/requirements.txt | 1 + tools/dtoc/setup.py | 2 +- 3 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/tools/binman/setup.py b/tools/binman/setup.py index 9a9206eb044a..bec078a3d9b1 100644 --- a/tools/binman/setup.py +++ b/tools/binman/setup.py @@ -1,6 +1,6 @@ # SPDX-License-Identifier: GPL-2.0+

-from distutils.core import setup +from setuptools import setup setup(name='binman', version='1.0', license='GPL-2.0+', diff --git a/tools/buildman/requirements.txt b/tools/buildman/requirements.txt index 052d0ed5c6f7..0dadbf3fa2c5 100644 --- a/tools/buildman/requirements.txt +++ b/tools/buildman/requirements.txt @@ -3,3 +3,4 @@ jsonschema==4.17.3 pycryptodome==3.20 pyyaml==6.0 yamllint==1.26.3 +setuptools==65.5.1 diff --git a/tools/dtoc/setup.py b/tools/dtoc/setup.py index 5e092fe0872a..ae9ad043b013 100644 --- a/tools/dtoc/setup.py +++ b/tools/dtoc/setup.py @@ -1,6 +1,6 @@ # SPDX-License-Identifier: GPL-2.0+

-from distutils.core import setup +from setuptools import setup setup(name='dtoc', version='1.0', license='GPL-2.0+',

-- 2.45.2

Simon Glass

19 Jul 19 Jul

5:05 p.m.

New subject: [PATCH 02/18] py: Replace distutils.core with setuptools

On Wed, 17 Jul 2024 at 15:29, Jiaxun Yang jiaxun.yang@flygoat.com wrote:

...

distutils is deprecated long ago and being removed in python 3.12.

Signed-off-by: Jiaxun Yang jiaxun.yang@flygoat.com

tools/binman/setup.py | 2 +- tools/buildman/requirements.txt | 1 + tools/dtoc/setup.py | 2 +- 3 files changed, 3 insertions(+), 2 deletions(-)

Reviewed-by: Simon Glass sjg@chromium.org

But please separate out the requirements update into a separate patch.

Jiaxun Yang

17 Jul 17 Jul

4:29 p.m.

New subject: [PATCH 03/18] doc/sphinx: Remove usage of six

We don't support python2 any more so there is no point to use six here.

Signed-off-by: Jiaxun Yang jiaxun.yang@flygoat.com --- doc/sphinx/kfigure.py | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/doc/sphinx/kfigure.py b/doc/sphinx/kfigure.py index dea7f91ef5ab..9467e8d52ac0 100644 --- a/doc/sphinx/kfigure.py +++ b/doc/sphinx/kfigure.py @@ -58,7 +58,6 @@ from docutils.parsers.rst.directives import images import sphinx

from sphinx.util.nodes import clean_astext -from six import iteritems

import kernellog

@@ -540,7 +539,7 @@ def add_kernel_figure_to_std_domain(app, doctree): docname = app.env.docname labels = std.data["labels"]

- for name, explicit in iteritems(doctree.nametypes): + for name, explicit in doctree.nametypes.items(): if not explicit: continue labelid = doctree.nameids[name]

-- 2.45.2

Simon Glass

19 Jul 19 Jul

5:05 p.m.

New subject: [PATCH 03/18] doc/sphinx: Remove usage of six

On Wed, 17 Jul 2024 at 15:29, Jiaxun Yang jiaxun.yang@flygoat.com wrote:

...

We don't support python2 any more so there is no point to use six here.

Signed-off-by: Jiaxun Yang jiaxun.yang@flygoat.com

doc/sphinx/kfigure.py | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)

Reviewed-by: Simon Glass sjg@chromium.org

Jiaxun Yang

17 Jul 17 Jul

4:29 p.m.

New subject: [PATCH 04/18] py: Bump pylint version and clear warnings

Bump pylint to 3.2.3 as old versions are not working with python 3.12.

Clear warnings, mostly E0606: (possibly-used-before-assignment).

diff --git a/.azure-pipelines.yml b/.azure-pipelines.yml index e1b2f87b974a..4119ca7ff849 100644 --- a/.azure-pipelines.yml +++ b/.azure-pipelines.yml @@ -163,7 +163,7 @@ stages: export USER=azure pip install -r test/py/requirements.txt pip install -r tools/buildman/requirements.txt - pip install asteval pylint==2.12.2 pyopenssl + pip install asteval pylint==3.2.3 pyopenssl export PATH=${PATH}:~/.local/bin echo "[MASTER]" >> .pylintrc echo "load-plugins=pylint.extensions.docparams" >> .pylintrc diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 0a15b7352cdd..bdaf5db1da67 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -217,7 +217,7 @@ Run pylint: - git config --global --add safe.directory "${CI_PROJECT_DIR}" - pip install -r test/py/requirements.txt - pip install -r tools/buildman/requirements.txt - - pip install asteval pylint==2.12.2 pyopenssl + - pip install asteval pylint==3.2.3 pyopenssl - export PATH=${PATH}:~/.local/bin - echo "[MASTER]" >> .pylintrc - echo "load-plugins=pylint.extensions.docparams" >> .pylintrc diff --git a/doc/develop/python_cq.rst b/doc/develop/python_cq.rst index 1e209ff197d6..c8a75a5b7a7b 100644 --- a/doc/develop/python_cq.rst +++ b/doc/develop/python_cq.rst @@ -23,7 +23,7 @@ regressions in any module. To run this locally you should use this version of pylint::

# pylint --version - pylint 2.11.1 + pylint 3.2.3 astroid 2.8.6 Python 3.8.10 (default, Sep 28 2021, 16:10:42) [GCC 9.3.0] @@ -31,7 +31,7 @@ To run this locally you should use this version of pylint::

You should be able to select and this install other required tools with::

- pip install pylint==2.11.1 + pip install pylint==3.2.3 pip install -r test/py/requirements.txt pip install asteval pyopenssl

diff --git a/test/py/tests/test_ums.py b/test/py/tests/test_ums.py index 749b1606235c..55372e42a928 100644 --- a/test/py/tests/test_ums.py +++ b/test/py/tests/test_ums.py @@ -118,6 +118,7 @@ def test_ums(u_boot_console, env__usb_dev_port, env__block_devs):

test_f = u_boot_utils.PersistentRandomFile(u_boot_console, 'ums.bin', 1024 * 1024); + mounted_test_fn = None if have_writable_fs_partition: mounted_test_fn = mount_point + '/' + mount_subdir + test_f.fn

diff --git a/test/py/tests/test_usb.py b/test/py/tests/test_usb.py index fb3d20f0826b..27105cd1d5e1 100644 --- a/test/py/tests/test_usb.py +++ b/test/py/tests/test_usb.py @@ -564,6 +564,7 @@ def test_usb_load(u_boot_console): part_detect = 1 addr = u_boot_utils.find_ram_base(u_boot_console)

+ file, size = 0, 0 if fs == 'fat': file, size = test_usb_fatload_fatwrite(u_boot_console) elif fs == 'ext4': diff --git a/tools/binman/etype/fdtmap.py b/tools/binman/etype/fdtmap.py index f1f6217940f2..6b4ca497f871 100644 --- a/tools/binman/etype/fdtmap.py +++ b/tools/binman/etype/fdtmap.py @@ -106,6 +106,7 @@ class Entry_fdtmap(Entry): Returns: FDT map binary data """ + fsw = None def _AddNode(node): """Add a node to the FDT map""" for pname, prop in node.props.items(): diff --git a/tools/binman/etype/fit.py b/tools/binman/etype/fit.py index 2c14b15b03cd..dfbb6de7b63e 100644 --- a/tools/binman/etype/fit.py +++ b/tools/binman/etype/fit.py @@ -808,6 +808,7 @@ class Entry_fit(Entry_section): data_size = fdt_util.GetInt(node, "data-size")

# Contents are inside the FIT + offset, size = 0, 0 if data_prop is not None: # GetOffset() returns offset of a fdt_property struct, # which has 3 fdt32_t members before the actual data. diff --git a/tools/binman/etype/image_header.py b/tools/binman/etype/image_header.py index 240118849580..3db8e61d23a2 100644 --- a/tools/binman/etype/image_header.py +++ b/tools/binman/etype/image_header.py @@ -62,6 +62,7 @@ class Entry_image_header(Entry):

def _GetHeader(self): image_pos = self.GetSiblingImagePos('fdtmap') + offset = 0xffffffff if image_pos == False: self.Raise("'image_header' section must have an 'fdtmap' sibling") elif image_pos is None: diff --git a/tools/binman/etype/pre_load.py b/tools/binman/etype/pre_load.py index 2e4c72359ff3..c095cf425c93 100644 --- a/tools/binman/etype/pre_load.py +++ b/tools/binman/etype/pre_load.py @@ -112,6 +112,8 @@ class Entry_pre_load(Entry_collection): # Compute the signature if padding_name is None: padding_name = "pkcs-1.5" + + padding, padding_args = None, {} if padding_name == "pss": salt_len = key.size_in_bytes() - hash_image.digest_size - 2 padding = pss diff --git a/tools/binman/etype/ti_board_config.py b/tools/binman/etype/ti_board_config.py index c10d66edcb15..33c7a351c4ea 100644 --- a/tools/binman/etype/ti_board_config.py +++ b/tools/binman/etype/ti_board_config.py @@ -118,6 +118,7 @@ class Entry_ti_board_config(Entry_section): Returns: array of bytes representing value """ + br = None size = 0 if (data_type == '#/definitions/u8'): size = 1 diff --git a/tools/binman/etype/x509_cert.py b/tools/binman/etype/x509_cert.py index 29630d1b86c8..763cb506399b 100644 --- a/tools/binman/etype/x509_cert.py +++ b/tools/binman/etype/x509_cert.py @@ -83,6 +83,7 @@ class Entry_x509_cert(Entry_collection): output_fname = tools.get_output_filename('cert.%s' % uniq) input_fname = tools.get_output_filename('input.%s' % uniq) config_fname = tools.get_output_filename('config.%s' % uniq) + stdout = None tools.write_file(input_fname, input_data) if type == 'generic': stdout = self.openssl.x509_cert( diff --git a/tools/binman/ftest.py b/tools/binman/ftest.py index e4da04030a51..e9ad145e790d 100644 --- a/tools/binman/ftest.py +++ b/tools/binman/ftest.py @@ -6293,6 +6293,7 @@ fdt fdtmap Extract the devicetree blob from the fdtmap re_name = re.compile('_binman_(u_boot_(.*))_prop_(.*)') for name, sym in syms.items(): msg = 'test' + expect_val = None val = elf.GetSymbolValue(sym, edata, msg) entry_m = re_name.match(name) if entry_m: diff --git a/tools/binman/state.py b/tools/binman/state.py index 45bae40c525a..f43be6d4ce6e 100644 --- a/tools/binman/state.py +++ b/tools/binman/state.py @@ -406,6 +406,7 @@ def CheckSetHashValue(node, get_data_func): hash_node = node.FindNode('hash') if hash_node: algo = hash_node.props.get('algo').value + data = None if algo == 'sha256': m = hashlib.sha256() m.update(get_data_func()) diff --git a/tools/buildman/builder.py b/tools/buildman/builder.py index c4384f53e8dc..88ee1f34b54e 100644 --- a/tools/buildman/builder.py +++ b/tools/buildman/builder.py @@ -1068,6 +1068,7 @@ class Builder: printed_target = False for name in sorted(result): diff = result[name] + color = None if name.startswith('_'): continue if diff != 0: @@ -1328,6 +1329,7 @@ class Builder: for line in lines: if not line: continue + col = None if line[0] == '+': col = self.col.GREEN elif line[0] == '-': diff --git a/tools/microcode-tool.py b/tools/microcode-tool.py index 24c02c4fca14..5f0287736dc7 100755 --- a/tools/microcode-tool.py +++ b/tools/microcode-tool.py @@ -277,6 +277,7 @@ def MicrocodeTool(): if cmd not in commands: parser.error("Unknown command '%s'" % cmd)

+ date, license_text, microcodes = None, None, None if (not not options.mcfile) != (not not options.mcfile): parser.error("You must specify either header files or a microcode file, not both") if options.headerfile: diff --git a/tools/patman/test_checkpatch.py b/tools/patman/test_checkpatch.py index db7860f551d0..e2f596940d3d 100644 --- a/tools/patman/test_checkpatch.py +++ b/tools/patman/test_checkpatch.py @@ -530,4 +530,6 @@ index 0000000..2234c87

if __name__ == "__main__": unittest.main() + # pylint doesn't seem to find this + # pylint: disable=E1101 gitutil.RunTests() diff --git a/tools/qconfig.py b/tools/qconfig.py index 04118d942da6..2492b37444a3 100755 --- a/tools/qconfig.py +++ b/tools/qconfig.py @@ -873,6 +873,7 @@ def read_database(): all_defconfigs = set()

defconfig_db = collections.defaultdict(set) + defconfig = None for line in read_file(CONFIG_DATABASE): line = line.rstrip() if not line: # Separator between defconfigs

-- 2.45.2

Jiaxun Yang

4:29 p.m.

New subject: [PATCH 05/18] binman: Workaround lz4 cli padding in test cases

Newer lz4 util is not happy with any padding at end of file, it would abort with error message like:

Stream followed by undecodable data at position 43.

Workaround by skipping testCompUtilPadding test case and manually strip padding in testCompressSectionSize test case.

Signed-off-by: Jiaxun Yang jiaxun.yang@flygoat.com --- tools/binman/ftest.py | 7 +++++-- tools/binman/test/184_compress_section_size.dts | 1 + 2 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/tools/binman/ftest.py b/tools/binman/ftest.py index e9ad145e790d..0143d5e41737 100644 --- a/tools/binman/ftest.py +++ b/tools/binman/ftest.py @@ -4518,6 +4518,8 @@ class TestFunctional(unittest.TestCase): dtb.Scan() props = self._GetPropTree(dtb, ['offset', 'image-pos', 'size', 'uncomp-size']) + data = data[:0x30] + data = data.rstrip(b'\xff') orig = self._decompress(data) self.assertEqual(COMPRESS_DATA + U_BOOT_DATA, orig) expected = { @@ -6118,8 +6120,9 @@ fdt fdtmap Extract the devicetree blob from the fdtmap

def testCompUtilPadding(self): """Test padding of compression algorithms""" - # Skip zstd because it doesn't support padding - for bintool in [v for k,v in self.comp_bintools.items() if k != 'zstd']: + # Skip zstd and lz4 because they doesn't support padding + for bintool in [v for k,v in self.comp_bintools.items() + if not k in ['zstd', 'lz4']]: self._CheckBintool(bintool) data = bintool.compress(COMPRESS_DATA) self.assertNotEqual(COMPRESS_DATA, data) diff --git a/tools/binman/test/184_compress_section_size.dts b/tools/binman/test/184_compress_section_size.dts index 95ed30add1aa..1c1dbd5f580f 100644 --- a/tools/binman/test/184_compress_section_size.dts +++ b/tools/binman/test/184_compress_section_size.dts @@ -6,6 +6,7 @@ section { size = <0x30>; compress = "lz4"; + pad-byte = <0xff>; blob { filename = "compress"; };

-- 2.45.2

Simon Glass

19 Jul 19 Jul

5:05 p.m.

New subject: [PATCH 05/18] binman: Workaround lz4 cli padding in test cases

On Wed, 17 Jul 2024 at 15:29, Jiaxun Yang jiaxun.yang@flygoat.com wrote:

...

Newer lz4 util is not happy with any padding at end of file, it would abort with error message like:

Stream followed by undecodable data at position 43.

Workaround by skipping testCompUtilPadding test case and manually strip padding in testCompressSectionSize test case.

Signed-off-by: Jiaxun Yang jiaxun.yang@flygoat.com

tools/binman/ftest.py | 7 +++++-- tools/binman/test/184_compress_section_size.dts | 1 + 2 files changed, 6 insertions(+), 2 deletions(-)

Reviewed-by: Simon Glass sjg@chromium.org

...

diff --git a/tools/binman/ftest.py b/tools/binman/ftest.py index e9ad145e790d..0143d5e41737 100644 --- a/tools/binman/ftest.py +++ b/tools/binman/ftest.py @@ -4518,6 +4518,8 @@ class TestFunctional(unittest.TestCase): dtb.Scan() props = self._GetPropTree(dtb, ['offset', 'image-pos', 'size', 'uncomp-size'])
   data = data[:0x30]
   data = data.rstrip(b'\xff')
   orig = self._decompress(data)
   self.assertEqual(COMPRESS_DATA + U_BOOT_DATA, orig)
   expected = {
@@ -6118,8 +6120,9 @@ fdt fdtmap Extract the devicetree blob from the fdtmap
 def testCompUtilPadding(self):
     """Test padding of compression algorithms"""
   # Skip zstd because it doesn't support padding
   for bintool in [v for k,v in self.comp_bintools.items() if k != 'zstd']:
   # Skip zstd and lz4 because they doesn't support padding
   for bintool in [v for k,v in self.comp_bintools.items()
                   if not k in ['zstd', 'lz4']]:
       self._CheckBintool(bintool)
       data = bintool.compress(COMPRESS_DATA)
       self.assertNotEqual(COMPRESS_DATA, data)
diff --git a/tools/binman/test/184_compress_section_size.dts b/tools/binman/test/184_compress_section_size.dts index 95ed30add1aa..1c1dbd5f580f 100644 --- a/tools/binman/test/184_compress_section_size.dts +++ b/tools/binman/test/184_compress_section_size.dts @@ -6,6 +6,7 @@ section { size = <0x30>; compress = "lz4";
                  pad-byte = <0xff>;
                  blob {
                          filename = "compress";
                  };
-- 2.45.2

Jiaxun Yang

17 Jul 17 Jul

4:29 p.m.

New subject: [PATCH 06/18] tests/test_event_dump: Relax match rule for output

event_dump.py relies on addr2line to obtain source location information, however newer addr2line is unable to determine line numbers for some functions.

With addr2line from binutils 2.34 we got:

Event type Id Source location -------------------- ------------------------------ ------------------------------ EVT_FT_FIXUP bootmeth_vbe_ft_fixup :? EVT_FT_FIXUP bootmeth_vbe_simple_ft_fixup :? EVT_LAST_STAGE_INIT install_smbios_table :? EVT_MISC_INIT_F sandbox_early_getopt_check arch/sandbox/cpu/start.c:61 EVT_TEST h_adder_simple :?

Which will fail the test.

Relax the source location regex to .*:.*, this is sufficent to show that addr2line is being called and returned a possible line number.

Signed-off-by: Jiaxun Yang jiaxun.yang@flygoat.com --- test/py/tests/test_event_dump.py | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/test/py/tests/test_event_dump.py b/test/py/tests/test_event_dump.py index e282c67335cd..e87825abcd1a 100644 --- a/test/py/tests/test_event_dump.py +++ b/test/py/tests/test_event_dump.py @@ -16,9 +16,9 @@ def test_event_dump(u_boot_console): out = util.run_and_log(cons, ['scripts/event_dump.py', sandbox]) expect = '''.*Event type Id Source location -------------------- ------------------------------ ------------------------------ -EVT_FT_FIXUP bootmeth_vbe_ft_fixup .*boot/vbe_request.c:.* -EVT_FT_FIXUP bootmeth_vbe_simple_ft_fixup .*boot/vbe_simple_os.c:.* -EVT_LAST_STAGE_INIT install_smbios_table .*lib/efi_loader/efi_smbios.c:.* -EVT_MISC_INIT_F sandbox_early_getopt_check .*arch/sandbox/cpu/start.c:.* -EVT_TEST h_adder_simple .*test/common/event.c:''' +EVT_FT_FIXUP bootmeth_vbe_ft_fixup .*:.* +EVT_FT_FIXUP bootmeth_vbe_simple_ft_fixup .*:.* +EVT_LAST_STAGE_INIT install_smbios_table .*:.* +EVT_MISC_INIT_F sandbox_early_getopt_check .*:.* +EVT_TEST h_adder_simple .*:''' assert re.match(expect, out, re.MULTILINE) is not None

-- 2.45.2

Simon Glass

19 Jul 19 Jul

5:05 p.m.

New subject: [PATCH 06/18] tests/test_event_dump: Relax match rule for output

Hi Jiaxun,

On Wed, 17 Jul 2024 at 15:29, Jiaxun Yang jiaxun.yang@flygoat.com wrote:

...

event_dump.py relies on addr2line to obtain source location information, however newer addr2line is unable to determine line numbers for some functions.

Is this a bug in the tool?

...

With addr2line from binutils 2.34 we got:

Event type Id Source location

EVT_FT_FIXUP bootmeth_vbe_ft_fixup :? EVT_FT_FIXUP bootmeth_vbe_simple_ft_fixup :? EVT_LAST_STAGE_INIT install_smbios_table :? EVT_MISC_INIT_F sandbox_early_getopt_check arch/sandbox/cpu/start.c:61 EVT_TEST h_adder_simple :?

Which will fail the test.

Relax the source location regex to .*:.*, this is sufficent to show that addr2line is being called and returned a possible line number.

Signed-off-by: Jiaxun Yang jiaxun.yang@flygoat.com

test/py/tests/test_event_dump.py | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)

Reviewed-by: Simon Glass sjg@chromium.org

...

diff --git a/test/py/tests/test_event_dump.py b/test/py/tests/test_event_dump.py index e282c67335cd..e87825abcd1a 100644 --- a/test/py/tests/test_event_dump.py +++ b/test/py/tests/test_event_dump.py @@ -16,9 +16,9 @@ def test_event_dump(u_boot_console): out = util.run_and_log(cons, ['scripts/event_dump.py', sandbox]) expect = '''.*Event type Id Source location

-EVT_FT_FIXUP bootmeth_vbe_ft_fixup .*boot/vbe_request.c:.* -EVT_FT_FIXUP bootmeth_vbe_simple_ft_fixup .*boot/vbe_simple_os.c:.* -EVT_LAST_STAGE_INIT install_smbios_table .*lib/efi_loader/efi_smbios.c:.* -EVT_MISC_INIT_F sandbox_early_getopt_check .*arch/sandbox/cpu/start.c:.* -EVT_TEST h_adder_simple .*test/common/event.c:''' +EVT_FT_FIXUP bootmeth_vbe_ft_fixup .*:.* +EVT_FT_FIXUP bootmeth_vbe_simple_ft_fixup .*:.* +EVT_LAST_STAGE_INIT install_smbios_table .*:.* +EVT_MISC_INIT_F sandbox_early_getopt_check .*:.* +EVT_TEST h_adder_simple .*:''' assert re.match(expect, out, re.MULTILINE) is not None

-- 2.45.2

Regards, Simon

Jiaxun Yang

17 Jul 17 Jul

4:29 p.m.

New subject: [PATCH 07/18] lib/charset & efi: Fix possible unaligned accesses

As per armv7 arch spec, for A-profile CPU if translation is disabled, then the default memory type is Device(-nGnRnE) instead of Normal, which requires that alignment be enforced.

This means in some cases we can't perform unaligned access even after allow_unaligned is called.

We do have many platforms that didn't have translation enabled in U-Boot, and QEMU started to enforce this since 9.0.

Fix by using unaligned access helper for UTF-16 memory read/write to ensure we don't do any unaligned access in U-Boot.

Signed-off-by: Jiaxun Yang jiaxun.yang@flygoat.com --- lib/charset.c | 21 ++++++++++++--------- lib/efi_loader/efi_device_path.c | 11 ++--------- 2 files changed, 14 insertions(+), 18 deletions(-)

diff --git a/lib/charset.c b/lib/charset.c index 182c92a50c48..af5f3ad16d9b 100644 --- a/lib/charset.c +++ b/lib/charset.c @@ -11,6 +11,7 @@ #include <efi_loader.h> #include <errno.h> #include <malloc.h> +#include <asm/unaligned.h>

/** * codepage_437 - Unicode to codepage 437 translation table @@ -215,7 +216,7 @@ s32 utf16_get(const u16 **src) return -1; if (!**src) return 0; - code = **src; + code = get_unaligned_le16(*src); ++*src; if (code >= 0xDC00 && code <= 0xDFFF) return -1; @@ -242,12 +243,12 @@ int utf16_put(s32 code, u16 **dst) if ((code >= 0xD800 && code <= 0xDFFF) || code >= 0x110000) return -1; if (code < 0x10000) { - **dst = code; + put_unaligned_le16(code, *dst); } else { code -= 0x10000; - **dst = code >> 10 | 0xD800; + put_unaligned_le16(code >> 10 | 0xD800, *dst); ++*dst; - **dst = (code & 0x3ff) | 0xDC00; + put_unaligned_le16((code & 0x3ff) | 0xDC00, *dst); } ++*dst; return 0; @@ -392,7 +393,7 @@ int __efi_runtime u16_strncmp(const u16 *s1, const u16 *s2, size_t n) int ret = 0;

for (; n; --n, ++s1, ++s2) { - ret = *s1 - *s2; + ret = get_unaligned_le16(s1) - get_unaligned_le16(s2); if (ret || !*s1) break; } @@ -403,7 +404,7 @@ int __efi_runtime u16_strncmp(const u16 *s1, const u16 *s2, size_t n) size_t __efi_runtime u16_strnlen(const u16 *in, size_t count) { size_t i; - for (i = 0; count-- && in[i]; i++); + for (i = 0; count-- && get_unaligned_le16(in + i); i++); return i; }

@@ -417,8 +418,10 @@ u16 *u16_strcpy(u16 *dest, const u16 *src) u16 *tmp = dest;

for (;; dest++, src++) { - *dest = *src; - if (!*src) + u16 code = get_unaligned_le16(src); + + put_unaligned_le16(code, dest); + if (!code) break; }

@@ -463,7 +466,7 @@ uint8_t *utf16_to_utf8(uint8_t *dest, const uint16_t *src, size_t size) uint32_t code_high = 0;

while (size--) { - uint32_t code = *src++; + uint32_t code = get_unaligned_le16(src++);

if (code_high) { if (code >= 0xDC00 && code <= 0xDFFF) { diff --git a/lib/efi_loader/efi_device_path.c b/lib/efi_loader/efi_device_path.c index 0f684590f22a..97a3200574a1 100644 --- a/lib/efi_loader/efi_device_path.c +++ b/lib/efi_loader/efi_device_path.c @@ -18,7 +18,7 @@ #include <efi_loader.h> #include <part.h> #include <uuid.h> -#include <asm-generic/unaligned.h> +#include <asm/unaligned.h> #include <linux/compat.h> /* U16_MAX */

/* template END node: */ @@ -873,13 +873,6 @@ static void path_to_uefi(void *uefi, const char *src) { u16 *pos = uefi;

- /* - * efi_set_bootdev() calls this routine indirectly before the UEFI - * subsystem is initialized. So we cannot assume unaligned access to be - * enabled. - */ - allow_unaligned(); - while (*src) { s32 code = utf8_get(&src);

@@ -889,7 +882,7 @@ static void path_to_uefi(void *uefi, const char *src) code = '\'; utf16_put(code, &pos); } - *pos = 0; + put_unaligned_le16(0, pos); }

/**

-- 2.45.2

Jiaxun Yang

4:29 p.m.

New subject: [PATCH 08/18] CI: Ensure pip install is always performed in venv

Since Ubuntu focal it's nolonger permitted to perform global pip install.

Ensure that pip install is always performed in venv. For buildman alone, all dependencies are already in docker so there is no need to perform pip install.

Signed-off-by: Jiaxun Yang jiaxun.yang@flygoat.com --- .azure-pipelines.yml | 16 ++++++++++------ .gitlab-ci.yml | 13 ++++++++----- 2 files changed, 18 insertions(+), 11 deletions(-)

diff --git a/.azure-pipelines.yml b/.azure-pipelines.yml index 4119ca7ff849..e2346ace3097 100644 --- a/.azure-pipelines.yml +++ b/.azure-pipelines.yml @@ -161,6 +161,8 @@ stages: - script: | git config --global --add safe.directory $(work_dir) export USER=azure + virtualenv -p /usr/bin/python3 /tmp/venv + . /tmp/venv/bin/activate pip install -r test/py/requirements.txt pip install -r tools/buildman/requirements.txt pip install asteval pylint==3.2.3 pyopenssl @@ -194,7 +196,10 @@ stages: image: $(ci_runner_image) options: $(container_option) steps: - - script: make pip + - script: | + virtualenv -p /usr/bin/python3 /tmp/venv + . /tmp/venv/bin/activate + make pip

- job: count_built_machines displayName: 'Ensure we build all possible machines' @@ -256,7 +261,11 @@ stages: if [ -n "${BUILD_ENV}" ]; then export ${BUILD_ENV}; fi + virtualenv -p /usr/bin/python3 /tmp/venv + . /tmp/venv/bin/activate pip install -r tools/buildman/requirements.txt + pip install -r test/py/requirements.txt + pip install pytest-azurepipelines tools/buildman/buildman -o ${UBOOT_TRAVIS_BUILD_DIR} -w -E -W -e --board ${TEST_PY_BD} ${OVERRIDE} cp ~/grub_x86.efi ${UBOOT_TRAVIS_BUILD_DIR}/ cp ~/grub_x64.efi ${UBOOT_TRAVIS_BUILD_DIR}/ @@ -280,10 +289,6 @@ stages: /opt/coreboot/cbfstool ${UBOOT_TRAVIS_BUILD_DIR}/coreboot.rom remove -n fallback/payload; /opt/coreboot/cbfstool ${UBOOT_TRAVIS_BUILD_DIR}/coreboot.rom add-flat-binary -f ${UBOOT_TRAVIS_BUILD_DIR}/u-boot.bin -n fallback/payload -c LZMA -l 0x1110000 -e 0x1110000; fi - virtualenv -p /usr/bin/python3 /tmp/venv - . /tmp/venv/bin/activate - pip install -r test/py/requirements.txt - pip install pytest-azurepipelines export PATH=/opt/qemu/bin:/tmp/uboot-test-hooks/bin:${PATH} export PYTHONPATH=/tmp/uboot-test-hooks/py/travis-ci # "${var:+"-k $var"}" expands to "" if $var is empty, "-k $var" if not @@ -556,7 +561,6 @@ stages: # make environment variables available as tests are running inside a container export BUILDMAN="${BUILDMAN}" git config --global --add safe.directory ${WORK_DIR} - pip install -r tools/buildman/requirements.txt EOF cat << "EOF" >> build.sh if [[ "${BUILDMAN}" != "" ]]; then diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index bdaf5db1da67..b38342117434 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -50,6 +50,10 @@ stages: - if [ -n "${BUILD_ENV}" ]; then export ${BUILD_ENV}; fi + - virtualenv -p /usr/bin/python3 /tmp/venv + - . /tmp/venv/bin/activate + - pip install -r tools/buildman/requirements.txt + - pip install -r test/py/requirements.txt - tools/buildman/buildman -o ${UBOOT_TRAVIS_BUILD_DIR} -w -E -W -e --board ${TEST_PY_BD} ${OVERRIDE} - cp ~/grub_x86.efi $UBOOT_TRAVIS_BUILD_DIR/ @@ -74,9 +78,6 @@ stages: /opt/coreboot/cbfstool ${UBOOT_TRAVIS_BUILD_DIR}/coreboot.rom remove -n fallback/payload; /opt/coreboot/cbfstool ${UBOOT_TRAVIS_BUILD_DIR}/coreboot.rom add-flat-binary -f ${UBOOT_TRAVIS_BUILD_DIR}/u-boot.bin -n fallback/payload -c LZMA -l 0x1110000 -e 0x1110000; fi - - virtualenv -p /usr/bin/python3 /tmp/venv - - . /tmp/venv/bin/activate - - pip install -r test/py/requirements.txt # "${var:+"-k $var"}" expands to "" if $var is empty, "-k $var" if not - export PATH=/opt/qemu/bin:/tmp/uboot-test-hooks/bin:${PATH}; export PYTHONPATH=/tmp/uboot-test-hooks/py/travis-ci; @@ -103,7 +104,6 @@ build all 32bit ARM platforms: script: - ret=0; git config --global --add safe.directory "${CI_PROJECT_DIR}"; - pip install -r tools/buildman/requirements.txt; ./tools/buildman/buildman -o /tmp -PEWM arm -x aarch64 || ret=$?; if [[ $ret -ne 0 ]]; then ./tools/buildman/buildman -o /tmp -seP; @@ -117,7 +117,6 @@ build all 64bit ARM platforms: - . /tmp/venv/bin/activate - ret=0; git config --global --add safe.directory "${CI_PROJECT_DIR}"; - pip install -r tools/buildman/requirements.txt; ./tools/buildman/buildman -o /tmp -PEWM aarch64 || ret=$?; if [[ $ret -ne 0 ]]; then ./tools/buildman/buildman -o /tmp -seP; @@ -215,6 +214,8 @@ Run pylint: extends: .testsuites script: - git config --global --add safe.directory "${CI_PROJECT_DIR}" + - virtualenv -p /usr/bin/python3 /tmp/venv; + - . /tmp/venv/bin/activate; - pip install -r test/py/requirements.txt - pip install -r tools/buildman/requirements.txt - pip install asteval pylint==3.2.3 pyopenssl @@ -243,6 +244,8 @@ Check for pre-schema tags: Check packing of Python tools: extends: .testsuites script: + - virtualenv -p /usr/bin/python3 /tmp/venv; + - . /tmp/venv/bin/activate; - make pip

# Test sandbox with test.py

-- 2.45.2

Jiaxun Yang

4:29 p.m.

New subject: [PATCH 09/18] CI: Dockerfile: Set global git name & email config

Set global git name & email config so we don't have to setup it for every project.

Signed-off-by: Jiaxun Yang jiaxun.yang@flygoat.com --- tools/docker/Dockerfile | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-)

diff --git a/tools/docker/Dockerfile b/tools/docker/Dockerfile index cda87354566d..5824d371f259 100644 --- a/tools/docker/Dockerfile +++ b/tools/docker/Dockerfile @@ -121,6 +121,10 @@ RUN apt-get update && apt-get install -y \ zip \ && rm -rf /var/lib/apt/lists/*

+# Setup Git +RUN git config --global user.name "U-Boot CI" && \ + git config --global user.email u-boot@denx.de + # Make kernels readable for libguestfs tools to work correctly RUN chmod +r /boot/vmlinu*

@@ -128,8 +132,6 @@ RUN chmod +r /boot/vmlinu* RUN git clone git://git.savannah.gnu.org/grub.git /tmp/grub && \ cd /tmp/grub && \ git checkout grub-2.06 && \ - git config --global user.name "GitLab CI Runner" && \ - git config --global user.email trini@konsulko.com && \ git cherry-pick 049efdd72eb7baa7b2bf8884391ee7fe650da5a0 && \ git cherry-pick 403d6540cd608b2706cfa0cb4713f7e4b490ff45 && \ ./bootstrap && \ @@ -180,9 +182,6 @@ RUN git clone git://git.savannah.gnu.org/grub.git /tmp/grub && \ RUN git clone https://gitlab.com/qemu-project/qemu.git /tmp/qemu && \ cd /tmp/qemu && \ git checkout v8.2.0 && \ - # config user.name and user.email to make 'git am' happy - git config user.name u-boot && \ - git config user.email u-boot@denx.de && \ git format-patch 0c7ffc977195~..0c7ffc977195 && \ git am 0001-hw-net-cadence_gem-Fix-MDIO_OP_xxx-values.patch && \ git cherry-pick d3c79c3974 && \

-- 2.45.2

Jiaxun Yang

4:29 p.m.

New subject: [PATCH 10/18] CI: Dockerfile: Bump base OS version

Bump base os to latest jammy release and install more required dependencies.

Signed-off-by: Jiaxun Yang jiaxun.yang@flygoat.com --- tools/docker/Dockerfile | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/tools/docker/Dockerfile b/tools/docker/Dockerfile index 5824d371f259..0bf2d180e255 100644 --- a/tools/docker/Dockerfile +++ b/tools/docker/Dockerfile @@ -2,7 +2,7 @@ # This Dockerfile is used to build an image containing basic stuff to be used # to build U-Boot and run our test suites.

-FROM ubuntu:jammy-20240227 +FROM ubuntu:jammy-20240627.1 MAINTAINER Tom Rini trini@konsulko.com LABEL Description=" This image is for building U-Boot inside a container"

@@ -55,6 +55,7 @@ RUN apt-get update && apt-get install -y \ gawk \ gdisk \ git \ + gnat \ gnu-efi \ gnutls-dev \ graphviz \ @@ -98,10 +99,13 @@ RUN apt-get update && apt-get install -y \ python2.7 \ python3 \ python3-dev \ + python3-jsonschema \ python3-pip \ python3-pyelftools \ python3-sphinx \ + python3-venv \ python3-virtualenv \ + python3-yaml \ rpm2cpio \ sbsigntool \ socat \ @@ -118,6 +122,7 @@ RUN apt-get update && apt-get install -y \ vboot-utils \ xilinx-bootgen \ xxd \ + yamllint \ zip \ && rm -rf /var/lib/apt/lists/*

-- 2.45.2

Jiaxun Yang

4:29 p.m.

New subject: [PATCH 11/18] CI: Dockerfile: Bump GRUB to 2.12

Signed-off-by: Jiaxun Yang jiaxun.yang@flygoat.com --- tools/docker/Dockerfile | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/tools/docker/Dockerfile b/tools/docker/Dockerfile index 0bf2d180e255..8e144a2b64e9 100644 --- a/tools/docker/Dockerfile +++ b/tools/docker/Dockerfile @@ -136,9 +136,7 @@ RUN chmod +r /boot/vmlinu* # Build GRUB UEFI targets for ARM & RISC-V, 32-bit and 64-bit RUN git clone git://git.savannah.gnu.org/grub.git /tmp/grub && \ cd /tmp/grub && \ - git checkout grub-2.06 && \ - git cherry-pick 049efdd72eb7baa7b2bf8884391ee7fe650da5a0 && \ - git cherry-pick 403d6540cd608b2706cfa0cb4713f7e4b490ff45 && \ + git checkout grub-2.12 && \ ./bootstrap && \ mkdir -p /opt/grub && \ ./configure --target=aarch64 --with-platform=efi \

-- 2.45.2

Jiaxun Yang

4:29 p.m.

New subject: [PATCH 12/18] CI: Dockerfile: Bump QEMU to 9.0.1

Previous patches had been applied in 9.0.1 so remove cherry-picks as well.

Signed-off-by: Jiaxun Yang jiaxun.yang@flygoat.com --- tools/docker/Dockerfile | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-)

diff --git a/tools/docker/Dockerfile b/tools/docker/Dockerfile index 8e144a2b64e9..8e242d08297a 100644 --- a/tools/docker/Dockerfile +++ b/tools/docker/Dockerfile @@ -184,10 +184,7 @@ RUN git clone git://git.savannah.gnu.org/grub.git /tmp/grub && \

RUN git clone https://gitlab.com/qemu-project/qemu.git /tmp/qemu && \ cd /tmp/qemu && \ - git checkout v8.2.0 && \ - git format-patch 0c7ffc977195~..0c7ffc977195 && \ - git am 0001-hw-net-cadence_gem-Fix-MDIO_OP_xxx-values.patch && \ - git cherry-pick d3c79c3974 && \ + git checkout v9.0.1 && \ ./configure --prefix=/opt/qemu --target-list="aarch64-softmmu,arm-softmmu,i386-softmmu,m68k-softmmu,mips-softmmu,mips64-softmmu,mips64el-softmmu,mipsel-softmmu,ppc-softmmu,riscv32-softmmu,riscv64-softmmu,sh4-softmmu,x86_64-softmmu,xtensa-softmmu" && \ make -j$(nproc) all install && \ rm -rf /tmp/qemu

-- 2.45.2

Jiaxun Yang

4:29 p.m.

New subject: [PATCH 13/18] CI: Dockerfile: Bump fiptool to 2.10.4

Signed-off-by: Jiaxun Yang jiaxun.yang@flygoat.com --- tools/docker/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/docker/Dockerfile b/tools/docker/Dockerfile index 8e242d08297a..91256f0248e4 100644 --- a/tools/docker/Dockerfile +++ b/tools/docker/Dockerfile @@ -192,7 +192,7 @@ RUN git clone https://gitlab.com/qemu-project/qemu.git /tmp/qemu && \ # Build fiptool RUN git clone https://git.trustedfirmware.org/TF-A/trusted-firmware-a.git /tmp/tf-a && \ cd /tmp/tf-a/ && \ - git checkout v2.10.0 && \ + git checkout lts-v2.10.4 && \ cd tools/fiptool && \ make && \ mkdir -p /usr/local/bin && \

-- 2.45.2

Jiaxun Yang

4:29 p.m.

New subject: [PATCH 14/18] CI: Dockerfile: Bump genimage to 17

Signed-off-by: Jiaxun Yang jiaxun.yang@flygoat.com --- tools/docker/Dockerfile | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/tools/docker/Dockerfile b/tools/docker/Dockerfile index 91256f0248e4..cc6e9dc4812d 100644 --- a/tools/docker/Dockerfile +++ b/tools/docker/Dockerfile @@ -200,12 +200,12 @@ RUN git clone https://git.trustedfirmware.org/TF-A/trusted-firmware-a.git /tmp/t rm -rf /tmp/tf-a

# Build genimage (required by some targets to generate disk images) -RUN wget -O - https://github.com/pengutronix/genimage/releases/download/v14/genimage-14.ta... | tar -C /tmp -xJ && \ - cd /tmp/genimage-14 && \ +RUN wget -O - https://github.com/pengutronix/genimage/releases/download/v17/genimage-17.ta... | tar -C /tmp -xJ && \ + cd /tmp/genimage-17 && \ ./configure && \ make -j$(nproc) && \ make install && \ - rm -rf /tmp/genimage-14 + rm -rf /tmp/genimage-17

# Build libtpms RUN git clone https://github.com/stefanberger/libtpms /tmp/libtpms && \

-- 2.45.2

Jiaxun Yang

4:29 p.m.

New subject: [PATCH 15/18] CI: Dockerfile: Use kernel.org upstream for trace-cmd

This is project's upstream URL.

Signed-off-by: Jiaxun Yang jiaxun.yang@flygoat.com --- tools/docker/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/docker/Dockerfile b/tools/docker/Dockerfile index cc6e9dc4812d..f45283d72441 100644 --- a/tools/docker/Dockerfile +++ b/tools/docker/Dockerfile @@ -236,7 +236,7 @@ RUN mkdir /tmp/trace && \ cd /tmp/trace/libtracefs && \ make -j$(nproc) && \ sudo make install && \ - git clone https://github.com/rostedt/trace-cmd.git /tmp/trace/trace-cmd && \ + git clone https://git.kernel.org/pub/scm/utils/trace-cmd/trace-cmd.git /tmp/trace/trace-cmd && \ cd /tmp/trace/trace-cmd && \ make -j$(nproc) && \ sudo make install && \

-- 2.45.2

Jiaxun Yang

4:29 p.m.

New subject: [PATCH 16/18] CI: Dockerfile: Bump coreboot to 24.05

Signed-off-by: Jiaxun Yang jiaxun.yang@flygoat.com --- tools/docker/Dockerfile | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/tools/docker/Dockerfile b/tools/docker/Dockerfile index f45283d72441..5ff1ac449e1f 100644 --- a/tools/docker/Dockerfile +++ b/tools/docker/Dockerfile @@ -243,15 +243,17 @@ RUN mkdir /tmp/trace && \ rm -rf /tmp/trace

# Build coreboot -RUN wget -O - https://coreboot.org/releases/coreboot-4.22.01.tar.xz | tar -C /tmp -xJ && \ - cd /tmp/coreboot-4.22.01 && \ +RUN wget -O - https://coreboot.org/releases/coreboot-24.05.tar.xz | tar -C /tmp -xJ && \ + cd /tmp/coreboot-24.05 && \ make crossgcc-i386 CPUS=$(nproc) && \ make -C payloads/coreinfo olddefconfig && \ make -C payloads/coreinfo && \ make olddefconfig && \ make -j $(nproc) && \ sudo mkdir /opt/coreboot && \ - sudo cp build/coreboot.rom build/cbfstool /opt/coreboot/ + sudo cp build/coreboot.rom build/cbfstool /opt/coreboot/ && \ + rm -rf /tmp/coreboot-24.05 +

# Create our user/group RUN echo uboot ALL=NOPASSWD: ALL > /etc/sudoers.d/uboot

-- 2.45.2

Jiaxun Yang

4:29 p.m.

New subject: [PATCH 17/18] CI: Dockerfile: Add LoongArch64 support

Install LoongArch64 toolchains, build LoongArch64 QEMU, build LoongArch64 GRUB.

Signed-off-by: Jiaxun Yang jiaxun.yang@flygoat.com --- tools/docker/Dockerfile | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-)

diff --git a/tools/docker/Dockerfile b/tools/docker/Dockerfile index 5ff1ac449e1f..3522c3627af9 100644 --- a/tools/docker/Dockerfile +++ b/tools/docker/Dockerfile @@ -19,6 +19,7 @@ RUN wget -O - https://mirrors.edge.kernel.org/pub/tools/crosstool/files/bin/x86_ RUN wget -O - https://mirrors.edge.kernel.org/pub/tools/crosstool/files/bin/x86_64/13.2.0/... | tar -C /opt -xJ RUN wget -O - https://mirrors.edge.kernel.org/pub/tools/crosstool/files/bin/x86_64/13.2.0/... | tar -C /opt -xJ RUN wget -O - https://mirrors.edge.kernel.org/pub/tools/crosstool/files/bin/x86_64/13.2.0/... | tar -C /opt -xJ +RUN wget -O - https://mirrors.edge.kernel.org/pub/tools/crosstool/files/bin/x86_64/13.2.0/... | tar -C /opt -xJ RUN wget -O - https://mirrors.edge.kernel.org/pub/tools/crosstool/files/bin/x86_64/13.2.0/... | tar -C /opt -xJ RUN wget -O - https://mirrors.edge.kernel.org/pub/tools/crosstool/files/bin/x86_64/13.2.0/... | tar -C /opt -xJ RUN wget -O - https://mirrors.edge.kernel.org/pub/tools/crosstool/files/bin/x86_64/13.2.0/... | tar -C /opt -xJ @@ -133,7 +134,7 @@ RUN git config --global user.name "U-Boot CI" && \ # Make kernels readable for libguestfs tools to work correctly RUN chmod +r /boot/vmlinu*

-# Build GRUB UEFI targets for ARM & RISC-V, 32-bit and 64-bit +# Build GRUB UEFI targets for ARM & LoongArch64 & RISC-V, 32-bit and 64-bit RUN git clone git://git.savannah.gnu.org/grub.git /tmp/grub && \ cd /tmp/grub && \ git checkout grub-2.12 && \ @@ -167,6 +168,20 @@ RUN git clone git://git.savannah.gnu.org/grub.git /tmp/grub && \ search search_fs_file search_fs_uuid search_label serial sleep test \ true && \ make clean && \ + ./configure --target=loongarch64 --with-platform=efi \ + CC=gcc \ + TARGET_CC=/opt/gcc-13.2.0-nolibc/loongarch64-linux/bin/loongarch64-linux-gcc \ + TARGET_OBJCOPY=/opt/gcc-13.2.0-nolibc/loongarch64-linux/bin/loongarch64-linux-objcopy \ + TARGET_STRIP=/opt/gcc-13.2.0-nolibc/loongarch64-linux/bin/loongarch64-linux-strip \ + TARGET_NM=/opt/gcc-13.2.0-nolibc/loongarch64-linux/bin/loongarch64-linux-nm \ + TARGET_RANLIB=/opt/gcc-13.2.0-nolibc/loongarch64-linux/bin/loongarch64-linux-ranlib && \ + make && \ + ./grub-mkimage -O loongarch64-efi -o /opt/grub/grubloongarch64.efi --prefix= -d \ + grub-core cat chain configfile echo efinet ext2 fat halt help linux \ + lsefisystab loadenv lvm minicmd normal part_msdos part_gpt reboot \ + search search_fs_file search_fs_uuid search_label serial sleep test \ + true && \ + make clean && \ ./configure --target=riscv64 --with-platform=efi \ CC=gcc \ TARGET_CC=/opt/gcc-13.2.0-nolibc/riscv64-linux/bin/riscv64-linux-gcc \ @@ -185,7 +200,7 @@ RUN git clone git://git.savannah.gnu.org/grub.git /tmp/grub && \ RUN git clone https://gitlab.com/qemu-project/qemu.git /tmp/qemu && \ cd /tmp/qemu && \ git checkout v9.0.1 && \ - ./configure --prefix=/opt/qemu --target-list="aarch64-softmmu,arm-softmmu,i386-softmmu,m68k-softmmu,mips-softmmu,mips64-softmmu,mips64el-softmmu,mipsel-softmmu,ppc-softmmu,riscv32-softmmu,riscv64-softmmu,sh4-softmmu,x86_64-softmmu,xtensa-softmmu" && \ + ./configure --prefix=/opt/qemu --target-list="aarch64-softmmu,arm-softmmu,i386-softmmu,loongarch64-softmmu,m68k-softmmu,mips-softmmu,mips64-softmmu,mips64el-softmmu,mipsel-softmmu,ppc-softmmu,riscv32-softmmu,riscv64-softmmu,sh4-softmmu,x86_64-softmmu,xtensa-softmmu" && \ make -j$(nproc) all install && \ rm -rf /tmp/qemu

-- 2.45.2

Jiaxun Yang

4:29 p.m.

New subject: [PATCH NFC 18/18] Use Jiaxun's CI Image

Use Jiaxun's CI Image for demonstration.

NOT FOR COMMIT!

Signed-off-by: Jiaxun Yang jiaxun.yang@flygoat.com --- .azure-pipelines.yml | 4 ++-- .gitlab-ci.yml | 4 ++-- tools/docker/Dockerfile | 6 +++--- 3 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/.azure-pipelines.yml b/.azure-pipelines.yml index e2346ace3097..e5fa57c3a222 100644 --- a/.azure-pipelines.yml +++ b/.azure-pipelines.yml @@ -2,7 +2,7 @@ variables: windows_vm: windows-2019 ubuntu_vm: ubuntu-22.04 macos_vm: macOS-12 - ci_runner_image: trini/u-boot-gitlab-ci-runner:jammy-20240227-14Mar2024 + ci_runner_image: ghcr.io/flygoat/u-boot-ci-docker:main # Add '-u 0' options for Azure pipelines, otherwise we get "permission # denied" error when it tries to "useradd -m -u 1001 vsts_azpcontainer", # since our $(ci_runner_image) user is not root. @@ -242,7 +242,7 @@ stages: # the below corresponds to .gitlab-ci.yml "before_script" cd ${WORK_DIR} git config --global --add safe.directory ${WORK_DIR} - git clone --depth=1 https://source.denx.de/u-boot/u-boot-test-hooks /tmp/uboot-test-hooks + git clone --depth=1 https://github.com/FlyGoat/u-boot-test-hooks.git /tmp/uboot-test-hooks ln -s travis-ci /tmp/uboot-test-hooks/bin/`hostname` ln -s travis-ci /tmp/uboot-test-hooks/py/`hostname` grub-mkimage --prefix="" -o ~/grub_x86.efi -O i386-efi normal echo lsefimmap lsefi lsefisystab efinet tftp minicmd diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index b38342117434..570ced808830 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -10,7 +10,7 @@ default:

# Grab our configured image. The source for this is found # in the u-boot tree at tools/docker/Dockerfile -image: ${MIRROR_DOCKER}/trini/u-boot-gitlab-ci-runner:jammy-20240227-14Mar2024 +image: ghcr.io/flygoat/u-boot-ci-docker:main

# We run some tests in different order, to catch some failures quicker. stages: @@ -26,7 +26,7 @@ stages: before_script: # Clone uboot-test-hooks - git config --global --add safe.directory "${CI_PROJECT_DIR}" - - git clone --depth=1 https://source.denx.de/u-boot/u-boot-test-hooks /tmp/uboot-test-hooks + - git clone --depth=1 https://github.com/FlyGoat/u-boot-test-hooks.git /tmp/uboot-test-hooks - ln -s travis-ci /tmp/uboot-test-hooks/bin/`hostname` - ln -s travis-ci /tmp/uboot-test-hooks/py/`hostname` - grub-mkimage --prefix="" -o ~/grub_x86.efi -O i386-efi normal echo lsefimmap lsefi lsefisystab efinet tftp minicmd diff --git a/tools/docker/Dockerfile b/tools/docker/Dockerfile index 3522c3627af9..bcc7f619f2ec 100644 --- a/tools/docker/Dockerfile +++ b/tools/docker/Dockerfile @@ -277,9 +277,9 @@ USER uboot:uboot

# Populate the cache for pip to use. Get these via wget as the # COPY / ADD directives don't work as we need them to. -RUN wget -O /tmp/pytest-requirements.txt https://source.denx.de/u-boot/u-boot/-/raw/master/test/py/requirements.txt -RUN wget -O /tmp/sphinx-requirements.txt https://source.denx.de/u-boot/u-boot/-/raw/master/doc/sphinx/requirements.tx... -RUN wget -O /tmp/buildman-requirements.txt https://source.denx.de/u-boot/u-boot/-/raw/master/tools/buildman/requirement... +RUN wget -O /tmp/pytest-requirements.txt https://gitlab.com/FlyGoat/u-boot/-/raw/b4/docker-image/test/py/requirements... +RUN wget -O /tmp/sphinx-requirements.txt https://gitlab.com/FlyGoat/u-boot/-/raw/b4/docker-image/doc/sphinx/requireme... +RUN wget -O /tmp/buildman-requirements.txt https://gitlab.com/FlyGoat/u-boot/-/raw/b4/docker-image/tools/buildman/requi... RUN virtualenv -p /usr/bin/python3 /tmp/venv && \ . /tmp/venv/bin/activate && \ pip install -r /tmp/pytest-requirements.txt \

-- 2.45.2

Tom Rini

18 Jul 18 Jul

4:36 p.m.

On Wed, Jul 17, 2024 at 10:29:11PM +0800, Jiaxun Yang wrote:

...

Hi all,

This series build a new CI image based on Ubuntu focal with LoongArch64 support, fixed various python scripts for python 3.12, fixed various problems popped up when testing againt latest software.

Last two commits are for demonstration purpose and not for commit into repo.

CI runs passed at azure [1].

Thanks [1]: https://flygoat.visualstudio.com/u-boot/_build/results?buildId=64&view=r...

Signed-off-by: Jiaxun Yang jiaxun.yang@flygoat.com

Overall this looks good but please split this up as:

...

Jiaxun Yang (18): binman: Replace pkg_resources with importlib.resources py: Replace distutils.core with setuptools doc/sphinx: Remove usage of six py: Bump pylint version and clear warnings binman: Workaround lz4 cli padding in test cases tests/test_event_dump: Relax match rule for output lib/charset & efi: Fix possible unaligned accesses

The binman patches (and change py: Replace.. to tool: Replace..). The doc/sphinx patch, standalone. Break the pylint patch up in to chunks fixing various tools, and then standalone bump the version. The dump test change by itself. The unaligned access change by itself.

...

  CI: Ensure pip install is always performed in venv
  CI: Dockerfile: Set global git name & email config
  CI: Dockerfile: Bump base OS version
  CI: Dockerfile: Bump GRUB to 2.12
  CI: Dockerfile: Bump QEMU to 9.0.1
  CI: Dockerfile: Bump fiptool to 2.10.4
  CI: Dockerfile: Bump genimage to 17
  CI: Dockerfile: Use kernel.org upstream for trace-cmd
  CI: Dockerfile: Bump coreboot to 24.05
  CI: Dockerfile: Add LoongArch64 support

These are their own series.

...

  [NFC] Use Jiaxun's CI Image

Just don't send this part next time.

Thanks!

-- Tom

291

Age (days ago)

293

Last active (days ago)

List overview

Download

24 comments

3 participants

tags (0)

participants (3)

Jiaxun Yang
Simon Glass
Tom Rini