Convert test/py/README.md to restructured text and add it to the generated HTML documentation.
Signed-off-by: Heinrich Schuchardt xypron.glpk@gmx.de --- doc/develop/index.rst | 1 + doc/develop/py_testing.rst | 426 +++++++++++++++++++++++++++++++++++++ test/py/README.md | 389 --------------------------------- 3 files changed, 427 insertions(+), 389 deletions(-) create mode 100644 doc/develop/py_testing.rst delete mode 100644 test/py/README.md
diff --git a/doc/develop/index.rst b/doc/develop/index.rst index ed2e73bf56..83eb1307ed 100644 --- a/doc/develop/index.rst +++ b/doc/develop/index.rst @@ -30,3 +30,4 @@ Testing
coccinelle testing + py_testing diff --git a/doc/develop/py_testing.rst b/doc/develop/py_testing.rst new file mode 100644 index 0000000000..f71e837aa9 --- /dev/null +++ b/doc/develop/py_testing.rst @@ -0,0 +1,426 @@ +U-Boot pytest suite +=================== + +Introduction +------------ + +This tool aims to test U-Boot by executing U-Boot shell commands using the +console interface. A single top-level script exists to execute or attach to the +U-Boot console, run the entire script of tests against it, and summarize the +results. Advantages of this approach are: + +- Testing is performed in the same way a user or script would interact with + U-Boot; there can be no disconnect. +- There is no need to write or embed test-related code into U-Boot itself. + It is asserted that writing test-related code in Python is simpler and more + flexible than writing it all in C. +- It is reasonably simple to interact with U-Boot in this way. + +Requirements +------------ + +The test suite is implemented using pytest. Interaction with the U-Boot console +involves executing some binary and interacting with its stdin/stdout. You will +need to implement various "hook" scripts that are called by the test suite at +the appropriate time. + +In order to run the test suite at a minimum we require that both Python 3 and +pip for Python 3 are installed. All of the required python modules are +described in the requirements.txt file in the /test/py/ directory and can be +installed via the command + +.. code-block:: bash + + pip install -r requirements.txt + +In order to execute certain tests on their supported platforms other tools +will be required. The following is an incomplete list: + +* gdisk +* dfu-util +* dtc +* openssl +* sudo OR guestmount +* e2fsprogs +* util-linux +* coreutils +* dosfstools +* efitools +* mount +* mtools +* sbsigntool +* udisks2 + +Please use the appropriate commands for your distribution to match these tools +up with the package that provides them. + +The test script supports either: + +- Executing a sandbox port of U-Boot on the local machine as a sub-process, + and interacting with it over stdin/stdout. +- Executing an external "hook" scripts to flash a U-Boot binary onto a + physical board, attach to the board's console stream, and reset the board. + Further details are described later. + +Using `virtualenv` to provide requirements +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The recommended way to run the test suite, in order to ensure reproducibility +is to use `virtualenv` to set up the necessary environment. This can be done +via the following commands: + + +.. code-block:: console + + $ cd /path/to/u-boot + $ sudo apt-get install python3 python3-virtualenv + $ virtualenv -p /usr/bin/python3 venv + $ . ./venv/bin/activate + $ pip install -r test/py/requirements.txt + +Testing sandbox +--------------- + +To run the test suite on the sandbox port (U-Boot built as a native user-space +application), simply execute: + +.. code-block:: bash + + ./test/py/test.py --bd sandbox --build + +The `--bd` option tells the test suite which board type is being tested. This +lets the test suite know which features the board has, and hence exactly what +can be tested. + +The `--build` option tells U-Boot to compile U-Boot. Alternatively, you may +omit this option and build U-Boot yourself, in whatever way you choose, before +running the test script. + +The test script will attach to U-Boot, execute all valid tests for the board, +then print a summary of the test process. A complete log of the test session +will be written to `${build_dir}/test-log.html`. This is best viewed in a web +browser, but may be read directly as plain text, perhaps with the aid of the +`html2text` utility. + +Testing under a debugger +~~~~~~~~~~~~~~~~~~~~~~~~ + +If you need to run sandbox under a debugger, you may pass the command-line +option `--gdbserver COMM`. This causes two things to happens: + +- Instead of running U-Boot directly, it will be run under gdbserver, with + debug communication via the channel `COMM`. You can attach a debugger to the + sandbox process in order to debug it. See `man gdbserver` and the example + below for details of valid values for `COMM`. +- All timeouts in tests are disabled, allowing U-Boot an arbitrary amount of + time to execute commands. This is useful if U-Boot is stopped at a breakpoint + during debugging. + +A usage example is: + +Window 1: + +.. code-block:: bash + + ./test/py/test.py --bd sandbox --gdbserver localhost:1234 + +Window 2: + +.. code-block:: bash + + gdb ./build-sandbox/u-boot -ex 'target remote localhost:1234' + +Alternatively, you could leave off the `-ex` option and type the command +manually into gdb once it starts. + +You can use any debugger you wish, as long as it speaks the gdb remote +protocol, or any graphical wrapper around gdb. + +Some tests deliberately cause the sandbox process to exit, e.g. to test the +reset command, or sandbox's CTRL-C handling. When this happens, you will need +to attach the debugger to the new sandbox instance. If these tests are not +relevant to your debugging session, you can skip them using pytest's -k +command-line option; see the next section. + +Command-line options +-------------------- + +--board-type, --bd, -B + set the type of the board to be tested. For example, `sandbox` or `seaboard`. + +--board-identity`, --id + sets the identity of the board to be tested. This allows differentiation + between multiple instances of the same type of physical board that are + attached to the same host machine. This parameter is not interpreted by th + test script in any way, but rather is simply passed to the hook scripts + described below, and may be used in any site-specific way deemed necessary. + +--build + indicates that the test script should compile U-Boot itself before running + the tests. If using this option, make sure that any environment variables + required by the build process are already set, such as `$CROSS_COMPILE`. + +--buildman + indicates that `--build` should use buildman to build U-Boot. There is no need + to set $CROSS_COMPILE` in this case since buildman handles it. + +--build-dir + sets the directory containing the compiled U-Boot binaries. If omitted, this + is `${source_dir}/build-${board_type}`. + +--result-dir + sets the directory to write results, such as log files, into. + If omitted, the build directory is used. + +--persistent-data-dir + sets the directory used to store persistent test data. This is test data that + may be re-used across test runs, such as file-system images. + +`pytest` also implements a number of its own command-line options. Commonly used +options are mentioned below. Please see `pytest` documentation for complete +details. Execute `py.test --version` for a brief summary. Note that U-Boot's +test.py script passes all command-line arguments directly to `pytest` for +processing. + +-k + selects which tests to run. The default is to run all known tests. This + option takes a single argument which is used to filter test names. Simple + logical operators are supported. For example: + + - `'-k ums'` runs only tests with "ums" in their name. + - `'-k ut_dm'` runs only tests with "ut_dm" in their name. Note that in this + case, "ut_dm" is a parameter to a test rather than the test name. The full + test name is e.g. "test_ut[ut_dm_leak]". + - `'-k not reset'` runs everything except tests with "reset" in their name. + - `'-k ut or hush'` runs only tests with "ut" or "hush" in their name. + - `'-k not (ut or hush)'` runs everything except tests with "ut" or "hush" in + their name. + +-s + prevents pytest from hiding a test's stdout. This allows you to see + U-Boot's console log in real time on pytest's stdout. + +Testing real hardware +--------------------- + +The tools and techniques used to interact with real hardware will vary +radically between different host and target systems, and the whims of the user. +For this reason, the test suite does not attempt to directly interact with real +hardware in any way. Rather, it executes a standardized set of "hook" scripts +via `$PATH`. These scripts implement certain actions on behalf of the test +suite. This keeps the test suite simple and isolated from system variances +unrelated to U-Boot features. + +Hook scripts +~~~~~~~~~~~~ + +Environment variables +''''''''''''''''''''' + +The following environment variables are set when running hook scripts: + +- `UBOOT_BOARD_TYPE` the board type being tested. +- `UBOOT_BOARD_IDENTITY` the board identity being tested, or `na` if none was + specified. +- `UBOOT_SOURCE_DIR` the U-Boot source directory. +- `UBOOT_TEST_PY_DIR` the full path to `test/py/` in the source directory. +- `UBOOT_BUILD_DIR` the U-Boot build directory. +- `UBOOT_RESULT_DIR` the test result directory. +- `UBOOT_PERSISTENT_DATA_DIR` the test persistent data directory. + +u-boot-test-console +''''''''''''''''''' + +This script provides access to the U-Boot console. The script's stdin/stdout +should be connected to the board's console. This process should continue to run +indefinitely, until killed. The test suite will run this script in parallel +with all other hooks. + +This script may be implemented e.g. by executing `cu`, `kermit`, `conmux`, etc. +via exec(). + +If you are able to run U-Boot under a hardware simulator such as QEMU, then +you would likely spawn that simulator from this script. However, note that +`u-boot-test-reset` may be called multiple times per test script run, and must +cause U-Boot to start execution from scratch each time. Hopefully your +simulator includes a virtual reset button! If not, you can launch the +simulator from `u-boot-test-reset` instead, while arranging for this console +process to always communicate with the current simulator instance. + +u-boot-test-flash +''''''''''''''''' + +Prior to running the test suite against a board, some arrangement must be made +so that the board executes the particular U-Boot binary to be tested. Often +this involves writing the U-Boot binary to the board's flash ROM. The test +suite calls this hook script for that purpose. + +This script should perform the entire flashing process synchronously; the +script should only exit once flashing is complete, and a board reset will +cause the newly flashed U-Boot binary to be executed. + +It is conceivable that this script will do nothing. This might be useful in +the following cases: + +- Some other process has already written the desired U-Boot binary into the + board's flash prior to running the test suite. +- The board allows U-Boot to be downloaded directly into RAM, and executed + from there. Use of this feature will reduce wear on the board's flash, so + may be preferable if available, and if cold boot testing of U-Boot is not + required. If this feature is used, the `u-boot-test-reset` script should + perform this download, since the board could conceivably be reset multiple + times in a single test run. + +It is up to the user to determine if those situations exist, and to code this +hook script appropriately. + +This script will typically be implemented by calling out to some SoC- or +board-specific vendor flashing utility. + +u-boot-test-reset +''''''''''''''''' + +Whenever the test suite needs to reset the target board, this script is +executed. This is guaranteed to happen at least once, prior to executing the +first test function. If any test fails, the test infra-structure will execute +this script again to restore U-Boot to an operational state before running the +next test function. + +This script will likely be implemented by communicating with some form of +relay or electronic switch attached to the board's reset signal. + +The semantics of this script require that when it is executed, U-Boot will +start running from scratch. If the U-Boot binary to be tested has been written +to flash, pulsing the board's reset signal is likely all this script needs to +do. However, in some scenarios, this script may perform other actions. For +example, it may call out to some SoC- or board-specific vendor utility in order +to download the U-Boot binary directly into RAM and execute it. This would +avoid the need for `u-boot-test-flash` to actually write U-Boot to flash, thus +saving wear on the flash chip(s). + +Examples +'''''''' + +https://github.com/swarren/uboot-test-hooks contains some working example hook +scripts, and may be useful as a reference when implementing hook scripts for +your platform. These scripts are not considered part of U-Boot itself. + +Board-type-specific configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Each board has a different configuration and behaviour. Many of these +differences can be automatically detected by parsing the `.config` file in the +build directory. However, some differences can't yet be handled automatically. + +For each board, an optional Python module `u_boot_board_${board_type}` may exist +to provide board-specific information to the test script. Any global value +defined in these modules is available for use by any test function. The data +contained in these scripts must be purely derived from U-Boot source code. +Hence, these configuration files are part of the U-Boot source tree too. + +Execution environment configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Each user's hardware setup may enable testing different subsets of the features +implemented by a particular board's configuration of U-Boot. For example, a +U-Boot configuration may support USB device mode and USB Mass Storage, but this +can only be tested if a USB cable is connected between the board and the host +machine running the test script. + +For each board, optional Python modules `u_boot_boardenv_${board_type}` and +`u_boot_boardenv_${board_type}_${board_identity}` may exist to provide +board-specific and board-identity-specific information to the test script. Any +global value defined in these modules is available for use by any test +function. The data contained in these is specific to a particular user's +hardware configuration. Hence, these configuration files are not part of the +U-Boot source tree, and should be installed outside of the source tree. Users +should set `$PYTHONPATH` prior to running the test script to allow these +modules to be loaded. + +Board module parameter usage +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The test scripts rely on the following variables being defined by the board +module: + +- none at present + +U-Boot `.config` feature usage +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The test scripts rely on various U-Boot `.config` features, either directly in +order to test those features, or indirectly in order to query information from +the running U-Boot instance in order to test other features. + +One example is that testing of the `md` command requires knowledge of a RAM +address to use for the test. This data is parsed from the output of the +`bdinfo` command, and hence relies on CONFIG_CMD_BDI being enabled. + +For a complete list of dependencies, please search the test scripts for +instances of: + +- `buildconfig.get(...` +- `@pytest.mark.buildconfigspec(...` +- `@pytest.mark.notbuildconfigspec(...` + +Complete invocation example +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Assuming that you have installed the hook scripts into $HOME/ubtest/bin, and +any required environment configuration Python modules into $HOME/ubtest/py, +then you would likely invoke the test script as follows: + +If U-Boot has already been built: + +.. code-block:: bash + + PATH=$HOME/ubtest/bin:$PATH \ + PYTHONPATH=${HOME}/ubtest/py/${HOSTNAME}:${PYTHONPATH} \ + ./test/py/test.py --bd seaboard + +If you want the test script to compile U-Boot for you too, then you likely +need to set `$CROSS_COMPILE` to allow this, and invoke the test script as +follows: + +.. code-block:: bash + + CROSS_COMPILE=arm-none-eabi- \ + PATH=$HOME/ubtest/bin:$PATH \ + PYTHONPATH=${HOME}/ubtest/py/${HOSTNAME}:${PYTHONPATH} \ + ./test/py/test.py --bd seaboard --build + +or, using buildman to handle it: + +.. code-block:: bash + + PATH=$HOME/ubtest/bin:$PATH \ + PYTHONPATH=${HOME}/ubtest/py/${HOSTNAME}:${PYTHONPATH} \ + ./test/py/test.py --bd seaboard --build --buildman + +Writing tests +------------- + +Please refer to the pytest documentation for details of writing pytest tests. +Details specific to the U-Boot test suite are described below. + +A test fixture named `u_boot_console` should be used by each test function. This +provides the means to interact with the U-Boot console, and retrieve board and +environment configuration information. + +The function `u_boot_console.run_command()` executes a shell command on the +U-Boot console, and returns all output from that command. This allows +validation or interpretation of the command output. This function validates +that certain strings are not seen on the U-Boot console. These include shell +error messages and the U-Boot sign-on message (in order to detect unexpected +board resets). See the source of `u_boot_console_base.py` for a complete list of +"bad" strings. Some test scenarios are expected to trigger these strings. Use +`u_boot_console.disable_check()` to temporarily disable checking for specific +strings. See `test_unknown_cmd.py` for an example. + +Board- and board-environment configuration values may be accessed as sub-fields +of the `u_boot_console.config` object, for example +`u_boot_console.config.ram_base`. + +Build configuration values (from `.config`) may be accessed via the dictionary +`u_boot_console.config.buildconfig`, with keys equal to the Kconfig variable +names. diff --git a/test/py/README.md b/test/py/README.md deleted file mode 100644 index fddc104b26..0000000000 --- a/test/py/README.md +++ /dev/null @@ -1,389 +0,0 @@ -# U-Boot pytest suite - -## Introduction - -This tool aims to test U-Boot by executing U-Boot shell commands using the -console interface. A single top-level script exists to execute or attach to the -U-Boot console, run the entire script of tests against it, and summarize the -results. Advantages of this approach are: - -- Testing is performed in the same way a user or script would interact with - U-Boot; there can be no disconnect. -- There is no need to write or embed test-related code into U-Boot itself. - It is asserted that writing test-related code in Python is simpler and more - flexible than writing it all in C. -- It is reasonably simple to interact with U-Boot in this way. - -## Requirements - -The test suite is implemented using pytest. Interaction with the U-Boot console -involves executing some binary and interacting with its stdin/stdout. You will -need to implement various "hook" scripts that are called by the test suite at -the appropriate time. - -In order to run the testsuite at a minimum we require that both python3 and -pip for python3 be installed. All of the required python modules are -described in the requirements.txt file in this directory and can be installed -with the command ```pip install -r requirements.txt``` - -In order to execute certain tests on their supported platforms other tools -will be required. The following is an incomplete list: - -| Package | -| -------------- | -| gdisk | -| dfu-util | -| dtc | -| openssl | -| sudo OR guestmount | -| e2fsprogs | -| util-linux | -| coreutils | -| dosfstools | -| efitools | -| mount | -| mtools | -| sbsigntool | -| udisks2 | - - -Please use the apporirate commands for your distribution to match these tools -up with the package that provides them. - -The test script supports either: - -- Executing a sandbox port of U-Boot on the local machine as a sub-process, - and interacting with it over stdin/stdout. -- Executing an external "hook" scripts to flash a U-Boot binary onto a - physical board, attach to the board's console stream, and reset the board. - Further details are described later. - -### Using `virtualenv` to provide requirements - -The recommended way to run the test suite, in order to ensure reproducibility -is to use `virtualenv` to set up the necessary environment. This can be done -via the following commands: - -```bash -$ cd /path/to/u-boot -$ sudo apt-get install python3 python3-virtualenv -$ virtualenv -p /usr/bin/python3 venv -$ . ./venv/bin/activate -$ pip install -r test/py/requirements.txt -``` - -## Testing sandbox - -To run the testsuite on the sandbox port (U-Boot built as a native user-space -application), simply execute: - -``` -./test/py/test.py --bd sandbox --build -``` - -The `--bd` option tells the test suite which board type is being tested. This -lets the test suite know which features the board has, and hence exactly what -can be tested. - -The `--build` option tells U-Boot to compile U-Boot. Alternatively, you may -omit this option and build U-Boot yourself, in whatever way you choose, before -running the test script. - -The test script will attach to U-Boot, execute all valid tests for the board, -then print a summary of the test process. A complete log of the test session -will be written to `${build_dir}/test-log.html`. This is best viewed in a web -browser, but may be read directly as plain text, perhaps with the aid of the -`html2text` utility. - -### Testing under a debugger - -If you need to run sandbox under a debugger, you may pass the command-line -option `--gdbserver COMM`. This causes two things to happens: - -- Instead of running U-Boot directly, it will be run under gdbserver, with - debug communication via the channel `COMM`. You can attach a debugger to the - sandbox process in order to debug it. See `man gdbserver` and the example - below for details of valid values for `COMM`. -- All timeouts in tests are disabled, allowing U-Boot an arbitrary amount of - time to execute commands. This is useful if U-Boot is stopped at a breakpoint - during debugging. - -A usage example is: - -Window 1: -```shell -./test/py/test.py --bd sandbox --gdbserver localhost:1234 -``` - -Window 2: -```shell -gdb ./build-sandbox/u-boot -ex 'target remote localhost:1234' -``` - -Alternatively, you could leave off the `-ex` option and type the command -manually into gdb once it starts. - -You can use any debugger you wish, so long as it speaks the gdb remote -protocol, or any graphical wrapper around gdb. - -Some tests deliberately cause the sandbox process to exit, e.g. to test the -reset command, or sandbox's CTRL-C handling. When this happens, you will need -to attach the debugger to the new sandbox instance. If these tests are not -relevant to your debugging session, you can skip them using pytest's -k -command-line option; see the next section. - -## Command-line options - -- `--board-type`, `--bd`, `-B` set the type of the board to be tested. For - example, `sandbox` or `seaboard`. -- `--board-identity`, `--id` set the identity of the board to be tested. - This allows differentiation between multiple instances of the same type of - physical board that are attached to the same host machine. This parameter is - not interpreted by the test script in any way, but rather is simply passed - to the hook scripts described below, and may be used in any site-specific - way deemed necessary. -- `--build` indicates that the test script should compile U-Boot itself - before running the tests. If using this option, make sure that any - environment variables required by the build process are already set, such as - `$CROSS_COMPILE`. -- `--buildman` indicates that `--build` should use buildman to build U-Boot. - There is no need to set $CROSS_COMPILE` in this case since buildman handles - it. -- `--build-dir` sets the directory containing the compiled U-Boot binaries. - If omitted, this is `${source_dir}/build-${board_type}`. -- `--result-dir` sets the directory to write results, such as log files, - into. If omitted, the build directory is used. -- `--persistent-data-dir` sets the directory used to store persistent test - data. This is test data that may be re-used across test runs, such as file- - system images. - -`pytest` also implements a number of its own command-line options. Commonly used -options are mentioned below. Please see `pytest` documentation for complete -details. Execute `py.test --version` for a brief summary. Note that U-Boot's -test.py script passes all command-line arguments directly to `pytest` for -processing. - -- `-k` selects which tests to run. The default is to run all known tests. This - option takes a single argument which is used to filter test names. Simple - logical operators are supported. For example: - - `'ums'` runs only tests with "ums" in their name. - - `'ut_dm'` runs only tests with "ut_dm" in their name. Note that in this - case, "ut_dm" is a parameter to a test rather than the test name. The full - test name is e.g. "test_ut[ut_dm_leak]". - - `'not reset'` runs everything except tests with "reset" in their name. - - `'ut or hush'` runs only tests with "ut" or "hush" in their name. - - `'not (ut or hush)'` runs everything except tests with "ut" or "hush" in - their name. -- `-s` prevents pytest from hiding a test's stdout. This allows you to see - U-Boot's console log in real time on pytest's stdout. - -## Testing real hardware - -The tools and techniques used to interact with real hardware will vary -radically between different host and target systems, and the whims of the user. -For this reason, the test suite does not attempt to directly interact with real -hardware in any way. Rather, it executes a standardized set of "hook" scripts -via `$PATH`. These scripts implement certain actions on behalf of the test -suite. This keeps the test suite simple and isolated from system variances -unrelated to U-Boot features. - -### Hook scripts - -#### Environment variables - -The following environment variables are set when running hook scripts: - -- `UBOOT_BOARD_TYPE` the board type being tested. -- `UBOOT_BOARD_IDENTITY` the board identity being tested, or `na` if none was - specified. -- `UBOOT_SOURCE_DIR` the U-Boot source directory. -- `UBOOT_TEST_PY_DIR` the full path to `test/py/` in the source directory. -- `UBOOT_BUILD_DIR` the U-Boot build directory. -- `UBOOT_RESULT_DIR` the test result directory. -- `UBOOT_PERSISTENT_DATA_DIR` the test persistent data directory. - -#### `u-boot-test-console` - -This script provides access to the U-Boot console. The script's stdin/stdout -should be connected to the board's console. This process should continue to run -indefinitely, until killed. The test suite will run this script in parallel -with all other hooks. - -This script may be implemented e.g. by exec()ing `cu`, `kermit`, `conmux`, etc. - -If you are able to run U-Boot under a hardware simulator such as qemu, then -you would likely spawn that simulator from this script. However, note that -`u-boot-test-reset` may be called multiple times per test script run, and must -cause U-Boot to start execution from scratch each time. Hopefully your -simulator includes a virtual reset button! If not, you can launch the -simulator from `u-boot-test-reset` instead, while arranging for this console -process to always communicate with the current simulator instance. - -#### `u-boot-test-flash` - -Prior to running the test suite against a board, some arrangement must be made -so that the board executes the particular U-Boot binary to be tested. Often, -this involves writing the U-Boot binary to the board's flash ROM. The test -suite calls this hook script for that purpose. - -This script should perform the entire flashing process synchronously; the -script should only exit once flashing is complete, and a board reset will -cause the newly flashed U-Boot binary to be executed. - -It is conceivable that this script will do nothing. This might be useful in -the following cases: - -- Some other process has already written the desired U-Boot binary into the - board's flash prior to running the test suite. -- The board allows U-Boot to be downloaded directly into RAM, and executed - from there. Use of this feature will reduce wear on the board's flash, so - may be preferable if available, and if cold boot testing of U-Boot is not - required. If this feature is used, the `u-boot-test-reset` script should - perform this download, since the board could conceivably be reset multiple - times in a single test run. - -It is up to the user to determine if those situations exist, and to code this -hook script appropriately. - -This script will typically be implemented by calling out to some SoC- or -board-specific vendor flashing utility. - -#### `u-boot-test-reset` - -Whenever the test suite needs to reset the target board, this script is -executed. This is guaranteed to happen at least once, prior to executing the -first test function. If any test fails, the test infra-structure will execute -this script again to restore U-Boot to an operational state before running the -next test function. - -This script will likely be implemented by communicating with some form of -relay or electronic switch attached to the board's reset signal. - -The semantics of this script require that when it is executed, U-Boot will -start running from scratch. If the U-Boot binary to be tested has been written -to flash, pulsing the board's reset signal is likely all this script need do. -However, in some scenarios, this script may perform other actions. For -example, it may call out to some SoC- or board-specific vendor utility in order -to download the U-Boot binary directly into RAM and execute it. This would -avoid the need for `u-boot-test-flash` to actually write U-Boot to flash, thus -saving wear on the flash chip(s). - -#### Examples - -https://github.com/swarren/uboot-test-hooks contains some working example hook -scripts, and may be useful as a reference when implementing hook scripts for -your platform. These scripts are not considered part of U-Boot itself. - -### Board-type-specific configuration - -Each board has a different configuration and behaviour. Many of these -differences can be automatically detected by parsing the `.config` file in the -build directory. However, some differences can't yet be handled automatically. - -For each board, an optional Python module `u_boot_board_${board_type}` may exist -to provide board-specific information to the test script. Any global value -defined in these modules is available for use by any test function. The data -contained in these scripts must be purely derived from U-Boot source code. -Hence, these configuration files are part of the U-Boot source tree too. - -### Execution environment configuration - -Each user's hardware setup may enable testing different subsets of the features -implemented by a particular board's configuration of U-Boot. For example, a -U-Boot configuration may support USB device mode and USB Mass Storage, but this -can only be tested if a USB cable is connected between the board and the host -machine running the test script. - -For each board, optional Python modules `u_boot_boardenv_${board_type}` and -`u_boot_boardenv_${board_type}_${board_identity}` may exist to provide -board-specific and board-identity-specific information to the test script. Any -global value defined in these modules is available for use by any test -function. The data contained in these is specific to a particular user's -hardware configuration. Hence, these configuration files are not part of the -U-Boot source tree, and should be installed outside of the source tree. Users -should set `$PYTHONPATH` prior to running the test script to allow these -modules to be loaded. - -### Board module parameter usage - -The test scripts rely on the following variables being defined by the board -module: - -- None at present. - -### U-Boot `.config` feature usage - -The test scripts rely on various U-Boot `.config` features, either directly in -order to test those features, or indirectly in order to query information from -the running U-Boot instance in order to test other features. - -One example is that testing of the `md` command requires knowledge of a RAM -address to use for the test. This data is parsed from the output of the -`bdinfo` command, and hence relies on CONFIG_CMD_BDI being enabled. - -For a complete list of dependencies, please search the test scripts for -instances of: - -- `buildconfig.get(...` -- `@pytest.mark.buildconfigspec(...` -- `@pytest.mark.notbuildconfigspec(...` - -### Complete invocation example - -Assuming that you have installed the hook scripts into $HOME/ubtest/bin, and -any required environment configuration Python modules into $HOME/ubtest/py, -then you would likely invoke the test script as follows: - -If U-Boot has already been built: - -```bash -PATH=$HOME/ubtest/bin:$PATH \ - PYTHONPATH=${HOME}/ubtest/py/${HOSTNAME}:${PYTHONPATH} \ - ./test/py/test.py --bd seaboard -``` - -If you want the test script to compile U-Boot for you too, then you likely -need to set `$CROSS_COMPILE` to allow this, and invoke the test script as -follows: - -```bash -CROSS_COMPILE=arm-none-eabi- \ - PATH=$HOME/ubtest/bin:$PATH \ - PYTHONPATH=${HOME}/ubtest/py/${HOSTNAME}:${PYTHONPATH} \ - ./test/py/test.py --bd seaboard --build -``` - -or, using buildman to handle it: - -```bash - PATH=$HOME/ubtest/bin:$PATH \ - PYTHONPATH=${HOME}/ubtest/py/${HOSTNAME}:${PYTHONPATH} \ - ./test/py/test.py --bd seaboard --build --buildman -``` - -## Writing tests - -Please refer to the pytest documentation for details of writing pytest tests. -Details specific to the U-Boot test suite are described below. - -A test fixture named `u_boot_console` should be used by each test function. This -provides the means to interact with the U-Boot console, and retrieve board and -environment configuration information. - -The function `u_boot_console.run_command()` executes a shell command on the -U-Boot console, and returns all output from that command. This allows -validation or interpretation of the command output. This function validates -that certain strings are not seen on the U-Boot console. These include shell -error messages and the U-Boot sign-on message (in order to detect unexpected -board resets). See the source of `u_boot_console_base.py` for a complete list of -"bad" strings. Some test scenarios are expected to trigger these strings. Use -`u_boot_console.disable_check()` to temporarily disable checking for specific -strings. See `test_unknown_cmd.py` for an example. - -Board- and board-environment configuration values may be accessed as sub-fields -of the `u_boot_console.config` object, for example -`u_boot_console.config.ram_base`. - -Build configuration values (from `.config`) may be accessed via the dictionary -`u_boot_console.config.buildconfig`, with keys equal to the Kconfig variable -names. -- 2.29.2