[PATCH v1] test: Allow simple glob pattern in the test name
When run `ut dm [test name]` allow to use simple pattern to run all tests started with given prefix. For example, to run all ACPI test cases: ut dm acpi*
Signed-off-by: Andy Shevchenko andriy.shevchenko@linux.intel.com --- test/dm/test-main.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)
diff --git a/test/dm/test-main.c b/test/dm/test-main.c index 69458d62c869..f15527e13d7b 100644 --- a/test/dm/test-main.c +++ b/test/dm/test-main.c @@ -129,17 +129,23 @@ static bool dm_test_run_on_flattree(struct unit_test *test)
static bool test_matches(const char *test_name, const char *find_name) { - if (!find_name) + size_t len = find_name ? strlen(find_name) : 0; + + /* Allow glob expansion in the test name */ + if (len && find_name[len - 1] == '*') + len--; + + if (!len) return true;
- if (!strcmp(test_name, find_name)) + if (!strncmp(test_name, find_name, len)) return true;
/* All tests have this prefix */ if (!strncmp(test_name, "dm_test_", 8)) test_name += 8;
- if (!strcmp(test_name, find_name)) + if (!strncmp(test_name, find_name, len)) return true;
return false;
Hi Andy,
On Wed, 3 Feb 2021 at 08:32, Andy Shevchenko andriy.shevchenko@linux.intel.com wrote:
When run `ut dm [test name]` allow to use simple pattern to run all tests started with given prefix. For example, to run all ACPI test cases: ut dm acpi*
Signed-off-by: Andy Shevchenko andriy.shevchenko@linux.intel.com
test/dm/test-main.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)
This seems like a good idea to me.
Can you please send it rebased to u-boot-dm/test-working though? I have done a bit of refactoring and this function is now in test/test-main.c
Regards, Simon
On Thu, Feb 04, 2021 at 08:17:24PM -0700, Simon Glass wrote:
On Wed, 3 Feb 2021 at 08:32, Andy Shevchenko andriy.shevchenko@linux.intel.com wrote:
When run `ut dm [test name]` allow to use simple pattern to run all tests started with given prefix. For example, to run all ACPI test cases: ut dm acpi*
Signed-off-by: Andy Shevchenko andriy.shevchenko@linux.intel.com
test/dm/test-main.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)
This seems like a good idea to me.
Can you please send it rebased to u-boot-dm/test-working though? I have done a bit of refactoring and this function is now in test/test-main.c
I will look at it.
Btw, you have an issue there, i.e. if test case failed, all percentage after it goes red, which is wrong.
On Fri, Feb 05, 2021 at 07:34:49PM +0200, Andy Shevchenko wrote:
On Thu, Feb 04, 2021 at 08:17:24PM -0700, Simon Glass wrote:
...
Btw, you have an issue there, i.e. if test case failed, all percentage after it goes red, which is wrong.
One more thing, is it known bug that either in the original code, or in your new branch the following test case is 100% failed?
/* Non-existent in DTB */ ut_asserteq(FDT_ADDR_T_NONE, dev_read_addr(dev));
Can you fix this sooner than later, please?
On Fri, Feb 05, 2021 at 08:15:25PM +0200, Andy Shevchenko wrote:
On Fri, Feb 05, 2021 at 07:34:49PM +0200, Andy Shevchenko wrote:
On Thu, Feb 04, 2021 at 08:17:24PM -0700, Simon Glass wrote:
...
Btw, you have an issue there, i.e. if test case failed, all percentage after it goes red, which is wrong.
One more thing, is it known bug that either in the original code, or in your new branch the following test case is 100% failed?
/* Non-existent in DTB */ ut_asserteq(FDT_ADDR_T_NONE, dev_read_addr(dev));Can you fix this sooner than later, please?
Another annoying issue is the 2+ run on uncleaned build folder, i.e. on the second and following runs it starts out of the sudden to ask my sudo password.
The later to me is a highest priority, now I have to run rm -rf build-sandbox each time, and if I forget - wasted time!
Can you fix this as soon as possible, please?
On Fri, Feb 05, 2021 at 08:15:25PM +0200, Andy Shevchenko wrote:
On Fri, Feb 05, 2021 at 07:34:49PM +0200, Andy Shevchenko wrote:
On Thu, Feb 04, 2021 at 08:17:24PM -0700, Simon Glass wrote:
...
Btw, you have an issue there, i.e. if test case failed, all percentage after it goes red, which is wrong.
One more thing, is it known bug that either in the original code, or in your new branch the following test case is 100% failed?
/* Non-existent in DTB */ ut_asserteq(FDT_ADDR_T_NONE, dev_read_addr(dev));Can you fix this sooner than later, please?
Actually it seems this very patch makes the issue visible (I suppose something with test case names).
On Fri, Feb 05, 2021 at 09:17:27PM +0200, Andy Shevchenko wrote:
On Fri, Feb 05, 2021 at 08:15:25PM +0200, Andy Shevchenko wrote:
On Fri, Feb 05, 2021 at 07:34:49PM +0200, Andy Shevchenko wrote:
On Thu, Feb 04, 2021 at 08:17:24PM -0700, Simon Glass wrote:
...
Btw, you have an issue there, i.e. if test case failed, all percentage after it goes red, which is wrong.
One more thing, is it known bug that either in the original code, or in your new branch the following test case is 100% failed?
/* Non-existent in DTB */ ut_asserteq(FDT_ADDR_T_NONE, dev_read_addr(dev));Can you fix this sooner than later, please?
Actually it seems this very patch makes the issue visible (I suppose something with test case names).
Okay, actually there *is* a problem with the test suite, i.e. you may not run some test cases twice during the same session.
Hi Andy,
On Fri, 5 Feb 2021 at 13:46, Andy Shevchenko andriy.shevchenko@linux.intel.com wrote:
On Fri, Feb 05, 2021 at 09:17:27PM +0200, Andy Shevchenko wrote:
On Fri, Feb 05, 2021 at 08:15:25PM +0200, Andy Shevchenko wrote:
On Fri, Feb 05, 2021 at 07:34:49PM +0200, Andy Shevchenko wrote:
On Thu, Feb 04, 2021 at 08:17:24PM -0700, Simon Glass wrote:
...
Btw, you have an issue there, i.e. if test case failed, all percentage after it goes red, which is wrong.
One more thing, is it known bug that either in the original code, or in your new branch the following test case is 100% failed?
/* Non-existent in DTB */ ut_asserteq(FDT_ADDR_T_NONE, dev_read_addr(dev));Can you fix this sooner than later, please?
Actually it seems this very patch makes the issue visible (I suppose something with test case names).
Okay, actually there *is* a problem with the test suite, i.e. you may not run some test cases twice during the same session.
If this is related to the squashfs tests? I have disabled them for now in my local tree.
Regards, Simon
On Sun, Feb 07, 2021 at 07:37:55AM -0700, Simon Glass wrote:
On Fri, 5 Feb 2021 at 13:46, Andy Shevchenko andriy.shevchenko@linux.intel.com wrote:
On Fri, Feb 05, 2021 at 09:17:27PM +0200, Andy Shevchenko wrote:
On Fri, Feb 05, 2021 at 08:15:25PM +0200, Andy Shevchenko wrote:
On Fri, Feb 05, 2021 at 07:34:49PM +0200, Andy Shevchenko wrote:
On Thu, Feb 04, 2021 at 08:17:24PM -0700, Simon Glass wrote:
...
Btw, you have an issue there, i.e. if test case failed, all percentage after it goes red, which is wrong.
One more thing, is it known bug that either in the original code, or in your new branch the following test case is 100% failed?
/* Non-existent in DTB */ ut_asserteq(FDT_ADDR_T_NONE, dev_read_addr(dev));Can you fix this sooner than later, please?
Actually it seems this very patch makes the issue visible (I suppose something with test case names).
Okay, actually there *is* a problem with the test suite, i.e. you may not run some test cases twice during the same session.
If this is related to the squashfs tests? I have disabled them for now in my local tree.
I don't think it's related to the certain test case, it's a test suite design issue and how some of the test cases have been written. So, if you run u-boot application and manually run `ut dm` in there, - first time: Failures: 0 - second time: Failures: 9 - third and next time: Failures: 12
Means that 12 test cases are written badly.
Hi Andy,
On Mon, 8 Feb 2021 at 04:34, Andy Shevchenko andriy.shevchenko@linux.intel.com wrote:
On Sun, Feb 07, 2021 at 07:37:55AM -0700, Simon Glass wrote:
On Fri, 5 Feb 2021 at 13:46, Andy Shevchenko andriy.shevchenko@linux.intel.com wrote:
On Fri, Feb 05, 2021 at 09:17:27PM +0200, Andy Shevchenko wrote:
On Fri, Feb 05, 2021 at 08:15:25PM +0200, Andy Shevchenko wrote:
On Fri, Feb 05, 2021 at 07:34:49PM +0200, Andy Shevchenko wrote:
On Thu, Feb 04, 2021 at 08:17:24PM -0700, Simon Glass wrote:
...
Btw, you have an issue there, i.e. if test case failed, all percentage after it goes red, which is wrong.
One more thing, is it known bug that either in the original code, or in your new branch the following test case is 100% failed?
/* Non-existent in DTB */ ut_asserteq(FDT_ADDR_T_NONE, dev_read_addr(dev));Can you fix this sooner than later, please?
Actually it seems this very patch makes the issue visible (I suppose something with test case names).
Okay, actually there *is* a problem with the test suite, i.e. you may not run some test cases twice during the same session.
If this is related to the squashfs tests? I have disabled them for now in my local tree.
I don't think it's related to the certain test case, it's a test suite design issue and how some of the test cases have been written. So, if you run u-boot application and manually run `ut dm` in there,
- first time: Failures: 0
- second time: Failures: 9
- third and next time: Failures: 12
Means that 12 test cases are written badly.
OK I see. It did not use to be possible to pass all tests without running through pytest, which does some setup. You still need to create a 'spi.bin' file for the SPI tests to pass.
I'm not sure what prevents multiple runs without quitting U-Boot, but I suspect it is just some state hanging around, as we don't reset everything. For example, sandbox provides control of what happens when a sysreset is performed, and perhaps that is not reset to initial values correct by state_reset_for_test()..
Regards, Simon
On Mon, Feb 08, 2021 at 10:07:55AM -0700, Simon Glass wrote:
On Mon, 8 Feb 2021 at 04:34, Andy Shevchenko andriy.shevchenko@linux.intel.com wrote:
On Sun, Feb 07, 2021 at 07:37:55AM -0700, Simon Glass wrote:
On Fri, 5 Feb 2021 at 13:46, Andy Shevchenko andriy.shevchenko@linux.intel.com wrote:
On Fri, Feb 05, 2021 at 09:17:27PM +0200, Andy Shevchenko wrote:
On Fri, Feb 05, 2021 at 08:15:25PM +0200, Andy Shevchenko wrote:
On Fri, Feb 05, 2021 at 07:34:49PM +0200, Andy Shevchenko wrote: > On Thu, Feb 04, 2021 at 08:17:24PM -0700, Simon Glass wrote:
...
> Btw, you have an issue there, i.e. if test case failed, all percentage after it > goes red, which is wrong.
One more thing, is it known bug that either in the original code, or in your new branch the following test case is 100% failed?
/* Non-existent in DTB */ ut_asserteq(FDT_ADDR_T_NONE, dev_read_addr(dev));Can you fix this sooner than later, please?
Actually it seems this very patch makes the issue visible (I suppose something with test case names).
Okay, actually there *is* a problem with the test suite, i.e. you may not run some test cases twice during the same session.
If this is related to the squashfs tests? I have disabled them for now in my local tree.
I don't think it's related to the certain test case, it's a test suite design issue and how some of the test cases have been written. So, if you run u-boot application and manually run `ut dm` in there,
- first time: Failures: 0
- second time: Failures: 9
- third and next time: Failures: 12
Means that 12 test cases are written badly.
OK I see. It did not use to be possible to pass all tests without running through pytest, which does some setup. You still need to create a 'spi.bin' file for the SPI tests to pass.
I run it _after_ the official pytest passed. So, every file is prepared.
I'm not sure what prevents multiple runs without quitting U-Boot, but I suspect it is just some state hanging around, as we don't reset everything. For example, sandbox provides control of what happens when a sysreset is performed, and perhaps that is not reset to initial values correct by state_reset_for_test()..
I believe this is the case in most of the cases.
participants (2)
-
Andy Shevchenko -
Simon Glass