[U-Boot] [RFC PATCH] test: py: Disable sleep test for qemu targets
Qemu for arm32/arm64 has a problem with time setup. That's why sleep test is failing. Add boardidentity marker to remove specific boards from running that test.
Signed-off-by: Michal Simek michal.simek@xilinx.com ---
If you want to add this checking in one patch and then put it to one test then it is fine for me.
--- test/py/conftest.py | 28 ++++++++++++++++++++++++++++ test/py/pytest.ini | 1 + test/py/tests/test_sleep.py | 1 + 3 files changed, 30 insertions(+)
diff --git a/test/py/conftest.py b/test/py/conftest.py index 6e66a48c15fd..1812ff340e6a 100644 --- a/test/py/conftest.py +++ b/test/py/conftest.py @@ -436,6 +436,33 @@ def setup_boardspec(item): if required_boards and ubconfig.board_type not in required_boards: pytest.skip('board "%s" not supported' % ubconfig.board_type)
+def setup_boardidentity(item): + """Process any 'boardidentity' marker for a test. + + Such a marker lists the set of board identity that a test does/doesn't + support. If tests are being executed on an unsupported board, the test is + marked to be skipped. + + Args: + item: The pytest test item. + + Returns: + Nothing. + """ + mark = item.get_marker('boardidentity') + if not mark: + return + required_boards = [] + for board in mark.args: + if board.startswith('!'): + if ubconfig.board_identity == board[1:]: + pytest.skip('board identity not supported') + return + else: + required_boards.append(board) + if required_boards and ubconfig.board_identity not in required_boards: + pytest.skip('board identity not supported') + def setup_buildconfigspec(item): """Process any 'buildconfigspec' marker for a test.
@@ -503,6 +530,7 @@ def pytest_runtest_setup(item):
start_test_section(item) setup_boardspec(item) + setup_boardidentity(item) setup_buildconfigspec(item) setup_requiredtool(item)
diff --git a/test/py/pytest.ini b/test/py/pytest.ini index 67e514f42058..9d64671814de 100644 --- a/test/py/pytest.ini +++ b/test/py/pytest.ini @@ -8,4 +8,5 @@ [pytest] markers = boardspec: U-Boot: Describes the set of boards a test can/can't run on. + boardidentity: U-Boot: Describes the board identity a test can/can't run on. buildconfigspec: U-Boot: Describes Kconfig/config-header constraints. diff --git a/test/py/tests/test_sleep.py b/test/py/tests/test_sleep.py index 64e057132622..02a8a85b0e22 100644 --- a/test/py/tests/test_sleep.py +++ b/test/py/tests/test_sleep.py @@ -5,6 +5,7 @@ import pytest import time
+@pytest.mark.boardidentity("!qemu") def test_sleep(u_boot_console): """Test the sleep command, and validate that it sleeps for approximately the correct amount of time."""
On 12/01/2017 03:46 PM, Michal Simek wrote:
Qemu for arm32/arm64 has a problem with time setup.
Wouldn't it be preferable to fix the root cause?
That's why sleep test is failing. Add boardidentity marker to remove specific boards from running that test.
Isn't this what 'boardspec' is used for?
test/py/pytest.ini: boardspec: U-Boot: Describes the set of boards a test can/can't run on.
Signed-off-by: Michal Simek michal.simek@xilinx.com
If you want to add this checking in one patch and then put it to one test then it is fine for me.
test/py/conftest.py | 28 ++++++++++++++++++++++++++++ test/py/pytest.ini | 1 + test/py/tests/test_sleep.py | 1 + 3 files changed, 30 insertions(+)
diff --git a/test/py/conftest.py b/test/py/conftest.py index 6e66a48c15fd..1812ff340e6a 100644 --- a/test/py/conftest.py +++ b/test/py/conftest.py @@ -436,6 +436,33 @@ def setup_boardspec(item): if required_boards and ubconfig.board_type not in required_boards: pytest.skip('board "%s" not supported' % ubconfig.board_type)
+def setup_boardidentity(item):
- """Process any 'boardidentity' marker for a test.
- Such a marker lists the set of board identity that a test does/doesn't
- support. If tests are being executed on an unsupported board, the test is
- marked to be skipped.
- Args:
item: The pytest test item.- Returns:
Nothing.- """
- mark = item.get_marker('boardidentity')
- if not mark:
return- required_boards = []
- for board in mark.args:
if board.startswith('!'):if ubconfig.board_identity == board[1:]:pytest.skip('board identity not supported')returnelse:required_boards.append(board)- if required_boards and ubconfig.board_identity not in required_boards:
pytest.skip('board identity not supported')- def setup_buildconfigspec(item): """Process any 'buildconfigspec' marker for a test.
@@ -503,6 +530,7 @@ def pytest_runtest_setup(item):
start_test_section(item) setup_boardspec(item)
- setup_boardidentity(item) setup_buildconfigspec(item) setup_requiredtool(item)
diff --git a/test/py/pytest.ini b/test/py/pytest.ini index 67e514f42058..9d64671814de 100644 --- a/test/py/pytest.ini +++ b/test/py/pytest.ini @@ -8,4 +8,5 @@ [pytest] markers = boardspec: U-Boot: Describes the set of boards a test can/can't run on.
- boardidentity: U-Boot: Describes the board identity a test can/can't run on. buildconfigspec: U-Boot: Describes Kconfig/config-header constraints.
diff --git a/test/py/tests/test_sleep.py b/test/py/tests/test_sleep.py index 64e057132622..02a8a85b0e22 100644 --- a/test/py/tests/test_sleep.py +++ b/test/py/tests/test_sleep.py @@ -5,6 +5,7 @@ import pytest import time
+@pytest.mark.boardidentity("!qemu")
According to your commit message you don't want to exclude qemu-x86 here.
Best regards
Heinrich Schuchardt
def test_sleep(u_boot_console): """Test the sleep command, and validate that it sleeps for approximately the correct amount of time."""
Hi,
On 1.12.2017 16:06, Heinrich Schuchardt wrote:
On 12/01/2017 03:46 PM, Michal Simek wrote:
Qemu for arm32/arm64 has a problem with time setup.
Wouldn't it be preferable to fix the root cause?
Definitely that would be the best and IIRC I have tried to convince our qemu guy to do that but they have never done that.
That's why sleep test is failing. Add boardidentity marker to remove specific boards from running that test.
Isn't this what 'boardspec' is used for?
There are two things here.
./test/py/test.py --bd zynq_zc706 --build --board-identity zc706 -s
boardspec is zynq_z706 -> u-boot config and I use board identity (zc706) as actual HW I have here
And I use this ./test/py/test.py --bd zynq_zc706 --build --board-identity qemu -s which means run the same config for zc706 but on qemu
etc.
test/py/pytest.ini: boardspec: U-Boot: Describes the set of boards a test can/can't run on.
as above. This file should be also update if this "feature" is fine.
Signed-off-by: Michal Simek michal.simek@xilinx.com
If you want to add this checking in one patch and then put it to one test then it is fine for me.
test/py/conftest.py | 28 ++++++++++++++++++++++++++++ test/py/pytest.ini | 1 + test/py/tests/test_sleep.py | 1 + 3 files changed, 30 insertions(+)
diff --git a/test/py/conftest.py b/test/py/conftest.py index 6e66a48c15fd..1812ff340e6a 100644 --- a/test/py/conftest.py +++ b/test/py/conftest.py @@ -436,6 +436,33 @@ def setup_boardspec(item): if required_boards and ubconfig.board_type not in required_boards: pytest.skip('board "%s" not supported' % ubconfig.board_type) +def setup_boardidentity(item): + """Process any 'boardidentity' marker for a test.
+ Such a marker lists the set of board identity that a test does/doesn't + support. If tests are being executed on an unsupported board, the test is + marked to be skipped.
+ Args: + item: The pytest test item.
+ Returns: + Nothing. + """ + mark = item.get_marker('boardidentity') + if not mark: + return + required_boards = [] + for board in mark.args: + if board.startswith('!'): + if ubconfig.board_identity == board[1:]: + pytest.skip('board identity not supported') + return + else: + required_boards.append(board) + if required_boards and ubconfig.board_identity not in required_boards: + pytest.skip('board identity not supported')
def setup_buildconfigspec(item): """Process any 'buildconfigspec' marker for a test. @@ -503,6 +530,7 @@ def pytest_runtest_setup(item): start_test_section(item) setup_boardspec(item) + setup_boardidentity(item) setup_buildconfigspec(item) setup_requiredtool(item) diff --git a/test/py/pytest.ini b/test/py/pytest.ini index 67e514f42058..9d64671814de 100644 --- a/test/py/pytest.ini +++ b/test/py/pytest.ini @@ -8,4 +8,5 @@ [pytest] markers = boardspec: U-Boot: Describes the set of boards a test can/can't run on. + boardidentity: U-Boot: Describes the board identity a test can/can't run on. buildconfigspec: U-Boot: Describes Kconfig/config-header constraints. diff --git a/test/py/tests/test_sleep.py b/test/py/tests/test_sleep.py index 64e057132622..02a8a85b0e22 100644 --- a/test/py/tests/test_sleep.py +++ b/test/py/tests/test_sleep.py @@ -5,6 +5,7 @@ import pytest import time +@pytest.mark.boardidentity("!qemu")
According to your commit message you don't want to exclude qemu-x86 here.
It is really just qemu board identity not qemu target.
The issue with this patch is that everybody can select whatever they want to for board identification.
Maybe it would be much simpler to create variable and check it in that test and skip that test.
Anyway this is what I have in the tree and it is time to find proper solution that's why RFC.
Thanks, Michal
On 12/01/2017 08:19 AM, Michal Simek wrote:
Hi,
On 1.12.2017 16:06, Heinrich Schuchardt wrote:
On 12/01/2017 03:46 PM, Michal Simek wrote:
Qemu for arm32/arm64 has a problem with time setup.
Wouldn't it be preferable to fix the root cause?
Definitely that would be the best and IIRC I have tried to convince our qemu guy to do that but they have never done that.
What is the exact failure condition? Is it simply that the test is still slightly too strict about which delays it accepts, or is sleep outright broken?
You can use command-line option -k to avoid some tests. For example "-k not sleep". That way, we don't have to hard-code the dependency into the test source. Depending on the root cause (issue in U-Boot, or issue in just your local version of qemu, or something that will never work) this might be better?
On Fri, Dec 01, 2017 at 10:07:54AM -0700, Stephen Warren wrote:
On 12/01/2017 08:19 AM, Michal Simek wrote:
Hi,
On 1.12.2017 16:06, Heinrich Schuchardt wrote:
On 12/01/2017 03:46 PM, Michal Simek wrote:
Qemu for arm32/arm64 has a problem with time setup.
Wouldn't it be preferable to fix the root cause?
Definitely that would be the best and IIRC I have tried to convince our qemu guy to do that but they have never done that.
What is the exact failure condition? Is it simply that the test is still slightly too strict about which delays it accepts, or is sleep outright broken?
You can use command-line option -k to avoid some tests. For example "-k not sleep". That way, we don't have to hard-code the dependency into the test source. Depending on the root cause (issue in U-Boot, or issue in just your local version of qemu, or something that will never work) this might be better?
Even with the most recent relaxing of the sleep test requirements, I can still (depending on overall system load) have 'sleep' take too long, on QEMU. I think it might have been half a second slow, but I don't have the log handy anymore. Both locally and in travis we -k not sleep all of the qemu instances.
On 1.12.2017 23:44, Tom Rini wrote:
On Fri, Dec 01, 2017 at 10:07:54AM -0700, Stephen Warren wrote:
On 12/01/2017 08:19 AM, Michal Simek wrote:
Hi,
On 1.12.2017 16:06, Heinrich Schuchardt wrote:
On 12/01/2017 03:46 PM, Michal Simek wrote:
Qemu for arm32/arm64 has a problem with time setup.
Wouldn't it be preferable to fix the root cause?
Definitely that would be the best and IIRC I have tried to convince our qemu guy to do that but they have never done that.
What is the exact failure condition? Is it simply that the test is still slightly too strict about which delays it accepts, or is sleep outright broken?
You can use command-line option -k to avoid some tests. For example "-k not sleep". That way, we don't have to hard-code the dependency into the test source. Depending on the root cause (issue in U-Boot, or issue in just your local version of qemu, or something that will never work) this might be better?
Even with the most recent relaxing of the sleep test requirements, I can still (depending on overall system load) have 'sleep' take too long, on QEMU. I think it might have been half a second slow, but I don't have the log handy anymore. Both locally and in travis we -k not sleep all of the qemu instances.
ok. By locally do you mean just using -k not sleep? Or is there any other way how to put this to uboot-test-hooks/py/<hostname>/*.py
We are talking about one test here but there could be different issues with different tests in connection to qemu.
Thanks, Michal
On Mon, Dec 04, 2017 at 02:55:45PM +0100, Michal Simek wrote:
On 1.12.2017 23:44, Tom Rini wrote:
On Fri, Dec 01, 2017 at 10:07:54AM -0700, Stephen Warren wrote:
On 12/01/2017 08:19 AM, Michal Simek wrote:
Hi,
On 1.12.2017 16:06, Heinrich Schuchardt wrote:
On 12/01/2017 03:46 PM, Michal Simek wrote:
Qemu for arm32/arm64 has a problem with time setup.
Wouldn't it be preferable to fix the root cause?
Definitely that would be the best and IIRC I have tried to convince our qemu guy to do that but they have never done that.
What is the exact failure condition? Is it simply that the test is still slightly too strict about which delays it accepts, or is sleep outright broken?
You can use command-line option -k to avoid some tests. For example "-k not sleep". That way, we don't have to hard-code the dependency into the test source. Depending on the root cause (issue in U-Boot, or issue in just your local version of qemu, or something that will never work) this might be better?
Even with the most recent relaxing of the sleep test requirements, I can still (depending on overall system load) have 'sleep' take too long, on QEMU. I think it might have been half a second slow, but I don't have the log handy anymore. Both locally and in travis we -k not sleep all of the qemu instances.
ok. By locally do you mean just using -k not sleep?
Yes, I have that in my CI scripts and similar.
On 4.12.2017 15:03, Tom Rini wrote:
On Mon, Dec 04, 2017 at 02:55:45PM +0100, Michal Simek wrote:
On 1.12.2017 23:44, Tom Rini wrote:
On Fri, Dec 01, 2017 at 10:07:54AM -0700, Stephen Warren wrote:
On 12/01/2017 08:19 AM, Michal Simek wrote:
Hi,
On 1.12.2017 16:06, Heinrich Schuchardt wrote:
On 12/01/2017 03:46 PM, Michal Simek wrote: > Qemu for arm32/arm64 has a problem with time setup.
Wouldn't it be preferable to fix the root cause?
Definitely that would be the best and IIRC I have tried to convince our qemu guy to do that but they have never done that.
What is the exact failure condition? Is it simply that the test is still slightly too strict about which delays it accepts, or is sleep outright broken?
You can use command-line option -k to avoid some tests. For example "-k not sleep". That way, we don't have to hard-code the dependency into the test source. Depending on the root cause (issue in U-Boot, or issue in just your local version of qemu, or something that will never work) this might be better?
Even with the most recent relaxing of the sleep test requirements, I can still (depending on overall system load) have 'sleep' take too long, on QEMU. I think it might have been half a second slow, but I don't have the log handy anymore. Both locally and in travis we -k not sleep all of the qemu instances.
ok. By locally do you mean just using -k not sleep?
Yes, I have that in my CI scripts and similar.
Wouldn't be easier to keep this in uboot-test-hooks repo with other target setting? What we are trying to do is that our testing group will run these tests for me that's why it is just easier for me to change local uboot-test-hooks repo instead of communicate with them what -k not XXX parameters to add to certain scripts.
It means in loop they will just run all tests on qemu, local targets and in boardfarm. It is probably not big deal to tell them to add -k not sleep for all qemu runs but I know that for some i2c testing qemu doesn't emulate these devices that's why these tests fails. And the amount of tests which we shouldn't run on qemu will probably grow.
Thanks, Michal
On Mon, Dec 04, 2017 at 03:21:04PM +0100, Michal Simek wrote:
On 4.12.2017 15:03, Tom Rini wrote:
On Mon, Dec 04, 2017 at 02:55:45PM +0100, Michal Simek wrote:
On 1.12.2017 23:44, Tom Rini wrote:
On Fri, Dec 01, 2017 at 10:07:54AM -0700, Stephen Warren wrote:
On 12/01/2017 08:19 AM, Michal Simek wrote:
Hi,
On 1.12.2017 16:06, Heinrich Schuchardt wrote: > > > On 12/01/2017 03:46 PM, Michal Simek wrote: >> Qemu for arm32/arm64 has a problem with time setup. > > Wouldn't it be preferable to fix the root cause?
Definitely that would be the best and IIRC I have tried to convince our qemu guy to do that but they have never done that.
What is the exact failure condition? Is it simply that the test is still slightly too strict about which delays it accepts, or is sleep outright broken?
You can use command-line option -k to avoid some tests. For example "-k not sleep". That way, we don't have to hard-code the dependency into the test source. Depending on the root cause (issue in U-Boot, or issue in just your local version of qemu, or something that will never work) this might be better?
Even with the most recent relaxing of the sleep test requirements, I can still (depending on overall system load) have 'sleep' take too long, on QEMU. I think it might have been half a second slow, but I don't have the log handy anymore. Both locally and in travis we -k not sleep all of the qemu instances.
ok. By locally do you mean just using -k not sleep?
Yes, I have that in my CI scripts and similar.
Wouldn't be easier to keep this in uboot-test-hooks repo with other target setting?
Or do as you did did and mark the tests as not allowed for qemu, yes.
What we are trying to do is that our testing group will run these tests for me that's why it is just easier for me to change local uboot-test-hooks repo instead of communicate with them what -k not XXX parameters to add to certain scripts.
It means in loop they will just run all tests on qemu, local targets and in boardfarm. It is probably not big deal to tell them to add -k not sleep for all qemu runs but I know that for some i2c testing qemu doesn't emulate these devices that's why these tests fails. And the amount of tests which we shouldn't run on qemu will probably grow.
Well, I'm still open to possibly tweaking the allowed variance in the sleep test. OTOH, if we just say "no QEMU" here, we can then go back to "sleep should be pretty darn accurate on HW" for the test too, and perhaps that's best.
On 12/04/2017 08:30 AM, Tom Rini wrote:
On Mon, Dec 04, 2017 at 03:21:04PM +0100, Michal Simek wrote:
On 4.12.2017 15:03, Tom Rini wrote:
On Mon, Dec 04, 2017 at 02:55:45PM +0100, Michal Simek wrote:
On 1.12.2017 23:44, Tom Rini wrote:
On Fri, Dec 01, 2017 at 10:07:54AM -0700, Stephen Warren wrote:
On 12/01/2017 08:19 AM, Michal Simek wrote: > Hi, > > On 1.12.2017 16:06, Heinrich Schuchardt wrote: >> >> >> On 12/01/2017 03:46 PM, Michal Simek wrote: >>> Qemu for arm32/arm64 has a problem with time setup. >> >> Wouldn't it be preferable to fix the root cause? > > Definitely that would be the best and IIRC I have tried to convince our > qemu guy to do that but they have never done that.
What is the exact failure condition? Is it simply that the test is still slightly too strict about which delays it accepts, or is sleep outright broken?
You can use command-line option -k to avoid some tests. For example "-k not sleep". That way, we don't have to hard-code the dependency into the test source. Depending on the root cause (issue in U-Boot, or issue in just your local version of qemu, or something that will never work) this might be better?
Even with the most recent relaxing of the sleep test requirements, I can still (depending on overall system load) have 'sleep' take too long, on QEMU. I think it might have been half a second slow, but I don't have the log handy anymore. Both locally and in travis we -k not sleep all of the qemu instances.
ok. By locally do you mean just using -k not sleep?
Yes, I have that in my CI scripts and similar.
Wouldn't be easier to keep this in uboot-test-hooks repo with other target setting?
Or do as you did did and mark the tests as not allowed for qemu, yes.
What we are trying to do is that our testing group will run these tests for me that's why it is just easier for me to change local uboot-test-hooks repo instead of communicate with them what -k not XXX parameters to add to certain scripts.
It means in loop they will just run all tests on qemu, local targets and in boardfarm. It is probably not big deal to tell them to add -k not sleep for all qemu runs but I know that for some i2c testing qemu doesn't emulate these devices that's why these tests fails. And the amount of tests which we shouldn't run on qemu will probably grow.
Well, I'm still open to possibly tweaking the allowed variance in the sleep test. OTOH, if we just say "no QEMU" here, we can then go back to "sleep should be pretty darn accurate on HW" for the test too, and perhaps that's best.
The fundamental problem of "over-sleeping" due to host system load/.. exists with all boards. There's nothing specific to qemu here except that running U-Boot on qemu on the host rather than on separate HW might more easily trigger the "high load on the host" condition; I see the issue now and then and manually retry that test, although that is a bit annoying.
The original test was mostly intended to make sure that e U-Boot clock didn't run at a significantly different rate to the host, since I had seen that issue during development of some board support or as a regression sometime. Perhaps the definition of "significantly different" should be more like "1/2 rate or twice rate or more" rather than "off by a small fraction of a second". That might avoid so many false positives.
On Mon, Dec 04, 2017 at 10:14:06AM -0700, Stephen Warren wrote:
On 12/04/2017 08:30 AM, Tom Rini wrote:
On Mon, Dec 04, 2017 at 03:21:04PM +0100, Michal Simek wrote:
On 4.12.2017 15:03, Tom Rini wrote:
On Mon, Dec 04, 2017 at 02:55:45PM +0100, Michal Simek wrote:
On 1.12.2017 23:44, Tom Rini wrote:
On Fri, Dec 01, 2017 at 10:07:54AM -0700, Stephen Warren wrote: >On 12/01/2017 08:19 AM, Michal Simek wrote: >>Hi, >> >>On 1.12.2017 16:06, Heinrich Schuchardt wrote: >>> >>> >>>On 12/01/2017 03:46 PM, Michal Simek wrote: >>>>Qemu for arm32/arm64 has a problem with time setup. >>> >>>Wouldn't it be preferable to fix the root cause? >> >>Definitely that would be the best and IIRC I have tried to convince our >>qemu guy to do that but they have never done that. > >What is the exact failure condition? Is it simply that the test is still >slightly too strict about which delays it accepts, or is sleep outright >broken? > >You can use command-line option -k to avoid some tests. For example "-k not >sleep". That way, we don't have to hard-code the dependency into the test >source. Depending on the root cause (issue in U-Boot, or issue in just your >local version of qemu, or something that will never work) this might be >better?
Even with the most recent relaxing of the sleep test requirements, I can still (depending on overall system load) have 'sleep' take too long, on QEMU. I think it might have been half a second slow, but I don't have the log handy anymore. Both locally and in travis we -k not sleep all of the qemu instances.
ok. By locally do you mean just using -k not sleep?
Yes, I have that in my CI scripts and similar.
Wouldn't be easier to keep this in uboot-test-hooks repo with other target setting?
Or do as you did did and mark the tests as not allowed for qemu, yes.
What we are trying to do is that our testing group will run these tests for me that's why it is just easier for me to change local uboot-test-hooks repo instead of communicate with them what -k not XXX parameters to add to certain scripts.
It means in loop they will just run all tests on qemu, local targets and in boardfarm. It is probably not big deal to tell them to add -k not sleep for all qemu runs but I know that for some i2c testing qemu doesn't emulate these devices that's why these tests fails. And the amount of tests which we shouldn't run on qemu will probably grow.
Well, I'm still open to possibly tweaking the allowed variance in the sleep test. OTOH, if we just say "no QEMU" here, we can then go back to "sleep should be pretty darn accurate on HW" for the test too, and perhaps that's best.
The fundamental problem of "over-sleeping" due to host system load/.. exists with all boards. There's nothing specific to qemu here except that running U-Boot on qemu on the host rather than on separate HW might more easily trigger the "high load on the host" condition; I see the issue now and then and manually retry that test, although that is a bit annoying.
The original test was mostly intended to make sure that e U-Boot clock didn't run at a significantly different rate to the host, since I had seen that issue during development of some board support or as a regression sometime. Perhaps the definition of "significantly different" should be more like "1/2 rate or twice rate or more" rather than "off by a small fraction of a second". That might avoid so many false positives.
I've pushed this up to 10 seconds and 0.5s worth of overrun and on qemu-mips here I see a 13.2s sleep. That's pretty close to 1/3rd fast and to me a wrong-clocking value, yes?
On 12/04/2017 04:21 PM, Tom Rini wrote:
On Mon, Dec 04, 2017 at 10:14:06AM -0700, Stephen Warren wrote:
On 12/04/2017 08:30 AM, Tom Rini wrote:
On Mon, Dec 04, 2017 at 03:21:04PM +0100, Michal Simek wrote:
On 4.12.2017 15:03, Tom Rini wrote:
On Mon, Dec 04, 2017 at 02:55:45PM +0100, Michal Simek wrote:
On 1.12.2017 23:44, Tom Rini wrote: > On Fri, Dec 01, 2017 at 10:07:54AM -0700, Stephen Warren wrote: >> On 12/01/2017 08:19 AM, Michal Simek wrote: >>> Hi, >>> >>> On 1.12.2017 16:06, Heinrich Schuchardt wrote: >>>> >>>> >>>> On 12/01/2017 03:46 PM, Michal Simek wrote: >>>>> Qemu for arm32/arm64 has a problem with time setup. >>>> >>>> Wouldn't it be preferable to fix the root cause? >>> >>> Definitely that would be the best and IIRC I have tried to convince our >>> qemu guy to do that but they have never done that. >> >> What is the exact failure condition? Is it simply that the test is still >> slightly too strict about which delays it accepts, or is sleep outright >> broken? >> >> You can use command-line option -k to avoid some tests. For example "-k not >> sleep". That way, we don't have to hard-code the dependency into the test >> source. Depending on the root cause (issue in U-Boot, or issue in just your >> local version of qemu, or something that will never work) this might be >> better? > > Even with the most recent relaxing of the sleep test requirements, I can > still (depending on overall system load) have 'sleep' take too long, on > QEMU. I think it might have been half a second slow, but I don't have > the log handy anymore. Both locally and in travis we -k not sleep all > of the qemu instances.
ok. By locally do you mean just using -k not sleep?
Yes, I have that in my CI scripts and similar.
Wouldn't be easier to keep this in uboot-test-hooks repo with other target setting?
Or do as you did did and mark the tests as not allowed for qemu, yes.
What we are trying to do is that our testing group will run these tests for me that's why it is just easier for me to change local uboot-test-hooks repo instead of communicate with them what -k not XXX parameters to add to certain scripts.
It means in loop they will just run all tests on qemu, local targets and in boardfarm. It is probably not big deal to tell them to add -k not sleep for all qemu runs but I know that for some i2c testing qemu doesn't emulate these devices that's why these tests fails. And the amount of tests which we shouldn't run on qemu will probably grow.
Well, I'm still open to possibly tweaking the allowed variance in the sleep test. OTOH, if we just say "no QEMU" here, we can then go back to "sleep should be pretty darn accurate on HW" for the test too, and perhaps that's best.
The fundamental problem of "over-sleeping" due to host system load/.. exists with all boards. There's nothing specific to qemu here except that running U-Boot on qemu on the host rather than on separate HW might more easily trigger the "high load on the host" condition; I see the issue now and then and manually retry that test, although that is a bit annoying.
The original test was mostly intended to make sure that e U-Boot clock didn't run at a significantly different rate to the host, since I had seen that issue during development of some board support or as a regression sometime. Perhaps the definition of "significantly different" should be more like "1/2 rate or twice rate or more" rather than "off by a small fraction of a second". That might avoid so many false positives.
I've pushed this up to 10 seconds and 0.5s worth of overrun and on qemu-mips here I see a 13.2s sleep. That's pretty close to 1/3rd fast and to me a wrong-clocking value, yes?
For me the qemu-x86 build of mid-Nov commit of U-Boot running under the same qemu version that U-Boot's Travis CI builds use, "sleep 10" takes about 10.5 seconds (including my reaction time), so ~13.2 does sound like it's probably a bug. Or maybe qemu just isn't fast enough in its emulation to keep up with real-time? I'd hope not for something simple like this, assuming you're using a recent CPU, but maybe.
On Tue, Dec 05, 2017 at 11:20:57AM -0700, Stephen Warren wrote:
On 12/04/2017 04:21 PM, Tom Rini wrote:
On Mon, Dec 04, 2017 at 10:14:06AM -0700, Stephen Warren wrote:
On 12/04/2017 08:30 AM, Tom Rini wrote:
On Mon, Dec 04, 2017 at 03:21:04PM +0100, Michal Simek wrote:
On 4.12.2017 15:03, Tom Rini wrote:
On Mon, Dec 04, 2017 at 02:55:45PM +0100, Michal Simek wrote: >On 1.12.2017 23:44, Tom Rini wrote: >>On Fri, Dec 01, 2017 at 10:07:54AM -0700, Stephen Warren wrote: >>>On 12/01/2017 08:19 AM, Michal Simek wrote: >>>>Hi, >>>> >>>>On 1.12.2017 16:06, Heinrich Schuchardt wrote: >>>>> >>>>> >>>>>On 12/01/2017 03:46 PM, Michal Simek wrote: >>>>>>Qemu for arm32/arm64 has a problem with time setup. >>>>> >>>>>Wouldn't it be preferable to fix the root cause? >>>> >>>>Definitely that would be the best and IIRC I have tried to convince our >>>>qemu guy to do that but they have never done that. >>> >>>What is the exact failure condition? Is it simply that the test is still >>>slightly too strict about which delays it accepts, or is sleep outright >>>broken? >>> >>>You can use command-line option -k to avoid some tests. For example "-k not >>>sleep". That way, we don't have to hard-code the dependency into the test >>>source. Depending on the root cause (issue in U-Boot, or issue in just your >>>local version of qemu, or something that will never work) this might be >>>better? >> >>Even with the most recent relaxing of the sleep test requirements, I can >>still (depending on overall system load) have 'sleep' take too long, on >>QEMU. I think it might have been half a second slow, but I don't have >>the log handy anymore. Both locally and in travis we -k not sleep all >>of the qemu instances. > >ok. By locally do you mean just using -k not sleep?
Yes, I have that in my CI scripts and similar.
Wouldn't be easier to keep this in uboot-test-hooks repo with other target setting?
Or do as you did did and mark the tests as not allowed for qemu, yes.
What we are trying to do is that our testing group will run these tests for me that's why it is just easier for me to change local uboot-test-hooks repo instead of communicate with them what -k not XXX parameters to add to certain scripts.
It means in loop they will just run all tests on qemu, local targets and in boardfarm. It is probably not big deal to tell them to add -k not sleep for all qemu runs but I know that for some i2c testing qemu doesn't emulate these devices that's why these tests fails. And the amount of tests which we shouldn't run on qemu will probably grow.
Well, I'm still open to possibly tweaking the allowed variance in the sleep test. OTOH, if we just say "no QEMU" here, we can then go back to "sleep should be pretty darn accurate on HW" for the test too, and perhaps that's best.
The fundamental problem of "over-sleeping" due to host system load/.. exists with all boards. There's nothing specific to qemu here except that running U-Boot on qemu on the host rather than on separate HW might more easily trigger the "high load on the host" condition; I see the issue now and then and manually retry that test, although that is a bit annoying.
The original test was mostly intended to make sure that e U-Boot clock didn't run at a significantly different rate to the host, since I had seen that issue during development of some board support or as a regression sometime. Perhaps the definition of "significantly different" should be more like "1/2 rate or twice rate or more" rather than "off by a small fraction of a second". That might avoid so many false positives.
I've pushed this up to 10 seconds and 0.5s worth of overrun and on qemu-mips here I see a 13.2s sleep. That's pretty close to 1/3rd fast and to me a wrong-clocking value, yes?
For me the qemu-x86 build of mid-Nov commit of U-Boot running under the same qemu version that U-Boot's Travis CI builds use, "sleep 10" takes about 10.5 seconds (including my reaction time), so ~13.2 does sound like it's probably a bug. Or maybe qemu just isn't fast enough in its emulation to keep up with real-time? I'd hope not for something simple like this, assuming you're using a recent CPU, but maybe.
Yeah, I can do x86, ARM and PowerPC but it fails on MIPS. And my build box isn't super new but an 8core/16thread E5-2670 should be good enough :)
Hey Daniel, do you have test.py running on real MIPS hardware? Thanks!
On 05.12.2017 19:38, Tom Rini wrote:
On Tue, Dec 05, 2017 at 11:20:57AM -0700, Stephen Warren wrote:
On 12/04/2017 04:21 PM, Tom Rini wrote:
On Mon, Dec 04, 2017 at 10:14:06AM -0700, Stephen Warren wrote:
On 12/04/2017 08:30 AM, Tom Rini wrote:
On Mon, Dec 04, 2017 at 03:21:04PM +0100, Michal Simek wrote:
On 4.12.2017 15:03, Tom Rini wrote: > On Mon, Dec 04, 2017 at 02:55:45PM +0100, Michal Simek wrote: >> On 1.12.2017 23:44, Tom Rini wrote: >>> On Fri, Dec 01, 2017 at 10:07:54AM -0700, Stephen Warren wrote: >>>> On 12/01/2017 08:19 AM, Michal Simek wrote: >>>>> Hi, >>>>> >>>>> On 1.12.2017 16:06, Heinrich Schuchardt wrote: >>>>>> >>>>>> >>>>>> On 12/01/2017 03:46 PM, Michal Simek wrote: >>>>>>> Qemu for arm32/arm64 has a problem with time setup. >>>>>> >>>>>> Wouldn't it be preferable to fix the root cause? >>>>> >>>>> Definitely that would be the best and IIRC I have tried to convince our >>>>> qemu guy to do that but they have never done that. >>>> >>>> What is the exact failure condition? Is it simply that the test is still >>>> slightly too strict about which delays it accepts, or is sleep outright >>>> broken? >>>> >>>> You can use command-line option -k to avoid some tests. For example "-k not >>>> sleep". That way, we don't have to hard-code the dependency into the test >>>> source. Depending on the root cause (issue in U-Boot, or issue in just your >>>> local version of qemu, or something that will never work) this might be >>>> better? >>> >>> Even with the most recent relaxing of the sleep test requirements, I can >>> still (depending on overall system load) have 'sleep' take too long, on >>> QEMU. I think it might have been half a second slow, but I don't have >>> the log handy anymore. Both locally and in travis we -k not sleep all >>> of the qemu instances. >> >> ok. By locally do you mean just using -k not sleep? > > Yes, I have that in my CI scripts and similar.
Wouldn't be easier to keep this in uboot-test-hooks repo with other target setting?
Or do as you did did and mark the tests as not allowed for qemu, yes.
What we are trying to do is that our testing group will run these tests for me that's why it is just easier for me to change local uboot-test-hooks repo instead of communicate with them what -k not XXX parameters to add to certain scripts.
It means in loop they will just run all tests on qemu, local targets and in boardfarm. It is probably not big deal to tell them to add -k not sleep for all qemu runs but I know that for some i2c testing qemu doesn't emulate these devices that's why these tests fails. And the amount of tests which we shouldn't run on qemu will probably grow.
Well, I'm still open to possibly tweaking the allowed variance in the sleep test. OTOH, if we just say "no QEMU" here, we can then go back to "sleep should be pretty darn accurate on HW" for the test too, and perhaps that's best.
The fundamental problem of "over-sleeping" due to host system load/.. exists with all boards. There's nothing specific to qemu here except that running U-Boot on qemu on the host rather than on separate HW might more easily trigger the "high load on the host" condition; I see the issue now and then and manually retry that test, although that is a bit annoying.
The original test was mostly intended to make sure that e U-Boot clock didn't run at a significantly different rate to the host, since I had seen that issue during development of some board support or as a regression sometime. Perhaps the definition of "significantly different" should be more like "1/2 rate or twice rate or more" rather than "off by a small fraction of a second". That might avoid so many false positives.
I've pushed this up to 10 seconds and 0.5s worth of overrun and on qemu-mips here I see a 13.2s sleep. That's pretty close to 1/3rd fast and to me a wrong-clocking value, yes?
For me the qemu-x86 build of mid-Nov commit of U-Boot running under the same qemu version that U-Boot's Travis CI builds use, "sleep 10" takes about 10.5 seconds (including my reaction time), so ~13.2 does sound like it's probably a bug. Or maybe qemu just isn't fast enough in its emulation to keep up with real-time? I'd hope not for something simple like this, assuming you're using a recent CPU, but maybe.
Yeah, I can do x86, ARM and PowerPC but it fails on MIPS. And my build box isn't super new but an 8core/16thread E5-2670 should be good enough :)
the problem with MIPS is that the timer frequency is hard-coded with config option CONFIG_SYS_MIPS_TIMER_FREQ in all older boards and the boards supported by Qemu (qemu_mips, malta, boston). The reason is that the timer frequency can't be derived from some hardware register. This is only possible with modern MIPS SoCs, where one can derive the timer frequency directly (or indirectly via the CPU frequency) from a PLL.
Also I can't see if or how the relevant Qemu timers for MIPS (mips_gictimer.c, i8254.c) are throttled to a deterministic value. Therefore I assume the timers are running as fast as possible. Thus there always will be a discrepancy between the hard-coded U-Boot timer frequency and the Qemu timer frequency dependent on the host system where Qemu runs. Due to this it doesn't make sense to enable the sleep test on MIPS.
I can think of following options to fix this:
1) U-Boot gets the possibilty to read the actual timer frequency somehow from Qemu
2) estimate the timer frequency by reading the CPU counter register at different times. But than we need some reference time or emulated RTC. At least for the Malta board the kernel does this.
3) the Qemu timer frequency can be throttled to a deterministic value (maybe configurable via Qemu command line argument).
4) keep the sleep test disabled
Or does someone know if and how the QEMU_CLOCK_VIRTUAL frequency can be configured?
Hey Daniel, do you have test.py running on real MIPS hardware? Thanks!
unfortunately not. I don't have any of the boards supported in mainline. And the boards I have aren't in mainline yet ;)
On 5.12.2017 19:38, Tom Rini wrote:
On Tue, Dec 05, 2017 at 11:20:57AM -0700, Stephen Warren wrote:
On 12/04/2017 04:21 PM, Tom Rini wrote:
On Mon, Dec 04, 2017 at 10:14:06AM -0700, Stephen Warren wrote:
On 12/04/2017 08:30 AM, Tom Rini wrote:
On Mon, Dec 04, 2017 at 03:21:04PM +0100, Michal Simek wrote:
On 4.12.2017 15:03, Tom Rini wrote: > On Mon, Dec 04, 2017 at 02:55:45PM +0100, Michal Simek wrote: >> On 1.12.2017 23:44, Tom Rini wrote: >>> On Fri, Dec 01, 2017 at 10:07:54AM -0700, Stephen Warren wrote: >>>> On 12/01/2017 08:19 AM, Michal Simek wrote: >>>>> Hi, >>>>> >>>>> On 1.12.2017 16:06, Heinrich Schuchardt wrote: >>>>>> >>>>>> >>>>>> On 12/01/2017 03:46 PM, Michal Simek wrote: >>>>>>> Qemu for arm32/arm64 has a problem with time setup. >>>>>> >>>>>> Wouldn't it be preferable to fix the root cause? >>>>> >>>>> Definitely that would be the best and IIRC I have tried to convince our >>>>> qemu guy to do that but they have never done that. >>>> >>>> What is the exact failure condition? Is it simply that the test is still >>>> slightly too strict about which delays it accepts, or is sleep outright >>>> broken? >>>> >>>> You can use command-line option -k to avoid some tests. For example "-k not >>>> sleep". That way, we don't have to hard-code the dependency into the test >>>> source. Depending on the root cause (issue in U-Boot, or issue in just your >>>> local version of qemu, or something that will never work) this might be >>>> better? >>> >>> Even with the most recent relaxing of the sleep test requirements, I can >>> still (depending on overall system load) have 'sleep' take too long, on >>> QEMU. I think it might have been half a second slow, but I don't have >>> the log handy anymore. Both locally and in travis we -k not sleep all >>> of the qemu instances. >> >> ok. By locally do you mean just using -k not sleep? > > Yes, I have that in my CI scripts and similar.
Wouldn't be easier to keep this in uboot-test-hooks repo with other target setting?
Or do as you did did and mark the tests as not allowed for qemu, yes.
What we are trying to do is that our testing group will run these tests for me that's why it is just easier for me to change local uboot-test-hooks repo instead of communicate with them what -k not XXX parameters to add to certain scripts.
It means in loop they will just run all tests on qemu, local targets and in boardfarm. It is probably not big deal to tell them to add -k not sleep for all qemu runs but I know that for some i2c testing qemu doesn't emulate these devices that's why these tests fails. And the amount of tests which we shouldn't run on qemu will probably grow.
Well, I'm still open to possibly tweaking the allowed variance in the sleep test. OTOH, if we just say "no QEMU" here, we can then go back to "sleep should be pretty darn accurate on HW" for the test too, and perhaps that's best.
The fundamental problem of "over-sleeping" due to host system load/.. exists with all boards. There's nothing specific to qemu here except that running U-Boot on qemu on the host rather than on separate HW might more easily trigger the "high load on the host" condition; I see the issue now and then and manually retry that test, although that is a bit annoying.
The original test was mostly intended to make sure that e U-Boot clock didn't run at a significantly different rate to the host, since I had seen that issue during development of some board support or as a regression sometime. Perhaps the definition of "significantly different" should be more like "1/2 rate or twice rate or more" rather than "off by a small fraction of a second". That might avoid so many false positives.
I've pushed this up to 10 seconds and 0.5s worth of overrun and on qemu-mips here I see a 13.2s sleep. That's pretty close to 1/3rd fast and to me a wrong-clocking value, yes?
For me the qemu-x86 build of mid-Nov commit of U-Boot running under the same qemu version that U-Boot's Travis CI builds use, "sleep 10" takes about 10.5 seconds (including my reaction time), so ~13.2 does sound like it's probably a bug. Or maybe qemu just isn't fast enough in its emulation to keep up with real-time? I'd hope not for something simple like this, assuming you're using a recent CPU, but maybe.
Yeah, I can do x86, ARM and PowerPC but it fails on MIPS. And my build box isn't super new but an 8core/16thread E5-2670 should be good enough :)
I should spend some time to add also microblaze. :-)
Thanks, Michal
On 4.12.2017 18:14, Stephen Warren wrote:
On 12/04/2017 08:30 AM, Tom Rini wrote:
On Mon, Dec 04, 2017 at 03:21:04PM +0100, Michal Simek wrote:
On 4.12.2017 15:03, Tom Rini wrote:
On Mon, Dec 04, 2017 at 02:55:45PM +0100, Michal Simek wrote:
On 1.12.2017 23:44, Tom Rini wrote:
On Fri, Dec 01, 2017 at 10:07:54AM -0700, Stephen Warren wrote: > On 12/01/2017 08:19 AM, Michal Simek wrote: >> Hi, >> >> On 1.12.2017 16:06, Heinrich Schuchardt wrote: >>> >>> >>> On 12/01/2017 03:46 PM, Michal Simek wrote: >>>> Qemu for arm32/arm64 has a problem with time setup. >>> >>> Wouldn't it be preferable to fix the root cause? >> >> Definitely that would be the best and IIRC I have tried to >> convince our >> qemu guy to do that but they have never done that. > > What is the exact failure condition? Is it simply that the test > is still > slightly too strict about which delays it accepts, or is sleep > outright > broken? > > You can use command-line option -k to avoid some tests. For > example "-k not > sleep". That way, we don't have to hard-code the dependency into > the test > source. Depending on the root cause (issue in U-Boot, or issue in > just your > local version of qemu, or something that will never work) this > might be > better?
Even with the most recent relaxing of the sleep test requirements, I can still (depending on overall system load) have 'sleep' take too long, on QEMU. I think it might have been half a second slow, but I don't have the log handy anymore. Both locally and in travis we -k not sleep all of the qemu instances.
ok. By locally do you mean just using -k not sleep?
Yes, I have that in my CI scripts and similar.
Wouldn't be easier to keep this in uboot-test-hooks repo with other target setting?
Or do as you did did and mark the tests as not allowed for qemu, yes.
What we are trying to do is that our testing group will run these tests for me that's why it is just easier for me to change local uboot-test-hooks repo instead of communicate with them what -k not XXX parameters to add to certain scripts.
It means in loop they will just run all tests on qemu, local targets and in boardfarm. It is probably not big deal to tell them to add -k not sleep for all qemu runs but I know that for some i2c testing qemu doesn't emulate these devices that's why these tests fails. And the amount of tests which we shouldn't run on qemu will probably grow.
Well, I'm still open to possibly tweaking the allowed variance in the sleep test. OTOH, if we just say "no QEMU" here, we can then go back to "sleep should be pretty darn accurate on HW" for the test too, and perhaps that's best.
The fundamental problem of "over-sleeping" due to host system load/.. exists with all boards. There's nothing specific to qemu here except that running U-Boot on qemu on the host rather than on separate HW might more easily trigger the "high load on the host" condition; I see the issue now and then and manually retry that test, although that is a bit annoying.
The original test was mostly intended to make sure that e U-Boot clock didn't run at a significantly different rate to the host, since I had seen that issue during development of some board support or as a regression sometime. Perhaps the definition of "significantly different" should be more like "1/2 rate or twice rate or more" rather than "off by a small fraction of a second". That might avoid so many false positives.
We had this issue with silicon v1 and having accurate sleep measuring is definitely good thing to have (Probably make sense to enable margin setup via variable anyway).
But still I would extend this to more wider discussion how to disable just one particular test case which is verified that it is broken on certain target/target configuration. Using -k not XXX option is possible but as I said before it is not ideal to keep track of these problematic tests in two locations and share this between two teams.
Better would be to add to u_boot_boardenv...py file line like this skip_test_sleep = True
Which would be parsed and test won't run for specific board/configuration. The same logic can be generic that user can add for example skip_test_net_dhcp = True to skip dhcp test for whatever reason.
Then for travis-ci we can just put these lines to py/travis-ci/.
What do you think?
Thanks, Michal
On Tue, Dec 05, 2017 at 01:10:47PM +0100, Michal Simek wrote:
On 4.12.2017 18:14, Stephen Warren wrote:
On 12/04/2017 08:30 AM, Tom Rini wrote:
On Mon, Dec 04, 2017 at 03:21:04PM +0100, Michal Simek wrote:
On 4.12.2017 15:03, Tom Rini wrote:
On Mon, Dec 04, 2017 at 02:55:45PM +0100, Michal Simek wrote:
On 1.12.2017 23:44, Tom Rini wrote: > On Fri, Dec 01, 2017 at 10:07:54AM -0700, Stephen Warren wrote: >> On 12/01/2017 08:19 AM, Michal Simek wrote: >>> Hi, >>> >>> On 1.12.2017 16:06, Heinrich Schuchardt wrote: >>>> >>>> >>>> On 12/01/2017 03:46 PM, Michal Simek wrote: >>>>> Qemu for arm32/arm64 has a problem with time setup. >>>> >>>> Wouldn't it be preferable to fix the root cause? >>> >>> Definitely that would be the best and IIRC I have tried to >>> convince our >>> qemu guy to do that but they have never done that. >> >> What is the exact failure condition? Is it simply that the test >> is still >> slightly too strict about which delays it accepts, or is sleep >> outright >> broken? >> >> You can use command-line option -k to avoid some tests. For >> example "-k not >> sleep". That way, we don't have to hard-code the dependency into >> the test >> source. Depending on the root cause (issue in U-Boot, or issue in >> just your >> local version of qemu, or something that will never work) this >> might be >> better? > > Even with the most recent relaxing of the sleep test requirements, > I can > still (depending on overall system load) have 'sleep' take too > long, on > QEMU. I think it might have been half a second slow, but I don't > have > the log handy anymore. Both locally and in travis we -k not sleep > all > of the qemu instances.
ok. By locally do you mean just using -k not sleep?
Yes, I have that in my CI scripts and similar.
Wouldn't be easier to keep this in uboot-test-hooks repo with other target setting?
Or do as you did did and mark the tests as not allowed for qemu, yes.
What we are trying to do is that our testing group will run these tests for me that's why it is just easier for me to change local uboot-test-hooks repo instead of communicate with them what -k not XXX parameters to add to certain scripts.
It means in loop they will just run all tests on qemu, local targets and in boardfarm. It is probably not big deal to tell them to add -k not sleep for all qemu runs but I know that for some i2c testing qemu doesn't emulate these devices that's why these tests fails. And the amount of tests which we shouldn't run on qemu will probably grow.
Well, I'm still open to possibly tweaking the allowed variance in the sleep test. OTOH, if we just say "no QEMU" here, we can then go back to "sleep should be pretty darn accurate on HW" for the test too, and perhaps that's best.
The fundamental problem of "over-sleeping" due to host system load/.. exists with all boards. There's nothing specific to qemu here except that running U-Boot on qemu on the host rather than on separate HW might more easily trigger the "high load on the host" condition; I see the issue now and then and manually retry that test, although that is a bit annoying.
The original test was mostly intended to make sure that e U-Boot clock didn't run at a significantly different rate to the host, since I had seen that issue during development of some board support or as a regression sometime. Perhaps the definition of "significantly different" should be more like "1/2 rate or twice rate or more" rather than "off by a small fraction of a second". That might avoid so many false positives.
We had this issue with silicon v1 and having accurate sleep measuring is definitely good thing to have (Probably make sense to enable margin setup via variable anyway).
But still I would extend this to more wider discussion how to disable just one particular test case which is verified that it is broken on certain target/target configuration. Using -k not XXX option is possible but as I said before it is not ideal to keep track of these problematic tests in two locations and share this between two teams.
Better would be to add to u_boot_boardenv...py file line like this skip_test_sleep = True
Which would be parsed and test won't run for specific board/configuration. The same logic can be generic that user can add for example skip_test_net_dhcp = True to skip dhcp test for whatever reason.
Then for travis-ci we can just put these lines to py/travis-ci/.
What do you think?
Ah, good idea! We have a few cases like this already, so how about env__sleep_accurate, default it to True and let the board files set it to false, and have test_sleep check for and use that?
On 5.12.2017 16:13, Tom Rini wrote:
On Tue, Dec 05, 2017 at 01:10:47PM +0100, Michal Simek wrote:
On 4.12.2017 18:14, Stephen Warren wrote:
On 12/04/2017 08:30 AM, Tom Rini wrote:
On Mon, Dec 04, 2017 at 03:21:04PM +0100, Michal Simek wrote:
On 4.12.2017 15:03, Tom Rini wrote:
On Mon, Dec 04, 2017 at 02:55:45PM +0100, Michal Simek wrote: > On 1.12.2017 23:44, Tom Rini wrote: >> On Fri, Dec 01, 2017 at 10:07:54AM -0700, Stephen Warren wrote: >>> On 12/01/2017 08:19 AM, Michal Simek wrote: >>>> Hi, >>>> >>>> On 1.12.2017 16:06, Heinrich Schuchardt wrote: >>>>> >>>>> >>>>> On 12/01/2017 03:46 PM, Michal Simek wrote: >>>>>> Qemu for arm32/arm64 has a problem with time setup. >>>>> >>>>> Wouldn't it be preferable to fix the root cause? >>>> >>>> Definitely that would be the best and IIRC I have tried to >>>> convince our >>>> qemu guy to do that but they have never done that. >>> >>> What is the exact failure condition? Is it simply that the test >>> is still >>> slightly too strict about which delays it accepts, or is sleep >>> outright >>> broken? >>> >>> You can use command-line option -k to avoid some tests. For >>> example "-k not >>> sleep". That way, we don't have to hard-code the dependency into >>> the test >>> source. Depending on the root cause (issue in U-Boot, or issue in >>> just your >>> local version of qemu, or something that will never work) this >>> might be >>> better? >> >> Even with the most recent relaxing of the sleep test requirements, >> I can >> still (depending on overall system load) have 'sleep' take too >> long, on >> QEMU. I think it might have been half a second slow, but I don't >> have >> the log handy anymore. Both locally and in travis we -k not sleep >> all >> of the qemu instances. > > ok. By locally do you mean just using -k not sleep?
Yes, I have that in my CI scripts and similar.
Wouldn't be easier to keep this in uboot-test-hooks repo with other target setting?
Or do as you did did and mark the tests as not allowed for qemu, yes.
What we are trying to do is that our testing group will run these tests for me that's why it is just easier for me to change local uboot-test-hooks repo instead of communicate with them what -k not XXX parameters to add to certain scripts.
It means in loop they will just run all tests on qemu, local targets and in boardfarm. It is probably not big deal to tell them to add -k not sleep for all qemu runs but I know that for some i2c testing qemu doesn't emulate these devices that's why these tests fails. And the amount of tests which we shouldn't run on qemu will probably grow.
Well, I'm still open to possibly tweaking the allowed variance in the sleep test. OTOH, if we just say "no QEMU" here, we can then go back to "sleep should be pretty darn accurate on HW" for the test too, and perhaps that's best.
The fundamental problem of "over-sleeping" due to host system load/.. exists with all boards. There's nothing specific to qemu here except that running U-Boot on qemu on the host rather than on separate HW might more easily trigger the "high load on the host" condition; I see the issue now and then and manually retry that test, although that is a bit annoying.
The original test was mostly intended to make sure that e U-Boot clock didn't run at a significantly different rate to the host, since I had seen that issue during development of some board support or as a regression sometime. Perhaps the definition of "significantly different" should be more like "1/2 rate or twice rate or more" rather than "off by a small fraction of a second". That might avoid so many false positives.
We had this issue with silicon v1 and having accurate sleep measuring is definitely good thing to have (Probably make sense to enable margin setup via variable anyway).
But still I would extend this to more wider discussion how to disable just one particular test case which is verified that it is broken on certain target/target configuration. Using -k not XXX option is possible but as I said before it is not ideal to keep track of these problematic tests in two locations and share this between two teams.
Better would be to add to u_boot_boardenv...py file line like this skip_test_sleep = True
Which would be parsed and test won't run for specific board/configuration. The same logic can be generic that user can add for example skip_test_net_dhcp = True to skip dhcp test for whatever reason.
Then for travis-ci we can just put these lines to py/travis-ci/.
What do you think?
Ah, good idea! We have a few cases like this already, so how about env__sleep_accurate, default it to True and let the board files set it to false, and have test_sleep check for and use that?
ok. this is about that variable for sleep not about generic skipping testcases which are failing.
Do you see the value to implement this? If yes. Stephen is there an easy way to do it? The reason why I ask is that values from that boardenv files are taken via u_boot_console.config.env.get
Thanks, Michal
On Wed, Dec 06, 2017 at 10:53:08AM +0100, Michal Simek wrote:
On 5.12.2017 16:13, Tom Rini wrote:
On Tue, Dec 05, 2017 at 01:10:47PM +0100, Michal Simek wrote:
On 4.12.2017 18:14, Stephen Warren wrote:
On 12/04/2017 08:30 AM, Tom Rini wrote:
On Mon, Dec 04, 2017 at 03:21:04PM +0100, Michal Simek wrote:
On 4.12.2017 15:03, Tom Rini wrote: > On Mon, Dec 04, 2017 at 02:55:45PM +0100, Michal Simek wrote: >> On 1.12.2017 23:44, Tom Rini wrote: >>> On Fri, Dec 01, 2017 at 10:07:54AM -0700, Stephen Warren wrote: >>>> On 12/01/2017 08:19 AM, Michal Simek wrote: >>>>> Hi, >>>>> >>>>> On 1.12.2017 16:06, Heinrich Schuchardt wrote: >>>>>> >>>>>> >>>>>> On 12/01/2017 03:46 PM, Michal Simek wrote: >>>>>>> Qemu for arm32/arm64 has a problem with time setup. >>>>>> >>>>>> Wouldn't it be preferable to fix the root cause? >>>>> >>>>> Definitely that would be the best and IIRC I have tried to >>>>> convince our >>>>> qemu guy to do that but they have never done that. >>>> >>>> What is the exact failure condition? Is it simply that the test >>>> is still >>>> slightly too strict about which delays it accepts, or is sleep >>>> outright >>>> broken? >>>> >>>> You can use command-line option -k to avoid some tests. For >>>> example "-k not >>>> sleep". That way, we don't have to hard-code the dependency into >>>> the test >>>> source. Depending on the root cause (issue in U-Boot, or issue in >>>> just your >>>> local version of qemu, or something that will never work) this >>>> might be >>>> better? >>> >>> Even with the most recent relaxing of the sleep test requirements, >>> I can >>> still (depending on overall system load) have 'sleep' take too >>> long, on >>> QEMU. I think it might have been half a second slow, but I don't >>> have >>> the log handy anymore. Both locally and in travis we -k not sleep >>> all >>> of the qemu instances. >> >> ok. By locally do you mean just using -k not sleep? > > Yes, I have that in my CI scripts and similar.
Wouldn't be easier to keep this in uboot-test-hooks repo with other target setting?
Or do as you did did and mark the tests as not allowed for qemu, yes.
What we are trying to do is that our testing group will run these tests for me that's why it is just easier for me to change local uboot-test-hooks repo instead of communicate with them what -k not XXX parameters to add to certain scripts.
It means in loop they will just run all tests on qemu, local targets and in boardfarm. It is probably not big deal to tell them to add -k not sleep for all qemu runs but I know that for some i2c testing qemu doesn't emulate these devices that's why these tests fails. And the amount of tests which we shouldn't run on qemu will probably grow.
Well, I'm still open to possibly tweaking the allowed variance in the sleep test. OTOH, if we just say "no QEMU" here, we can then go back to "sleep should be pretty darn accurate on HW" for the test too, and perhaps that's best.
The fundamental problem of "over-sleeping" due to host system load/.. exists with all boards. There's nothing specific to qemu here except that running U-Boot on qemu on the host rather than on separate HW might more easily trigger the "high load on the host" condition; I see the issue now and then and manually retry that test, although that is a bit annoying.
The original test was mostly intended to make sure that e U-Boot clock didn't run at a significantly different rate to the host, since I had seen that issue during development of some board support or as a regression sometime. Perhaps the definition of "significantly different" should be more like "1/2 rate or twice rate or more" rather than "off by a small fraction of a second". That might avoid so many false positives.
We had this issue with silicon v1 and having accurate sleep measuring is definitely good thing to have (Probably make sense to enable margin setup via variable anyway).
But still I would extend this to more wider discussion how to disable just one particular test case which is verified that it is broken on certain target/target configuration. Using -k not XXX option is possible but as I said before it is not ideal to keep track of these problematic tests in two locations and share this between two teams.
Better would be to add to u_boot_boardenv...py file line like this skip_test_sleep = True
Which would be parsed and test won't run for specific board/configuration. The same logic can be generic that user can add for example skip_test_net_dhcp = True to skip dhcp test for whatever reason.
Then for travis-ci we can just put these lines to py/travis-ci/.
What do you think?
Ah, good idea! We have a few cases like this already, so how about env__sleep_accurate, default it to True and let the board files set it to false, and have test_sleep check for and use that?
ok. this is about that variable for sleep not about generic skipping testcases which are failing.
Right. I'm not sure we need to get too complicated on "skip these tests" logic. But I think we have enough technical information now to say it's reasonable to skip sleep in some cases.
On 6.12.2017 13:17, Tom Rini wrote:
On Wed, Dec 06, 2017 at 10:53:08AM +0100, Michal Simek wrote:
On 5.12.2017 16:13, Tom Rini wrote:
On Tue, Dec 05, 2017 at 01:10:47PM +0100, Michal Simek wrote:
On 4.12.2017 18:14, Stephen Warren wrote:
On 12/04/2017 08:30 AM, Tom Rini wrote:
On Mon, Dec 04, 2017 at 03:21:04PM +0100, Michal Simek wrote: > On 4.12.2017 15:03, Tom Rini wrote: >> On Mon, Dec 04, 2017 at 02:55:45PM +0100, Michal Simek wrote: >>> On 1.12.2017 23:44, Tom Rini wrote: >>>> On Fri, Dec 01, 2017 at 10:07:54AM -0700, Stephen Warren wrote: >>>>> On 12/01/2017 08:19 AM, Michal Simek wrote: >>>>>> Hi, >>>>>> >>>>>> On 1.12.2017 16:06, Heinrich Schuchardt wrote: >>>>>>> >>>>>>> >>>>>>> On 12/01/2017 03:46 PM, Michal Simek wrote: >>>>>>>> Qemu for arm32/arm64 has a problem with time setup. >>>>>>> >>>>>>> Wouldn't it be preferable to fix the root cause? >>>>>> >>>>>> Definitely that would be the best and IIRC I have tried to >>>>>> convince our >>>>>> qemu guy to do that but they have never done that. >>>>> >>>>> What is the exact failure condition? Is it simply that the test >>>>> is still >>>>> slightly too strict about which delays it accepts, or is sleep >>>>> outright >>>>> broken? >>>>> >>>>> You can use command-line option -k to avoid some tests. For >>>>> example "-k not >>>>> sleep". That way, we don't have to hard-code the dependency into >>>>> the test >>>>> source. Depending on the root cause (issue in U-Boot, or issue in >>>>> just your >>>>> local version of qemu, or something that will never work) this >>>>> might be >>>>> better? >>>> >>>> Even with the most recent relaxing of the sleep test requirements, >>>> I can >>>> still (depending on overall system load) have 'sleep' take too >>>> long, on >>>> QEMU. I think it might have been half a second slow, but I don't >>>> have >>>> the log handy anymore. Both locally and in travis we -k not sleep >>>> all >>>> of the qemu instances. >>> >>> ok. By locally do you mean just using -k not sleep? >> >> Yes, I have that in my CI scripts and similar. > > Wouldn't be easier to keep this in uboot-test-hooks repo with other > target setting?
Or do as you did did and mark the tests as not allowed for qemu, yes.
> What we are trying to do is that our testing group will run these tests > for me that's why it is just easier for me to change local > uboot-test-hooks repo instead of communicate with them what -k not XXX > parameters to add to certain scripts. > > It means in loop they will just run all tests on qemu, local targets and > in boardfarm. It is probably not big deal to tell them to add -k not > sleep for all qemu runs but I know that for some i2c testing qemu > doesn't emulate these devices that's why these tests fails. And the > amount of tests which we shouldn't run on qemu will probably grow.
Well, I'm still open to possibly tweaking the allowed variance in the sleep test. OTOH, if we just say "no QEMU" here, we can then go back to "sleep should be pretty darn accurate on HW" for the test too, and perhaps that's best.
The fundamental problem of "over-sleeping" due to host system load/.. exists with all boards. There's nothing specific to qemu here except that running U-Boot on qemu on the host rather than on separate HW might more easily trigger the "high load on the host" condition; I see the issue now and then and manually retry that test, although that is a bit annoying.
The original test was mostly intended to make sure that e U-Boot clock didn't run at a significantly different rate to the host, since I had seen that issue during development of some board support or as a regression sometime. Perhaps the definition of "significantly different" should be more like "1/2 rate or twice rate or more" rather than "off by a small fraction of a second". That might avoid so many false positives.
We had this issue with silicon v1 and having accurate sleep measuring is definitely good thing to have (Probably make sense to enable margin setup via variable anyway).
But still I would extend this to more wider discussion how to disable just one particular test case which is verified that it is broken on certain target/target configuration. Using -k not XXX option is possible but as I said before it is not ideal to keep track of these problematic tests in two locations and share this between two teams.
Better would be to add to u_boot_boardenv...py file line like this skip_test_sleep = True
Which would be parsed and test won't run for specific board/configuration. The same logic can be generic that user can add for example skip_test_net_dhcp = True to skip dhcp test for whatever reason.
Then for travis-ci we can just put these lines to py/travis-ci/.
What do you think?
Ah, good idea! We have a few cases like this already, so how about env__sleep_accurate, default it to True and let the board files set it to false, and have test_sleep check for and use that?
ok. this is about that variable for sleep not about generic skipping testcases which are failing.
Right. I'm not sure we need to get too complicated on "skip these tests" logic. But I think we have enough technical information now to say it's reasonable to skip sleep in some cases.
Ok. Patch send please review.
Thanks, Michal
participants (6)
-
Daniel Schwierzeck -
Heinrich Schuchardt -
Michal Simek -
Michal Simek -
Stephen Warren -
Tom Rini