Pass a test failure reason text from harness to the instance
to convey more details on the failure.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Set missing TestCase statuses when a test under the BinaryHandler failed
(crashed), so remaining 'STARTED' and 'NONE' are now 'BLOCK' instead.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Use with statement to open stdout/stderr and resolve current warning
when running twister:
ResourceWarning: unclosed file....
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This fixes ruff linting error UP007,
where one would use Optional or Union
instead of X | Y notation.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Fixes ruff linting errors UP030, UP031, UP032,
which make you use format specifiers (fstrings)
instead of printf formatting or str.format().
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
This fixes ruff linting error F541,
where an fstring is used when a normal
string would suffice.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
This fixes ruff linting error SIM115,
where files were opened without the use of
a context handler.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
This fixes ruff linting error SIM105,
where try-except-pass construct was used
instead of contextlib.suppress().
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
This fixes ruff linting error SIM102,
where multiple ifs were used instead
of combining them with and or similar
operators.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
This fixes ruff linting error SIM114,
where there are multiple arms in an
if-elif-else construct with the same result.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
This fixes ruff linting error SIM201,
where not is used instead of a more
appropriate comparison operator.
SIM202 exclusion was removed, as it
didn't apply to anything.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
This change introduces the ability in twister to select which
emulation/simulation tool to use on the command line.
If none is specified, it will select the first in the list.
Signed-off-by: Wilfried Chauveau <wilfried.chauveau@arm.com>
Extended hardware map to share a single board between variants.
To run tests for different variants on the same board
without re-configuring the hardware map file for each variant,
one can use a `platform` atribute as a list of names.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Fix trailing `\\r\\n` (escaped CR/LF) didn't cut off because of rstrip()
removed by #58338, so the CR/LF suffix was never found as the actual line
end was `\\r\\n\n`.
Add ANSI code sequence to `test_handlers` Twister unit test.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Running sysbuilt tests fails because of missing "run" target.
This adds the default domain context to the command.
Signed-off-by: Wilfried Chauveau <wilfried.chauveau@arm.com>
This changes how some arguments are set in the `Handler`s.
`options`, `generator_cmd` and `suite_name_check` are now passed as
arguments to the constructor rather than injected from an other module.
Signed-off-by: Wilfried Chauveau <wilfried.chauveau@arm.com>
Readline method sometimes receives more lines in buffer.
Split them to avoid misinterpreting data in harness module.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
The serial number for debugger selection over USB
can be selected with the dev-id. This change
reflects also more the workflow of west flash
with J-Link.
The usage of SelectEmuBySN breaks the support for
J-Link over IP with twister.
Signed-off-by: Michael Arnold <marnold@baumer.com>
Improve DUT selection at DeviceHandler: for each DUT it counts
how many test instances have been failed on it during the current
twister execution, so the next available DUT will be chosen
ordering the eligible DUTs by less failures occured so far.
The new selection mechanism should increase chances to retry failed
tests on different DUTs, for instance to resolve ploblems when some
DUTs have connectivity or HW issues slowing down test plan execution,
or even block the execution when only one test suite runs whereas
the same first DUT candidate in the list is not working and others
were not chosen.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Twister DeviceHandler - add test failure counter for how many
test instances have been failed on each DUT (Device Under Test)
when it executes the current test plan.
Output DUT falure counter summary at the end of Twister run.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Fix Twister DeviceHandler exit on SerialException when
it connects to the serial device in 'flash before' mode.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Several improvements at Twister DeviceHandler when it releases
current DUT (Device Under Test):
- release the exact DUT which is used for the test instance
instead of all configured DUTs which happened to have
the same serial device configured.
- Twister PyTtest harness plugin adjustment to the above.
- additional debug logging to track DUT waiting/retain/release.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Various different Statuses were joined into a single class,
TwisterStatus. This change anticipates further streamlining
of the Twister's approach to Status.
Code guarding Twister's Properties was shortened to a
value check only.
QEMUOutputStatus was left separate, as doubts were cast
whether it should remain a status. Leaving it separate makes
its removal easier.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Now statuses are not just a str that can be easily mistyped
or assigned wrong. Now they are an Enum.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
The current version of scipts do not consider OOT boards use cases and
the tests with robot now are strict to only one robot file, which is
not realistic for real environment. This address those issues and allow
multiple testsuits at command line and lists at tests entries. It add
another test parameter to allow configure robotframework options.
Fixes: #74563
Signed-off-by: Gerson Fernando Budke <gerson.budke@ossystems.com.br>
More complex platforms require sysbuild to use always, even for
such "simple" samples like hello_world. Such platforms can have
`sysbuild: true` entry in their board_name.yaml used by twister.
Using such entry will tell twister, that sysbuild must always be used
on a given platform.
Twister is aligned to have information about need of sysbuild at
instance (platform + suite) level (was only at suite level before).
Instance.sysbuild is true whenever a test suite or a platform requires
sysbuild.
Twister pytest unit tests are aligned with changes.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Currently we launch qemu (well, "ninja run" usually) using Popen and
request stdout and stderr to be redirected into a pipe. However we never
read that pipe so the information is not captured.
Instead log directly into files that can be inspected after a failed
to to find out why qemu run failed.
Note that this is really only useful in cases where qemu either fails to
launch or crashes.
Regular test data is still handled via the qemu fifo.
Signed-off-by: Juha Kuikka <juha.kuikka@gmail.com>
we use reason for a failure to indicate state and then set the status
later and reason for the failure, in case of the failure is taken from
the handler status. Clean this up by setting status and reason coming
from the handler very early, so we do not have to go through replacing
meaning later.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Make sure Twster DeviceHandler serial-pty process is terminated
with all its remaining children to avoid Twister hanging on it
infinitely.
The reolved issue occurs sometimes, for example when serial-pty
script is used for serial port tunneling over network.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Allow twister fixtures to contain extra information, which can be used for
test suite configuration. The extra information can be appended to existing
fixtures separated by a colon (i.e. <fixture>:<configuration>).
This is especially useful for the pytest harness, where a fixture of a
given type may need to refer to an instance of a particular piece of host
hardware needed by the pytest suite (e.g. a network interface, a UART, or a
CAN interface connected to the device under test).
Signed-off-by: Henrik Brix Andersen <hebad@vestas.com>
Initial integration with renode-test was introduced in commit bdf02ff,
which added support for calling the `renode-test` command from both west
and twister.
This commit removes the custom run_renode_test target used for running
Robot tests with the `west build` command and makes twister call
`renode-test` directly instead.
Signed-off-by: Michał Szprejda <mszprejda@antmicro.com>
Terminate QEMU earlier if the test finishes and no output data is
received for 1 second. This new timeout operates in parallel with the
global test timeout.
For coverage testing, this can reduce the time spent running individual
tests by up to 29 seconds, while still giving the full 30 extra seconds
to dump gcov data if needed.
Signed-off-by: Jordan Yates <jordan@embeint.com>
In the case where a test causes the test executor to crash, the stderr
is currently lost, making it hard to debug failures. This changes it
so that the process' stderr gets captured to 'handler_stderr.log' for
inspection.
Signed-off-by: Benjamin Gwin <bgwin@google.com>
PR #72100 broke execution of unit type tests. This commit fixes
it by bringing back the "old" logic for selecting a binary path
for unit type tests.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
in many platforms, before the debuger flash, the former testsuite
many already executed, so we will see some old testsuite name.
and we should not compare them directly, instead if we compare in
reversed order them we can avoid such mis-judge.
Signed-off-by: Hake Huang <hake.huang@oss.nxp.com>
Fix issue 72083. Update path to zephyr.exe binary using
default domain from domains.yaml file.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Refactor Twister 'recording' feature moving it from Handler class
to TestInstance class and enable it also for other Harness child
classes other than Console.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Removed following fields from relevant Handlers,
as they were unused in code:
* Handler's state
* Handler's generator
* BinaryHandler's call_west_flash
* QEMUHandler's results
* QEMUWinHandler's results
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
ANSI color is great for viewing output in a terminal, but currently it
also gets written out to the `handler.log` file. Text editors usually
don't render these, resulting in a hard-to-read file with a lot of
gibberish interspered. This commit strips ANSI sequences from lines
before writing them to the handler log file. This change does not
affect what is printed in Twister's console output, so one would still
see the colors there.
Signed-off-by: Tristan Honscheid <honscheid@google.com>
Add option to flash board before attach serial.
Current implementation performs the following sequence:
1. Open serial port to listen to board log output
2. Flash device
In case of ESP32 where it uses the same serial port
for both operations, flashing needs to come first.
This PR adds a twister option named --flash-before
which enables the process above, allowing tests to be
performed properly.
Signed-off-by: Lucas Tamborrino <lucas.tamborrino@espressif.com>
