Cleanup notrun tests with the -M option. This is used in the weekly
build and fails because devices will run out of space.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Fixes: #69548
Support extending an existing board with new board variants.
This commit introduces the following changes to allow a board to be
extended out-of-tree.
The board yaml schema is extended to support an extend field which
will be used to identify the board to be extended.
A board 'plank' can be extended like this:
> board:
> extend: plank
> variants:
> - name: ext
> qualifier: soc1
For the rest of the build system this means that there is no longer a
single board directory.
The existing CMake variable BOARD_DIR is kept and reference the
directory which defines the board.
A new CMake variable BOARD_DIRECTORIES provides a list of all
directories which defines board targets for the board.
This means the directory which defines the board as well as all
directories that extends the board.
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
if alias or shorthand name is provided on the command line, we need
convert this to complete target name for the filters to work.
Fixes#80332
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
--device-testing implies runnable tests, this was the case before the
rework and was missed in one spot, so instead do that directly in the
argument parser.
Fixes#80428
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
We now use hwmv2 to list boards instead of relying on twister specific
config files.
One yaml files (twister.yaml for now) will have all the data needed for
all possible targets and variations of a board reusing most of the data
where possible and variations can override the top level data.
Twister keeps track of 'aliases' of boards and identifies that for
example native_sim is the same as native_sim/native, so either names
will be possible in both test yaml files or on the command line,
however, the reporting will always use the full name, so no there is no
confusion about what is being tested/built.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Adds a new NOTRUN status, which indicates
that a test was successfully built, but
not run on account of being not
runnable in given test instance.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
like extra_confs, add extr_args filterable by
soc:<arch>:<extra_args> or
platfrom:<platform name>: <extra_args>
simulation:<simulator name>: <extra_args>
which will only apply to given platform or arch
Signed-off-by: Hake Huang <hake.huang@oss.nxp.com>
Testcase execution time doesn't match between twister.xml and
twister.log. Testcase execution time is the sum of the previous
testcases' execution time plus its own execution time in
twister.xml.
This patch fixes the issue above.
Signed-off-by: Ioannis Damigos <ioannis.damigos.uj@renesas.com>
Fix trailing `\\r\\n` (escaped CR/LF) didn't cut off because of rstrip()
removed by #58338, so the CR/LF suffix was never found as the actual line
end was `\\r\\n\n`.
Add ANSI code sequence to `test_handlers` Twister unit test.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Separate the pickled EDT generation from the C-Macro header
generation in gen_defines.py to have a more clear responsibility
of the scripts in the DTS parsing process.
Signed-off-by: Benedikt Schmidt <benedikt.schmidt@embedded-solutions.at>
Currently, debug logging in the console and verbosity
are tightly coupled - verbosity of level 2 and higher
enables logging at the debug level.
This change introduces a separate Twister flag
responsible for controlling the debug logging,
while leaving the rest of verbosity unchanged.
This allows for controlling the verbosity on
both logging levels, according to one's needs.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
In case of problem with parsing hex data from coverage dump,
do not create empty gcda file.
Such empty file will break gcovr parsing.
Signed-off-by: Piotr Kosycarz <piotr.kosycarz@nordicsemi.no>
This small change concerns the following filter functions:
1. `dt_compat_enabled(C)`:
There's a node with compatible `C` and status "okay".
2. `dt_enabled_alias_with_parent_compat(A, C)`:
There's a node with alias `A` and status "okay", and its parent
has compatible `C`.
3. `dt_label_with_parent_compat_enabled(L, C)`:
There's a node with label `L`, and its parent has compatible `C`
and status "okay".
All three functions involve checking whether some node or its parent has
a given compatible, but the way this has been checked is inconsistent.
Function (1) has done it with this Python conditional:
compat in node.compats
while (2) and (3) have used:
parent.matching_compat == compat
The first check works well with nodes that have multiple compatibles,
and it is more aligned with the notion of "has_compat" as seen in the
devicetree macros for C, CMake, and Kconfig.
Arguably, `matching_compat` shouldn't have been used here, because it is
actually a property of a node's binding, moreso than of the node itself.
In practice, it's usually equal to the first compatible for which edtlib
has found a binding, which at first glance is just more constrained than
the `node.compats` check. However, there also exist obscure cases where
the `node.compats` are empty, while the `node.matching_compat` is not.
For now, the three functions can use a combined check, to improve
consistency and utility while avoiding breakage:
node.matching_compat == compat or compat in node.compats
Signed-off-by: Grzegorz Swiderski <grzegorz.swiderski@nordicsemi.no>
It has been deprecated since Zephyr v2.6.0.
The filter that replaced it - `dt_enabled_alias_with_parent_compat` -
had shared code with it, which can now be inlined into `ast_expr()` to
match the parser's overall code structure.
Signed-off-by: Grzegorz Swiderski <grzegorz.swiderski@nordicsemi.no>
When at verbosity 1, we print out the status of TestInstances.
This makes it harder to notice changes at TestCase level,
which require perusing the logs.
This adds TestCase status and reason printing
if verbosity level is 2 or more.
Reason printing is suppressed if the reason is empty or None.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Running sysbuilt tests fails because of missing "run" target.
This adds the default domain context to the command.
Signed-off-by: Wilfried Chauveau <wilfried.chauveau@arm.com>
Some dict.get() calls did not use a TwisterStatus
as a default value, thus using a NoneType where
TwisterStatus should appear.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Fixes bad usage of single backticks in lieu of double backticks for
rendering inline literals, or simple '*' for italics.
When appropriate, a better construct than double backticks has been
selected (ex. :file:, :kconfig:option:, :c:func:, ...), or proper :ref:
have been used if the original intention was to have a link.
Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
If you interrupt process() operation, we want Twister
to exit as gracefully as it can. This avoids the
UnboundLocalError that could appear e.g.
when interrupting the operation via SIGINT.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
This changes how some arguments are set in the `Handler`s.
`options`, `generator_cmd` and `suite_name_check` are now passed as
arguments to the constructor rather than injected from an other module.
Signed-off-by: Wilfried Chauveau <wilfried.chauveau@arm.com>
Python 3.12 warns that
> Testing an element's truth value will raise an exception in future
> versions. Use specific 'len(elem)' or 'elem is not None' test instead.
> if elem_ts := root.find('testsuite'):
Signed-off-by: Wilfried Chauveau <wilfried.chauveau@arm.com>
Space separated lists are deprecated but this notice is not checked for.
extract_fields_from_arg_list also converts lists back to space-separated
lists causing a warning on get_scenario
Signed-off-by: Wilfried Chauveau <wilfried.chauveau@arm.com>
Status errors previously logged an error, but didn't fail the running test.
This commit changes that
and introduces a new StatusAttributeError to use there.
One test is modified so it follows proper status form.
One test for the new error has been added.
Status errors now will properly mark the Instance as ERROR
and not run TestCases as SKIP.
This necessitated some code layout changes in runner.py
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Get data for various roots from modules and do not hardcode location
when retrieving list of boards in testplan.
Fixes#71761
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Re-arrange Twister command line options for test plan reporting
to a dedicated group with mutally-exclusive options to reflect
actual implementation and its dry-run execution mode.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
To specify the serial number, JLink expects either one argument
('--dev-id=xxx') or two (e.g. '--dev-id' 'xxx'), but it can not deal
with a single one that is '--dev-id xxx'.
The problem has been introduced (or just made visible?) by commit
5ee4284320 (twister: runner: j-link: use
dev-id instead of SelectEmuBySN) in PR #76931.
How to reproduce:
1. Create a HW configuration map, e.g.:
```
$ cat zephyr-hw-map-nrf52840dk-1.yml
- connected: true
id: '683517317'
platform: nrf52840dk/nrf52840
product: nRF52840 DK 1
runner: jlink
serial: /dev/ttyACM-nrf-dk-1
```
2. Run test `logging.dictionary` with Twister:
```
$ west twister --platform nrf52840dk/nrf52840 --device-testing \
--hardware-map zephyr-hw-map-nrf52840dk-1.yml -s logging.dictionary
```
3. The build will fail, and the `twister_harness.log` contains:
```
10:21:24.375:DEBUG:twister_harness.device.factory: Get device type
"hardware"
10:21:24.375:DEBUG:twister_harness.device.hardware_adapter: Opening
serial connection for /dev/ttyACM-nrf-dk-1
10:21:24.376:DEBUG:twister_harness.device.hardware_adapter: Flashing
device 683517317
10:21:24.376:DEBUG:twister_harness.device.hardware_adapter: Flashing
command: <snip>/bin/west flash --skip-rebuild --build-dir
twister-out/<snip>/tests/subsys/logging/dictionary/logging.dictionary
--runner jlink '--dev-id 683517317'
10:21:24.590:ERROR:twister_harness.device.hardware_adapter: Could not
flash device 683517317
10:21:24.592:DEBUG:twister_harness.device.hardware_adapter: Closed
serial connection for /dev/ttyACM-nrf-dk-1
```
(note the '--dev-id 683517317' part)
4. Running the stated `west flash` command shows the following error
message:
```
-- west flash: using runner jlink
FATAL ERROR: runner jlink received unknown arguments: ['--dev-id
683517317']
```
Signed-off-by: Reto Schneider <reto.schneider@husqvarnagroup.com>
Readline method sometimes receives more lines in buffer.
Split them to avoid misinterpreting data in harness module.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Use pytest-args parameters provided in command line after
parameters taken from yaml file. The last occurence is
considered by argparse, so parameters from command line
will be used.
Fixes: #77319
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
The gtest 1.15 has slightly different output from the original one that
was used to set up the harness. With this change we support both output
formats.
Fixes#72318
Signed-off-by: Yuval Peress <peress@google.com>
The serial number for debugger selection over USB
can be selected with the dev-id. This change
reflects also more the workflow of west flash
with J-Link.
The usage of SelectEmuBySN breaks the support for
J-Link over IP with twister.
Signed-off-by: Michael Arnold <marnold@baumer.com>
Extend pytest-args from configuration yaml file with args
from command line instead of overwriting.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Fix confusing `TypeError: 'type' object is not subscriptable`
at _get_installed_packages() if twister runs on python version < 3.9
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Improve DUT selection at DeviceHandler: for each DUT it counts
how many test instances have been failed on it during the current
twister execution, so the next available DUT will be chosen
ordering the eligible DUTs by less failures occured so far.
The new selection mechanism should increase chances to retry failed
tests on different DUTs, for instance to resolve ploblems when some
DUTs have connectivity or HW issues slowing down test plan execution,
or even block the execution when only one test suite runs whereas
the same first DUT candidate in the list is not working and others
were not chosen.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Change Twister PyTest plugin's test finalizing sequence to release
the DUT it is used as the very last operation, after the Test Instance
status becomes fully updated from the execution results.
This also fix a race condition possible when pytest plugin releases
the DUT and it becomes acquired by another test while the current test
is not yet finalized completely.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Twister DeviceHandler - add test failure counter for how many
test instances have been failed on each DUT (Device Under Test)
when it executes the current test plan.
Output DUT falure counter summary at the end of Twister run.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Fix Twister DeviceHandler exit on SerialException when
it connects to the serial device in 'flash before' mode.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Several improvements at Twister DeviceHandler when it releases
current DUT (Device Under Test):
- release the exact DUT which is used for the test instance
instead of all configured DUTs which happened to have
the same serial device configured.
- Twister PyTtest harness plugin adjustment to the above.
- additional debug logging to track DUT waiting/retain/release.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Updated the Pytest-Harness methods to capture and log error messages from
the subprocess, when pytest command fails.
Ensured that error messages are logged with `--inline-log` option
and placed in twister log reports.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
While the option to inform pytest about which platform a test is for
(platform being the board, such as mps2/an385) is available, it wasn't
being used. This patch fixes that, by building a pytest command which
includes the platform.
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
Add a new fixture, `unlaunched_dut`, which is basically the `dut`
one, but without launching the device - thus also not building the
application. It is useful for tests who need a finer control of
the building of the application, before the dut can be launched
to run it.
It will be used on a future patch, which will use it to enable LLEXT EDK
tests.
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
Fix after #71401 merge at Twister Reporting:
* filter status value type mismatch;
* tesplan.json generation incorrect error logs.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Incorrect status assignments were previously just warnings in logs.
Now they are elevated to errors.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Various different Statuses were joined into a single class,
TwisterStatus. This change anticipates further streamlining
of the Twister's approach to Status.
Code guarding Twister's Properties was shortened to a
value check only.
QEMUOutputStatus was left separate, as doubts were cast
whether it should remain a status. Leaving it separate makes
its removal easier.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
TestInstanceStatus of TIMEOUT and FLASH were never really used.
They were checked for, but never assigned.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Now statuses are not just a str that can be easily mistyped
or assigned wrong. Now they are an Enum.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
This change is removing some deprecation warnings
which for some reason causing failing tests with
Python 3.12 on CI #76877.
Also, it fixes warnings from pytest like:
PytestCollectionWarning: cannot collect test class 'TestPlan'
because it has a __init__ constructor
(from: scripts/tests/twister/test_testplan.py)
Signed-off-by: Lukasz Fundakowski <lukasz.fundakowski@nordicsemi.no>
When presenting an example of how to run a failing test case with
`west build`, provide the source dir path relative to the current
working directory, not the zephyr root directory.
Signed-off-by: Jordan Yates <jordan@embeint.com>
Read CONFIG_SHELL_PROMPT_UART from config file and use them
in shell fixture in pytest-harness package.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Moved helper methods from tests/boot/with_mcumgr to pytest-harness
package. It can be reused by other tests.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Twister scans C-files to find testcases that are implemented
using ZTest framework. Also runs scanning of Elf files
after building. Skip scanning files if it is not required.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
The default base timeout for pytest is statically set by
the TwisterHarnessConfig class to be 60 seconds. However,
sometimes it takes longer than 60s before the app starts
to run, especially on emulator/simulator where it takes
quite some time to start. So pass the test timeout as
the base timeout via pytest command line argument.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Disable branch coverage for the `__ASSERT` family of macros. Covering
all of the assertion branches by definition means triggering the
assertion, which can be either challenging or impossible to exercise,
and in either case results in the immediate termination of the test.
Signed-off-by: Jordan Yates <jordan@embeint.com>
Multiple values for `--exclude-branches-by-pattern` will result in only
the last value taking effect. Resolve this by merging all the provided
regex patterns into a single pattern with the `|` operator.
Signed-off-by: Jordan Yates <jordan@embeint.com>
The gTest harness asssumed that the lines end with the test name, but
some gTest implementations include the test duration in the line. Update
both the tests and regex to allow this and also avoid capturing
characters into the `test_name` that cannot be valid test name chars.
Fixes#72318
Signed-off-by: Yuval Peress <peress@google.com>
The current version of scipts do not consider OOT boards use cases and
the tests with robot now are strict to only one robot file, which is
not realistic for real environment. This address those issues and allow
multiple testsuits at command line and lists at tests entries. It add
another test parameter to allow configure robotframework options.
Fixes: #74563
Signed-off-by: Gerson Fernando Budke <gerson.budke@ossystems.com.br>
The flag won't work with sysbuild since there is no way to
reliably tell to a parser which data came from which image.
fixes: #74092
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Execute 'zephyr.exe' in application build directory as 'current working
directory' (cwd). This makes sure that native_sim specific drivers (like
flash simulator with file backend in 'flash.bin') are using unique context
for external resources with relative paths.
This fixes executing native_sim tests in twister with flash simulator.
Previously a shared 'flash.bin' was used for all executed 'zephyr.exe'
processes in twister. After this patch a unique 'flash.bin' file is used
for each tested sample, since those 'flash.bin' is placed in application
build directory instead of twister root directory.
Signed-off-by: Marcin Niestroj <m.niestroj@emb.dev>
Stop Twister if there are too many backup copies of the output
directory already.
Before this fix, Twister silently kept artifacts from the last run,
unless `--clobber-output` was explicitly given.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Extend the coverage tool to handle applications that generate multiple
gcov dumps in a single execution. This can happen when the application
calls `sys_reboot`.
Handling multiple dumps enables coverage testing of exception handlers.
Signed-off-by: Jordan Yates <jordan@embeint.com>
Pass Twister pytest plugin's log output, as well as output from
a test image running with pytest, up to Twister log irregardless
of the current verbosity level set at Twister.
This allows to collect full test output and also ensures that
recording data embedded in the test log is passed to the Twister
pytest harness for export.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Extend Twister Harness recording feature to allow selected data fields,
extracted from the log by a regular expression, to be parsed into JSON
objects and eventually reported in `twister.json` as `recording` list
property of the test suite.
With this extension, log records can convey layered data structures
passed from a test image as summary results, traces, statistics, etc.
This extension also allows flexible recording structure: a test image
can output different types of data records incapsulated into a fixed
set of fields, so `recording.csv` file columns are respected, whereas
some of the columns keep strings with json-encoded semi-structured data.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
More complex platforms require sysbuild to use always, even for
such "simple" samples like hello_world. Such platforms can have
`sysbuild: true` entry in their board_name.yaml used by twister.
Using such entry will tell twister, that sysbuild must always be used
on a given platform.
Twister is aligned to have information about need of sysbuild at
instance (platform + suite) level (was only at suite level before).
Instance.sysbuild is true whenever a test suite or a platform requires
sysbuild.
Twister pytest unit tests are aligned with changes.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Currently we launch qemu (well, "ninja run" usually) using Popen and
request stdout and stderr to be redirected into a pipe. However we never
read that pipe so the information is not captured.
Instead log directly into files that can be inspected after a failed
to to find out why qemu run failed.
Note that this is really only useful in cases where qemu either fails to
launch or crashes.
Regular test data is still handled via the qemu fifo.
Signed-off-by: Juha Kuikka <juha.kuikka@gmail.com>
we use reason for a failure to indicate state and then set the status
later and reason for the failure, in case of the failure is taken from
the handler status. Clean this up by setting status and reason coming
from the handler very early, so we do not have to go through replacing
meaning later.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
When dumping logs from the `--inline-logs` option, remove any coverage
information that may be contained in those logs. Coverage dumps are
unrelated to any test failures and make it harder to find the failing
test information. In extreme cases the relevant information is lost due
to terminal scrollback limits.
If the raw dump information is required, it is still present in the
original `handler.log` file.
Signed-off-by: Jordan Yates <jordan@embeint.com>
New Twister option `--footprint-report` is introduced to collect and
write detailed memory footprint results for symbols as an additional
JSON file. By default, the new option is disabled.
The new option implies and extends `--create-rom-ram-report`, so there
are three choices: 'ROM', 'RAM', and 'all' to select what memory area
symbols to report in `twister_footprint.json`.
In case of the custom report name, or per-platform report, it is always
composed with the rightmost '_footprint.json' suffix.
The memory footprint report has similar structure as `twister.json`
and compelements it having reduced set of test suite properties:
- instead of `testcases` it contains `footprint` object with
`rom.json` and `ram.json` artifacts embedded there;
- other properites are limited to represent only the essential test
suite context, thus to allow further data processing consistently
and independently from the `twister.json`.
- 'filtered' test instances are not included into the footprint report.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
When Twister composes resulting twister.json reports, add optional
filtering by a test instance resulting status and/or its individual
properties to be allowed/denied on output to the JSON file.
This internal feature is introduced to facilitate JSON reports with
a custom data schema.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Make sure Twster DeviceHandler serial-pty process is terminated
with all its remaining children to avoid Twister hanging on it
infinitely.
The reolved issue occurs sometimes, for example when serial-pty
script is used for serial port tunneling over network.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Filtered testcases are removed by default from Twister tests.
Older functionality is preserved via a new Twister flag:
--report-filtered.
Old tests were adjusted and a new test for that flag added.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Allow twister fixtures to contain extra information, which can be used for
test suite configuration. The extra information can be appended to existing
fixtures separated by a colon (i.e. <fixture>:<configuration>).
This is especially useful for the pytest harness, where a fixture of a
given type may need to refer to an instance of a particular piece of host
hardware needed by the pytest suite (e.g. a network interface, a UART, or a
CAN interface connected to the device under test).
Signed-off-by: Henrik Brix Andersen <hebad@vestas.com>
Pass the list of supported twister fixtures for a given platform to pytest
via DeviceConfig. This allows for the pytest suites to use knowledge of the
fixtures for test suite configuration.
Signed-off-by: Henrik Brix Andersen <hebad@vestas.com>
When something goes wrong with execution, due to twister issues and
bugs, do not continue with execution and abort the process and report
and return an error code.
Fixes#72807
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Read default domain from domains.yaml file and update
paths to proper build directory. It fixes native and qemu
pytest scenarios, when application is build with sysbuild.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Remove the deprecated uart_mux and gsm_mux modules and all of
their configurations/dependencies across zephyr.
Optimally uart_mux and gsm_mux would be removed in their own
respective PRs, but the two modules are directly coupled, so
to preserve bisectability, they must be removed together.
Signed-off-by: Bjarki Arge Andreasen <bjarki@arge-andreasen.me>
Initial integration with renode-test was introduced in commit bdf02ff,
which added support for calling the `renode-test` command from both west
and twister.
This commit removes the custom run_renode_test target used for running
Robot tests with the `west build` command and makes twister call
`renode-test` directly instead.
Signed-off-by: Michał Szprejda <mszprejda@antmicro.com>
The --flash-before flag allows devices to be flashed before the serial
connection is established. However, the implementation was incomplete and
only worked if the port address already existed at the start of the run.
This is incompatible with devices that implement the USB in software
(eg: USB-CDC).
This commit fixes the implementation to delay setting up the connection
until after the device is flashed, and to retry the connection for two
seconds after flashing to give the device time to enumerate as a USB
device.
Signed-off-by: Mike Szczys <mike@golioth.io>
Terminate QEMU earlier if the test finishes and no output data is
received for 1 second. This new timeout operates in parallel with the
global test timeout.
For coverage testing, this can reduce the time spent running individual
tests by up to 29 seconds, while still giving the full 30 extra seconds
to dump gcov data if needed.
Signed-off-by: Jordan Yates <jordan@embeint.com>
In the case where a test causes the test executor to crash, the stderr
is currently lost, making it hard to debug failures. This changes it
so that the process' stderr gets captured to 'handler_stderr.log' for
inspection.
Signed-off-by: Benjamin Gwin <bgwin@google.com>
PR #72100 broke execution of unit type tests. This commit fixes
it by bringing back the "old" logic for selecting a binary path
for unit type tests.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Fix default command line options check dependency on `--device-testing`
when it is provided to run Twister and turn device command options
as mandatory.
Fixes after changes introduced with #72399.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Store Twister command line options in twister.json report as
`environment.options` object. It allows to keep the actual
execution context for test results analysis and history comparison.
A new command line option `--report-all-options` enables to report all
command line options applied, including these set as default values.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
in many platforms, before the debuger flash, the former testsuite
many already executed, so we will see some old testsuite name.
and we should not compare them directly, instead if we compare in
reversed order them we can avoid such mis-judge.
Signed-off-by: Hake Huang <hake.huang@oss.nxp.com>
Fix issue 72083. Update path to zephyr.exe binary using
default domain from domains.yaml file.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Added a switch that show failed/error report from the last run.
Default shows all items found. However, you can specify the number of items
(e.g. --report-summary 15).
It also works well with the --outdir switch
Signed-off-by: Kamil Paszkiet <kamilx.paszkiet@intel.com>
Extend Pytest Harness to support 'recording' feature to parse
test log by a regular expression and collect as records the same
way as Console Harness do.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Refactor Twister 'recording' feature moving it from Handler class
to TestInstance class and enable it also for other Harness child
classes other than Console.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Using `erase` with west-flash was blocked as it was
messing with sysbuild. With #69748 the issue is fixed, hence
'erase' is no longer blocked.
Remove obsolete twister test
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Fix Twister to generate platform reports (`--platform-reports`)
only once for each platform instead of repeating it multiple times
for each instance executed.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Disable the branch coverage calculations on the `LOG_*` family of macros.
Branch misses are due to the implementation of `Z_LOG2` and cannot be
reasonably covered in library code. The internal paths through the
`Z_LOG2` macro are not of interest when inspecting files that happen to
use the Zephyr logging API.
Signed-off-by: Jordan Yates <jordan@embeint.com>
sometime we need use quarantine at runtime when --test-only
add this will help to skip problematic cases in test cycle
to reenable the quarantined case you need add
--quarantine-list quarantine.yaml --quarantine-verify
Signed-off-by: Hake Huang <hake.huang@oss.nxp.com>
Current find_v2_boards() has a lot of unnecessarry fluff.
We can make a new version of that function without it
given its use in TestPlan.add_configurations().
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Sped up Twister via name prefix loading.
If the only thing influencing the platform selection
is the platform filter (-p), then we only load *.yaml
files that start the same way as those selected platforms.
We split the platform name to the first '/' or '@'.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Spell checking tools do not recognize "iff", replace with "if and only if".
See https://en.wikipedia.org/wiki/If_and_only_if
Signed-off-by: Pieter De Gendt <pieter.degendt@basalte.be>
Removed Harness's repeat field.
Added to schemas in e722db14ad,
together with the rest of harness_config.
Added to Harness when it was created.
Now present only in one sample -
mec15xxevb_assy6853 power management sample.
Could not find definitive proof whether it has ever been used.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
esp32 platforms can overflow its dram0_0_seg and dram0_1_seg.
So update current dram section to meet both cases.
Signed-off-by: Sylvio Alves <sylvio.alves@espressif.com>
Removed following fields from relevant Handlers,
as they were unused in code:
* Handler's state
* Handler's generator
* BinaryHandler's call_west_flash
* QEMUHandler's results
* QEMUWinHandler's results
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Pytest harness in Twister tries to get custom parameters to
the pytest harnesses. It is required although those do not exist
(e.g. Twister uses devices without hw map). This change checks
if the custom parameters to the pytest harnesses exists.
Fixes#71817
Signed-off-by: Katarzyna Giądła <katarzyna.giadla@nordicsemi.no>
Make the --list-tests and --tests-tree options outputs
respect the --tag and --exclude-tag options,
so that only the tests for the specified tags are listed.
Also update the TestPlan report testcases for this change.
Signed-off-by: Declan Snyder <declan.snyder@nxp.com>
If the src_dir_path is a child of testsuite_path,
some c files might be scanned twice.
To prevent that, we check for parentage.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Add error handling for 'gather_metrics' stage and separate
build issues from 'scripts/footprint/size_report' issues.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
In some cases, Twister would report twice the amount
of testcases run than expected, with half of them
lacking any status.
This change fixes erroneous ELF testcase name extraction,
which deleted every instance of 'test_' in the name,
rather than just the first.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Fail a test run gracefully when an unknown test level
is requested to execute with the current test plan.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Use 'runner_params' specified in hardware map YAML file. This allows to
configure custom params (like openocd's adapter configuration for
FT232-based probe) when used with pytest harness.
Signed-off-by: Marcin Niestroj <m.niestroj@emb.dev>
Some NXP boards are not detected by twister when creating hardware map,
because manufacture name is NXP instead of NXP Semiconductors expected
by twister.
Fix it by adding NXP to manufacturers list.
Signed-off-by: Michal Smola <michal.smola@nxp.com>
Improve Twister command line arguments for memory footprint:
* group and order footprint arguments meaningfully,
* clearer help descriptions,
* resolve logical inconsistences for combinations of arguments,
in particular:
`--last-metrics` now forces `--enable-size-report`,
`--show-footprint` now controls only detailed log output of
footprint deltas in comparison modes.
* align twister tests accordingly.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
When sending commands containing something like "AT+CFUN" the expected
regex was not escaped and as a result such regex never matched what has
been echoed over serial.
Escape all commands with re.escape() so that echoed shell commands are
always matched, even when they contain special characters (like '+' in the
example above).
Signed-off-by: Marcin Niestroj <m.niestroj@emb.dev>
ANSI color is great for viewing output in a terminal, but currently it
also gets written out to the `handler.log` file. Text editors usually
don't render these, resulting in a hard-to-read file with a lot of
gibberish interspered. This commit strips ANSI sequences from lines
before writing them to the handler log file. This change does not
affect what is printed in Twister's console output, so one would still
see the colors there.
Signed-off-by: Tristan Honscheid <honscheid@google.com>
Winodws user may use the `\` as path, but in twister we use the common `/`
as path separated, to avoid the mis-use, convert it to `/` in twister
first normpath, and then replace the os.sep
tested by:
For Linux Like:
west twister -p disco_l475_iot1 -s samples/hello_world/...
For Windows:
west twister -p disco_l475_iot1 -s samples\hello_world\...
fixing: #70310
Signed-off-by: Hake Huang <hake.huang@oss.nxp.com>
Add option to flash board before attach serial.
Current implementation performs the following sequence:
1. Open serial port to listen to board log output
2. Flash device
In case of ESP32 where it uses the same serial port
for both operations, flashing needs to come first.
This PR adds a twister option named --flash-before
which enables the process above, allowing tests to be
performed properly.
Signed-off-by: Lucas Tamborrino <lucas.tamborrino@espressif.com>
For using --device-testing with --device-serial or --device-serial-pty
it is necessary to provide the platform to be used for testing.
Not specifying a platform or specifying more than one is an error.
Signed-off-by: Luca Burelli <l.burelli@arduino.cc>
tfm_integration samples fail on Windows on mps2_an521 platform,
because output from ninja process running QEMU freezes in re-build
phase.
Fix it by implementing named pipes to read output from QEMU process
directly using os.open and os.read methods.
Signed-off-by: Michal Smola <michal.smola@nxp.com>
Follow-up: fe25709a9c twister: add unit_testing soc and board
Moving the soc.yml and board.yml to subsys/testsuite tree.
Introduce subsys/testsuite/arch/archs.yml with unit_testing arch.
Update the Zephyr CMake module unittest.cmake to include HWMv2 and
reorder includes to follow same order as zephyr_default.cmake.
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
The new hwmv2 platform names have "/" and "@" in
their names, we need to replace those with "_".
Otherwise the harness will produce a FileNotFoundError
exception when trying to create the executable.
Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
in hardwaremap the dev-id is used to select debug probe
and linkserver accept the --probe. so and this support
Signed-off-by: Hake Huang <hake.huang@oss.nxp.com>
If sysbuild is used a flag SB_CONFIG_COMPILER_WARNINGS_AS_ERRORS
has to be used in order to turn warings to errors on all images.
Align pytest tests with the change.
Fixes: #67360
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
https://github.com/zephyrproject-rtos/zephyr/pull/67595
introduces a bug where if QEMU_BIN_PATH is not set on a Windows
PC, on-device tests are skipped. This fixes that issue.
Signed-off-by: Glenn Andrews <glenn.andrews.42@gmail.com>
Requirements added for bz2.
Blackbox test for --package-artifacts added.
package.py no longer includes twister-out no matter the outdir.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Twister does not support running tests on QEMU on Windows.
Add a new twister handler called QEMUWinHandler which supports running
tests on Windows. Enable running tests in testinstance, only if
QEMU_BIN_PATH environment variable exists on Windows.
Modify pytest test for testinstance to reflect the change in
runnability of QEMU on Windows.
Tested by running several Ztest and console harness tests.
Signed-off-by: Michal Smola <michal.smola@nxp.com>
