Store Twister command line options in twister.json report as
`environment.options` object. It allows to keep the actual
execution context for test results analysis and history comparison.
A new command line option `--report-all-options` enables to report all
command line options applied, including these set as default values.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
in many platforms, before the debuger flash, the former testsuite
many already executed, so we will see some old testsuite name.
and we should not compare them directly, instead if we compare in
reversed order them we can avoid such mis-judge.
Signed-off-by: Hake Huang <hake.huang@oss.nxp.com>
Fix issue 72083. Update path to zephyr.exe binary using
default domain from domains.yaml file.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Added a switch that show failed/error report from the last run.
Default shows all items found. However, you can specify the number of items
(e.g. --report-summary 15).
It also works well with the --outdir switch
Signed-off-by: Kamil Paszkiet <kamilx.paszkiet@intel.com>
Extend Pytest Harness to support 'recording' feature to parse
test log by a regular expression and collect as records the same
way as Console Harness do.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Refactor Twister 'recording' feature moving it from Handler class
to TestInstance class and enable it also for other Harness child
classes other than Console.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Using `erase` with west-flash was blocked as it was
messing with sysbuild. With #69748 the issue is fixed, hence
'erase' is no longer blocked.
Remove obsolete twister test
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Fix Twister to generate platform reports (`--platform-reports`)
only once for each platform instead of repeating it multiple times
for each instance executed.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Disable the branch coverage calculations on the `LOG_*` family of macros.
Branch misses are due to the implementation of `Z_LOG2` and cannot be
reasonably covered in library code. The internal paths through the
`Z_LOG2` macro are not of interest when inspecting files that happen to
use the Zephyr logging API.
Signed-off-by: Jordan Yates <jordan@embeint.com>
sometime we need use quarantine at runtime when --test-only
add this will help to skip problematic cases in test cycle
to reenable the quarantined case you need add
--quarantine-list quarantine.yaml --quarantine-verify
Signed-off-by: Hake Huang <hake.huang@oss.nxp.com>
Current find_v2_boards() has a lot of unnecessarry fluff.
We can make a new version of that function without it
given its use in TestPlan.add_configurations().
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Sped up Twister via name prefix loading.
If the only thing influencing the platform selection
is the platform filter (-p), then we only load *.yaml
files that start the same way as those selected platforms.
We split the platform name to the first '/' or '@'.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Spell checking tools do not recognize "iff", replace with "if and only if".
See https://en.wikipedia.org/wiki/If_and_only_if
Signed-off-by: Pieter De Gendt <pieter.degendt@basalte.be>
Removed Harness's repeat field.
Added to schemas in e722db14ad,
together with the rest of harness_config.
Added to Harness when it was created.
Now present only in one sample -
mec15xxevb_assy6853 power management sample.
Could not find definitive proof whether it has ever been used.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
esp32 platforms can overflow its dram0_0_seg and dram0_1_seg.
So update current dram section to meet both cases.
Signed-off-by: Sylvio Alves <sylvio.alves@espressif.com>
Removed following fields from relevant Handlers,
as they were unused in code:
* Handler's state
* Handler's generator
* BinaryHandler's call_west_flash
* QEMUHandler's results
* QEMUWinHandler's results
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Pytest harness in Twister tries to get custom parameters to
the pytest harnesses. It is required although those do not exist
(e.g. Twister uses devices without hw map). This change checks
if the custom parameters to the pytest harnesses exists.
Fixes#71817
Signed-off-by: Katarzyna Giądła <katarzyna.giadla@nordicsemi.no>
Make the --list-tests and --tests-tree options outputs
respect the --tag and --exclude-tag options,
so that only the tests for the specified tags are listed.
Also update the TestPlan report testcases for this change.
Signed-off-by: Declan Snyder <declan.snyder@nxp.com>
If the src_dir_path is a child of testsuite_path,
some c files might be scanned twice.
To prevent that, we check for parentage.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Add error handling for 'gather_metrics' stage and separate
build issues from 'scripts/footprint/size_report' issues.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
In some cases, Twister would report twice the amount
of testcases run than expected, with half of them
lacking any status.
This change fixes erroneous ELF testcase name extraction,
which deleted every instance of 'test_' in the name,
rather than just the first.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Fail a test run gracefully when an unknown test level
is requested to execute with the current test plan.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Use 'runner_params' specified in hardware map YAML file. This allows to
configure custom params (like openocd's adapter configuration for
FT232-based probe) when used with pytest harness.
Signed-off-by: Marcin Niestroj <m.niestroj@emb.dev>
Some NXP boards are not detected by twister when creating hardware map,
because manufacture name is NXP instead of NXP Semiconductors expected
by twister.
Fix it by adding NXP to manufacturers list.
Signed-off-by: Michal Smola <michal.smola@nxp.com>
Improve Twister command line arguments for memory footprint:
* group and order footprint arguments meaningfully,
* clearer help descriptions,
* resolve logical inconsistences for combinations of arguments,
in particular:
`--last-metrics` now forces `--enable-size-report`,
`--show-footprint` now controls only detailed log output of
footprint deltas in comparison modes.
* align twister tests accordingly.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
When sending commands containing something like "AT+CFUN" the expected
regex was not escaped and as a result such regex never matched what has
been echoed over serial.
Escape all commands with re.escape() so that echoed shell commands are
always matched, even when they contain special characters (like '+' in the
example above).
Signed-off-by: Marcin Niestroj <m.niestroj@emb.dev>
ANSI color is great for viewing output in a terminal, but currently it
also gets written out to the `handler.log` file. Text editors usually
don't render these, resulting in a hard-to-read file with a lot of
gibberish interspered. This commit strips ANSI sequences from lines
before writing them to the handler log file. This change does not
affect what is printed in Twister's console output, so one would still
see the colors there.
Signed-off-by: Tristan Honscheid <honscheid@google.com>
Winodws user may use the `\` as path, but in twister we use the common `/`
as path separated, to avoid the mis-use, convert it to `/` in twister
first normpath, and then replace the os.sep
tested by:
For Linux Like:
west twister -p disco_l475_iot1 -s samples/hello_world/...
For Windows:
west twister -p disco_l475_iot1 -s samples\hello_world\...
fixing: #70310
Signed-off-by: Hake Huang <hake.huang@oss.nxp.com>
Add option to flash board before attach serial.
Current implementation performs the following sequence:
1. Open serial port to listen to board log output
2. Flash device
In case of ESP32 where it uses the same serial port
for both operations, flashing needs to come first.
This PR adds a twister option named --flash-before
which enables the process above, allowing tests to be
performed properly.
Signed-off-by: Lucas Tamborrino <lucas.tamborrino@espressif.com>
For using --device-testing with --device-serial or --device-serial-pty
it is necessary to provide the platform to be used for testing.
Not specifying a platform or specifying more than one is an error.
Signed-off-by: Luca Burelli <l.burelli@arduino.cc>
tfm_integration samples fail on Windows on mps2_an521 platform,
because output from ninja process running QEMU freezes in re-build
phase.
Fix it by implementing named pipes to read output from QEMU process
directly using os.open and os.read methods.
Signed-off-by: Michal Smola <michal.smola@nxp.com>
Follow-up: fe25709a9c twister: add unit_testing soc and board
Moving the soc.yml and board.yml to subsys/testsuite tree.
Introduce subsys/testsuite/arch/archs.yml with unit_testing arch.
Update the Zephyr CMake module unittest.cmake to include HWMv2 and
reorder includes to follow same order as zephyr_default.cmake.
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
The new hwmv2 platform names have "/" and "@" in
their names, we need to replace those with "_".
Otherwise the harness will produce a FileNotFoundError
exception when trying to create the executable.
Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
in hardwaremap the dev-id is used to select debug probe
and linkserver accept the --probe. so and this support
Signed-off-by: Hake Huang <hake.huang@oss.nxp.com>
If sysbuild is used a flag SB_CONFIG_COMPILER_WARNINGS_AS_ERRORS
has to be used in order to turn warings to errors on all images.
Align pytest tests with the change.
Fixes: #67360
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
https://github.com/zephyrproject-rtos/zephyr/pull/67595
introduces a bug where if QEMU_BIN_PATH is not set on a Windows
PC, on-device tests are skipped. This fixes that issue.
Signed-off-by: Glenn Andrews <glenn.andrews.42@gmail.com>
Requirements added for bz2.
Blackbox test for --package-artifacts added.
package.py no longer includes twister-out no matter the outdir.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Twister does not support running tests on QEMU on Windows.
Add a new twister handler called QEMUWinHandler which supports running
tests on Windows. Enable running tests in testinstance, only if
QEMU_BIN_PATH environment variable exists on Windows.
Modify pytest test for testinstance to reflect the change in
runnability of QEMU on Windows.
Tested by running several Ztest and console harness tests.
Signed-off-by: Michal Smola <michal.smola@nxp.com>
We were missing mocking for normalized_name and board detection was not
working correctly after moving to hwmv2.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
So that we can stop using the artificial riscv32/64 dual naming. This
patch temporarily allows using both, riscv or riscv32/64.
Signed-off-by: Gerard Marull-Paretas <gerard@teslabs.com>
Add integration_platforms to the list of the default platforms per test
for improved coverage when not running in integration mode.
Some tests when run without any option do not exercise any tests and
skip the integration platforms completely.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Enhance the -s option of twister, used to point to a single scenario in
a testsuite.
- Now accept --scenario on the command line, --test still supported.
- If no testsuite is provided, extract testsuite from scenario
identifier and avoid parsing the whole tree if we only want to address
one scenario in a testsuite.
- If just the identifier of the scenario is provided to
--test/--scenario option, try to find this as well, do not need the
full path for that.
Something like this is now possible:
twister --scenario kernel.threads.init --list-tests
twister -T <path> --scenario kernel.threads.init --list-tests
twister -T <path>/kernel.threads.init --list-tests
All should print the same output.
Fixes#67307
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Binaries are located in multiple build directory subfolders when built for
multiple domains (sysbuild is one example). Parse domains.yaml and preserve
files across all domains for testing when --prep-artifacts-for-testing
option is used.
Signed-off-by: Mike Szczys <szczys@hotmail.com>
Updated the instance with the device id to have it in the summary
report also when using pytest harness.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
On the hardware, after booting up the device, on the
console might appear additional logs after receiving first
prompt. Wait and clear the buffer to avoid unexpected
messages when verifying output.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Using ztest predicate feature, testcases are skipped, however, the skip
is only reported as part of the summary and not through normal
execution. Until now such tests were reported as blocked or had a null
status in the twister json output.
This changes will look into summary results and use them to confirm
parsed results and add any missing results that either were not reported
or not captured through the serial console.
The issue can be observed with the drivers.can.api test for example on
the frdm_k64f board.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
As saving tests writes to file, rather than appending to it,
we should indicate that in the --save-tests help.
--load-tests help changed so its grammar is the same as --save-tests's.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
When pytest harness test is run on harware, messages sent from target
right after application start-up are lost, because connection to COM
port is not established yet. It can cause unexpected
behavior of a test. Fix it by flashing and running application after
connecting to COM port when testing on hardware.
Signed-off-by: Michal Smola <michal.smola@nxp.com>
Import of pty module causes exception when pytest harness is used
for device testing on Windows. Fix it by importing pty module
on non-windows hosts only. Add logger message for case pty
is used by mistake on Windows.
Signed-off-by: Michal Smola <michal.smola@nxp.com>
Update helper method used by pytest fixtures. Extended upload
method with 'slot' parameter. Added searching of uploaded images
that can be tested or confirmed.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
When twister is run on Windows with --short-build-path option, mklink
fails to create link, because path to build dir contains forward
slashes, which are not handled correctly by mklink.
Fix it by using os.path.normpath in mklink call.
Added os.path.join mock in twister unit test to handle path join
consistently.
Signed-off-by: Michal Smola <michal.smola@nxp.com>
Automatically populate the snippet roots from Zephyr modules, instead of
only looking in `ZEPHYR_BASE`.
Signed-off-by: Jordan Yates <jordan.yates@data61.csiro.au>
Harness is freeform right now in the yaml file and if the harness is not
implemented in class, things fail. While we cleanup and enforce
implementations, this should serve as a quick fix dealing with such
unimplemented harnesses.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
To make possible to build bsim tests by Twister, it is necessary to
copy executables to BabbleSim bin directory.
Signed-off-by: Piotr Golyzniak <metody159@gmail.com>
Twister DeviceHandler now checks `--enable-coverage` command line argument
instead of `--coverage` when it deals with device output.
This resolves potential problem when only `--enable-coverage` argument
is given and the coverage report is not needed. In this case the test image
which is built for code coverage works slower also producing additional
console output, so the additional DeviceHandler timeout still have to be
applied and the output with coverage data correctly processed by Harness.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
lcov 2.0 added support for processing coverage data in parallel, which
provides a large speedup when processing many files, at the cost of some
additional overhead. When running the Chrome EC tests with coverage,
parallel reporting on a 36C72T machine reduces the time spent generating
coverage reports by 40 minutes (from approximately 1 hour to 20 minutes
total runtime), at the cost of about 3x greater CPU time overall
(assumed to be overhead for parallel processing, likely from spawning
much larger numbers of subprocesses).
The level of lcov parallelism is taken from the --jobs option passed to
twister, allowing lcov to choose if unspecified.
Signed-off-by: Peter Marheine <pmarheine@chromium.org>
This moves handling of changed lcov arguments in lcov 2.0 into helper
functions, significantly simplifying Lcov._generate() to reduce the
visual noise of extra arguments that are required but unimportant to the
task at hand.
Signed-off-by: Peter Marheine <pmarheine@chromium.org>
Make sure we set the gcov tool in a consistent way and avoid issues
where path is set as Path instead of a string.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
BinaryHandler.hanlde() method was trying to call GCOV and fails
silently because of incorrect call parameters. Moreover, even
being fixed, this call is not needed here as it attempts to
find and process .gcno and .gcna files (using wrong paths)
to create .gcov coverage text reports.
Currently the CoverageTool class does all data processing
and report generation using GCOVR or LCOV tools instead of
direct calls to GCOV.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Verify we have the coverage tool we want to use, otherwise we will end
up with many warnings and errors during coverage data collection.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Additional checks for Twister command line options `--coverage-tool`
and `--coverage-formats`.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Twister now uses GCOVR by default as the more reliable code
coverage reporting tool instead of LCOV.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
When scanning ports the manufacture field is not always
filled. It must be checked before using.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
In some cases genhtml incorrectly built reports from LCOV
coverage data using full path for some of the source files
and relative paths for other files.
This fix adds `--prefix` parameter to shorten paths explicitly
relative to the ZEPHYR_BASE directory.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
This gives us the flexibility to exclude certain platforms'
simulator that's known to fail from running in Twister, but
allows real hardware to be tested.
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
NXP boards with CMSID-DAP are not detected by twister
--generate-hardware-map, because serial device name 'mbed' is compared
with upper case 'MBED' in a list of supported manufacturers.
Fix it by making the comparison case-insensitive.
Tested using mimxrt1020_evk.
Fixes#63765
Signed-off-by: Michal Smola <michal.smola@nxp.com>
Fixes#65477
Platform key checking seemed to be erroneous;
now the variable names, comments and code seem in line with each other.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Support lcov > 2.0 tool which has strict error checking and some new
configuration options deprecating syntax used in 1.4 versions.
Fixes#62202
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Improve documentation about pytest integration with Twister. Add
examples of usage, improve description of available options and
introduce automatic doc generation of two plugin classes (DeviceAdapter
and Shell) basing on their docstrings from source code.
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
as in windows platfrom the format by default is not utf-8,
and we will see below error
Traceback (most recent call last):
File "...zephyr\scripts\twister", line 211, in <module>
ret = main(options)
^^^^^^^^^^^^^
File "...scripts/pylib/twister\twisterlib\twister_main.py",
tplan.load()
File "...scripts/pylib/twister\twisterlib\testplan.py",
self.load_from_file(last_run, filter_platform=connected_list)
File "...scripts/pylib/twister\twisterlib\testplan.py",
instance.create_overlay(platform, self.options.enable_asan,
self.options.enable_ubsan, self.options.enable_coverage,
self.options.coverage_platform)
File "...scripts/pylib/twister\twisterlib\testinstance.py"
f.write(content)
UnicodeEncodeError: 'gbk' codec can't encode character '\xf8'
in position 64: illegal multibyte sequence
Signed-off-by: Hake Huang <hake.huang@oss.nxp.com>
The Console Harness is able to parse its log with patterns to compose
extracted fields into records in 'recording.csv' file in the test's build
directory. This feature allows to extract custom test results like
performance counters.
With this change the extracted records are also written into 'twister.json'
as a part of each test suite object. This makes easier to store
all the data collected by the test for its further processing.
Other improvements:
- compile parsing pattern only once instead of at each input line;
- quote fields in '.csv' to avoid unexpected field separators;
- make 'regex' a required schema field of 'harness_config';
- Twister documentation update.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
A race condition was possible at QEMUHandler which didn't wait for
its harness to complete causing false negatives and test suite
retries, for example when the console harness has got rather
long output from a test application to check for patterns.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Using --platform-reports will also generate json reports
with results from a single platform. Needed for on-target
results publishing process.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
One can not even replace sucessfully pytest basic sample `pytest-args`
with command line "--pytest-args", as all it does is to append a single
string to current list of commands, making it impossible to send several
arguments.
This patch fixes that by allowing several instances of `--pytest-args`
to compose the whole list of arguments to be passed to pytest.
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
If a platform is fitlered, do not add it to keyed test map, otherwise we
will end up skipping all platforms from the same class without any
coverage.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Useful option during development and when applying smaller change to the
code and verifying changes using tests. This works fine as long as no
dependencies or major changes are done, i.e. when editing few files and
in a subsystem. Use with caution and use on your own risk :-)
When building multiple tests, this provide significant boost, up to 300%
faster than when rebuilding everything from scratch or when re-running
cmake every single time.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Measure both cmake and make/ninja build times and log them in debug mode
and additionally put the result in the json report for tracking build
times.
Cleanup what build/cmake function return and remove unused keys in the
result map.
Remove some excessive logging of launched jobs or default platforms that
gets in the way when in debug mode.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Extend Twister command line with --pytest-args. This parameter
is passed to pytest subprocess. It allows to select a specific
testcase from a test suite.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
In the docs replace references to native_posix with native_sim
Switch the default test platform to native_sim from native_posix
Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
On MacOS, ps utils raises a `NoSuchProcess` error rather than a
`ProcessLookupError` when a pid no longer exists.
Signed-off-by: Wilfried Chauveau <wilfried.chauveau@arm.com>
Implement a workaround for Console harness to compose TestCase
identifier correctly when a Ztest suite with a single testcase
uses this harness type. Normally, a Ztest suite should use the
Ztest Twister harness.
Without this workaround each Ztest TestCase result on Console is
duplicated (and written into twister.json) with its 'identifier'
attribute set to TestSuite id only, no TestCase suffix added;
the resulting entry with the full TestCase id is also stored,
but its values are empty with defaults.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
If Console Harness 'harness_config' properties have no 'regex'
patterns or no correct 'type' set, then ConfigurationError exception
is raised, handled, and the test instance error is accounted
in the summary results.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Fix the Twister Console harness unordered pattern matching
to treat the ztest as failed when not all of the
expected patterns were found in the console output, but the ztest
application itself reports 'PROJECT EXECUTION SUCCESSFUL'.
Unify debug logging on pattern match for ordered and unordered patterns.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
* Replace native_posix references with native_sim
in arguments help messages
* For the seed parameter, correct platform check to accept
native_sim
* Use native_sim in twister tests
Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
Create copys of the common configuration values when constructing test
scenarios, to avoid mutating the common value with test specific
appends.
Signed-off-by: Jordan Yates <jordan.yates@data61.csiro.au>
when load test plan it is possible the plan is built in another os
so the case key would be
'samples/hello_world/samples.baseic.hello_world'
but the testsuite is scaned in current os may in window
and the key is like
'samples\\hello_world\\samples.baseic.hello_world'
so update the uniq path with only backslash in path
Signed-off-by: Hake Huang <hake.huang@oss.nxp.com>
Some platforms prefix extra logging information before the standard
[] blocks so I've added `.\*` to the regex. Also, removed the static
values so they're only referenced using 'self.' and stopped parsing
lines after the FINISHED_PATTERN is matched since some versions of
gTest also print out a test summary after and it's not useful for
the processing.
Signed-off-by: Yuval Peress <peress@google.com>
When pytest scenario fails, then 'handler.log' is printed.
Changed to print 'twister_harness.log' that is a log from
pytest execution. That file tells much more when test fails.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Apply a workaround for an issue introduced with bc8b7dd
for code coverage collection at qemu_x86 when gcov data
was broken because of optimized out memcpy() in write_u64()
static inline function, so gcovr parser fails with
"gcovr.gcov_parser.NegativeHits: Got negative hit value
in gcov line 'branch 1 taken -1'".
Add gcovr command line option to ignore such errors (since v.6.0):
"--gcov-ignore-parse-errors=negative_hits.warn_once_per_file"
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Added pytest_dut_scope keyword under harness_config section.
New keyword is used to determine the scope of dut and shell
fixtures in pytest-twister-harness plugin.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Do not break processing when incorrect coverage data are received.
Instead, report failures but still process valid data.
Include coverage processing status within twister retrun code.
Signed-off-by: Piotr Kosycarz <piotr.kosycarz@nordicsemi.no>
When coverage is enabled, gather console logs longer
to receive gcov data which are send after test output.
Signed-off-by: Piotr Kosycarz <piotr.kosycarz@nordicsemi.no>
Very verbose log message without too much context that should be a debug
message rather than a info().
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
An urforseen error was introduce with #62713. When "--device-testing"
or "--filter runnable" is used in CLI, twister will skip tests which
cannot be executed (not just built) on a given platform. If a given
platform is among integration platforms it will cause an error.
However, it shouldn't be the case. Such skip is intentional and
shouldn't be an error. Fixed by assigning an existing Filter.CMD_LINE,
which is exempted from being turned to error.
fixes: #63844
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Reset TestInstance status when ProjectBuilder.run() is called
to run the test after its successful build, so the build status
will not propagate as the run status if the run fails.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Fix the Twister Console harness ordered 'multi_line' and 'one_line'
pattern matching to treat the ztest as failed when not all of the
expected patterns were found in the console output, but the ztest
application itself reports 'PROJECT EXECUTION SUCCESSFUL'.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
If flash_order were to be missing from the YAML, the Domains init
would crash instead of substituting an empty list.
This commit fixes that.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
By requiring a full-on YAML fragment, the Domains class can rely on
pykwalify to do the heavy lifting with validating types and structure.
Remove support for loading domains from a dictionary, so that we don't
ever have to replicate the same checks on the input, and we can instead
have a single entry point.
Signed-off-by: Grzegorz Swiderski <grzegorz.swiderski@nordicsemi.no>
Do extra checks to reject inconsistent `domains.yaml` contents early:
default: app
build_dir: /path/to/build
domains:
- name: image_1
build_dir: /path/to/build/image_1
- name: image_2
build_dir: /path/to/build/image_2
flash_order:
- image_1
- image_2
- unknown
This example lists "image_1" and "image_2" as known domains, but an
entry for the default "app" is missing, which should never be allowed.
A valid `domains.yaml` file shall also not contain extra domains under
`flash_order`, such as "unknown", which doesn't appear under `domains`.
Either of these cases can now be caught while initializing an instance
of the Domains class.
It follows that a valid `domains.yaml` file must always contain a list
of domains with at least one entry - the default image. Since `default`
is a required key in the YAML schema, `domains` should be required too.
Thus, empty lists will be rejected by pykwalify, so the questionable
warning for "no domains defined; this probably won't work" can be axed.
Fixes#63166
Signed-off-by: Grzegorz Swiderski <grzegorz.swiderski@nordicsemi.no>
When using a quarantine file with more than 512 unique
entries, than time of matching quarantine increases
significantly. This is because regexp cache size is
512. Add precompiled regexp entries to the quarantine
as a fix.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
All handlers except QEMU were right-stripping console lines before
checking them. In certain cases where the regex is expecting the
whitespace, this could make the test fail on qemu when it is passing
on something else. Change QEMUHandler to us .rstrip instead of .strip()
Signed-off-by: Øyvind Rønningstad <oyvind.ronningstad@nordicsemi.no>
In PR #62947 a timeout variable was replaced with a method,
but this timeout was used in another place to start
pytest subprogram. This commit provides a fix.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Test suites names are not being handled uniformly for tests not
in zephyr tree. Their names depend on -T arg used in twister's CLI.
The newly added options allow to select if twister should add paths
to suite names. This is needed if test plans are to be used for tests
outside of zephyr tree.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
"Path" is a path (relative to zephyr) to a directory with test suite's
definition. Such entry helps to locate a given test.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
In test-only mode with a hardware map, all tests with fixtures were
skipped, even when the hardware map has appropriate fixtures defined.
TestPlan.load_from_file had the check for fixtures missing, but actually
best is that the TestInstance checks the fixtures in the hardware map.
Signed-off-by: Manoel Brunnen <manoel.brunnen@ithinx.io>
Twister allows us to control maximum execution time for each test
with timeout value in test's .yaml configuration and
platform level timeout multiplier which allows us to tweak
timeout value for specific platform.
However, sometimes we want to additionally adjust tests timeouts
when running twister. This is especially useful in case of
simulation platform as simulation time may depend on the host
speed & load, we may select different (i.e. cycle accurate but
slower one) simulation method, etc...
Let's introduce global (twister-level) timeout multiplier option.
Signed-off-by: Eugeniy Paltsev <Eugeniy.Paltsev@synopsys.com>
Signed-off-by: Evgeniy Paltsev <PaltsevEvgeniy@gmail.com>
The recent support for BUILD_ONLY images was implemented by excluding
them from `domains.yaml`, in order to crudely prevent them from being
picked up by `west flash`. Arguably, this is incorrect or unexpected,
because the sysbuild documentation defines a "domain" as:
Every Zephyr CMake build system managed by sysbuild.
Another consequence is that, given a build-only `<image>`, this makes it
impossible to pass `--domain <image>` to `west flash`, `west debug`, and
ironically `west build`.
To fix that, `domains.yaml` should again represent all domains, and the
build-only ones should be indicated in another way. Enter `flash_order`:
a new top-level key in the domains YAML schema. It contains the default
sequence of images used by `west flash`, where the build-only images are
excluded, and the order is influenced by `sysbuild_add_dependencies()`.
Signed-off-by: Grzegorz Swiderski <grzegorz.swiderski@nordicsemi.no>
Added application based on SMP Server Sample. Application is built together
with MCUboot using sysbuild and is flashed onto device in one step.
Tests are automated with pytest - new harness of Twister.
The image for upgrade is prepared using west sign command
then is uploaded by mcumgr into device and tested.
Automated scenarios to test upgrade (image upload, test, revert, confirm),
to test downgrade prevention mechanism and to test upgrade with image,
that is signed with an invalid key.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Currently, when a Robot test is run, the test case status value is
reported as `None`, which causes the XML to contain information about
the test being skipped due to misconfiguration.
This commit fixes the value reported in Twister XML test result by
assigning the test result to the `status` variable of a testcase
instance after the test has finished running.
Signed-off-by: Franciszek Pindel <fpindel@internships.antmicro.com>
Signed-off-by: Piotr Zierhoffer <pzierhoffer@antmicro.com>
This is coming from devicetree and corrosponds to what we have in the
dts/bindings/vendor-prefixes.txt file.
This will allow for static filtering, especially with twister, i.e. no
need to build anything to know the vendor of a board
All of the vendor data was extracted automatically from the devicetree,
so some platforms might not have the right vendor or no vendor at all
right now, we need to fix some of the DTS information or do this
manually to get this 100% correct. But we are close.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Add a new option --vendor which allows filtering by vendors tracked in
the board/platform yaml file. The vendor string is compatible with DTS
and is what we have in dts/bindings/vendor-prefixes.txt.
Providing multiple vendors is also supported.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Only impacts on twister-pytest scenarios. Exec command is waiting for
the prompt message, but in some cases, e.g. after reseting the DUT, prompt
can be followed by another strings. Solution is to clear internal serial
buffer and to not request the prompt string before executing a command.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Sometimes you might need exec_command() to return an value from
running device. If the output contains shell prompts or logger
output, it is harder to capture the value.
Filter away the lines we thing belongs to the shell prompt or
normal logging output.
Signed-off-by: Seppo Takalo <seppo.takalo@nordicsemi.no>
Skips are turned to errors on integration platforms. However, if
a user (or a CI) selects toolchain other than zephyr, unexpected
errors due to skips can appear. With this commit skips due to
toolchain incompatibilities are not treated as errors.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
If coverage is provided by a different platform not in the integration
list, it should not be reported as error.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Remove dead and erroneous (DUT is not a dict) if branch. Simply use the
self.options.platform list. Which is filled by either the hardware map
or the '--platform' option.
This change addresses issue #62723 and #62560.
Signed-off-by: Manoel Brunnen <manoel.brunnen@ithinx.io>
Before only with --integration such skips were changed to errors.
This hinders issues when twister is call with --all flag in CI
and new skips on integration platforms are not caught. Then those
skips cause errors in other PRs when --integration was used.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
The value of emulation_platforms is a bool corresponding to
--emulation-only CLI option. Therefore, it should be registered
as Filters.CMD_LINE. Without it, twister raises errors when an
integration platform is descoped due to this option.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Added an exception to scl.py module, to not process an empty
yaml file, because of error from pykwalify. New exception
is handled when processing quarantine files, to allow
an empty file.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Simple refactoring aiming to reduce the average method length.
Minor corrections of the handlers module, removing wholly unused
variables, etc.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Allow to specify a list of pytest directories, files or subtests
with pytest_root keyword in test yaml.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Fixtures in pytest-twister-harness plugin are moved to one
file to simplify adding new fixtures in the future - no
need to add pytest_plugins entry and register asserts.
Moved also shell fixture from sample dir, because that
fixture can be reused in new tests.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
If pylib twister harness reads bytes that can't be decoded to Unicode,
it would raise an exception at a separate "serial reader" process, that
would fail without any warning.
This patch uses "errors='replace'" parameter for Python str.decode()
method, so that this doesn't happen.
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
Previously, only the main src/ directories was scanned for
ZTest testsuite names and tests. This allows us to place
test sources in subdirectories as well.
Signed-off-by: Christopher Friedt <cfriedt@meta.com>
Add support handling quotes for configuration options in extra args by
escaping them properly instead of removing the quotes altogether. For
other options in extra_args quotes are removes as usual.
Add similar support in west build command also.
Add a unit test to check this functionality.
Signed-off-by: Chaitanya Tata <Chaitanya.Tata@nordicsemi.no>
Add tests which use pytester to call pytest with pytest-twister-harness
plugin on previously defined test. This gives a possibility to test
plugin in end-to-end variant.
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
According to new adapters API add new tests which verify their
correctness. Introduce some minor naming changes to describe better
their role.
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Add better description of available options and add actions which
should help to prevent from passing invalid options to plugin.
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Remove log.py and by this logging configuration from
pytest-harness-plugin. Instead use pytest options for setting verbosity
and format of printed logs. Thanks to this management of verbosity
level can be easier for users.
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Create helper class Shell which can be used in tests which require
interaction with shell application.
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Move helper.py file content to utils.py file in device directory,
where those helper functions are used.
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Divide dut fixture into three smaller fixtures to make it easier
to user to change dut fixture scope or to write her/his own dut
fixture. Add initialize_log_files method to make it possible to
initialize log files per each test.
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Pass pytest test name to handler.log header to make it easier to
analyze handler.log content.
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Add readlines_until method to DeviceAdapter class to simplify using
adapters in tests by give user possibility to read output from device
until following conditions:
- provided regex was found
- provided number of lines was already read
- timeout was occurred
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Introduce changes into FifoHandler class to make it more safe
(especially to avoid blocking all program when FIFO queue is not opened
by other application).
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Select and place common code of three adapters (HardwareAdapter,
BinaryAdapterBase and QemuAdapter) into basic DeviceAdapter class.
Introduce new way of reading device output by run separate thread which
try to read device output and place it into internal python queue.
Thanks to this, now it is possible to create readline method for all
adapters, which can be unblock when timeout occur.
Collect all common steps which have to be done before setup device in
launch method. The same was done for teardown operations which were
placed into close method.
Additionally some protection mechanisms were introduced to prevent for
undesirable side-effects when user could try to launch to already
launched device or try to send some data to disconnected device.
iter_stdout method was replaced by two new methods: readline and
readlines. To make it possible to remove all read output from internal
buffer (queue), clear_buffer method was introduced.
Also unit tests were rewritten to work properly with current version
of adapters.
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Change name of SimulatorAdapterBase class to more meaningful like
BinaryAdapterBase. This changes is also made to be more align with
Twister terminology.
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Add possibility of passing timeouts from Twister to pytest and unify
various timeouts used in adapters to point to one main timeout.
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
PytestAssertRewriteWarning is raised when pytest-twister-harness
plugin is called from twister. Call register method to get rid
of that warning
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
When parsing test names from symbol names, use a non-greedy match to
avoid parsing test names with __ in them.
The symbol looks like z_ztest_unit_test__SUITE__TEST. It is more likely
that a test name will contain __ than a suite name, so now something
z_ztest_unit_test__a_b__c_d__e_f will be parsed as suite:a_b
test:c_d__e_f, whereas it used to be parsed as suite:a_b__c_d test:e_f
Signed-off-by: Jeremy Bettis <jbettis@google.com>
If a test doesn't start with test_, twister currently emits an error:
suite_name.test_name: Unknown status 'started'
Following the suggestion by Maksim Masalski on PR#21191, change the
regex to exclude spaces in the test name, and remove the check that the
test_ prefix exists.
See also #21162
Signed-off-by: Jeremy Bettis <jbettis@google.com>
While the code already includes some provisions to allow running outside
of a git checkout, attempting that right now causes a failure to generate
the XML report, as the JSON report creates a null `zephyr_version` field.
The reason for that is that the original code doesn't set `self.version`
when the subprocess returns a non-zero status (and also doesn't log.)
Instead of having to set `self.version` for all failure cases, initialize
it to the failure state, and log if it hasn't changed from that.
Signed-off-by: Diego Elio Pettenò <flameeyes@meta.com>
When using the hardware map and testing on device, print out the name of
the dut (from the hardware map) on the script and report it in the json
output as well.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
If size is given then twister prnits the size information for the ELF
but returns one which if used in automation fails the command.
Return zero as the command is successful. Also, fix the unit test case.
Signed-off-by: Chaitanya Tata <Chaitanya.Tata@nordicsemi.no>
Let's not provide default toolchains for 'arc' intentionally: some
targets can't be built with GNU toolchain ("zephyr", "cross-compile",
"xtools" options) and for some targets we haven't provided MWDT
compiler / linker options in corresponding SoC file in Zephyr, so
these targets can't be built with ARC MWDT toolchain
("arcmwdt" option) by Zephyr build system.
Instead for 'arc' we rely on 'toolchain' option in board yaml
configuration.
Signed-off-by: Eugeniy Paltsev <Eugeniy.Paltsev@synopsys.com>
Signed-off-by: Evgeniy Paltsev <PaltsevEvgeniy@gmail.com>
Enable the possibility for boards to implement a custom `run` target in
its board.cmake to run any arbitrary commands. This is helpful for devs
who would like to add support for proprietary simulator to their boards
that can't be upstreamed.
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
We have been dealing with missing and multiple binaries the same way and
both would result in a build error, which is not accurate. multiple
binaries in the build directory are fine, we just need to pick the right
one for parsing.
If we get no binaries, raise an exception and report failure, however,
if we have multiple binaries, filter intermediate artifacts out and
parse what remains.
qemu binaries generated after a run are also being filtered here. Those
are not build artificats and appear only after running in qemu. However
they have been causing issues on retries.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
When no/too many elf files are detected after a build a BuildError
exception is raised. However, it was not being counted as an issue
with a test. With the patch satuses of tests' with such exception
are reported as errors. Whithout it, twister finished without
reported errors and was getting green CI checks.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
When using twister to generate coverage with the coverage tool lcov,
allow using --coverage-formats to pick if you want lcov or html,
defaulting to both. Picking html will also use lcov, since that is
required for geninfo. This will allow callers to avoid the potentially
slow and disk intensive html reports if they only wanted the lcov info
file.
Signed-off-by: Jeremy Bettis <jbettis@google.com>
Fixes Issue #60522 by guarding member access against None parameters.
Shortens the init by using the if...else assignment.
Fixes two typos.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Fix `--device-flash-with-test` command line options check logic
to be effective with `--device-testing`.
Remove `--device-flash-timeout` check for `--device-testing`
presence as marginal.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
hardware was referenced as return value, without being declared. When the
program goes into the 'except' path, hardware is unbound, but returned,
which throws an error:
`UnboundLocalError`: local variable 'hardware' referenced before assignment
Signed-off-by: Manoel Brunnen <manoel.brunnen@ithinx.io>
Extended parsing of report.xml file - produced by pytest-twister-harness
plugin. Now testcases are properly extracted and added to twister
report. Added unit tests.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
We have many tests that are marked as slow that do not run in CI and any
known workflow, to isolate them and be able to run them on their own,
add a new option --enable-slow-only which causes only those tests to build
and execute.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Add DeviceAbstract class to default imports from pytest-twister-harness
package to simplify importing DUT package, when creating tests.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Use the natural sort of list when generating a hardware map. The
list is sorted with a serial port as a key. When more than 10 ports
are active and some of devices use more than one port, the ports
of one device may be listed in wrong sequence, which may cause
futher problems with selecting the right port for listening to
the device.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Add protection timeout for pytest subprocess, to avoid situation of
suspending whole Twister in case of internal pytest test problem.
Co-authored-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Added waiting for prompt instead of calling time.sleep,
to fix issue with longer setting up platforms with TF-M
enabled. Wrapped iter_stdout generator, to avoid issues
when generator is created and called multiple times
in test scenarios.
Fixes: #58747
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
When calling twister with --coverage and no platform specified on the
command line, set the the value of coverage platforms to an empty list.
Fixes#57534
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
testinstance.py contains a string literal comparison to the
word 'platform', which has been misspelled as 'plaform'.
The change is verified by a test expansion.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Making the necessary changes to enable the new pytest plugin.
By default Twister should work without the pytest-twister-harness
plugin installed. To achieve this, each time Twister calls pytest,
the PYTHONPATH environment variable is expanded and the
`-p twister_harness.plugin` option is added to the pytest command.
Co-authored-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Adding pytest plugin dedicated to running pytest tests in Zephyr
project. This plugin provides a dut fixture which allows to handle
bidirectional communication with the device under test. This version
of plugin can be used for tests dedicated to real hardware, QEMU and
native_posix simulator.
Co-authored-by: Lukasz Fundakowski <lukasz.fundakowski@nordicsemi.no>
Co-authored-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Co-authored-by: Katarzyna Giadla <katarzyna.giadla@nordicsemi.no>
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Add support for calling the `renode-test` command from west and twister.
Enable running Robot Framework tests suites in Renode.
Signed-off-by: Michał Szprejda <mszprejda@antmicro.com>
Signed-off-by: Mateusz Hołenko <mholenko@antmicro.com>
Current implementation of ConfigurationError causes
an exception to be thrown.
It is because a PosixPath is being concatenated with an str with a '+'
operator, which is not permitted.
The path in question is now cast to an str.
Additional test has been added to test_twister.py to track regressions.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
For the arch's that have standard toolchains thare are supported
across the arch set them in twister rather than in <BOARD>.yaml.
Signed-off-by: Kumar Gala <kumar.gala@intel.com>
Some 3P logic (Pigweed for example) is already heavily invested in
gTest as a testing framework. Adding the `gtest` harness enables
running the existing 3P tests on various Zephyr platforms and configs.
Signed-off-by: Yuval Peress <peress@google.com>
Simics converts newline characters when writing console output to
stdout, so we need to remove them as a suffix string. Otherwise the
console harness fails to match expected PASS/FAIL/RunID string patterns
and twister tests timeout.
Signed-off-by: Maureen Helm <maureen.helm@intel.com>
After generating a hardware-map a platform field is marked
as 'unknown'. If it is not changed, do not process them
when running --device-testing.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
In some cases CMake doesn't create build directory and in those cases
Twister should create this directory by itself.
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
We have been clearning the list of testcases in every iteration over the
symbols ending up with only one testcase in the instance.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Add a new option to Twister that allows to load alternative
configuration files, `--alt-config-root`. This new option takes an
arbitrary number of root folders where alternative configuration files
can be stored. Twister will check if a test configuration file exists in
any of the alternative test configuration folders. For example, given
`$test_root/tests/foo/testcase.yaml`, Twister will use
`$alt_config_root/tests/foo/testcase.yaml` if it exists.
This feature can be useful if an out-of-tree project needs to run
upstream tests in different configurations, potentially not available
upstream (e.g. an out-of-tree board, or Kconfig setting).
Note that overlaying has been discarded because we can't easily delete
fields from the original configuration file, something that in certain
cases could be required. Current approach may lead to some dupplication,
but guarantees full control of the test configuration.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
Make twister to raise error by default whenever duplicated test ids
are found. Remove redundant option to list duplicates.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Don't override a test case name check failed result by the following
RunID check failed result to report clearly about test case mismatch
situation as well as more specifically when the same test case appears
with unexpected RunID.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Fix suite-name-check bypassed at DeviceHandler on flash errors.
If the flash timeout is smaller than needed, but test images are
actually flashed successfully and appropriate tests executed,
then results from the previous test case were taken because
the flash error status makes the suite-name-check bypassed.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
We can now have situations like this when it comes to lists in Twister
config:
```yaml
common:
platform_exclude: foo
tests:
my.test:
platform_exclude: bar
```
```yaml
common:
platform_exclude: foo
tests:
my.test:
platform_exclude:
- bar
- baz
```
```yaml
common:
platform_exclude:
- foo
- bar
tests:
my.test:
platform_exclude: baz
```
```yaml
common:
platform_exclude:
- foo
- bar
tests:
my.test:
platform_exclude:
- baz
- bazz
```
This patch fixes handling of cases (2), (3) and (4).
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
This patch handles mixed types between common/tests sections in the
configuration, e.g. str+list, list+str, str+str.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
Since pykwalify does not support multi-type fields, we are forced to use
the 'any' type. This is used for fields that allow sets/lists written as
either YAML lists or space-separated strings. This patch checks for
str/list Python type, and raises a configuration error if not true so
that user obtains better error messages in case the configuration file
contains invalid format.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
Twister configuration files have a few fields defined with the "set"
type. The set needs to be specified as a space-separated list of
entries, for example:
```yaml
plaform_allow: boardx boardy
```
This patch allows users to write these fields as YAML lists, for
example:
```yaml
platform_allow:
- boardx
- boardy
```
The list gets converted to a Python set, therfore the same exact
behavior is kept in Twister.
In some cases like in the example above, using lists allows to have more
readable configuration files, specially if the list is long. It also
helps on diffs, provided we have one item per line.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
A change to the scope selection rules was introduced by #52715
but it doesn't comply with the description in the code. What's more,
the change introduces major issues i.e. non deterministic scope
selection, especially in a CI environment. More context at #57595
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Twister does not handle out-of-tree boards automatically. Modules like
e.g. example-application that define their own board root folder, are
required to inform twister via `--board-root`. In contrast, west is able
to to this automatically. This patch makes twister a bit smarter by
automatically filling the board root list with module paths as well.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
Add option to shuffle tests (randomly) before they are split into subsets.
This allow to fairly distribute tests across subsets,
so that long tests are not crowded in a single group.
This also allows to detect unwanted dependencies in test execution order.
Also, added option to provide custom seed to random generator
used to shuffle tests.
This will allow to reproduce certain test order if needed.
Used seed is printed at console.
Signed-off-by: Piotr Kosycarz <piotr.kosycarz@nordicsemi.no>
The initshell sections in the linker scripts where associated with theo
old shell code. The old shell code has been removed for some time so
remove references to initshell in the linker scripts and size_calc.py
Signed-off-by: Kumar Gala <kumar.gala@intel.com>
Convert following construction:
elif self.options.device_serial or self.options.device_serial_pty:
if self.options.device_serial:
...
else
...
to the construction:
elif self.options.device_serial:
...
else self.options.device_serial_pty:
...
Signed-off-by: Andrei Emeltchenko <andrei.emeltchenko@intel.com>
Use flash_timeout and flash_with_test options not only for
serial console but also for device_serial_pty.
Signed-off-by: Andrei Emeltchenko <andrei.emeltchenko@intel.com>
compliance script runs pylink on size_calc.py and reports:
C0325:Unnecessary parens after '=' keyword (superfluous-parens)
Fix warning by remove unnecessary parens.
Signed-off-by: Kumar Gala <kumar.gala@intel.com>
`TestInstance.get_elf_file()` does not include the binary name used for
unit testing (`testbinary`) in its search, causing unit tests to not
complete successfully. This does not cause tests to fail and thus goes
unnoticed.
Update the glob expressions used in `get_elf_file()` to also look for
files called `testbinary`
Signed-off-by: Tristan Honscheid <honscheid@google.com>
If we are doing a sysbuild get_elf_file needs to return the default
domain's build_dir. The standard build_dir will not any ELF builds
in it so will error out.
Signed-off-by: Kumar Gala <kumar.gala@intel.com>
This is usually used only when doing size calculation, but now it is
being used for extracting symbols from the ELF for testing purposes and
some issues arise with multiple ELF files found.
Simplify the code a bit to make it easy to maintain a lists of things to
exclude.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The DeviceHandler class method monitor_serial() has time-of-check
to time-of-use (TOCTOU) race condition possible on its serial port
which might be closed and its file descriptor cleared in between
of isOpen() and is_waiting status checks.
The issue is alleviated with controlled exception handling.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Fixes an issue when DeviceHandler awaited for a suitable DUT infinitely.
It was possible if the device map has no devices with required platform,
or no devices with fixture needed for the test case, or devices without
serial connection. A trivial case is a typo in platform name.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
For ztest twister would parse the symbol information that was
generated as part of the build (zephyr.symbols). However the format
of the zephyr.symbols files is highly dependant on the toolchain.
Move to using pyelf to parse the symbol information directly from
zephyr.elf instead so that this works regardless of toolchain.
Signed-off-by: Kumar Gala <kumar.gala@intel.com>
On native platforms we typically have CONFIG_BUILD_OUTPUT_EXE set so
we end up with both zephyr.elf and zephyr.exe. This would cause a
call to get_elf_file() to fail. There isn't any reason to error out
since zephyr.elf and zephyr.exe should be identical. So allow it
if the platform type is native.
Signed-off-by: Kumar Gala <kumar.gala@intel.com>
Use explicit flash command execution timeout at DeviceHandler
instead of the hardcoded value of 60 sec.
When a HW platform executes test cases right after the flash command,
the test case remaining timeout is affected by how much time the flash
operation consumes. Some simulation platforms need to spend significant
amount of time on each 'flash' cycle, usually adding the same delay on
each test case.
This improvement adds two new command line options and device map fields:
--device-flash-timeout - for the flash operation timeout
('flash-timeout' device map field).
--device-flash-with-test - to indicate that the platform flash
command also runs a test case, so the overall timeout should be
calculated as a sum of the flash timeout and the current test case
timeout to receive all console output from the platform
('flash-with-test' device map field).
The device map field values override command line values for the
particular platform where configured.
Default behavior is backward compatible: flash operation fixed timeout
is 60 sec. not including the test case timeout.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Rather than having to add 'armclang' to every <BOARD>.yaml, just list
it as a generally supported toolchain if the architecture is 'arm'.
Signed-off-by: Kumar Gala <kumar.gala@intel.com>
When a test start, mark it as such, if we do not get any results due to
a timeout, it will be finally marked as failed. Previously such tests
causing a freeze were marked as blocked (or not run), which is not
exactly right.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Skips due to overflows on integration_platforms during integretion
mode should be treated as errors.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
`self.instances[filtered_instance.name]` and `filtered_instance`
are already referancing the same object.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
If canonical zephyr base is already ended by trailing slash, then it
should not be added redundantly.
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
