Only impacts on twister-pytest scenarios. Exec command is waiting for
the prompt message, but in some cases, e.g. after reseting the DUT, prompt
can be followed by another strings. Solution is to clear internal serial
buffer and to not request the prompt string before executing a command.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Sometimes you might need exec_command() to return an value from
running device. If the output contains shell prompts or logger
output, it is harder to capture the value.
Filter away the lines we thing belongs to the shell prompt or
normal logging output.
Signed-off-by: Seppo Takalo <seppo.takalo@nordicsemi.no>
Skips are turned to errors on integration platforms. However, if
a user (or a CI) selects toolchain other than zephyr, unexpected
errors due to skips can appear. With this commit skips due to
toolchain incompatibilities are not treated as errors.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
If coverage is provided by a different platform not in the integration
list, it should not be reported as error.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Remove dead and erroneous (DUT is not a dict) if branch. Simply use the
self.options.platform list. Which is filled by either the hardware map
or the '--platform' option.
This change addresses issue #62723 and #62560.
Signed-off-by: Manoel Brunnen <manoel.brunnen@ithinx.io>
Before only with --integration such skips were changed to errors.
This hinders issues when twister is call with --all flag in CI
and new skips on integration platforms are not caught. Then those
skips cause errors in other PRs when --integration was used.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
The value of emulation_platforms is a bool corresponding to
--emulation-only CLI option. Therefore, it should be registered
as Filters.CMD_LINE. Without it, twister raises errors when an
integration platform is descoped due to this option.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Added an exception to scl.py module, to not process an empty
yaml file, because of error from pykwalify. New exception
is handled when processing quarantine files, to allow
an empty file.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Simple refactoring aiming to reduce the average method length.
Minor corrections of the handlers module, removing wholly unused
variables, etc.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Allow to specify a list of pytest directories, files or subtests
with pytest_root keyword in test yaml.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Fixtures in pytest-twister-harness plugin are moved to one
file to simplify adding new fixtures in the future - no
need to add pytest_plugins entry and register asserts.
Moved also shell fixture from sample dir, because that
fixture can be reused in new tests.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
If pylib twister harness reads bytes that can't be decoded to Unicode,
it would raise an exception at a separate "serial reader" process, that
would fail without any warning.
This patch uses "errors='replace'" parameter for Python str.decode()
method, so that this doesn't happen.
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
Previously, only the main src/ directories was scanned for
ZTest testsuite names and tests. This allows us to place
test sources in subdirectories as well.
Signed-off-by: Christopher Friedt <cfriedt@meta.com>
Add support handling quotes for configuration options in extra args by
escaping them properly instead of removing the quotes altogether. For
other options in extra_args quotes are removes as usual.
Add similar support in west build command also.
Add a unit test to check this functionality.
Signed-off-by: Chaitanya Tata <Chaitanya.Tata@nordicsemi.no>
Add tests which use pytester to call pytest with pytest-twister-harness
plugin on previously defined test. This gives a possibility to test
plugin in end-to-end variant.
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
According to new adapters API add new tests which verify their
correctness. Introduce some minor naming changes to describe better
their role.
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Add better description of available options and add actions which
should help to prevent from passing invalid options to plugin.
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Remove log.py and by this logging configuration from
pytest-harness-plugin. Instead use pytest options for setting verbosity
and format of printed logs. Thanks to this management of verbosity
level can be easier for users.
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Create helper class Shell which can be used in tests which require
interaction with shell application.
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Move helper.py file content to utils.py file in device directory,
where those helper functions are used.
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Divide dut fixture into three smaller fixtures to make it easier
to user to change dut fixture scope or to write her/his own dut
fixture. Add initialize_log_files method to make it possible to
initialize log files per each test.
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Pass pytest test name to handler.log header to make it easier to
analyze handler.log content.
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Add readlines_until method to DeviceAdapter class to simplify using
adapters in tests by give user possibility to read output from device
until following conditions:
- provided regex was found
- provided number of lines was already read
- timeout was occurred
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Introduce changes into FifoHandler class to make it more safe
(especially to avoid blocking all program when FIFO queue is not opened
by other application).
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Select and place common code of three adapters (HardwareAdapter,
BinaryAdapterBase and QemuAdapter) into basic DeviceAdapter class.
Introduce new way of reading device output by run separate thread which
try to read device output and place it into internal python queue.
Thanks to this, now it is possible to create readline method for all
adapters, which can be unblock when timeout occur.
Collect all common steps which have to be done before setup device in
launch method. The same was done for teardown operations which were
placed into close method.
Additionally some protection mechanisms were introduced to prevent for
undesirable side-effects when user could try to launch to already
launched device or try to send some data to disconnected device.
iter_stdout method was replaced by two new methods: readline and
readlines. To make it possible to remove all read output from internal
buffer (queue), clear_buffer method was introduced.
Also unit tests were rewritten to work properly with current version
of adapters.
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Change name of SimulatorAdapterBase class to more meaningful like
BinaryAdapterBase. This changes is also made to be more align with
Twister terminology.
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Add possibility of passing timeouts from Twister to pytest and unify
various timeouts used in adapters to point to one main timeout.
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
PytestAssertRewriteWarning is raised when pytest-twister-harness
plugin is called from twister. Call register method to get rid
of that warning
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
When parsing test names from symbol names, use a non-greedy match to
avoid parsing test names with __ in them.
The symbol looks like z_ztest_unit_test__SUITE__TEST. It is more likely
that a test name will contain __ than a suite name, so now something
z_ztest_unit_test__a_b__c_d__e_f will be parsed as suite:a_b
test:c_d__e_f, whereas it used to be parsed as suite:a_b__c_d test:e_f
Signed-off-by: Jeremy Bettis <jbettis@google.com>
If a test doesn't start with test_, twister currently emits an error:
suite_name.test_name: Unknown status 'started'
Following the suggestion by Maksim Masalski on PR#21191, change the
regex to exclude spaces in the test name, and remove the check that the
test_ prefix exists.
See also #21162
Signed-off-by: Jeremy Bettis <jbettis@google.com>
While the code already includes some provisions to allow running outside
of a git checkout, attempting that right now causes a failure to generate
the XML report, as the JSON report creates a null `zephyr_version` field.
The reason for that is that the original code doesn't set `self.version`
when the subprocess returns a non-zero status (and also doesn't log.)
Instead of having to set `self.version` for all failure cases, initialize
it to the failure state, and log if it hasn't changed from that.
Signed-off-by: Diego Elio Pettenò <flameeyes@meta.com>
When using the hardware map and testing on device, print out the name of
the dut (from the hardware map) on the script and report it in the json
output as well.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
If size is given then twister prnits the size information for the ELF
but returns one which if used in automation fails the command.
Return zero as the command is successful. Also, fix the unit test case.
Signed-off-by: Chaitanya Tata <Chaitanya.Tata@nordicsemi.no>
Let's not provide default toolchains for 'arc' intentionally: some
targets can't be built with GNU toolchain ("zephyr", "cross-compile",
"xtools" options) and for some targets we haven't provided MWDT
compiler / linker options in corresponding SoC file in Zephyr, so
these targets can't be built with ARC MWDT toolchain
("arcmwdt" option) by Zephyr build system.
Instead for 'arc' we rely on 'toolchain' option in board yaml
configuration.
Signed-off-by: Eugeniy Paltsev <Eugeniy.Paltsev@synopsys.com>
Signed-off-by: Evgeniy Paltsev <PaltsevEvgeniy@gmail.com>
Enable the possibility for boards to implement a custom `run` target in
its board.cmake to run any arbitrary commands. This is helpful for devs
who would like to add support for proprietary simulator to their boards
that can't be upstreamed.
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
We have been dealing with missing and multiple binaries the same way and
both would result in a build error, which is not accurate. multiple
binaries in the build directory are fine, we just need to pick the right
one for parsing.
If we get no binaries, raise an exception and report failure, however,
if we have multiple binaries, filter intermediate artifacts out and
parse what remains.
qemu binaries generated after a run are also being filtered here. Those
are not build artificats and appear only after running in qemu. However
they have been causing issues on retries.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
When no/too many elf files are detected after a build a BuildError
exception is raised. However, it was not being counted as an issue
with a test. With the patch satuses of tests' with such exception
are reported as errors. Whithout it, twister finished without
reported errors and was getting green CI checks.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
When using twister to generate coverage with the coverage tool lcov,
allow using --coverage-formats to pick if you want lcov or html,
defaulting to both. Picking html will also use lcov, since that is
required for geninfo. This will allow callers to avoid the potentially
slow and disk intensive html reports if they only wanted the lcov info
file.
Signed-off-by: Jeremy Bettis <jbettis@google.com>
Fixes Issue #60522 by guarding member access against None parameters.
Shortens the init by using the if...else assignment.
Fixes two typos.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Fix `--device-flash-with-test` command line options check logic
to be effective with `--device-testing`.
Remove `--device-flash-timeout` check for `--device-testing`
presence as marginal.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
hardware was referenced as return value, without being declared. When the
program goes into the 'except' path, hardware is unbound, but returned,
which throws an error:
`UnboundLocalError`: local variable 'hardware' referenced before assignment
Signed-off-by: Manoel Brunnen <manoel.brunnen@ithinx.io>
Extended parsing of report.xml file - produced by pytest-twister-harness
plugin. Now testcases are properly extracted and added to twister
report. Added unit tests.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
We have many tests that are marked as slow that do not run in CI and any
known workflow, to isolate them and be able to run them on their own,
add a new option --enable-slow-only which causes only those tests to build
and execute.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Add DeviceAbstract class to default imports from pytest-twister-harness
package to simplify importing DUT package, when creating tests.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Use the natural sort of list when generating a hardware map. The
list is sorted with a serial port as a key. When more than 10 ports
are active and some of devices use more than one port, the ports
of one device may be listed in wrong sequence, which may cause
futher problems with selecting the right port for listening to
the device.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Add protection timeout for pytest subprocess, to avoid situation of
suspending whole Twister in case of internal pytest test problem.
Co-authored-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Added waiting for prompt instead of calling time.sleep,
to fix issue with longer setting up platforms with TF-M
enabled. Wrapped iter_stdout generator, to avoid issues
when generator is created and called multiple times
in test scenarios.
Fixes: #58747
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
When calling twister with --coverage and no platform specified on the
command line, set the the value of coverage platforms to an empty list.
Fixes#57534
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
testinstance.py contains a string literal comparison to the
word 'platform', which has been misspelled as 'plaform'.
The change is verified by a test expansion.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Making the necessary changes to enable the new pytest plugin.
By default Twister should work without the pytest-twister-harness
plugin installed. To achieve this, each time Twister calls pytest,
the PYTHONPATH environment variable is expanded and the
`-p twister_harness.plugin` option is added to the pytest command.
Co-authored-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Adding pytest plugin dedicated to running pytest tests in Zephyr
project. This plugin provides a dut fixture which allows to handle
bidirectional communication with the device under test. This version
of plugin can be used for tests dedicated to real hardware, QEMU and
native_posix simulator.
Co-authored-by: Lukasz Fundakowski <lukasz.fundakowski@nordicsemi.no>
Co-authored-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Co-authored-by: Katarzyna Giadla <katarzyna.giadla@nordicsemi.no>
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Add support for calling the `renode-test` command from west and twister.
Enable running Robot Framework tests suites in Renode.
Signed-off-by: Michał Szprejda <mszprejda@antmicro.com>
Signed-off-by: Mateusz Hołenko <mholenko@antmicro.com>
Current implementation of ConfigurationError causes
an exception to be thrown.
It is because a PosixPath is being concatenated with an str with a '+'
operator, which is not permitted.
The path in question is now cast to an str.
Additional test has been added to test_twister.py to track regressions.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
For the arch's that have standard toolchains thare are supported
across the arch set them in twister rather than in <BOARD>.yaml.
Signed-off-by: Kumar Gala <kumar.gala@intel.com>
Some 3P logic (Pigweed for example) is already heavily invested in
gTest as a testing framework. Adding the `gtest` harness enables
running the existing 3P tests on various Zephyr platforms and configs.
Signed-off-by: Yuval Peress <peress@google.com>
Simics converts newline characters when writing console output to
stdout, so we need to remove them as a suffix string. Otherwise the
console harness fails to match expected PASS/FAIL/RunID string patterns
and twister tests timeout.
Signed-off-by: Maureen Helm <maureen.helm@intel.com>
After generating a hardware-map a platform field is marked
as 'unknown'. If it is not changed, do not process them
when running --device-testing.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
In some cases CMake doesn't create build directory and in those cases
Twister should create this directory by itself.
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
We have been clearning the list of testcases in every iteration over the
symbols ending up with only one testcase in the instance.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Add a new option to Twister that allows to load alternative
configuration files, `--alt-config-root`. This new option takes an
arbitrary number of root folders where alternative configuration files
can be stored. Twister will check if a test configuration file exists in
any of the alternative test configuration folders. For example, given
`$test_root/tests/foo/testcase.yaml`, Twister will use
`$alt_config_root/tests/foo/testcase.yaml` if it exists.
This feature can be useful if an out-of-tree project needs to run
upstream tests in different configurations, potentially not available
upstream (e.g. an out-of-tree board, or Kconfig setting).
Note that overlaying has been discarded because we can't easily delete
fields from the original configuration file, something that in certain
cases could be required. Current approach may lead to some dupplication,
but guarantees full control of the test configuration.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
Make twister to raise error by default whenever duplicated test ids
are found. Remove redundant option to list duplicates.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Don't override a test case name check failed result by the following
RunID check failed result to report clearly about test case mismatch
situation as well as more specifically when the same test case appears
with unexpected RunID.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Fix suite-name-check bypassed at DeviceHandler on flash errors.
If the flash timeout is smaller than needed, but test images are
actually flashed successfully and appropriate tests executed,
then results from the previous test case were taken because
the flash error status makes the suite-name-check bypassed.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
We can now have situations like this when it comes to lists in Twister
config:
```yaml
common:
platform_exclude: foo
tests:
my.test:
platform_exclude: bar
```
```yaml
common:
platform_exclude: foo
tests:
my.test:
platform_exclude:
- bar
- baz
```
```yaml
common:
platform_exclude:
- foo
- bar
tests:
my.test:
platform_exclude: baz
```
```yaml
common:
platform_exclude:
- foo
- bar
tests:
my.test:
platform_exclude:
- baz
- bazz
```
This patch fixes handling of cases (2), (3) and (4).
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
This patch handles mixed types between common/tests sections in the
configuration, e.g. str+list, list+str, str+str.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
Since pykwalify does not support multi-type fields, we are forced to use
the 'any' type. This is used for fields that allow sets/lists written as
either YAML lists or space-separated strings. This patch checks for
str/list Python type, and raises a configuration error if not true so
that user obtains better error messages in case the configuration file
contains invalid format.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
Twister configuration files have a few fields defined with the "set"
type. The set needs to be specified as a space-separated list of
entries, for example:
```yaml
plaform_allow: boardx boardy
```
This patch allows users to write these fields as YAML lists, for
example:
```yaml
platform_allow:
- boardx
- boardy
```
The list gets converted to a Python set, therfore the same exact
behavior is kept in Twister.
In some cases like in the example above, using lists allows to have more
readable configuration files, specially if the list is long. It also
helps on diffs, provided we have one item per line.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
A change to the scope selection rules was introduced by #52715
but it doesn't comply with the description in the code. What's more,
the change introduces major issues i.e. non deterministic scope
selection, especially in a CI environment. More context at #57595
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Twister does not handle out-of-tree boards automatically. Modules like
e.g. example-application that define their own board root folder, are
required to inform twister via `--board-root`. In contrast, west is able
to to this automatically. This patch makes twister a bit smarter by
automatically filling the board root list with module paths as well.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
Add option to shuffle tests (randomly) before they are split into subsets.
This allow to fairly distribute tests across subsets,
so that long tests are not crowded in a single group.
This also allows to detect unwanted dependencies in test execution order.
Also, added option to provide custom seed to random generator
used to shuffle tests.
This will allow to reproduce certain test order if needed.
Used seed is printed at console.
Signed-off-by: Piotr Kosycarz <piotr.kosycarz@nordicsemi.no>
The initshell sections in the linker scripts where associated with theo
old shell code. The old shell code has been removed for some time so
remove references to initshell in the linker scripts and size_calc.py
Signed-off-by: Kumar Gala <kumar.gala@intel.com>
Convert following construction:
elif self.options.device_serial or self.options.device_serial_pty:
if self.options.device_serial:
...
else
...
to the construction:
elif self.options.device_serial:
...
else self.options.device_serial_pty:
...
Signed-off-by: Andrei Emeltchenko <andrei.emeltchenko@intel.com>
Use flash_timeout and flash_with_test options not only for
serial console but also for device_serial_pty.
Signed-off-by: Andrei Emeltchenko <andrei.emeltchenko@intel.com>
compliance script runs pylink on size_calc.py and reports:
C0325:Unnecessary parens after '=' keyword (superfluous-parens)
Fix warning by remove unnecessary parens.
Signed-off-by: Kumar Gala <kumar.gala@intel.com>
`TestInstance.get_elf_file()` does not include the binary name used for
unit testing (`testbinary`) in its search, causing unit tests to not
complete successfully. This does not cause tests to fail and thus goes
unnoticed.
Update the glob expressions used in `get_elf_file()` to also look for
files called `testbinary`
Signed-off-by: Tristan Honscheid <honscheid@google.com>
If we are doing a sysbuild get_elf_file needs to return the default
domain's build_dir. The standard build_dir will not any ELF builds
in it so will error out.
Signed-off-by: Kumar Gala <kumar.gala@intel.com>
This is usually used only when doing size calculation, but now it is
being used for extracting symbols from the ELF for testing purposes and
some issues arise with multiple ELF files found.
Simplify the code a bit to make it easy to maintain a lists of things to
exclude.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The DeviceHandler class method monitor_serial() has time-of-check
to time-of-use (TOCTOU) race condition possible on its serial port
which might be closed and its file descriptor cleared in between
of isOpen() and is_waiting status checks.
The issue is alleviated with controlled exception handling.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Fixes an issue when DeviceHandler awaited for a suitable DUT infinitely.
It was possible if the device map has no devices with required platform,
or no devices with fixture needed for the test case, or devices without
serial connection. A trivial case is a typo in platform name.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
For ztest twister would parse the symbol information that was
generated as part of the build (zephyr.symbols). However the format
of the zephyr.symbols files is highly dependant on the toolchain.
Move to using pyelf to parse the symbol information directly from
zephyr.elf instead so that this works regardless of toolchain.
Signed-off-by: Kumar Gala <kumar.gala@intel.com>
On native platforms we typically have CONFIG_BUILD_OUTPUT_EXE set so
we end up with both zephyr.elf and zephyr.exe. This would cause a
call to get_elf_file() to fail. There isn't any reason to error out
since zephyr.elf and zephyr.exe should be identical. So allow it
if the platform type is native.
Signed-off-by: Kumar Gala <kumar.gala@intel.com>
Use explicit flash command execution timeout at DeviceHandler
instead of the hardcoded value of 60 sec.
When a HW platform executes test cases right after the flash command,
the test case remaining timeout is affected by how much time the flash
operation consumes. Some simulation platforms need to spend significant
amount of time on each 'flash' cycle, usually adding the same delay on
each test case.
This improvement adds two new command line options and device map fields:
--device-flash-timeout - for the flash operation timeout
('flash-timeout' device map field).
--device-flash-with-test - to indicate that the platform flash
command also runs a test case, so the overall timeout should be
calculated as a sum of the flash timeout and the current test case
timeout to receive all console output from the platform
('flash-with-test' device map field).
The device map field values override command line values for the
particular platform where configured.
Default behavior is backward compatible: flash operation fixed timeout
is 60 sec. not including the test case timeout.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Rather than having to add 'armclang' to every <BOARD>.yaml, just list
it as a generally supported toolchain if the architecture is 'arm'.
Signed-off-by: Kumar Gala <kumar.gala@intel.com>
When a test start, mark it as such, if we do not get any results due to
a timeout, it will be finally marked as failed. Previously such tests
causing a freeze were marked as blocked (or not run), which is not
exactly right.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Skips due to overflows on integration_platforms during integretion
mode should be treated as errors.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
`self.instances[filtered_instance.name]` and `filtered_instance`
are already referancing the same object.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
If canonical zephyr base is already ended by trailing slash, then it
should not be added redundantly.
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
In some cases built binary files have different names than "zephyr.hex"
files, and for those cases, information about names of file to keep
should be taken from runners.yaml file.
Moreover if in runners.yaml file exist absolute files paths to binary
files, then they should be changed to relative ones.
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Currently in binary handler and qemu handler we have status `failed`
in case of test timeout, but in device handler we have status `error`
This not only adds inconsistency between handlers, but also prevents
us from usage test retry functionality for the runs on HW.
Fix timeout status by changing it to `failed` instead of `error`.
Signed-off-by: Eugeniy Paltsev <Eugeniy.Paltsev@synopsys.com>
Signed-off-by: Evgeniy Paltsev <PaltsevEvgeniy@gmail.com>
In many of test specifications yaml files (testcase.yaml, sample.yaml)
section `platform_allow` is added. This change allows to test some
scenarios on platforms, that are not added yet to platform allow
list (or are not going to be added for some reasons).
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Fix how DeviceHandler class does 'serial leftover' cleanup
with loss of test output observed on fast console connections,
e.g. over telnet: either 1 sec. of output might be missed,
or up to 1000 bytes/chars, whatever happens first.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
We are wrongly claiming no console output, yet console output is there
and the failure is just a regular timeout, i.e. the test did not
complete within allocated time.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Add support test levels and the ability to assign a specific test to one
or more levels. Using command line options of twister it is then possible
to select a level and just execute the tests included in this level.
Additionally, a test configuration allows definiing level
dependencies and additional inclusion of tests into a specific level if
the test itself does not have this information already.
In the configuration file you can include complete components using
regular expressions and you can specify which test level to import from
the same file, making management of levels easier.
To help with testing outside of upstream CI infrastructure, additional
options are available in the configuration file, which can be hosted
locally. As of now, those options are available:
- Ability to ignore default platforms as defined in board definitions
(Those are mostly emulation platforms used to run tests in upstream
CI)
- Option to specify your own list of default platforms overriding what
upstream defines.
- Ability to override build_onl_all options used in some testscases.
This will treat tests or sample as any other just build for default
platforms you specify in the configuation file or on the command line.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Twister allows filtering based on kconfigs and dts, however the
filtration is a part of the cmake stage, i.e. the stage has to pass
first and then twister checks if required properties are available.
This causes problems, when the full cmake stage is unable to pass.
If so, other filtration methods had to be used, e.g. platform_allow.
The commit modifies the twister workflow:
if a test defines filters based on kconfig/dts first a cmake package
helper script is used to extract dt and/or kconfigs and if the
conditions are fulfilled it proceeds to a regular cmake stage.
If not, test is skipped.
Co-authored-by: Daniel DeGrasse <daniel.degrasse@nxp.com>
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
This is an implementation of issue #48334 and adds support for
specifying additional config and device tree overlays through fields in
the testcase.yaml file, which is more readable than having to cram these
in to `extra_args`.
Consider this example which shows the original and new ways to add
config and DT overlays:
```
common:
extra_args: "CONF_FILE=a.conf;b.conf
DTC_OVERLAY_FILE=w.overlay;x.overlay OVERLAY_CONFIG=e.conf
UNRELATED=abc"
tests:
utilities.base64:
extra_conf_files:
- "c.conf"
- "d.conf"
extra_overlay_confs:
- "extra_overlay.conf"
extra_dtc_overlay_files:
- "y.overlay"
- "z.overlay"
extra_configs:
- CONFIG_SAMPLE=y
tags: base64
type: unit
```
The new fields are `extra_conf_files`, `extra_overlay_confs,
`extra_dtc_overlay_files`. Files specified in these sections are appended
to any `CONF_FILE`, `OVERLAY_CONFIG`, or `DTC_OVERLAY_FILE` fields in
`extra_args`, causing the following args being passed in to
`self.run_cmake` at `runner.py:850`:
```
['-DUNRELATED=abc',
'-DCONF_FILE=a.conf;b.conf;c.conf;d.conf',
'-DDTC_OVERLAY_FILE=w.overlay;x.overlay;y.overlay;z.overlay',
'-DOVERLAY_CONFIG=e.conf extra_overlay.conf '
'<build_dir>/twister/testsuite_extra.conf']
```
These fields can be used in the common or scenario-specific YAML
sections and will be merged in order of least to most specific:
1. config files extracted from common's extra_args
2. files listed in common's {extra_conf_files or extra_overlay_confs
or extra_dtc_overlay_files}
3. config files extracted from test scenario's extra_args
4. files listed in test scenario's {extra_conf_files or
extra_overlay_confs or extra_dtc_overlay_files}
Specifying these files in extra_args now triggers a deprecation warning,
as the direct YAML fields are preferred for readability. They will still
function for now but support will be dropped in the future. One
testcase.yaml
(`zephyr/tests/cmake/overlays/var_expansions/testcase.yaml`) is
converted to use the new fields. A follow-up PR will convert the
remaining files to the new format.
Signed-off-by: Tristan Honscheid <honscheid@google.com>
Percentage calculation was off due to a recent fix in counting and lines
were not breaking correctly on errors.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Fixed runtime cleanup option. A typo in a conditional prevented the
cleanup from happened when a test has passed.
Fixes#54240
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Initialize 'use_system_gcov', so it is always set before being referenced
in "elif" statement.
Fixes: 1440b19bde ("twister: restore logic to determine which gcov we
should use")
Signed-off-by: Marcin Niestroj <m.niestroj@emb.dev>
Stop counting errors as failures and report them as is. This fixes
issues with counting when retrying failures and errors.
We have been adding errors to failures and then double count them when
displaying information on the screen instead of keep an accurate count
in the counter class.
This is now fixed and we do report errors seperately, ie. build errors.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
We do not want to use gcov from the SDK for native posix and unit tests,
instead we want the system gcov. Restore logic that determines that.
Fixes#54226
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Move extra warning option from generic twister script into
compiler-dependent config files.
ARCMWDT compiler doesn't support extra warning options ex.
"-Wl,--fatal-warnings". To avoid build fails flag
"disable_warnings_as_errors" should be passed to twister.
This allows all warning messages and make atomatic test useles.
Signed-off-by: Nikolay Agishev <agishev@synopsys.com>
Tests under quarantine are not treated as errors in the integration
mode. However --quarantine-verify argument allowing to execute just
test under quarantine and skipping others was not considered in
relation to integration mode. Those skips are wrongly threated as
errors. This commit fix this relation and makes thoses skips not
turned to errors.
fixes: #54516
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Take an instance that has failed and not whatever was last processed in
the for loop.
Also, change instructions to use `west twister` rather than
`./scripts/twister` to be consistent.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
If errors are found and tests are failing, print a brief summary showing
the top 10 items failing with instructions on how to reproduce either
with twister or west.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Disables allowing the python argparse library from automatically
shortening command line arguments, this prevents issues whereby
a new command is added and code that wrongly uses the shortened
command of an existing argument which is the same as the new
command being added will silently change script behaviour.
Signed-off-by: Jamie McCrae <jamie.mccrae@nordicsemi.no>
When twister is started from make, use the information in MAKEFLAGS to
participate in the job server started by make.
When twister is running standalone, start a jobserver to share with
child processes. If using the kitware version of ninja, then this will
prevent ninja+twister from starting more processes than XX when twister
is run with -jXX.
There is no implementation for windows yet.
Signed-off-by: Jeremy Bettis <jbettis@google.com>
This reverts commit 21eb27c5c0.
This change has been a source of much trouble and ends up preventing us
from tracking test results across tree. It uses the repo name as the
namespace, and that is not always the same and does not have to be
called 'zephyr', depending on where you are running and in which
environment.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Improves the message for tests being excluded using the platform key,
describes the key already found and the platform being used to run the test
in its place.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
Implementation ported from TwisterV2.
- quarantine handled by separate module
- multiple yaml allowed from args: --quarantine-list
- scenarios, platforms, architectures keywords in
quarantine yaml are optional, if not given - means take it all
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Some hardware boards are supported in qemu as simulation platforms. If
we are testing with the real hardware, make sure we treat this as a
device and not as a simulator. First priority when initializing a
handler is to check if device testing is requested, then we can handle
the rest without --device-testing.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Adds an option to inform twister a testsuite should only be built and
run for platforms with unique sets of attributes. This enables
for example keying on unique (arch, simulation) platforms to run the test
suite on.
The most common usage may be test suites configured to run once per
(arch, simulation) pair as being enough. Additional information about
platforms may enable running a test once per hardware IP block or once
per soc family or soc avoiding duplicated effort in building and running
tests when once suffices.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
We want to be able to have platform or architecture extra configs
without having to duplicate a whole section of the test specification.
This adds support for namespacing of extra configs, for example:
arch:nios2:CONFIG_SYS_CLOCK_TICKS_PER_SEC=1000
or
platform:qemu_x86:CONFIG_FOO=y
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Previous commit dealing with cleanup (c304db56f1) was not taking
--prep-artifacts-for-testing into consideration. Fix this and make sure
we do cleanup when this option is enabled.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Have boards define what binaries should be kept for device testing. The
purpose of this is to reduce the amount of artifacts and just keep the
binaries needed for flashing a device, reducing the size of the final
payload.
A board can have twister keep multiple binaries needed for testing when
--prep-artifacts-for-testing option is used.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
When re-generating hardware map for DKs with more than 1 serial port (more
enries in the hardware map), then we keep enfo only about the last entry.
For boards like nrf5340 it works, because the last serial port is used,
but for nrf52840 with two ports, the first one is important, so after
re-generating the hardware map it does not work.
The solution is to keep all entries after re-generating the hardware map.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
When scanning for available elfs, twister discards ones with `_pre`
in their name, to not include prebuilds like 'zephyr_pre0.elf`.
However, the scan looks at the whole test name. If _pre is in the name
(e.g. _precision) it would be wrongly discarded ending with twister
reporting error that no elfs were found. This commit makes the discard
to look only at the last part of the name (i.e. not including test name
but only a lowest level, where zephyr.elf etc are located.)
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
When using llvm we need to set the gcov-tool to "llvm-cov gcov" but
the lcov tool is incapable of passing arguments to the gcov-tool. i.e.
the following cannot work:
$ lcov --gcov-tool "llvm-cov gcov" ...
Instead, create a symlink to llvm-cov prefixed as `gcov` which by the
documentation of llvm-cov will alias to `llvm-cov gcov` subcommand.
Signed-off-by: Yuval Peress <peress@google.com>
This commit expands Twister's CLI so that arguments passed after
a double dash (`--`) get forwarded to the ztest test executable when
run. When built for native_posix, ztest provides a useful CLI that
allows filtering individual suites and tests, adjusting timing settings,
and controlling the shuffling seed. Currently there is no easy way to
use these (with the exception of `--seed`) without needing to dig out
the resulting `zephyr.exe` binary from the build dir and invoke it
manually.
### Examples
Run a specific ztest suite only (useful when writing tests and you don't
want to run a long testcase in its entirety)
```
$ scripts/twister \
-p native_posix \
-s
zephyr/tests/lib/cmsis_dsp/complexmath/libraries.cmsis_dsp.complexmath \
-- -test=complexmath_q31::*
```
Unrecognized arguments that precede the double dash will result in an
error message:
```
$ scripts/twister \
-p native_posix \
--foobar 123 \
-- -test=basic_math_q7::*
...
Unrecognized arguments found: '--foobar 123'. Use -- to delineate extra
arguments for test binary or pass -h for help.
```
Signed-off-by: Tristan Honscheid <honscheid@google.com>
Ignore unit tests and EFI binaries when calculating footprints with
--show-footprint, or we will get a build error.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The --snr (nrfjprog) --id (jlink) and --board-id (pyocd) options were
deprecated a long time ago in favor of --dev-id. It is time to remove
them.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
This commit calculate memory footprint from build.log and
proposes an alternative approach to #2465.
Signed-off-by: Katarzyna Giądła <katarzyna.giadla@nordicsemi.no>
non-simulation platforms set key to 'na', we need to check for that
instead of just checking for existance of key.
Fixes#52595
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
If a default platform is not able to run for whatever reason, treat it
as any other platform. Default platforms shall ALWAYS be runnable in CI,
or else we are just re-building for no good reason.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
the ser.in_wait sometime meet issue like serial handler error.
This is because the serial port is in reset status.
add final protection to retry later
error log:
Traceback (most recent call last):
File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/usr/lib/python3.10/threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
File "zephyr/scripts/pylib/twister/twisterlib/handlers.py", line 345,
in monitor_serial
if not ser.in_waiting:
File "~/python3.10/site-packages/serial/serialposix.py", line 549,
in in_waiting
s = fcntl.ioctl(self.fd, TIOCINQ, TIOCM_zero_str)
OSError: [Errno 9] Bad file descriptor
Signed-off-by: Hake Huang <hake.huang@oss.nxp.com>
add 2 choices for '--runtime-artifact-cleanup' option,
'passing' to delete artifacts of passing tests,
'all' to delete artifacts of both passing tests
and failing tests
keep testcase_extra.config, it contains some extra Kconfig
needed for the tests and they is only generated first time
when run retry failed
Signed-off-by: Cong Nguyen Huu <cong.nguyenhuu@nxp.com>
Convert domains.py to use basic python logging module implementation to
remove west logging dependency. This removes hard dependency on west from
twister.
Fixes#51731
Signed-off-by: Daniel DeGrasse <daniel.degrasse@nxp.com>
With the patch a relative path between a test and west's project top
directory is always calculated instead of being relative to zephyr
base. The test names and results structur change by addition of one
level. This way the names/structure conventon for tests from zephyr and
modules is alligned.
For external tests ".." parts are replaced with "external".
This allows to maintain unified folder structure also for results of
external tests.
Without this patch the folder/name allignment is lost for
external and module test becaming depandant on a way how
twister was called.
.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Version is not enough, we need the run date but also the commit date, to
ease reporting and use in dashboards.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
In some scenarios we might end up with a null version value in the json
file, if the version can't be determined, set it to Unkown instead.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
When twister is set to retry any failures, count the number of retries
and record the number in the json file. This will help us identify
unstable tests or tests requiring attention.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Update twister to support running QEMU platforms with sysbuild, by parsing
domains.yaml and executing the "run" target of the default application.
This will allow twister to test QEMU targets with sysbuild. It is assumed
that QEMU targets will add any external images they need in the build
phase
Signed-off-by: Daniel DeGrasse <daniel.degrasse@nxp.com>
Since twister can make use of domains.py as well, refactor west's use
this file so that domains.py can be moved to a generic library folder.
Signed-off-by: Daniel DeGrasse <daniel.degrasse@nxp.com>
When running twister -vv many blank lines would be logged.
Fix handing of empty lines from the reader thread. Real blank likes will
be b"\n" not b"".
Change open call to use with, and logger to use %s to fix pylint
warnings.
Signed-off-by: Jeremy Bettis <jbettis@google.com>
Modify the regex used to merge overlays specified for
test samples in twister tests.
Overlays without any prefixes will be merged and passed to cmake.
Overlays with prefixes won't be merged but passed to cmake as they are.
Signed-off-by: Piotr Węgliński <piotr.weglinski@nordicsemi.no>
The logic to drain the leftover logs of previous test should be
put before the monitor_serial(). It was misplaced when re-adding
this logic back.
Signed-off-by: Ming Shao <ming.shao@intel.com>
In BinaryHandler we process input from simulator with decode('utf-8')
to convert it to string. decode has strict error handling by default -
so it raises UnicodeDecodeError exception if it can't decode input
binary sequence.
So if test start to print some junk to uart console we get
UnicodeDecodeError exception which cause the whole twister crash.
To fix that switch decode to less strict error handling when it
just replace undecoded binary sequence with unicode replacement
character.
Signed-off-by: Eugeniy Paltsev <Eugeniy.Paltsev@synopsys.com>
Signed-off-by: Evgeniy Paltsev <PaltsevEvgeniy@gmail.com>
On windows, some board manufacturers have name that is "Microsoft"
instead of real its name. Need add it to manufacturer list.
Product information is missing. we can assign it "unknown" and
then add missing informations by hand.
Signed-off-by: Cong Nguyen Huu <cong.nguyenhuu@nxp.com>
Currently, support for Twister on Windows is only build.
The only thing that needs to be ported is that select()
cannot wait on file descriptors on Windows. Therefore
the serial monitor function needs to be reworked to
support both OSes.
Signed-off-by: Cong Nguyen Huu <cong.nguyenhuu@nxp.com>
Add support for building with sysbuild using twister, via the "sysbuild"
yaml property in testsuites. This will currently disable Kconfig and
devicetree filtering.
Signed-off-by: Daniel DeGrasse <daniel.degrasse@nxp.com>
Due to the diverse coding styles and lack of preprocessing when
scanning for test cases, there were many unintended combinations
of yaml test scenarios and C test functions with the regex-based
test case discovery, which caused an inaccurate test plan and test
result stats.
As the new ztest fx is used, the test cases of a test instance can
be accurately determined via zephyr.symbols file.
Because the zephyr.symbols file is not available until after build,
test cases determination is placed right after the build stage of
the pipeline and before the runtime test collection.
For those test instances that don't go through the build stage,
such as those with "Skip filtered" (statically filtered) reason
before entering the pipeline, they are not affected.
This patch also adjust the stats logic because the zephyr.symbols
file is generated after build. So ExecutionCounter update is split
and some must be postponed until test execution pipeline is completed.
Some concepts:
A test instance = a yaml scenario + a platform
"Test instance" and "test config" are synonyms in twister, unfortunately
excessive IMHO...
A test instance can be filtered or skipped in 3 ways.
Let's define "runtime" as "after entering the execution pipeline".
1) statically filtered (before runtime)
Such test instance is filtered by analyzing the yaml and never
enters the execution pipeline.
2) cmake filtered (runtime)
Such test instance enters pipeline and is filtered at cmake stage.
3) build skipped (also runtime)
Such test instance enters pipeline and is skipped at build stage.
All the test instances that enter the execution pipeline will go
through the report stage, where ExecutionCounter is further updated.
The meaning of the fields of ExecutionCounter are:
.skipped_configs = static filtered + cmake filtered + build skipped
.skipped_runtime = cmake filtered + build skipped
.skipped_filter = static filtered
.done = instances that enter the execution pipeline
.passed = instances that actually executed and passed
Definition of the overall stats:
total_complete = .done + .skipped_filter
total = yaml test scenario * applicable platforms
complete percentage = total_complete / total
pass rate = passed / (total - skipped_configs)
Signed-off-by: Ming Shao <ming.shao@intel.com>
This commit removes the stale reference to `k_mem_pool_area` for the
memory pool API, which was removed in the v2.5.0 release.
Signed-off-by: Stephanos Ioannidis <stephanos.ioannidis@nordicsemi.no>
This logic was added before but seems to be missed during the
twister refactoring.
Some tests can cause serial leftover logs buffered somewhere.
Such leftover can interfere with the next test case because the
serial log monitoring thread is started before a board is flashed.
And the monitoring thread can be fooled by such leftover logs and
make incorrect judgement of the test result.
A simple ser.flush() is not enough to eliminate such leftovers.
So add explicit readline() to drain such logs which ensures a clean
serial context for the case that follows.
An example from reel board captured with this patch:
leftover log of previous test: b'.287 seconds\r\n'
leftover log of previous test: b' - SKIP - [...test_coredump_backend]...
leftover log of previous test: b'\r\n'
leftover log of previous test: b'------ TESTSUITE SUMMARY END ------\r\n'
leftover log of previous test: b'\r\n'
leftover log of previous test: b'=====================================...
leftover log of previous test: b'RunID: 4e93757ad...53dcab9f0f5c6\r\n'
leftover log of previous test: b'PROJECT EXECUTION SUCCESSFUL\r\n
Signed-off-by: Ming Shao <ming.shao@intel.com>
Twister detects FLASH overflow and skips tests that trigger that
condition by default, but sometimes images are just on the flash size
limit and then overflows at a later stage when imgtool adds a trailer,
which gets detected as a build fail and fails the run.
This detects the imgtool flash overflow together with the normal build
ones, causing the build to be skipped on imgtool flash overflow as well.
Signed-off-by: Fabio Baltieri <fabiobaltieri@google.com>
A workaround to exclude boards from twister. Other solutions do not work,
setting twister fields to false in the board yaml file is not enough,
given that this board has many associated samples and tests, so it can't
be disabled on its own.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Follow-up: #41301
This commit is a rework and cleanup of the tools handling in Zephyr
CMake build system.
Instead of directly loading code a CMake modules for tool lookup, the
host tools now follows the CMake `find_package()` pattern for finding
programs / tools in module mode.
This makes it more clear which modules are responsible for finding tools
and which modules provides build integration / features.
The following tools can now be found using `find_package()`:
- Zephyr-sdk : find_package(Zephyr-sdk <version>)
- Generic host tools: find_package(HostTools)
This further allows us to decouple the `verify-toolchain` CMake script
part required by `twister` into a tool lookup module and a dedicated
CMake script which utilizes the lookup module.
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
`handlers.py` treats the `psutil` package as optional and quietly prints
a message to stdout if it is not present, but it is actually a hard
requirement for the base `Handler` class. This try/except block is bad
as it hides the requirement until a timeout occurs and the handler
attempts to use `psutil` in the terminate method, which will crash if
the module happens to be missing.
This PR removes the try/except guard so missing `psutil` will cause
twister to immediately fail. `psutil` is already specified in the
`requirements.txt` file, so this should only affect users who
unwittingly have an outdated/incorrect environment.
Also update the Github Action for twister tests to install from
requirements files.
Signed-off-by: Tristan Honscheid <honscheid@google.com>
stty operates on the file descriptor from stdin, but we want to operate
on stdout. If stdin and stdout are not the same tty then stty prints an
error.
Redirect stdout to stdin for stty calls.
Tested by running twister -T tests/subsys/shell </dev/null
Signed-off-by: Jeremy Bettis <jbettis@google.com>
With verbose output enabled, twister logs the cmake arguments as a
python array whenever it invokes cmake to generate a build system.
This is clunky output for manual reproduction, because it forces the
developer to manually join the arguments together for consumption by
the shell.
As an enhancement for this use case, use the standard library
shlex.join() function to produce a string that can be copy/pasted
directly into the shell on POSIX platforms.
We should use this function instead of str.join, because it correctly
handles things like shell quoting for options with spaces in them.
This function is not available on Windows, so we fall back on behavior
that is similar to the old one there.
The main difference on Windows is that the output now includes the
path to cmake as well, and not just its arguments. (This is the same
on POSIX, and is intended to help debug if multiple cmakes are
installed on the same machine.)
We are running cmake from a couple of different places, so doing this
cleanly requires adding a shared module with the platform abstraction.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
esp32 platforms can overflow their dram0_1_seg so add that section name
to the overflow check in twister. We were seeing this with the
samples/arch/smp/pi/sample.smp.pi test and would get:
zephyr_pre0.elf section `noinit' will not fit in region `dram0_1_seg'
region `dram0_1_seg' overflowed by 17456 bytes
Fixes#49164
Signed-off-by: Kumar Gala <galak@kernel.org>
This adds an extra layer to match the revision part where this
needs to be of permitted patterns as described in.
cmake/modules/extensions.cmake. Without this, the matching
may produce undesirable results. For example, if there is
a file named qemu_x86_tiny_1.conf, the testplan will create
a board name qemu_x86@tiny.1 (which is definitely not correct).
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
On qemu_cortex_a53 with icount enabled, the qemu process will spin
hard at the end of its run (common with icount), but will also ignore
the SIGTERM twister is sending to terminate the (successful!) test
run. The resulting zombie spinning processes end up accumulating and
poisoning the test run.
Just use a hard KILL instead of trying to be a good citizen (and in
lieu of fixing the upstream qemu bug). I can't see why this should be
a problem.
Signed-off-by: Andy Ross <andyross@google.com>
Fix the missing logic of setting the flash_error flag when
caught the exceptions. And we should let the serial thread
can be run first if flash error. Otherwise the twister will
hang up while an exception happens on closing serial, after
the #47820 got merged. This mostly happens on frdm_k64f.
Fixes#49086
Signed-off-by: Enjia Mai <enjia.mai@intel.com>
Add specifying the port for the remote cavstool remote service.
Ex.
west flash --remote-host {host}:{port} \
--pty {host}:{port}
Specify the port is optional when running west.
And another major change is now we have to specify the remote log
service by --pty to make it output the log message immediately.
Previously it will output directly. Why we make this change is
because this is more close to the behavior we use west flash.
Fixes. #46865
Signed-off-by: Enjia Mai <enjia.mai@intel.com>
Currently, twister keeps on monitoring serial output until timeout
even if we have a flash error. And the error reason will be reported
as 'No Console Output(Timeout)', which is not accurate.
So add a flash_error flag to control if we need to monitor the output.
Signed-off-by: Guo Lixin <lixinx.guo@intel.com>
The boards command was not properly using the zephyr_module
functionality to obtain the board roots of all modules. Fix that by
moving the functionality required to the core zephyr_module file and
reuse it from external scripts.
Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
support to use board@revision as platform filter when running
twister, like "twister -p nucleo_f030r8@1 ...".
Signed-off-by: Chen Peng1 <peng1.chen@intel.com>
Escape quotes provided in extra arguments on the command line.
example: ... -x "CONFIG_COMPILER_OPT=\"-fanalyzer\""
Fixes#46382
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Fix bug in verify_platforms_existence method - make it possible to
verify all platform names from list - not only first.
Fixes: #48321
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
This commit makes the `--ninja` argument default to `true` when neither
`-k` nor `--make` is specified (i.e. when Make generator is not
selected).
Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
When `--short-build-path` argument is specified to the twister, the
following error message is displayed because the Python argparse module
does not allow specifying a mutually exclusive argument as required:
ValueError: mutually exclusive arguments must be optional
This commit removes the `required` parameter when adding the `--ninja`
argument, which is a mutually exclusive argument, and adds a manual
check to validate this condition instead.
Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
Generates an error if --coverage-formats option is supplied without
coverage tool being set to gcovr.
Signed-off-by: Jason Wright <jwright@synchron.com>
The specific hardware instance is already got from
device_is_available function, we don't need to
iterate all hardwares again.
Signed-off-by: Chen Peng1 <peng1.chen@intel.com>
Somehow this change was dropped (rebase?), this is needed to parse new
ztest test correctly, otherwise will have many tests missing from the
plan.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Add option to prepare for device testing by only building and storing
the needed data that can then later be executed and run on DUT using
--test-only.
This adds --prep-artifacts-for-testing which is the same as --build-only
and --device-testing and other options that were needed to make this
operational and produce the right binaries and files.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
using serial as the group name is misleading, so call it 'device' which
is what all those options about.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This is all static functions that can maintained outside of the class
and called only in one place, so move this to the testsuite file.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Get options from passed environment class instead of using command line
options to the init function.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Move all code related to instance filtering and loading into the class
instead of having it in the main script. Simplify option passing.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
We have been scanning code for tests for every scenario defined in the
yaml file. This needs to only be parsed once and not more. We are
dealing with the same files for each scenario.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
If we already have recorded the reason during the qemu run, do not
override it at the final stage of status processing.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
