This commit updates all in-tree code to use `CONFIG_CPP` instead of
`CONFIG_CPLUSPLUS`, which is now deprecated.
Signed-off-by: Stephanos Ioannidis <stephanos.ioannidis@nordicsemi.no>
The old random timer test was not random-looking
enough on some platforms.
Replace with new test which is psuedo-xoshiro.
The generator is still deterministic
and does not depend on entropy at all,
but should look more random for testing.
Change name of generator tree-wide also.
Signed-off-by: Declan Snyder <declan.snyder@nxp.com>
Add a test to make sure that when a zassume fails, we mark the test
as failed and print an "Assumption failed" error for the test
Signed-off-by: Yuval Peress <peress@google.com>
Adding the new Kconfig (enabled by default) to make a failed assumption
mark the final result as failed. This change has the following benefits
which have been asked for by the Zephyr community:
1. A failed assumption does not go silent. In this example, the failed
assumption will still mark the test as skipped, but the final result
will be to mark the full test run as failed. This would allow
blocking the CI when an assume fails.
2. Normal test skipping via the ztest_test_skip() is unaffected by this
change. Those tests will be marked as skipped, but the binary will
still pass.
Signed-off-by: Yuval Peress <peress@google.com>
I've update FFF to run tests in the native CI for GitHub, so we no
longer need to run these extra tests. Remove the fff module from
west.yml since the module was only used in the CI and the header
was directly included in the Zephyr tree.
See https://github.com/zephyrproject-rtos/fff/pull/2
Signed-off-by: Yuval Peress <peress@google.com>
When CONFIG_ZTEST_SHUFFLE is enabled and ASSERTS are disabled
`start_pos` becomes an unused variable leading to a compile error.
Cleaned-up shuffling algorithm to not need a `start_pos` check.
Signed-off-by: Al Semjonovs <asemjonovs@google.com>
This commit disables the `testing.ztest.ztress` test on
`qemu_cortex_r5` because it fails at run-time when compiled with
GCC 12.
Revert this commit when the GitHub issue #49494, which tracks this bug,
is fixed.
Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
Add specialized tests that execute another test indirectly and ensure
that the results report errors correctly. See the README.rst file
provided with this commit for details.
Signed-off-by: Yuval Peress <peress@google.com>
With the support of Kconfig in unittest CMake module then prj.conf is
now supported.
Add a prj.conf with ZTEST config settings and remove hardcoded values
from CMakeLists.txt file.
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
Add test summary after all test suites finish running.
The summary can be one-line or verbose, which is configured
with CONFIG_ZTEST_VERBOSE_SUMMARY. The one-line summary covers
overall suite stats. The verbose summary covers each test
function within the suite besides the one-line summary.
The new ztest output ultimately go through the printk. If
printk go through the logging subsystem, there may be log
messages dropped. And if log_panic is invoked, log messages
can be flushed in a mess. So several explicit log flush
are used when printing summary to ensure no content is lost
and content is in good shape.
Some macros are shared between old and new ztests. Such as
TC_START_PRINT and TC_END_PRINT. The are defined accordingly.
Signed-off-by: Ming Shao <ming.shao@intel.com>
This commit enables the assert test mode (`CONFIG_ASSERT_TEST`) for the
ztest error hook test because it implements a custom post assert fail
hook (`ztest_post_assert_fail_hook`) that returns without aborting to
faciliate the testing of the assert functions.
Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
Updates the ztest_test_fail() function to allow failures in setup.
When executed, a failed assert will fail every test in the suite owning
the setup function. This was verified by adding a suite which asserts
in the setup function and has a test that should pass. During
exeuction, ztest marks the test as failing.
In order to verify exection I also added 2 new APIs:
- ZTEST_EXPECT_FAIL(suite_name, test_name)
- ZTEST_EXPECT_SKIP(suite_name, test_name)
Signed-off-by: Yuval Peress <peress@google.com>
Add a bunch of missing "zephyr/" prefixes to #include statements in
various test and test framework files.
Signed-off-by: Fabio Baltieri <fabiobaltieri@google.com>
Add an assume API which works like JUnit's. Assumptions can be made
at any point before your test returns (setup, before, and during the
test). If an assumption fails, the test will be marked as skipped.
This helps avoid a cascading affect of failed tests when a base
feature breaks. The feature is expected to have tests and the tests
which depend on it should be skipped (not failed) if that feature
is broken.
Issue #42472
Signed-off-by: Yuval Peress <peress@google.com>
Fix the deprecated calls to loading ZephyrUnittest by replacing them
with `Zephyr COMPONENTS unittest`.
Signed-off-by: Yuval Peress <peress@google.com>
Since output can be cumstomized, so slightly change the output format,
so that twister can parse the output and collect the passrate correctly.
Signed-off-by: Guo Lixin <lixinx.guo@intel.com>
Adds few missing zephyr/ prefixes to leftover #include statements that
either got added recently or were using double quote format.
Signed-off-by: Fabio Baltieri <fabiobaltieri@google.com>
Update the new API to use K_USER as the flags for both
CONFIG_USERSPACE and CONFIG_TEST_USERSPACE. Also, fix the linker
script to properly include the suites, tests, and rules.
Fixes#44108
Signed-off-by: Yuval Peress <peress@google.com>
1.Remove filter to allow platforms(e.g. qemu_x86) supporting
CONFIG_ARCH_HAS_USERSPACE run testcases when userspace is disabled.
2.Define all testcases in the yaml for consistency and issue parsing
them during setup.
Signed-off-by: Guo Lixin <lixinx.guo@intel.com>
Instead of relying on runtime filter to limit scope to emulation
platforms, use the type attribute for each platform and do the filtering
very early on. This will speed things up for tests where we only run on
emulation platforms.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Enable ZTEST_DO_THE_SHUFFLE to shuffle the order tests are ran.
Additional configs ZTEST_DO_THE_SHUFFLE_SUITE_REPEAT_COUNT
ZTEST_DO_THE_SHUFFLE_TEST_REPEAT_COUNT specify the number of times
the test or suite is executed.
Signed-off-by: Al Semjonovs <asemjonovs@google.com>
In order to bring consistency in-tree, migrate all tests to the new
prefix <zephyr/...>. Note that the conversion has been scripted, refer
to #45388 for more details.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
Some names of the test cases are duplicated within the project.
This commit contains the proposed names of the test scenarios.
Signed-off-by: Katarzyna Giadla <katarzyna.giadla@nordicsemi.no>
The filter in the testcase.yaml made it impossible for these to run.
Also, the Kconfig names for the options didn't really make much sense
so they were renamed to avoid conflict.
This fixes an issue found by #36433.
Signed-off-by: Yuval Peress <peress@google.com>
Test were executed on single CPU only and with qemu_cortex_a9
excluded. Removing those limitations after fixes.
Signed-off-by: Krzysztof Chruscinski <krzysztof.chruscinski@nordicsemi.no>
Seen failures on some platforms. No harm to relax the
check for test timeout.
Signed-off-by: Krzysztof Chruscinski <krzysztof.chruscinski@nordicsemi.no>
Change the CPU_CORTEX_R kconfig option to CPU_AARCH32_CORTEX_R to
distinguish the armv7 version from the armv8 version of Cortex-R.
Signed-off-by: Bradley Bolen <bbolen@lexmark.com>
1. Test suites in prior ztest serve no purpose other than logical
ordering of tests into a named-group. Move the construct of setup and
teardown into the test suite and away from individual tests.
Additionally, add the constructs of before/after to the test suites.
This model more closely resembels other testing frameworks such as gTest
and Junit.
2. Test can be added to a suite by using ZTEST() or ZTEST_F() where _F
stands for fixture. In the case where _F is used, the argument `this`
will be provided with the type `struct suite_name##_fixture*`. Again,
this models other modern testing frameworks and allows the test to
directly access the already set up data related to the test suite.
3. Add the concept of test rules (from Junit). Rules are similar to the
before/after functions of the test suites but are global and run on all
suites. An example of a test rule can be to check that nothing was
logged to ERROR. The rule can cause the test to fail if anything was
logged to ERROR during an integration test. Another example would be a
rule that verifies that tests ran within some defined timeout.
Signed-off-by: Yuval Peress <peress@google.com>
Because RISC-V arch (M extension) doesn't trigger exception for
division-by-zero, this test can't support RISC-V and is disabled in
RISC-V.
Signed-off-by: Jim Shu <cwshu@andestech.com>
Introduce a weak implementation of test_main() which calls:
* ztest_run_registered_test_suites(NULL);
* ztest_verify_all_registered_test_suites_ran();
This will attempt to run all registered test suites and verify that
they each ran.
Signed-off-by: Yuval Peress <peress@chromium.org>
Add new functionality to ztest to improve test modularity. The two
primary new entry points are:
* ztest_register_test_suite
* ztest_run_registered_test_suites
When registering a new test suite, users provide the name as well as
an optional predicate used to filter the tests for each run. Using NULL
as the predicate ensures that the test is run exactly once (after which
it is automatically filtered from future runs).
Calls to ztest_run_registered_test_suites take a state pointer as an
argument. This allows the the pragma functions to decide whether the
test should be run.
The biggest benefit of this system (other than the ability to filter
tests and maintain a larger test state) is the ability to better
modularize the test source code. Instead of all the various tests
having to coordinate and the main function having to know which tests
to run, each source file manages registering its own test
suite and handling the conditions for running the suite.
Signed-off-by: Yuval Peress <peress@chromium.org>
Issue #38643
Introduce a more powerful and well established mocking framework
into Zephyr. It also allows running the actual FFF tests using the
zephyr SDK and ztest framework to ensure compatibility.
As per TSC meeting, the fff.h header was directly added to
subsys/testsuite/include/. As per the guidelines, the file is exactly
the same as it is in FFF's library, but re-styled with clang-format.
The west.yml entry was added using the "ci" group and filtered by
default. (note that the tests will break until the CI actually
specifies that the group is needed).
Signed-off-by: Yuval Peress <peress@google.com>
Fix the trigger_fault_access. The fix is similar to cortex m.
Fix the trigger_fault_divide_zero. The fix is similar to qemu, i.e.
skip this testcase, because FVP would not trigger the exception for
dividing zero either.
Signed-off-by: Jaxson Han <jaxson.han@arm.com>
Move to CMake 3.20.0.
At the Toolchain WG it was decided to move to CMake 3.20.0.
The main reason for increasing CMake version is better toolchain
support.
Better toolchain support is added in the following CMake versions:
- armclang, CMake 3.15
- Intel oneAPI, CMake 3.20
- IAR, CMake 3.15 and 3.20
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
The Cortex-R platforms may no use XIP which means userspace code with
have read and execute access to some of the RAM so that it can execute
code. However, userspace will not have access the the protected
read/write parts of RAM so it is safe to use the Cortex-M version of
trigger_fault_access for this test.
Signed-off-by: Bradley Bolen <bbolen@lexmark.com>
On the Cortex M0+, M0, M23 the div-by-zero is not mapped
as an hardware exception but triggers a HardFault.
The test case is skipped for any ARMV6_M_ARMV8_M_BASELINE.
Signed-off-by: Francois Ramu <francois.ramu@st.com>
This compile test should be checking if a symbol has been defined,
otherwise it is using the kconfig value directly. This fixes a warning
../src/main.c:115:37: warning: "CONFIG_SOC_QEMU_ARC" is not defined,
evaluates to 0 [-Wundef]
when using the -Wundef flag.
Signed-off-by: Bradley Bolen <bbolen@lexmark.com>
qemu arc has limited support for DIV/REM, no DIVZERO exception
throw, so we need to skip trigger_fault_divide_zero test case.
Signed-off-by: Watson Zeng <zhiwei@synopsys.com>
Put the testcase test_catch_assert_in_isr() to execute last, to prevent
it affects other test cases. Because when we caught an assert failure
in the ISR handler, it cannot be guaranteed that all the current
program status would be recovered.
Fixes#34844.
Signed-off-by: Enjia Mai <enjiax.mai@intel.com>
Make the divide instruction not being optimized of the ztest example
test case. Do this because it will be replaced by invalid opcode
exception instruction generated by GCC.
Fixes#33403.
Signed-off-by: Enjia Mai <enjiax.mai@intel.com>
em_starterkit has ICCM at 0x0 address, access to 0x0
address doesn't generate any exception, so we access
to 0xFFFFFFFF address instead to trigger exception.
Signed-off-by: Watson Zeng <zhiwei@synopsys.com>
When running the test on Cortex-M platforms, use access to
_current to trigger a memory access fault, as this address
is guaranteed to be in kernel ram and will not trigger
SecureFault exceptions.
Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
This functions is being called across the tree, no reason why it should
not be a public API.
The current usage violates a few MISRA rules.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The quirky behavior when accessing of 0x0 address is related to
exact ARC SoCs/boards and not to entire ARC EM CPUs family. So
check for exact SoCs configs (CONFIG_SOC_ARC_IOT and CONFIG_SOC_NSIM)
instead of checking against CONFIG_CPU_ARCEM.
Signed-off-by: Eugeniy Paltsev <Eugeniy.Paltsev@synopsys.com>
Minor fixes to the sample output in the README file,
reflecting the correct error reason (3: oops) instead
of reason 0 (generic CPU error).
Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
test_catch_z_oops test case should not run in user mode,
since Z_OOPS needs to be called in supervisor mode, otherwise
we will be getting a reguser user memory access error that is
irrelevant to the test case (and simply is triggered by Z_OOPS
accessing the thread's syscall frame pointer).
In addition to that, we fix the argument of Z_OOPS, because, it
was triggering a null-pointer dereferencing, which results in an
error thrown before Z_OOPS is even executed (if null-pointer
exception is caught).
We also need to generate a dummy thread->syscall_frame, that
Z_OOPS implementation will access. We fix this to
_image_ram_start, because this memory is always part of the
image memory and is always available to supervisor mode. So,
accessing it won't trigger a security fault, for example.
Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
The test case suggests that "For most arch which support
userspace, derefencing NULL pointer will be caught by
exception.". This is certainly not true for Cortex-M, where
read access to address 0x0 is generally allowed. The reason
the test had been passing was either 1) because in many
Cortex-M platforms, including QEMU, address 0x0 is unmapped,
or 2) GCC is generating an undefining instruction. However,
now that we have activated the null pointer derefrencing
detection, we may end up with two exceptions and the test
would fail.
Change illegal access to something outside the mapped memory
area, e.g. 0xFFFFFFFF.
Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
Enlarge the test thread stack size, to fix test case fail
in some board which need more stack size for testing, when
CONFIG_EXCEPTION_DEBUG is enabled.
Signed-off-by: Enjia Mai <enjiax.mai@intel.com>
Fix issue #31339 and #31419, test case fail due to access NULL did not
trigger a fatal error in some platform such as nsim_em and iotdk.
Signed-off-by: Enjia Mai <enjiax.mai@intel.com>
This commit is a followup to PR #25808 which updates the tests to
ensure the REQUIRED keyword is also used for the ZephyrUnitest package.
This provides a better error message when building with CMake and
forgetting ZEPHYR_BASE or not registering Zephyr in the CMake package
registry.
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
... because it is (required).
This makes a difference when building with CMake and forgetting
ZEPHYR_BASE or not registering Zephyr in the CMake package registry.
In this particular case, REQUIRED turns this harmless looking log
statement:
-- Could NOT find Zephyr (missing: Zephyr_DIR)
-- The C compiler identification is GNU 9.3.0
-- The CXX compiler identification is GNU 9.3.0
-- Check for working C compiler: /usr/bin/cc
-- ...
-- ...
-- ...
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMake Error at CMakeLists.txt:8 (target_sources):
Cannot specify sources for target "app" which is not built by
this project.
... into this louder, clearer, faster and (last but not least) final
error:
CMake Error at CMakeLists.txt:5 (find_package):
Could not find a package configuration file provided by "Zephyr" with
any of the following names:
ZephyrConfig.cmake
zephyr-config.cmake
Add the installation prefix of "Zephyr" to CMAKE_PREFIX_PATH or set
"Zephyr_DIR" to a directory containing one of the above files. If
"Zephyr" provides a separate development package or SDK, be sure it
has been installed.
-- Configuring incomplete, errors occurred!
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
The following commit:
commit 407b49b35c (refs/bisect/bad)
Author: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
Date: Wed Feb 12 15:00:46 2020 +0100
cmake: use find_package to locate Zephyr
breaks as we don't find the ZephyrUnittest package. For now revert to
the old means until a proper fix can be made.
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
Using find_package to locate Zephyr.
Old behavior was to use $ENV{ZEPHYR_BASE} for inclusion of boiler plate
code.
Whenever an automatic run of CMake happend by the build system / IDE
then it was required that ZEPHYR_BASE was defined.
Using ZEPHYR_BASE only to locate the Zephyr package allows CMake to
cache the base variable and thus allowing subsequent invocation even
if ZEPHYR_BASE is not set in the environment.
It also removes the risk of strange build results if a user switchs
between different Zephyr based project folders and forgetting to reset
ZEPHYR_BASE before running ninja / make.
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
A Kconfig boolean is added to allow users to provide their own
output strings when running tests via ztest.
This allows changing e.g. the PASS/FAIL/SKIPPED strings,
add counters, change separators, and similar.
A test using the feature and relevant documentation is added.
Signed-off-by: Torstein Grindvik <torstein.grindvik@nordicsemi.no>
This commit changes zassert_mem_equal as a macro instead
of a function implementation.
In the previous implementation when an assertion fails
the location inside ztest_assert.h file was displayed.
This modification displays the location where zassert_mem_equal
was used.
Signed-off-by: Radoslaw Koppel <radoslaw.koppel@nordicsemi.no>
Update the files which contain no license information with the
'Apache-2.0' SPDX license identifier. Many source files in the tree are
missing licensing information, which makes it harder for compliance
tools to determine the correct license.
By default all files without license information are under the default
license of Zephyr, which is Apache version 2.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Move test related code and the testsuite away from tests/ and make it a
proper subsystem.
The way tests were integrate in the tree was not obvious and actual
tests were intermixed with the testsuite code.
This will allow us to have trees with the testcode and without the
samples by just remove the folders tests/ and samples, needed for
isolating actual code from test/sample code.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This was never a long-term solution, more of a gross hack
to get test cases working until we could figure out a good
end-to-end solution for memory domains that generated
appropriate linker sections. Now that we have this with
the app shared memory feature, and have converted all tests
to remove it, delete this feature.
To date all userspace APIs have been tagged as 'experimental'
which sidesteps deprecation policies.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
* Newlib now defines a special z_newlib_partition containing
all globals relevant to newlib. Most of these are in libc.a
with a heap tracking variable in newlib's hooks.
* Both C libraries now expose a k_mem_partition containing the
bounds of the malloc heap arena. Threads that want to use
libc malloc() will need to add this to their memory domain.
* z_newlib_get_heap_bounds has been removed, in favor of the
memory partition for the heap arena
* ztest now includes the C library partitions in its memory
domain.
* The mem_alloc test now runs in user mode to prove that this
all works for both C libraries.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
If CONFIG_APP_SHARED_MEM is enabled, ztest will set up a
default memory domain with a single partition on startup.
Test case globals may be added to the partition via
ZTEST_BMEM/ZTEST_DMEM macros, add their own partitions,
or leave the domain and join another one.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
The public APIs for application shared memory are now
properly documented and conform to zephyr naming
conventions.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
