Removes the ISR restriction from k_thread_priority_set().
Since the first commit, the routine k_thread_priority_set() has
had a restriction preventing it from being called from within
the context of an ISR. As there does not (any longer) appear to
be a reason why the restriction exists, it is being removed.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
Add functions k_thread_foreach_unlocked_filter_by_cpu() and
k_thread_foreach_filter_by_cpu() to loop through the threads on the
specified cpu only.
Signed-off-by: Jyri Sarha <jyri.sarha@linux.intel.com>
Reduce `k_spin_unlock` calls in `k_obj_core_stats_xxx` functions by
consolidating error handling into an `if-else if-else` structure and
using a single return variable `rv`.
Signed-off-by: Pisit Sawangvonganan <pisit@ndrsolution.com>
Removes the ALWAYS_INLINE attribute from the definition of the
routine z_unpend_thread_no_timeout() to fix an unresolved symbol
error that was occurring with some compilers.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
PR #63973 namespaced generated headers with zephyr/, including generated
syscall headers.
Since then, some new generated syscall header includes have been added
without the zephyr/ prefix, breaking builds when
CONFIG_LEGACY_GENERATED_INCLUDE_PATH is disabled.
This commit adds the zephyr/ prefix to includes for generated syscall
headers where it has been missed.
Signed-off-by: Ben Marsh <ben.marsh@helvar.com>
Utilize a code spell-checking tool to scan for and correct spelling errors
in all files within the `kernel` directory.
Signed-off-by: Pisit Sawangvonganan <pisit@ndrsolution.com>
native_posix (unlike native_sim) or its breathen (NATIVE_APPLICATION)
link together the "runner" code and the embedded code.
This means that when CONFIG_STATIC_INIT_GNU is set,
any host library code (like the llvm fuzzer) constructors will get
postponed to the Zephyr initialization.
These libraries are unlikely to work if we do this.
(the llvm fuzzer does not)
So let's instead not enable STATIC_INIT_GNU for these targets.
This means possible embedded library constructors will continue to be
picked during the link and be still called during the native runner
initialization instead of during the Zephyr OS initialization as they
were just before we introduced STATIC_INIT_GNU in
6e977ae2d5
Note that native_posix will be deprecated shortly and its users
are strongly encouraged to move to native_sim.
Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
In device init phase, it will call _mbedtls_init before malloc_prepare
as mbedtls has higher priority defined in SYS_INIT..
_mbedtls_init() will call psa_crypto_init() and malloc buffer,
but z_malloc_heap is not initialized, which will cause device hang.
Should call malloc_prepare() before _mbedtls_init to fix this issue,
so add new Kconfig to increase the priority of libc to deafult 30.
Signed-off-by: Maochen Wang <maochen.wang@nxp.com>
* Move ctors and init_array from the CPP library
to the kernel library, as this is common for both C
and C++ and it is the kernel who is running it.
* Rename the hidden kconfig option CPP_STATIC_INIT_GNU
STATIC_INIT_GNU instead.
* If STATIC_INIT_GNU is not selected verify there is
constructors left behind.
* Rename common-rom-cpp.ld to common-rom-init.ld
* Rename z_cpp_init_static to z_init_static,
and have the kernel always call it.
Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
Signed-off-by: Keith Packard <keithp@keithp.com>
When this new Kconfig option is enabled, preempted threads on an SMP
system may generate additional IPIs when they are switched out.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
Created tracing APIs to trace the enter and (exit + result) of
SYS_INIT and DEVICE_DT_DEFINE (and friends).
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
The deadline of deadline scheduler should lager than zero
because if deadline is negative, it menas the task should
be finished in past.
Signed-off-by: TaiJu Wu <tjwu1217@gmail.com>
Let eviction algorithms be notified when a given page frame:
- should be considered as possible candidate
- should no longer be considered as candidate
- has just been marked as "accessed"
The NRU algorithm is unchanged so it implements those as empty stubs.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Extern declaration of __stack_chk_guard added volatile to
the type while the declaration was non-volatile. This cause
type check errors with compilers that declares the
__stack_chk_guard variable in an internal pre-include
header file (IAR).
While I think the volatile keyword is unnecessary, I decided
on keep it and add it to the declaration in
kernel/compiler_stack_protect.c
Tested with IAR ICCARM and the Zephyr SDK GCC.
Signed-off-by: Lars-Ove Karlsson <lars-ove.karlsson@iar.com>
On each reboot, this option causes the serial output to start top-left
on the users terminal, simplifying (human) parsing.
Signed-off-by: Reto Schneider <reto.schneider@husqvarnagroup.com>
If a page is paged out or paged in but unaccessible for the purpose of
tracking the "accessed" flag then k_mem_unmap() may fails. Add the code
needed to support those cases.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
This is part of a series of moving memory management related
stuff out of the Z_ namespace and into its own namespace.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This is part of a series of move memory management related
stuff out of Z_ namespace into its own namespace.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Also any demand paging and page frame related bits are
renamed.
This is part of a series to move memory management related
stuff out of the Z_ namespace into its own namespace.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This is part of a series to move memory management related
stuff from Z_ namespace into its own namespace.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This is part of a series to move memory management related
stuff from Z_ namespace into its own namespace.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This is part of a series to move memory management related
stuff from Z_ namespace into its own namespace.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Renames:
Z_KERNEL_VIRT_START to K_MEM_KERNEL_VIRT_START
Z_KERNEL_VIRT_SIZE to K_MEM_KERNEL_VIRT_SIZE
Z_KERNEL_VIRT_END to K_MEM_KERNEL_VIRT_END
This is part of a series to move memory management related
stuff from Z_ namespace into its own namespace.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Renames:
Z_VIRT_RAM_START to K_MEM_VIRT_RAM_START
Z_VIRT_RAM_SIZE to K_MEM_VIRT_RAM_SIZE
Z_VIRT_RAM_END to K_MEM_VIRT_RAM_END
This is part of a series to move memory management related
stuff from Z_ namespace into its own namespace.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Renames:
Z_PHYS_RAM_START to K_MEM_PHYS_RAM_START
Z_PHYS_RAM_SIZE to K_MEM_PHYS_RAM_SIZE
Z_PHYS_RAM_END to K_MEM_PHYS_RAM_END
This is part of a series to move memory management related
stuff from Z_ namespace into its own namespace.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This is part of a series to move memory management related
stuff from the Z_ namespace into its own namespace.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Rename Z_BOOT_VIRT_TO_PHYS() and Z_BOOT_PHYS_TO_VIRT() to
K_MEM_BOOT_VIRT_TO_PHYS() and K_MEM_BOOT_PHYS_TO_VIRT()
respectively. This is part of a series to move memory management
functions away from the Z_ namespace and into its own namespace.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This is part of a series to move memory management functions
away from the z_ namespace and into its own namespace. Also
make documentation available via doxygen.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This renames z_phys_map() and z_phys_unmap() to
k_mem_map_phys_bare() and k_mem_unmap_phys_bare()
respectively. This is part of the series to move memory
management functions away from the z_ namespace.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
These functions were introduced alongside with the memory mapped
stack feature, and are currently only being used there only.
To avoid potential confusion with k_mem_un/map(), remove them
and use k_mem_map/unmap_phys_guard() directly instead.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
The internal functions k_mem_map_impl() and k_mem_unmap_impl()
are renamed to k_mem_map_phys_guard() and
k_mem_unmap_phys_guard() respectively to better clarify
their usage.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
As their stacks are defined by zephyr's kernel/thread stack definition
macro, better use zephyr's kernel/thread stack size macro for their stack
size, ensuring consistency and preventing potenial issues related to stack
size misconfiguration.
Signed-off-by: Dong Wang <dong.d.wang@intel.com>
Move this to a call in the init process. arch_* calls are no services
and should be called consistently during initialization.
Place it between PRE_KERNEL_1 and PRE_KERNEL_2 as some drivers
initialized in PRE_KERNEL_2 might depend on SMP being setup.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This option allows you to look up a struct device from any of the
node labels that were attached to the devicetree node used to create
the device, etc.
This is helpful because node labels are a much more human-friendly set
of unique identifiers than the node names we are currently relying on
for use with device_get_binding(). Adding this infrastructure in the
device core allows anyone to make use of it without having to
replicate node label storage and search functions in various places in
the tree. The main use case, however, is for looking up devices by
node label in the shell.
Since there is a footprint penalty associated with storing the node
label metadata, leave this option disabled by default.
Signed-off-by: Martí Bolívar <mbolivar@amperecomputing.com>
Add a __ASSERT_ON guard around slab_ptr_is_good, as that is only used in
assertions and leaving it on seems to generate a build warning with some
clang versions:
kernel/mem_slab.c:207:20: error: unused function 'slab_ptr_is_good'
207 | static inline bool slab_ptr_is_good(struct k_mem_slab *slab,...
| ^~~~~~~~~~~~~~~~
Signed-off-by: Fabio Baltieri <fabiobaltieri@google.com>
When the CONFIG_BOOT_BANNER flag is set to "n", but CONFIG_BOOT_DELAY
is enabled, there is a delay message printed at boot time.
This allows for the whole boot banner to be disabled.
Signed-off-by: Krzysztof Sychla <ksychla@antmicro.com>
Abstract slab pointer validation and apply it to block dequeue during
allocation in addition to the existing block freeing. This should help
catching some buffer overflow induced corruptions.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
As it is, blocks are allocated going backward within the buffer.
There is nothing fundamentally wrong with that, but it makes debugging
unnatural with the successively descending addresses. Create the free
list so pointers are oriented forward, at least initially.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Platforms that support IPIs allow them to be broadcast via the
new arch_sched_broadcast_ipi() routine (replacing arch_sched_ipi()).
Those that also allow IPIs to be directed to specific CPUs may
use arch_sched_directed_ipi() to do so.
As the kernel has the capability to track which CPUs may need an IPI
(see CONFIG_IPI_OPTIMIZE), this commit updates the signalling of
tracked IPIs to use the directed version if supported; otherwise
they continue to use the broadcast version.
Platforms that allow directed IPIs may see a significant reduction
in the number of IPI related ISRs when CONFIG_IPI_OPTIMIZE is
enabled and the number of CPUs increases. These platforms can be
identified by the Kconfig option CONFIG_ARCH_HAS_DIRECTED_IPIS.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
The CONFIG_IPI_OPTIMIZE configuration option allows for the flagging
and subsequent signaling of IPIs to be optimized.
It does this by making each bit in the kernel's pending_ipi field
a flag that indicates whether the corresponding CPU might need an IPI
to trigger the scheduling of a new thread on that CPU.
When a new thread is made ready, we compare that thread against each
of the threads currently executing on the other CPUs. If there is a
chance that that thread should preempt the thread on the other CPU
then we flag that an IPI is needed for that CPU. That is, a clear bit
indicates that the CPU absolutely will not need to reschedule, while a
set bit indicates that the target CPU must make that determination for
itself.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
1. The flagging of IPIs is moved out of k_thread_priority_set() into
z_thread_prio_set(). This allows for an IPI to be done for a thread
that had its priority bumped due to the handling of priority
inheritance from a mutex.
2. k_thread_priority_set()'s check for sched_locked only applies to
non-SMP builds that are using the old arch_swap() framework to switch
between threads.
Incidentally, nearly all calls to flag_ipi() are now performed with
sched_spinlock being locked. The only exception is in slice_timeout().
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
Updates the CONFIG_PIPES Kconfig description to add a note that
enabling it will cause a slight increase to the thread structure.
This mirrors a similar comment in CONFIG_EVENTS.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
Created `GEN_OFFSET_STRUCT` & `GEN_NAMED_OFFSET_STRUCT` that
works for `struct`, and remove the use of `z_arch_esf_t`
completely.
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
Make `struct arch_esf` compulsory for all architectures by
declaring it in the `arch_interface.h` header.
After this commit, the named struct `z_arch_esf_t` is only used
internally to generate offsets, and is slated to be removed
from the `arch_interface.h` header in the future.
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
This duplicates the functionality of device_is_ready.
Calls for z_device_is_ready are being done in kernel mode, so it is
safe to call its implementation directly.
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
Namespaced the generated headers with `zephyr` to prevent
potential conflict with other headers.
Introduce a temporary Kconfig `LEGACY_GENERATED_INCLUDE_PATH`
that is enabled by default. This allows the developers to
continue the use of the old include paths for the time being
until it is deprecated and eventually removed. The Kconfig will
generate a build-time warning message, similar to the
`CONFIG_TIMER_RANDOM_GENERATOR`.
Updated the includes path of in-tree sources accordingly.
Most of the changes here are scripted, check the PR for more
info.
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
There is no need to this function be defined inside the kernel since
all places using it are protecting the call under ifdef PM guards.
This way we can also remove the ifdef condition inside the implementation.
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
Updates z_smp_global_lock() to follow the pattern used in spinlocks
to relax the loop between atomic_cas() attempts.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
k_thread_stack_free syscall was not checking if the caller
had permission to given stack object.
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
This adds a new kconfig to indicate if architecture code
supports isolating thread stacks within the same domain,
and another new kconfig to selectively enable this
behavior.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This value isn't used outside of the PM subsystem, so don't build it.
More important than the four bytes of .bss was the use of an
atomic_inc(). Some platforms are forced to use
CONFIG_ATOMIC_OPERATIONS_C (but in almost all cases are single-core
devices that won't use atomics at runtime). There, this turns into a
function call that pulls in the whole atomics implementation.
Signed-off-by: Andy Ross <andyross@google.com>
The sys_bitfield_(clear/set)_bit() work on pointer size element.
However, _thread_idx_map[] is a byte array. On little endian
systems, the bitops should work fine. However, on big endian
systems, changing the lower bits may actually be manipulating
memory outside the array when CONFIG_MAX_THREAD_BYTES is not
multiple of 4. So modify the code to perform bit ops on
a per-byte basis.
Fixes#72430
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
The struct z_page_frame is marked __packed to avoid extra padding as
such padding may represent significant memory waste when lots of page
frames are used. However this is a bad strategy.
The code contained this somewhat dubious comment and code in
free_page_frame_list_put():
/* The structure is packed, which ensures that this is true */
void *node = pf;
sys_slist_append(&free_page_frame_list, node);
This is bad for many reasons:
- type checking is completely bypassed;
- if the sys_snode_t node member is no longer located at the front of
struct z_page_frame then the code will still compile and possibly run
but be broken with memory corruption as a likely outcome;
- the sys_slist_append() code is completely unaware of the packed
attribute which breaks architectures with alignment restrictions.
Let's improve code efficiency as well as memory usage by removing the
packed attribute and manually packing the flags in the unused virtual
address bits. This way the page frame array remains naturally aligned,
data access becomes optimal and the actual array size gets even smaller.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Introduce z_page_frame_set() and z_page_frame_clear() to manipulate
flags. Obtain the virtual address using the existing
z_page_frame_to_virt(). This will make changes to the page frame
structure easier.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Dynamic code execution applications not using LLEXT for "extension"
loading are subject to the same linker optimization symbol resolution
issue described in commit 321e395 (in summary, libkernel.a syscalls
not used directly by the application result in weak symbol resolution
of their z_mrsh_ wrapper).
To support usecases where an application is using alternative methods
to load and execute code calling syscalls (likely from userspace) or
is using a mechanism where the linker may not be aware, the configuration
option has been decoupled from CONFIG_LLEXT (who is now a selector) to
KERNEL_WHOLE_ARCHIVE.
Signed-off-by: Daniel Apperloo <daniel.apperloo@intel.com>
- modified parameter types to receive a const pointer when a
non-const pointer is not needed
- avoided redundant casts
Signed-off-by: Hess Nathan <nhess@baumer.com>
limit is unsigned int and K_SEM_MAX_LIMIT is defined as UINT_MAX this
means that limit will never be greater K_SEM_MAX_LIMIT.
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
Spell checking tools do not recognize "iff", replace with "if and only if".
See https://en.wikipedia.org/wiki/If_and_only_if
Signed-off-by: Pieter De Gendt <pieter.degendt@basalte.be>
update kernel timeout logic based on retrieve system timer clock
frequency at runtime or static way based on Kconfig
TIMER_READS_ITS_FREQUENCY_AT_RUNTIME
Signed-off-by: Najumon B.A <najumon.ba@intel.com>
The previous abort-lifecycle fix missed a case: other threads can
enter k_thread_join(), see that the thread is already dead, and then
need to call z_thread_switch_spin() to wait for a context switch. But
the new "dummification" code was (by design!) terminating the thread
such that no context would be saved to it. So switch_handle stayed
NULL and if you hit that timing case correctly[1] you'd deadlock
waiting for a switch that would never come.
Fix is just to set switch_handle when dummifying to any non-NULL
value.
Also add an assertion to catch the obvious case that a thread is
actually dead on the exit path of k_thread_abort() to make sure the
variant path continues to set flags correctly
[1] CI was doing it fairly reliably via tests/kernel/smp_abort on
qemu_cortex_a53 only. Only one of my dev systems could see it,
and then only about 15% of the time.
Signed-off-by: Andy Ross <andyross@google.com>
We've had threads spinning on the thread state bits, but weren't being
careful to ensure that those bits were the last things seen to change
in a halting thread. Move it to the end, and add a barrier for
correctness.
Signed-off-by: Andy Ross <andyross@google.com>
Nicolas Pitre points out that since these thread structs are just
dummies for the context swtiching, they can be presumed to be "write
only" and thus there's no point in having one per CPU, everyone can
share the same one.
The only gotcha is that we never really documented (nor really have a
place to document) that rule, so it's not theoretically impossible for
an architecture to read back what it might have written underneath
arch_switch(). Leave this in a separate commit for bisection
purposes, but the risk seems very low.
Signed-off-by: Andy Ross <andyross@google.com>
After a k_thread_abort(), the resulting thread struct is documented as
unused/free memory that may be re-used (for example, to respawn a new
thread).
But in the special case of aborting the current thread from within an
ISR, that wasn't quite happening. The scheduler cleanup would
complete, but the architecture layer would still try to context switch
away from the aborted thread on exit, and that can include writes to
the now-reused thread struct! The specifics will depend on
architecture (some do a full context save on entry, most don't), but
in the case of USE_SWITCH=y it will at the very least write the
switch_handle field.
Fix this simply, with a per-cpu "switch dummy" thread struct for use
as a target for context switches like this. There is some non-trivial
memory cost to that; thread structs on many architectures are large.
Pleasingly, this also addresses a known deadlock on SMP: because the
"spin in ISR" step now happens as the very last stage of
k_thread_abort() handling, the existing scheduler lock works to
serialize calls such that it's impossible for a cycle of threads to
independently decide to spin on each other: at least one will see
itself as "already aborting" and break the cycle.
Fixes#64646
Signed-off-by: Andy Ross <andyross@google.com>
Spinlocks taken in ISRs were storing the _current thread pointer of
the interrupted thread as the owner, which was never strictly correct
but was benign as the thread would never run until the lock was
released.
But now k_thread_abort(_current) in an ISR has been fixed to eliminate
all references to the (now aborted) thread struct, and _current points
to a dummy thread. Handle that edge case in the validation framework.
Signed-off-by: Andy Ross <andyross@google.com>
This will reset time slice in k_thread_time_slice_set()
when slice per thread api is used.
Currently it will reset it only in standard slice_set
Signed-off-by: Adrian Bonislawski <adrian.bonislawski@intel.com>
Updates z_get_next_switch_handle() to set the new thread's base.cpu
value as it is done in do_swap(). This helps to ensure that the
last CPU on which the thread executed remains current.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
There is a duplicate write in `z_sched_thread_usage()` that can be
removed. Also modified the docstrings to `k_thread_runtime_stats` to
help better describe the differences between execution_cycles and
total_cycles when getting stats for the CPU or a thread
Signed-off-by: Eric Johnson <eric@memfault.com>
This reverts commit 93dc7e7438.
This PR introduced 2 regressions in main CI:
71977 & 71978
Let's revert it by now to get main's CI passing again.
Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
This reverts commit 61c70626a5.
This PR introduced 2 regressions in main CI:
71977 & 71978
Let's revert it by now to get main's CI passing again.
Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
This reverts commit 20611f13ca.
This PR introduced 2 regressions in main CI:
71977 & 71978
Let's revert it by now to get main's CI passing again.
Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
This reverts commit 02b24911f7.
This PR introduced 2 regressions in main CI:
71977 & 71978
Let's revert it by now to get main's CI passing again.
Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
Marking sysworkq as essential, so when it fails, the system will halt
instead of continuously working, and dependent components stay
in a broken state.
Signed-off-by: Mohamed ElShahawi <ExtremeGTX@hotmail.com>
Allow the creator of a work_queue instance to choose whether
the work_queue thread should be marked as ESSENTIAL or not.
Signed-off-by: Mohamed ElShahawi <ExtremeGTX@hotmail.com>
We've had threads spinning on the thread state bits, but weren't being
careful to ensure that those bits were the last things seen to change
in a halting thread. Move it to the end, and add a barrier for
correctness.
Signed-off-by: Andy Ross <andyross@google.com>
Nicolas Pitre points out that since these thread structs are just
dummies for the context swtiching, they can be presumed to be "write
only" and thus there's no point in having one per CPU, everyone can
share the same one.
The only gotcha is that we never really documented (nor really have a
place to document) that rule, so it's not theoretically impossible for
an architecture to read back what it might have written underneath
arch_switch(). Leave this in a separate commit for bisection
purposes, but the risk seems very low.
Signed-off-by: Andy Ross <andyross@google.com>
After a k_thread_abort(), the resulting thread struct is documented as
unused/free memory that may be re-used (for example, to respawn a new
thread).
But in the special case of aborting the current thread from within an
ISR, that wasn't quite happening. The scheduler cleanup would
complete, but the architecture layer would still try to context switch
away from the aborted thread on exit, and that can include writes to
the now-reused thread struct! The specifics will depend on
architecture (some do a full context save on entry, most don't), but
in the case of USE_SWITCH=y it will at the very least write the
switch_handle field.
Fix this simply, with a per-cpu "switch dummy" thread struct for use
as a target for context switches like this. There is some non-trivial
memory cost to that; thread structs on many architectures are large.
Pleasingly, this also addresses a known deadlock on SMP: because the
"spin in ISR" step now happens as the very last stage of
k_thread_abort() handling, the existing scheduler lock works to
serialize calls such that it's impossible for a cycle of threads to
independently decide to spin on each other: at least one will see
itself as "already aborting" and break the cycle.
Fixes#64646
Signed-off-by: Andy Ross <andyross@google.com>
Spinlocks taken in ISRs were storing the _current thread pointer of
the interrupted thread as the owner, which was never strictly correct
but was benign as the thread would never run until the lock was
released.
But now k_thread_abort(_current) in an ISR has been fixed to eliminate
all references to the (now aborted) thread struct, and _current points
to a dummy thread. Handle that edge case in the validation framework.
Signed-off-by: Andy Ross <andyross@google.com>
Big change is to factor out a thread_halt_spin() utility to manage the
core complexity of this code: the situation where an ISR is asked to
abort a thread already running on another SMP CPU.
With that gone, things can be cleaned up quite a bit. Remove early
returns, most of the "#if CONFIG_SMP" usage was superfluous and will
optimize out, unify and clean up the comments, etc...
No behavioral changes (hopefully), just refactoring.
Signed-off-by: Andy Ross <andyross@google.com>
After the move to C files we got some drop in the performance when
running latency_measure. This patch declares the priority queue
functions as static inlines with minor optimizations.
The result for one metric (on qemu):
3.6 and before the anything was changed:
Get data from LIFO (w/ ctx switch): 13087 ns
after original change (46484da502):
Get data from LIFO (w/ ctx switch): 13663 ns
with this change:
Get data from LIFO (w/ ctx switch): 12543 ns
So overall, a net gain of ~ 500ns that can be seen across the board on many
of the metrics.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Check possible overflow in k_stack data struct. An overflow
can happens resulting in a much smaller amount of memory allocation.
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
Currently, all devices are initialized at boot time (following their
level and priority order). This patch introduces deferred
initialization: by setting the property `zephyr,deferred-init` on a
device on the devicetree, Zephyr will not initialized the device.
To initialize such devices, one has to call `device_init()`.
Deferred initialization is done by grouping all deferred devices on a
different ELF section. In this way, there's no need to consume more
memory to keep track of deferred devices. When `device_init()` is
called, Zephyr will scan the deferred devices section and call the
initialization function for the matching device. As this scanning is
done only during deferred device initialization, its cost should be
bearable.
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
This introduces support for memory mapped thread stacks,
where each thread stack is mapped into virtual memory
address space with two guard pages to catch
under-/over-flowing the stack. This is just on the kernel
side. Additional architecture code is required to fully
support this feature.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This is similar to k_mem_map()/_unmap(). But instead of using
anonymous memory, the provided physical region is mapped
into virtual address instead. In addition to simple mapping
physical ro virtual addresses, the mapping also adds two
guard pages before and after the virtual region to catch
buffer under-/over-flow.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This adds the mechanism to do cleanup after k_thread_abort()
is called with the current thread. This is mainly used for
cleaning up things when the thread cannot be running, e.g.,
cleanup the thread stack.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Next timeout was set unconditionally at the end of sys_clock_announce.
However, if one of the current expired timeouts was setting a new
timeout which is the first to execute then system clock was configured
twice. Lets configure system clock only once in the isr at the and of
sys_clock_announce.
If timeouts are frequent this optimization can reduce CPU load. In
many cases setting the new sys_clock timeout is the most time
consuming operation in the sys_clock isr handler. As an example,
on the target I used setting new sys_clock timeout is taking 6 uS of
9 uS spent in the isr and it takes 16 uS with the redundant call.
Signed-off-by: Krzysztof Chruściński <krzysztof.chruscinski@nordicsemi.no>
Do not build threading support when CONFIG_MULTITHREADING=n is set and
move needed calls to a new file with the changes needed instead of the
ifdef party in sched.c
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The `BUILD_VERSION` can be defined but empty when built
without git, causing version to be missing from the banner:
```
*** Booting Zephyr OS build ***
Hello World! qemu_riscv64
```
Let's check if it is empty before using it, so that
`KERNEL_VERSION_STRING`, which is generated independently
with cmake can be used as a fallback:
```
*** Booting Zephyr OS build 3.5.0 ***
Hello World! qemu_riscv64
```
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
When thread stack is defined as an array, K_THREAD_STACK_LEN()
is used to calculate the size for each stack in the array.
However, standalone thread stack has its size calculated by
Z_THREAD_STACK_SIZE_ADJUST() instead. Depending on the arch
alignment requirement, they may not be the same... which
could cause some confusions. So align them both to use
K_THREAD_STACK_LEN().
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
When kernel stack is defined as an array, K_KERNEL_STACK_LEN()
is used to calculate the size for each stack in the array.
However, standalone kernel stack has its size calculated by
Z_KERNEL_STACK_SIZE_ADJUST() instead. Depending on the arch
alignment requirement, they may not be the same... which
could cause some confusions. So align them both to use
K_KERNEL_STACK_LEN().
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Simple rename to align the kernel naming scheme. This is being
used throughout the tree, especially in the architecture code.
As this is not a private API internal to kernel, prefix it
appropriately with K_.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Simple rename to align the kernel naming scheme. This is being
used throughout the tree, especially in the architecture code.
As this is not a private API internal to kernel, prefix it
appropriately with K_.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Differently from other libraries, which are included whole in the final
Zephyr ELF, libkernel.a itself isn't. Assuming this is intended to
enable optimisations (if it isn't, this patch will break things) - linker
can remove parts of the kernel that are not used by the application.
However, when considering Linkable Loadable Extensions (llext), this
optimisations can be counterproductive: for instance, syscalls that are
not used by the application won't be available for extensions. It won't
matter if someone "EXPORT_SYMBOL" for them, or even try to keep them
using LINKER_KEEP, they'll be gone.
To avoid that, this patches includes, when CONFIG_LLEXT=y, libkernel.a
inside the linker "whole-archive" block. This ends up making it consider
libkernel.a as a library whose all symbols should be kept. Note this
doesn't mean that all symbols will be there - things compiled out via
Kconfig will naturally still be out.
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
Add a closing comment to the endif with the configuration
information to which the endif belongs too.
To make the code more clearer if the configs need adaptions.
Signed-off-by: Simon Hein <Shein@baumer.com>
arch_interface.h is for architecture and should not be
under sys/. So move it under include/zephyr/arch/.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
k_mem_map additionally allocates two guard pages that are not mapped.
These pages are not being accounted when checking the provided size and
when they are added an overflow can happen and the mapped memory is not
correct.
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
Linkable loadable extensions can only use syscalls if they are exported
via EXPORT_SYSCALL (or EXPORT_SYMBOL). Instead of enabling used syscalls
one by one, this patch exports all of them automatically via
`gen_syscalls.py`. If CONFIG_LLEXT=n, the section where the exported
symbols live is discarded, so it should be a non-op when llext is not
enabled.
This patch also removes the now redundant EXPORT_SYSCALL macro. Note
that EXPORT_SYMBOL is still useful on different situations (and is
indeed used by the code generated by `gen_syscalls.py`).
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
Modified bitmask to bitmask array, it can make multilevel queue remove
32 bit prioriry limit.
We can scan bitmask array to find which queue have ready thread.
Only need the number of queues as priority because the priority
is checked on create_thread.
Signed-off-by: TaiJu Wu <tjwu1217@gmail.com>
k_thread_deadline_set() would modify the thread's deadline and then,
if it was in the run queue, requeue it to put it at the right spot.
Sounds right, right?
It's wrong. The deadline field is part of the thread priority, so
this results in a mis-ordered list. For dlist backends, that's benign
as the removal works anyway, but if CONFIG_SCHED_SCALABLE=y we've now
broken the sorting order of an in-tree item and corrupted the rbtree!
Fixes#69935
Signed-off-by: Andy Ross <andyross@google.com>
This currently calls timeout_rem() then adds back the result of
elapsed() to cancel out the subtraction of another elapsed() call
within timeout_rem(). Better not to make any calls to elapsed() in
that case, especially given that elapsed() may incur hardware access
overhead through sys_clock_elapsed().
Let's move the elapsed() subtraction to the only user of timeout_rem()
that needs it and remove its invocation from the z_timeout_expires()
path entirely.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
The code `SYS_SLIST_FOR_EACH_CONTAINER_SAFE` just for remove work
from `pending_cancels`.
After removing work successfully, the function can return early.
It is unnecessary to iterate continuously.
Signed-off-by: TaiJu Wu <tjwu1217@gmail.com>
Rename private function to make it clear what priority we are setting
and to be consistent across the code.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Avoid single characker variables that renders code unreadable and might
cause conflicts in maing, similar to t for both timeout and thread in
some places.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Use thread wherever it makes sense, using 't' in some places can get
confused with 't' used for timeouts for example.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Move thread monitor related functions, not enabled in most cases outside
of thread.c and cleanup headers.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This function is only being used by a test, so instead of reimplementing
a syscall in the test, provide a Kconfig option to provide the
functionality that only works with tests and remove some of the
duplication and extra code.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Move out of thread and put directly in init.c where it is being used.
Also remove definition from kernel.h, this is an internal function and
should not be in a public header.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
We shouldn't be calling hooks from optional and upper layer subsystems
in the kernel, instead, just call the hook to set thread status in the
API where it is needed.
This now clears related bit in cmsis thread status bitarray when
terminating a thread in the cmsis rtos v1 layer directly and not in the
kenrel code.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
In an effort to cleanup sched.c, move sections of code that can be
compiled in based on options into own files. CPU mask here is managed by
a kconfig and is not widely used (SMP affinity on multicore systems).
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The kernel kconfig is becoming too big and unmanageable with too many
options scattered across the file. Move some areas out and reorg main
Kconfig slightly.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
