1. The flagging of IPIs is moved out of k_thread_priority_set() into
z_thread_prio_set(). This allows for an IPI to be done for a thread
that had its priority bumped due to the handling of priority
inheritance from a mutex.
2. k_thread_priority_set()'s check for sched_locked only applies to
non-SMP builds that are using the old arch_swap() framework to switch
between threads.
Incidentally, nearly all calls to flag_ipi() are now performed with
sched_spinlock being locked. The only exception is in slice_timeout().
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
Updates the CONFIG_PIPES Kconfig description to add a note that
enabling it will cause a slight increase to the thread structure.
This mirrors a similar comment in CONFIG_EVENTS.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
Created `GEN_OFFSET_STRUCT` & `GEN_NAMED_OFFSET_STRUCT` that
works for `struct`, and remove the use of `z_arch_esf_t`
completely.
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
Make `struct arch_esf` compulsory for all architectures by
declaring it in the `arch_interface.h` header.
After this commit, the named struct `z_arch_esf_t` is only used
internally to generate offsets, and is slated to be removed
from the `arch_interface.h` header in the future.
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
This duplicates the functionality of device_is_ready.
Calls for z_device_is_ready are being done in kernel mode, so it is
safe to call its implementation directly.
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
Namespaced the generated headers with `zephyr` to prevent
potential conflict with other headers.
Introduce a temporary Kconfig `LEGACY_GENERATED_INCLUDE_PATH`
that is enabled by default. This allows the developers to
continue the use of the old include paths for the time being
until it is deprecated and eventually removed. The Kconfig will
generate a build-time warning message, similar to the
`CONFIG_TIMER_RANDOM_GENERATOR`.
Updated the includes path of in-tree sources accordingly.
Most of the changes here are scripted, check the PR for more
info.
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
There is no need to this function be defined inside the kernel since
all places using it are protecting the call under ifdef PM guards.
This way we can also remove the ifdef condition inside the implementation.
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
Updates z_smp_global_lock() to follow the pattern used in spinlocks
to relax the loop between atomic_cas() attempts.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
k_thread_stack_free syscall was not checking if the caller
had permission to given stack object.
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
This adds a new kconfig to indicate if architecture code
supports isolating thread stacks within the same domain,
and another new kconfig to selectively enable this
behavior.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This value isn't used outside of the PM subsystem, so don't build it.
More important than the four bytes of .bss was the use of an
atomic_inc(). Some platforms are forced to use
CONFIG_ATOMIC_OPERATIONS_C (but in almost all cases are single-core
devices that won't use atomics at runtime). There, this turns into a
function call that pulls in the whole atomics implementation.
Signed-off-by: Andy Ross <andyross@google.com>
The sys_bitfield_(clear/set)_bit() work on pointer size element.
However, _thread_idx_map[] is a byte array. On little endian
systems, the bitops should work fine. However, on big endian
systems, changing the lower bits may actually be manipulating
memory outside the array when CONFIG_MAX_THREAD_BYTES is not
multiple of 4. So modify the code to perform bit ops on
a per-byte basis.
Fixes#72430
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
The struct z_page_frame is marked __packed to avoid extra padding as
such padding may represent significant memory waste when lots of page
frames are used. However this is a bad strategy.
The code contained this somewhat dubious comment and code in
free_page_frame_list_put():
/* The structure is packed, which ensures that this is true */
void *node = pf;
sys_slist_append(&free_page_frame_list, node);
This is bad for many reasons:
- type checking is completely bypassed;
- if the sys_snode_t node member is no longer located at the front of
struct z_page_frame then the code will still compile and possibly run
but be broken with memory corruption as a likely outcome;
- the sys_slist_append() code is completely unaware of the packed
attribute which breaks architectures with alignment restrictions.
Let's improve code efficiency as well as memory usage by removing the
packed attribute and manually packing the flags in the unused virtual
address bits. This way the page frame array remains naturally aligned,
data access becomes optimal and the actual array size gets even smaller.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Introduce z_page_frame_set() and z_page_frame_clear() to manipulate
flags. Obtain the virtual address using the existing
z_page_frame_to_virt(). This will make changes to the page frame
structure easier.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Dynamic code execution applications not using LLEXT for "extension"
loading are subject to the same linker optimization symbol resolution
issue described in commit 321e395 (in summary, libkernel.a syscalls
not used directly by the application result in weak symbol resolution
of their z_mrsh_ wrapper).
To support usecases where an application is using alternative methods
to load and execute code calling syscalls (likely from userspace) or
is using a mechanism where the linker may not be aware, the configuration
option has been decoupled from CONFIG_LLEXT (who is now a selector) to
KERNEL_WHOLE_ARCHIVE.
Signed-off-by: Daniel Apperloo <daniel.apperloo@intel.com>
- modified parameter types to receive a const pointer when a
non-const pointer is not needed
- avoided redundant casts
Signed-off-by: Hess Nathan <nhess@baumer.com>
limit is unsigned int and K_SEM_MAX_LIMIT is defined as UINT_MAX this
means that limit will never be greater K_SEM_MAX_LIMIT.
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
Spell checking tools do not recognize "iff", replace with "if and only if".
See https://en.wikipedia.org/wiki/If_and_only_if
Signed-off-by: Pieter De Gendt <pieter.degendt@basalte.be>
update kernel timeout logic based on retrieve system timer clock
frequency at runtime or static way based on Kconfig
TIMER_READS_ITS_FREQUENCY_AT_RUNTIME
Signed-off-by: Najumon B.A <najumon.ba@intel.com>
The previous abort-lifecycle fix missed a case: other threads can
enter k_thread_join(), see that the thread is already dead, and then
need to call z_thread_switch_spin() to wait for a context switch. But
the new "dummification" code was (by design!) terminating the thread
such that no context would be saved to it. So switch_handle stayed
NULL and if you hit that timing case correctly[1] you'd deadlock
waiting for a switch that would never come.
Fix is just to set switch_handle when dummifying to any non-NULL
value.
Also add an assertion to catch the obvious case that a thread is
actually dead on the exit path of k_thread_abort() to make sure the
variant path continues to set flags correctly
[1] CI was doing it fairly reliably via tests/kernel/smp_abort on
qemu_cortex_a53 only. Only one of my dev systems could see it,
and then only about 15% of the time.
Signed-off-by: Andy Ross <andyross@google.com>
We've had threads spinning on the thread state bits, but weren't being
careful to ensure that those bits were the last things seen to change
in a halting thread. Move it to the end, and add a barrier for
correctness.
Signed-off-by: Andy Ross <andyross@google.com>
Nicolas Pitre points out that since these thread structs are just
dummies for the context swtiching, they can be presumed to be "write
only" and thus there's no point in having one per CPU, everyone can
share the same one.
The only gotcha is that we never really documented (nor really have a
place to document) that rule, so it's not theoretically impossible for
an architecture to read back what it might have written underneath
arch_switch(). Leave this in a separate commit for bisection
purposes, but the risk seems very low.
Signed-off-by: Andy Ross <andyross@google.com>
After a k_thread_abort(), the resulting thread struct is documented as
unused/free memory that may be re-used (for example, to respawn a new
thread).
But in the special case of aborting the current thread from within an
ISR, that wasn't quite happening. The scheduler cleanup would
complete, but the architecture layer would still try to context switch
away from the aborted thread on exit, and that can include writes to
the now-reused thread struct! The specifics will depend on
architecture (some do a full context save on entry, most don't), but
in the case of USE_SWITCH=y it will at the very least write the
switch_handle field.
Fix this simply, with a per-cpu "switch dummy" thread struct for use
as a target for context switches like this. There is some non-trivial
memory cost to that; thread structs on many architectures are large.
Pleasingly, this also addresses a known deadlock on SMP: because the
"spin in ISR" step now happens as the very last stage of
k_thread_abort() handling, the existing scheduler lock works to
serialize calls such that it's impossible for a cycle of threads to
independently decide to spin on each other: at least one will see
itself as "already aborting" and break the cycle.
Fixes#64646
Signed-off-by: Andy Ross <andyross@google.com>
Spinlocks taken in ISRs were storing the _current thread pointer of
the interrupted thread as the owner, which was never strictly correct
but was benign as the thread would never run until the lock was
released.
But now k_thread_abort(_current) in an ISR has been fixed to eliminate
all references to the (now aborted) thread struct, and _current points
to a dummy thread. Handle that edge case in the validation framework.
Signed-off-by: Andy Ross <andyross@google.com>
This will reset time slice in k_thread_time_slice_set()
when slice per thread api is used.
Currently it will reset it only in standard slice_set
Signed-off-by: Adrian Bonislawski <adrian.bonislawski@intel.com>
Updates z_get_next_switch_handle() to set the new thread's base.cpu
value as it is done in do_swap(). This helps to ensure that the
last CPU on which the thread executed remains current.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
There is a duplicate write in `z_sched_thread_usage()` that can be
removed. Also modified the docstrings to `k_thread_runtime_stats` to
help better describe the differences between execution_cycles and
total_cycles when getting stats for the CPU or a thread
Signed-off-by: Eric Johnson <eric@memfault.com>
This reverts commit 93dc7e7438.
This PR introduced 2 regressions in main CI:
71977 & 71978
Let's revert it by now to get main's CI passing again.
Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
This reverts commit 61c70626a5.
This PR introduced 2 regressions in main CI:
71977 & 71978
Let's revert it by now to get main's CI passing again.
Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
This reverts commit 20611f13ca.
This PR introduced 2 regressions in main CI:
71977 & 71978
Let's revert it by now to get main's CI passing again.
Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
This reverts commit 02b24911f7.
This PR introduced 2 regressions in main CI:
71977 & 71978
Let's revert it by now to get main's CI passing again.
Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
Marking sysworkq as essential, so when it fails, the system will halt
instead of continuously working, and dependent components stay
in a broken state.
Signed-off-by: Mohamed ElShahawi <ExtremeGTX@hotmail.com>
Allow the creator of a work_queue instance to choose whether
the work_queue thread should be marked as ESSENTIAL or not.
Signed-off-by: Mohamed ElShahawi <ExtremeGTX@hotmail.com>
We've had threads spinning on the thread state bits, but weren't being
careful to ensure that those bits were the last things seen to change
in a halting thread. Move it to the end, and add a barrier for
correctness.
Signed-off-by: Andy Ross <andyross@google.com>
Nicolas Pitre points out that since these thread structs are just
dummies for the context swtiching, they can be presumed to be "write
only" and thus there's no point in having one per CPU, everyone can
share the same one.
The only gotcha is that we never really documented (nor really have a
place to document) that rule, so it's not theoretically impossible for
an architecture to read back what it might have written underneath
arch_switch(). Leave this in a separate commit for bisection
purposes, but the risk seems very low.
Signed-off-by: Andy Ross <andyross@google.com>
After a k_thread_abort(), the resulting thread struct is documented as
unused/free memory that may be re-used (for example, to respawn a new
thread).
But in the special case of aborting the current thread from within an
ISR, that wasn't quite happening. The scheduler cleanup would
complete, but the architecture layer would still try to context switch
away from the aborted thread on exit, and that can include writes to
the now-reused thread struct! The specifics will depend on
architecture (some do a full context save on entry, most don't), but
in the case of USE_SWITCH=y it will at the very least write the
switch_handle field.
Fix this simply, with a per-cpu "switch dummy" thread struct for use
as a target for context switches like this. There is some non-trivial
memory cost to that; thread structs on many architectures are large.
Pleasingly, this also addresses a known deadlock on SMP: because the
"spin in ISR" step now happens as the very last stage of
k_thread_abort() handling, the existing scheduler lock works to
serialize calls such that it's impossible for a cycle of threads to
independently decide to spin on each other: at least one will see
itself as "already aborting" and break the cycle.
Fixes#64646
Signed-off-by: Andy Ross <andyross@google.com>
Spinlocks taken in ISRs were storing the _current thread pointer of
the interrupted thread as the owner, which was never strictly correct
but was benign as the thread would never run until the lock was
released.
But now k_thread_abort(_current) in an ISR has been fixed to eliminate
all references to the (now aborted) thread struct, and _current points
to a dummy thread. Handle that edge case in the validation framework.
Signed-off-by: Andy Ross <andyross@google.com>
Big change is to factor out a thread_halt_spin() utility to manage the
core complexity of this code: the situation where an ISR is asked to
abort a thread already running on another SMP CPU.
With that gone, things can be cleaned up quite a bit. Remove early
returns, most of the "#if CONFIG_SMP" usage was superfluous and will
optimize out, unify and clean up the comments, etc...
No behavioral changes (hopefully), just refactoring.
Signed-off-by: Andy Ross <andyross@google.com>
After the move to C files we got some drop in the performance when
running latency_measure. This patch declares the priority queue
functions as static inlines with minor optimizations.
The result for one metric (on qemu):
3.6 and before the anything was changed:
Get data from LIFO (w/ ctx switch): 13087 ns
after original change (46484da502):
Get data from LIFO (w/ ctx switch): 13663 ns
with this change:
Get data from LIFO (w/ ctx switch): 12543 ns
So overall, a net gain of ~ 500ns that can be seen across the board on many
of the metrics.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Check possible overflow in k_stack data struct. An overflow
can happens resulting in a much smaller amount of memory allocation.
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
Currently, all devices are initialized at boot time (following their
level and priority order). This patch introduces deferred
initialization: by setting the property `zephyr,deferred-init` on a
device on the devicetree, Zephyr will not initialized the device.
To initialize such devices, one has to call `device_init()`.
Deferred initialization is done by grouping all deferred devices on a
different ELF section. In this way, there's no need to consume more
memory to keep track of deferred devices. When `device_init()` is
called, Zephyr will scan the deferred devices section and call the
initialization function for the matching device. As this scanning is
done only during deferred device initialization, its cost should be
bearable.
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
This introduces support for memory mapped thread stacks,
where each thread stack is mapped into virtual memory
address space with two guard pages to catch
under-/over-flowing the stack. This is just on the kernel
side. Additional architecture code is required to fully
support this feature.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This is similar to k_mem_map()/_unmap(). But instead of using
anonymous memory, the provided physical region is mapped
into virtual address instead. In addition to simple mapping
physical ro virtual addresses, the mapping also adds two
guard pages before and after the virtual region to catch
buffer under-/over-flow.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This adds the mechanism to do cleanup after k_thread_abort()
is called with the current thread. This is mainly used for
cleaning up things when the thread cannot be running, e.g.,
cleanup the thread stack.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Next timeout was set unconditionally at the end of sys_clock_announce.
However, if one of the current expired timeouts was setting a new
timeout which is the first to execute then system clock was configured
twice. Lets configure system clock only once in the isr at the and of
sys_clock_announce.
If timeouts are frequent this optimization can reduce CPU load. In
many cases setting the new sys_clock timeout is the most time
consuming operation in the sys_clock isr handler. As an example,
on the target I used setting new sys_clock timeout is taking 6 uS of
9 uS spent in the isr and it takes 16 uS with the redundant call.
Signed-off-by: Krzysztof Chruściński <krzysztof.chruscinski@nordicsemi.no>
Do not build threading support when CONFIG_MULTITHREADING=n is set and
move needed calls to a new file with the changes needed instead of the
ifdef party in sched.c
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The `BUILD_VERSION` can be defined but empty when built
without git, causing version to be missing from the banner:
```
*** Booting Zephyr OS build ***
Hello World! qemu_riscv64
```
Let's check if it is empty before using it, so that
`KERNEL_VERSION_STRING`, which is generated independently
with cmake can be used as a fallback:
```
*** Booting Zephyr OS build 3.5.0 ***
Hello World! qemu_riscv64
```
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
When thread stack is defined as an array, K_THREAD_STACK_LEN()
is used to calculate the size for each stack in the array.
However, standalone thread stack has its size calculated by
Z_THREAD_STACK_SIZE_ADJUST() instead. Depending on the arch
alignment requirement, they may not be the same... which
could cause some confusions. So align them both to use
K_THREAD_STACK_LEN().
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
When kernel stack is defined as an array, K_KERNEL_STACK_LEN()
is used to calculate the size for each stack in the array.
However, standalone kernel stack has its size calculated by
Z_KERNEL_STACK_SIZE_ADJUST() instead. Depending on the arch
alignment requirement, they may not be the same... which
could cause some confusions. So align them both to use
K_KERNEL_STACK_LEN().
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Simple rename to align the kernel naming scheme. This is being
used throughout the tree, especially in the architecture code.
As this is not a private API internal to kernel, prefix it
appropriately with K_.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Simple rename to align the kernel naming scheme. This is being
used throughout the tree, especially in the architecture code.
As this is not a private API internal to kernel, prefix it
appropriately with K_.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Differently from other libraries, which are included whole in the final
Zephyr ELF, libkernel.a itself isn't. Assuming this is intended to
enable optimisations (if it isn't, this patch will break things) - linker
can remove parts of the kernel that are not used by the application.
However, when considering Linkable Loadable Extensions (llext), this
optimisations can be counterproductive: for instance, syscalls that are
not used by the application won't be available for extensions. It won't
matter if someone "EXPORT_SYMBOL" for them, or even try to keep them
using LINKER_KEEP, they'll be gone.
To avoid that, this patches includes, when CONFIG_LLEXT=y, libkernel.a
inside the linker "whole-archive" block. This ends up making it consider
libkernel.a as a library whose all symbols should be kept. Note this
doesn't mean that all symbols will be there - things compiled out via
Kconfig will naturally still be out.
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
Add a closing comment to the endif with the configuration
information to which the endif belongs too.
To make the code more clearer if the configs need adaptions.
Signed-off-by: Simon Hein <Shein@baumer.com>
arch_interface.h is for architecture and should not be
under sys/. So move it under include/zephyr/arch/.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
k_mem_map additionally allocates two guard pages that are not mapped.
These pages are not being accounted when checking the provided size and
when they are added an overflow can happen and the mapped memory is not
correct.
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
Linkable loadable extensions can only use syscalls if they are exported
via EXPORT_SYSCALL (or EXPORT_SYMBOL). Instead of enabling used syscalls
one by one, this patch exports all of them automatically via
`gen_syscalls.py`. If CONFIG_LLEXT=n, the section where the exported
symbols live is discarded, so it should be a non-op when llext is not
enabled.
This patch also removes the now redundant EXPORT_SYSCALL macro. Note
that EXPORT_SYMBOL is still useful on different situations (and is
indeed used by the code generated by `gen_syscalls.py`).
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
Modified bitmask to bitmask array, it can make multilevel queue remove
32 bit prioriry limit.
We can scan bitmask array to find which queue have ready thread.
Only need the number of queues as priority because the priority
is checked on create_thread.
Signed-off-by: TaiJu Wu <tjwu1217@gmail.com>
k_thread_deadline_set() would modify the thread's deadline and then,
if it was in the run queue, requeue it to put it at the right spot.
Sounds right, right?
It's wrong. The deadline field is part of the thread priority, so
this results in a mis-ordered list. For dlist backends, that's benign
as the removal works anyway, but if CONFIG_SCHED_SCALABLE=y we've now
broken the sorting order of an in-tree item and corrupted the rbtree!
Fixes#69935
Signed-off-by: Andy Ross <andyross@google.com>
This currently calls timeout_rem() then adds back the result of
elapsed() to cancel out the subtraction of another elapsed() call
within timeout_rem(). Better not to make any calls to elapsed() in
that case, especially given that elapsed() may incur hardware access
overhead through sys_clock_elapsed().
Let's move the elapsed() subtraction to the only user of timeout_rem()
that needs it and remove its invocation from the z_timeout_expires()
path entirely.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
The code `SYS_SLIST_FOR_EACH_CONTAINER_SAFE` just for remove work
from `pending_cancels`.
After removing work successfully, the function can return early.
It is unnecessary to iterate continuously.
Signed-off-by: TaiJu Wu <tjwu1217@gmail.com>
Rename private function to make it clear what priority we are setting
and to be consistent across the code.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Avoid single characker variables that renders code unreadable and might
cause conflicts in maing, similar to t for both timeout and thread in
some places.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Use thread wherever it makes sense, using 't' in some places can get
confused with 't' used for timeouts for example.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Move thread monitor related functions, not enabled in most cases outside
of thread.c and cleanup headers.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This function is only being used by a test, so instead of reimplementing
a syscall in the test, provide a Kconfig option to provide the
functionality that only works with tests and remove some of the
duplication and extra code.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Move out of thread and put directly in init.c where it is being used.
Also remove definition from kernel.h, this is an internal function and
should not be in a public header.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
We shouldn't be calling hooks from optional and upper layer subsystems
in the kernel, instead, just call the hook to set thread status in the
API where it is needed.
This now clears related bit in cmsis thread status bitarray when
terminating a thread in the cmsis rtos v1 layer directly and not in the
kenrel code.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
In an effort to cleanup sched.c, move sections of code that can be
compiled in based on options into own files. CPU mask here is managed by
a kconfig and is not widely used (SMP affinity on multicore systems).
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The kernel kconfig is becoming too big and unmanageable with too many
options scattered across the file. Move some areas out and reorg main
Kconfig slightly.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The functions to manipulate the essential flag indeed operate on
threads, but they are misplaced in the thread implementation file. Put
them alongside other routines setting other thread flags and cleanup
headers a bit.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Move spin_lock validation outside of thread.c into own file. This code
really has nothing to do with the thread code.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
It wasn't saving adjusted stack size at either the private stack or the
k_object, thus failing subsequent checks.
Test added to check for this case and prevent regressions.
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
clean up headers under include/ and move handling of priority queue to
own file/header.
No need for the header include/zephyr/kernel/internal/sched_priq.h
anymore. Move the relevant structures where they are being used in
kernel_structs.h.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Instead of rounding up both __tdata_size and __tbss_size at runtime,
perform the calculation when the image is built.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
This commit does two things to k_wakeup():
1. It locks the scheduler before marking the thread as not suspended.
As the the clearing of the _THREAD_SUSPENDED bit is not atomic, this
helps ensure that neither another thread nor ISR interrupts this
action (resulting in a corrupted thread_state).
2. The call to flag_ipi() has been removed as it is already being
made within ready_thread().
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
cpu_start_fn is global, it's used by the initiator CPU to start or
resume secondary CPUs. However it's possible, that the initiator CPU
goes ahead and starts a second secondary CPU before the first one has
finished using the object. Fix this by creating a local copy of the
global object.
Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
When allowing dynamic thread stack allocation the stack may come from
the heap in coherent memory, trying to use cached memory is over
complicated because of heap meta data and cache line sizes.
Also when userspace is enabled, stacks have to be page aligned and the
address of the stack is used to track kernel objects.
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
It is observed that starting up CPU may result in other CPUs
crashing due to de-referencing NULL pointers. Note that this
happened on the up_squared board, but there was no way to
attach debugger to verify. It started working again after
moving z_dummy_thread_init() before smp_timer_init(), which
was the old behavior before commit
eefaeee061 where the issue
first appeared. So mimic the old behavior to workaround
the issue.
Fixes#68115
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Traditionally, k_thread_create() has required that the application
size the stack correctly. Zephyr doesn't detect or return errors and
treats stack overflow as an application bug (though obviously some
architectures have runtime features to trap on overflows).
At this one spot though, it's possible for the kernel to adjust the
stack for K_THREAD_STACK_RESERVED in such a way that the arch layer's
own stack initialization overflows. That failure can be seen by
static analysis, so we can't just sweep it under the rug as an
application failure.
Unfortunately there aren't any good options for handling it here (no
way to return failure, can't be a build assert as the size is a
runtime argument). A panic will have to do.
Fixes: #67106Fixes: #65584
Signed-off-by: Andy Ross <andyross@google.com>
In many cases suspending or resuming of a device is limited to
just a few register writes. Current solution assumes that those
operations may be blocking, asynchronous and take a lot of time.
Due to this assumption runtime PM API cannot be effectively used
from the interrupt context. Zephyr has few driver APIs which
can be used from an interrupt context and now use of runtime PM
is limited in those cases.
Patch introduces a new type of PM device - synchronous PM. If
device is specified as capable of synchronous PM operations then
device runtime getting and putting is executed in the critical
section. In that case, runtime API can be used from an interrupt
context. Additionally, this approach reduces RAM needed for
PM device (104 -> 20 bytes of RAM on ARM Cortex-M).
Signed-off-by: Krzysztof Chruściński <krzysztof.chruscinski@nordicsemi.no>
k_thread_stack_free() is designed to be called with any pointer
value. We return -EINVAL when an attempt is made to free an
invalid stack pointer.
This change reduces the verbosity in the degenerate case, when
the pointer is not obtained via k_thread_stack_alloc(), but
otherwise does not affect functionality.
If debug log verbosity is not enabled, we save a few bytes in
.text / .rodata / .strtab.
Signed-off-by: Christopher Friedt <cfriedt@meta.com>
This removes z_smp_thread_init() and z_smp_thread_swap() as
SOF has been updated to use k_smp_cpu_custom_start() instead.
This removes the need for SOF to mirror part of the SMP
power up code, and thus these two functions are no longer
needed.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This provides a path to resume a previously suspended CPU.
This differs from k_smp_cpu_start() where per-CPU kernel
structs are not initialized such that execution context
can be saved during suspend and restored during resume.
Though the actual context saving and restoring are
platform specific.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This renames z_smp_cpu_start() to k_smp_cpu_start().
This effectively promotes z_smp_cpu_start() into a public API
which allows out of tree usage. Since this is a new API,
we can afford to change it signature, where it allows
an additional initialization steps to be done before a newly
powered up CPU starts participating in scheduling threads
to run.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This extends the wording so that not only architecture code can
start secondary CPUs at a later time. Also adds a missing 'to'.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
The current z_tick_sleep return directly when building kernel for Single
Thread model. This reorganize the code to use k_busy_wait() to be time
coherent since subsystems may depend on it.
In the case of a K_FOREVER timeout is selected the Single Thread the
implementation will invoke k_cpu_idle() and the system will wait for
an interrupt saving power.
Signed-off-by: Gerson Fernando Budke <gerson.budke@ossystems.com.br>
After a call to k_work_flush returns the sync variable
may still be modified by the workq. This is because
the work queue thread continues to modify the flag in
sync even after k_work_flush returns. This commit adds
K_WORK_FLUSHING_BIT, and with this bit, we moved the
logic of waking up the caller from handle_flush to the
finalize_flush_locked in workq, so that after waking up
the caller, the workqueue will no longer operate on sync.
Fixes: #64530
Signed-off-by: Junfan Song <sjf221100@gmail.com>
It is possible that address + size will overflow the available
address space and the pointer wraps around back to zero. Some
of these have been fixed in previous commits. This fixes
the remaining ones with regard to Z_PHYS_RAM_START/_END,
and Z_VIRT_RAM_START/_END.
Fixes#65542
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Use the new HEAP_MEM_POOL_ADD_SIZE_ prefix to construct a minimum
requirement for posix message queue usage. This way we can remove the
"special case" default values from the HEAP_MEM_POOL_SIZE Kconfig
definition.
Signed-off-by: Johan Hedberg <johan.hedberg@intel.com>
There are several subsystems and boards which require a relatively large
system heap (used by k_malloc()) to function properly. This became even
more notable with the recent introduction of the ACPICA library, which
causes ACPI-using boards to require a system heap of up to several
megabytes in size.
Until now, subsystems and boards have tried to solve this by having
Kconfig overlays which modify the default value of HEAP_MEM_POOL_SIZE.
This works ok, except when applications start explicitly setting values
in their prj.conf files:
$ git grep CONFIG_HEAP_MEM_POOL_SIZE= tests samples|wc -l
157
The vast majority of values set by current sample or test applications
is much too small for subsystems like ACPI, which results in the
application not being able to run on such boards.
To solve this situation, we introduce support for subsystems to specify
their own custom system heap size requirement. Subsystems do
this by defining Kconfig options with the prefix HEAP_MEM_POOL_ADD_SIZE_.
The final value of the system heap is the sum of the custom
minimum requirements, or the value existing HEAP_MEM_POOL_SIZE option,
whichever is greater.
We also introduce a new HEAP_MEM_POOL_IGNORE_MIN Kconfig option which
applications can use to force a lower value than what subsystems have
specficied, however this behavior is disabled by default.
Whenever the minimum is greater than the requested value a CMake warning
will be issued in the build output.
This patch ends up modifying several places outside of kernel code,
since the presence of the system heap is no longer detected using a
non-zero CONFIG_HEAP_MEM_POOL_SIZE value, rather it's now detected using
a new K_HEAP_MEM_POOL_SIZE value that's evaluated at build.
Signed-off-by: Johan Hedberg <johan.hedberg@intel.com>
Remove converting bit to string and comparing the string instead of
ready helpers. The "Check if thread is in use" seems to check only
that parameters state_buf and sizeof(state_buf) not zero.
Signed-off-by: Andrei Emeltchenko <andrei.emeltchenko@intel.com>
The pointer parameter 'data' in the function 'k_pipe_put()' ought to
use the const modifier as the contents of the buffer to which it
points never change. Internally, that const modifier is dropped as
both 'k_pipe_get()' and 'k_pipe_put()' share common code for copying
data; however 'k_pipe_put()' never takes a path that modifies those
contents.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
Fix k_sleep compilation error:
[build] ... syscalls/kernel.h:135: undefined reference to `z_impl_k_sleep'
for single thread applications (CONFIG_MULTITHREADING = n).
The shed.c contains source code which must be present also
in single thread applications.
Signed-off-by: Andrej Butok <andrey.butok@nxp.com>
z_free_page_count is only used in one file, so there is
no need to expose it, even to other part of kernel.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Updates both the k_sleep() and k_usleep() return values so that if
the thread was woken up prematurely, they will return the time left
to sleep rounded up to the nearest millisecond (for k_sleep) or
microsecond (for k_usleep) instead of rounding down. This removes
ambiguity should there be a non-zero number of remaining ticks
that correlate to a time of less than 1 millisecond or 1 microsecond.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
Replace open-coded time conversion with the macro which as that will
usually use a constant divide or multiply.
Signed-off-by: Keith Packard <keithp@keithp.com>
Export some symbols for loadable modules. Also add an
EXPORT_SYSCALL() helper macro for exporting system calls by their
official names.
Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
The function _Cstart has already been renamed to z_cstart,
so change the remaining references of it in various docs.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
- issue found with Ztest case of test_thread_timeout_remaining_expires
on Intel ISH platform when adjust CONFIG_SYS_CLOCK_TICKS_PER_SEC
to 10k.
- timeout_rem() return exact remaining ticks which is calibrated by
decrease elapsed(), while z_timeout_expires try to get expire ticks
to be timeout using current tick as base, so need get exact current
ticks by plus elasped().
Signed-off-by: Qipeng Zha <qipeng.zha@intel.com>
Instead of performing a set of relative address comparisons using
pointers of type 'uint8_t *', we leverage the existing IN_RANGE()
macro and perform the comparisons with 'uintptr_t'.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
Add support for mutable devices. Mutable devices are those which
can be modified after declaration, in-place, in kernel mode.
In order for a device to be mutable, the following must be true
* `CONFIG_DEVICE_MUTABLE` must be y-selected
* the Devicetree bindings for the device must include
`mutable.yaml`
* the Devicetree node must include the `zephyr,mutable` property
Signed-off-by: Christopher Friedt <cfriedt@meta.com>
This moves including of demand_paging.h out of kernel/mm.h,
so that users of demand paging APIs must include the header
explicitly. Since the main user is kernel itself, we can be
more discipline about header inclusion.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Add a Kconfig option to tell whether or not using thread
local storage to store current thread.
The function using it can be called from ISR and using
TLS variables in this context may (should ???) not be
allowed
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
This separates demand paging related headers into its own file
instead of being stuffed inside the main kernel memory
management header file.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This moves the k_* memory management functions from sys/ into
kernel/ includes, as there are kernel public APIs. The z_*
functions are further separated into the kernel internal
header directory.
Also made a quick change to doxygen to group sys_mem_* into
the OS Memory Management group so they will appear in doc.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Some platforms already have .bss section zeroed-out externally before the
Zephyr initialization and there is no sence to zero it out the second time
from the SW.
Such boot-time optimization could be critical e.g. for RTL Simulation.
Signed-off-by: Alexander Razinkov <alexander.razinkov@syntacore.com>
Extends the concept of halting a thread from just aborting a thread
to both aborting and suspending a thread.
Part of this involves updating k_thread_suspend() to operate in a
similar fashion to that of k_thread_abort().
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
Extracts the essential thread synchronization logic when aborting
a thread from z_thread_abort() and moves it to its own routine
called z_thread_halt().
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
The routine halt_thread() acts nearly identical to end_thread()
except that instead of only halting the thread if the _THREAD_DEAD
state bit is not set, it will halt it if bit specified by the
parameter new_state is not set (which is always _THREAD_DEAD).
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
The halt queue will be used to identify threads that are waiting
for a thread on another CPU to finish suspending.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
Basic spinlock implementation is based on single
atomic variable and doesn't guarantee locking fairness
across multiple CPUs. It's even possible that single CPU
will win the contention every time which will result
in a live-lock.
Ticket spinlocks provide a FIFO order of lock aquisition
which resolves such unfairness issue at the cost of slightly
increased memory footprint.
Signed-off-by: Alexander Razinkov <alexander.razinkov@syntacore.com>
Move the syscall_handler.h header, used internally only to a dedicated
internal folder that should not be used outside of Zephyr.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
k_thread_name_get has inconsistent signature. In the function
declaration it uses k_tid_t but in the implementation it is using
struct k_thread *. Change the implementation to use k_tid_t.
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
virt_page_phys_get can be called with phy parameter NULL when
the intention is just checking if a virtual address is mapped.
This function is generally overwritten by a an arch API that checks if
phys is null before using it but this default implementation doesn't.
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
Moving the Zephyr specific config options from
modules/hostap/Kconfig to corresponding Kconfig where the
option is specified.
Signed-off-by: Jukka Rissanen <jukka.rissanen@nordicsemi.no>
Thread userspace local data is to be used with storing errno per
thread without thread local storage support. However, if the C
library has native errno support, there is no need to enable
thread userspace local data to store errno per thread. Therefore,
amend the default for CONFIG_THREAD_USERSPACE_LOCAL_DATA so that
it is not enabled if the C library has native errno support.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
The documentation suggests that k_timer_start can be invoked from ISR
and preemptive contexts, however, an assertion failure occurs if one
k_timer_start call preempts another for the same timer instance. This
commit mitigates the issue by implementing a spinlock throughout the
k_timer_start function, ensuring thread-safety.
Fixes: #62908
Signed-off-by: Pedro Sousa <sousapedro596@gmail.com>
Sometimes the generic address range checker is not adequate
(think Xtensa cached/uncached pointers). This provides a way
to implement custom memory range checkers for those
situations. When enabled, sys_mm_is_phys_addr_in_range()
and sys_mm_is_virt_addr_in_range() must be implemented.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
The early random get function was making many wrong assumptions
about random subsys and entropy drivers. First, it was assuming
that entropy_get_entropy() would be ISR safe, that is not right,
the driver has an ISR safe callback and if it is not implemented
or not working it is not ok using the other callback.
Second, the fallback to the random subsys is even more problematic
since they can use kernel services to protect internal states and be
thread-safe.
Another incorrect thing in this function was the guard around it.
It was needed by features like stack randomization and stack canaries,
and not when those conditions were match. Just remove it and in case
it is not needed the linker will take care of it.
The drawback of this change is that in the absence of an entropy
generator with support to be called from ISR the randomness is very
weak.
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
Allow targets come up with their own early random generator
since the default can be NOT so random due constraints.
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
Rename z_early_boot_rand_get with z_early_rand_get to get consistent
with other early functions.
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
Platforms that determine their basic timer frequency at runtime instead of
build time cannot compute thread initialization timeouts during
compilation.
Switch back to storing the init_delay value in milliseconds and perform the
conversion to a k_timeout_t at runtime.
Signed-off-by: Keith Packard <keithp@keithp.com>
Instead of adding every possible subsystem which places variables in the C
library memory partition in libc-hooks.h, place those conditions in the
related Kconfig files and simplify the libc-hooks.h to just looking at
CONFIG_NEED_LIBC_MEM_PARTITION.
Signed-off-by: Keith Packard <keithp@keithp.com>
rand32.h does not make much sense, since the random subsystem
provides more APIs than just getting a random 32 bits value.
Rename it to random.h and get consistently with other
subsystems.
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
The wording on deprecating arch_kernel_init() in favor of prep_c()
has never been materialized. Various architectures are using it to
perform initialization. So remove the wording.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Assert that the handler of a work is not NULL when submitting
it to the queue. This allows early detection of the
code that is submitting a non-NULL work with NULL handler to
the work queue (where it happens), rather than right before the
work item get executed in the queue (when it happens).
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
Zephyr's code base uses MP_MAX_NUM_CPUS to
know how many cores exists in the target. It is
also expected that both symbols MP_MAX_NUM_CPUS
and MP_NUM_CPUS have the same value, so lets
just use MP_MAX_NUM_CPUS and simplify it.
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
Integrates object core statistics framework into the following
kernel objects:
sys_mem_blocks, k_mem_slab
threads, _cpu, z_kernel
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
