Enabled by MICROPY_COMPILE_ALLOW_TOP_LEVEL_AWAIT. When enabled, this means
that scope such as module-level functions and REPL statements can yield.
The outer C code must then handle this yielded generator.
Signed-off-by: Damien George <damien@micropython.org>
This fixes a bug where a random Python object may become
un-garbage-collectable until an enclosing Python file (compiled on device)
finishes executing.
Details:
The mp_parse_tree_t structure is stored on the stack in top-level functions
such as parse_compile_execute() in pyexec.c (and others).
Although it quickly falls out of scope in these functions, it is usually
still in the current stack frame when the compiled code executes. (Compiler
dependent, but usually it's one stack push per function.)
This means if any Python object happens to allocate at the same address as
the (freed) root parse tree chunk, it's un-garbage-collectable as there's a
(dangling) pointer up the stack referencing this same address.
As reported by @GitHubsSilverBullet here:
https://github.com/orgs/micropython/discussions/14116#discussioncomment-8837214
This work was funded through GitHub Sponsors.
Signed-off-by: Angus Gratton <angus@redyak.com.au>
This commit implements the 'e' half-float format: 10-bit mantissa, 5-bit
exponent. It uses native _Float16 if supported by the compiler, otherwise
uses custom bitshifting encoding/decoding routines.
Signed-off-by: Matthias Urlichs <matthias@urlichs.de>
Signed-off-by: Damien George <damien@micropython.org>
ASM_NOT_REG is optional, it can be synthesised by xor(reg, -1).
ASM_NEG_REG can also be synthesised with a subtraction, but most
architectures have a dedicated instruction for it.
Signed-off-by: Damien George <damien@micropython.org>
Add `pop()`, `appendleft()`, and `extend()` methods, support iteration
and indexing, and initializing from an existing sequence.
Iteration and indexing (subscription) have independent configuration flags
to enable them. They are enabled by default at the same level that
collections.deque is enabled (the extra features level).
Also add tests for checking new behavior.
Signed-off-by: Damien George <damien@micropython.org>
This new machine-module driver provides a "USBDevice" singleton object and
a shim TinyUSB "runtime" driver that delegates the descriptors and all of
the TinyUSB callbacks to Python functions. This allows writing arbitrary
USB devices in pure Python. It's also possible to have a base built-in
USB device implemented in C (eg CDC, or CDC+MSC) and a Python USB device
added on top of that.
This work was funded through GitHub Sponsors.
Signed-off-by: Angus Gratton <angus@redyak.com.au>
Disabled by default, but enabled on all boards that previously had
`MICROPY_PY_MACHINE_BARE_METAL_FUNCS` enabled.
Signed-off-by: Damien George <damien@micropython.org>
The STATIC macro was introduced a very long time ago in commit
d5df6cd44a. The original reason for this was
to have the option to define it to nothing so that all static functions
become global functions and therefore visible to certain debug tools, so
one could do function size comparison and other things.
This STATIC feature is rarely (if ever) used. And with the use of LTO and
heavy inline optimisation, analysing the size of individual functions when
they are not static is not a good representation of the size of code when
fully optimised.
So the macro does not have much use and it's simpler to just remove it.
Then you know exactly what it's doing. For example, newcomers don't have
to learn what the STATIC macro is and why it exists. Reading the code is
also less "loud" with a lowercase static.
One other minor point in favour of removing it, is that it stops bugs with
`STATIC inline`, which should always be `static inline`.
Methodology for this commit was:
1) git ls-files | egrep '\.[ch]$' | \
xargs sed -Ei "s/(^| )STATIC($| )/\1static\2/"
2) Do some manual cleanup in the diff by searching for the word STATIC in
comments and changing those back.
3) "git-grep STATIC docs/", manually fixed those cases.
4) "rg -t python STATIC", manually fixed codegen lines that used STATIC.
This work was funded through GitHub Sponsors.
Signed-off-by: Angus Gratton <angus@redyak.com.au>
This call used to be needed when there was an `emit_bc_pre()` function that
needed to be called at the start of each emitted bytecode. But in
8e7745eb31 that function was removed and now
the call to `mp_emit_bc_adjust_stack_size()` does nothing when adjusting by
0 entries, so it can be removed.
Signed-off-by: Damien George <damien@micropython.org>
For boards with MICROPY_CONFIG_ROM_LEVEL_AT_LEAST_CORE_FEATURES and up.
This gets samd21 boards working (which need the vfs module in _boot.py),
B_L072Z_LRWAN1, and nrf boards with smaller MCUs that use CORE or BASIC
feature levels.
Signed-off-by: robert-hh <robert@hammelrath.com>
This is now easy to support, since the first machine-word of a native
function tells how to find the prelude, from which the function name can be
extracted in the same way as for bytecode.
Signed-off-by: Damien George <damien@micropython.org>
Now native functions and native generators have similar behaviour: the
first machine-word of their code is an index to get to the prelude. This
simplifies the handling of these types of functions, and also reduces the
size of the emitted native machine code by no longer requiring special code
at the start of the function to load a pointer to the prelude.
Signed-off-by: Damien George <damien@micropython.org>
Viper functions are quite different to native functions and benefit from
being a separate type. For example, viper functions don't have a bytecode-
style prelude, and don't support generators or default arguments.
Signed-off-by: Damien George <damien@micropython.org>
They are no longer used. The new `mp_obj_malloc_with_finaliser()` macros
should be used instead, which force the setting of the `base.type` field.
And there's always `m_malloc_with_finaliser()` if needed.
Signed-off-by: Damien George <damien@micropython.org>
Following 709e8328d9.
Using this helps to reduce code size. And it ensure that the type is
always set as soon as the object is allocated, which is important for the
GC to function correctly.
Signed-off-by: Damien George <damien@micropython.org>
MICROPY_GIT_HASH was removed in 69e34b6b6b
but it is useful for, and used by, third-party code to tell which hash of
MicroPython is used.
Signed-off-by: Damien George <damien@micropython.org>
Allows bytecode itself to be used instead of an mp_raw_code_t in the simple
and common cases of a bytecode function without any children.
This can be used to further reduce frozen code size, and has the potential
to optimise other areas like importing.
Signed-off-by: Damien George <damien@micropython.org>
To simplify their access and reduce code size.
The `scope_flags` member is only ever used to determine if a function is a
generator or not, so make it reflect that fact as a bool type.
Signed-off-by: Damien George <damien@micropython.org>
The `asm_n_pos_args` and `asm_type_sig` members of `mp_raw_code_t` are only
used for raw codes of type MP_CODE_NATIVE_ASM, which are rare, for example
in frozen code. So using a truncated `mp_raw_code_t` in these cases helps
to reduce frozen code size on targets that have MICROPY_EMIT_INLINE_ASM
enabled.
With this, change in firmware size of RPI_PICO builds is -648.
Signed-off-by: Damien George <damien@micropython.org>
The mp_raw_code_t struct has been reordered and some members resized. The
`n_pos_args` member is renamed to `asm_n_pos_args`, and `type_sig` renamed
to `asm_type_sig` to indicate that these are used only for the inline-asm
emitters. These two members are also grouped together in the struct.
The justifications for resizing the members are:
- `fun_data_len` can be 32-bits without issue
- `n_children` is already limited to 16-bits by
`mp_emit_common_t::ct_cur_child`
- `scope_flags` is already limited to 16-bits by `scope_t::scope_flags`
- `prelude_offset` is already limited to 16-bits by the argument to
`mp_emit_glue_assign_native()`
- it's reasonable to limit the maximim number of inline-asm arguments to 12
(24 bits for `asm_type_sig` divided by 2)
This change helps to reduce frozen code size (and in some cases RAM usage)
in the following cases:
- 64-bit targets
- builds with MICROPY_PY_SYS_SETTRACE enabled
- builds with MICROPY_EMIT_MACHINE_CODE enabled but MICROPY_EMIT_INLINE_ASM
disabled
With this change, unix 64-bit builds are -4080 bytes in size. Bare-metal
ports like rp2 are unchanged (because mp_raw_code_t is still 32 bytes on
those 32-bit targets).
Signed-off-by: Damien George <damien@micropython.org>
If a return is executed within the try block of a try-finally then the
return value is stored on the top of the Python stack during the execution
of the finally block. In this case the Python stack is one larger than it
normally would be in the finally block.
Prior to this commit, the compiler was not taking this case into account
and could have a Python stack overflow if the Python stack used by the
finally block was more than that used elsewhere in the function. In such
a scenario the last argument of the function would be clobbered by the
top-most temporary value used in the deepest Python expression/statement.
This commit fixes that case by making sure enough Python stack is allocated
to the function.
Fixes issue #13562.
Signed-off-by: Damien George <damien@micropython.org>
stat_path is only called with stringified vstr_t objects.
Thus, pulling the stringification into the function replaces three
function calls with one, saving a few bytes.
Signed-off-by: Matthias Urlichs <matthias@urlichs.de>
This will apply to bare-arm and minimal, as well as the minimal unix
variant.
Change the default to MICROPY_QSTR_BYTES_IN_HASH=1 for the CORE,BASIC
levels, 2 for >=EXTRA.
Removes explicit setting of MICROPY_QSTR_BYTES_IN_HASH==1 in ports that
don't set the feature level (because 1 is implied by the default level,
CORE). Applies to cc3200, pic16bt, powerpc.
Removes explicit setting for nRF (which sets feature level). Also for samd,
which sets CORE for d21 and FULL for d51. This means that d21 is unchanged
with MICROPY_QSTR_BYTES_IN_HASH==1, but d51 now moves from 1 to 2 (roughly
adds 1kiB).
The only remaining port which explicitly set bytes-in-hash is rp2 because
it's high-flash (hence CORE level) but lowish-SRAM, so it's worthwhile
saving the RAM for runtime qstrs.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This disables using qstr hashes altogether, which saves RAM and flash
(two bytes per interned string on a typical build) as well as code size.
On PYBV11 this is worth over 3k flash.
qstr comparison will now be done just by length then data. This affects
qstr_find_strn although this has a negligible performance impact as, for a
given comparison, the length and first character will ~usually be
different anyway.
String hashing (e.g. builtin `hash()` and map.c) now need to compute the
hash dynamically, and for the map case this does come at a performance
cost.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Make can't handle paths with spaces, see https://savannah.gnu.org/bugs/?712
The following workarounds exist:
- When using make's built-in functions:
- Use relative paths wherever possible to avoid spaces in the first
place.
- All spaces in paths can be escaped with backslashes; quotes don't
work.
- Some users use the shell to temporarily rename directories, or to
create symlinks without spaces.
- When using make to pass commands to the system's shell, enclose paths in
quotes. While make will still interpret quoted strings with spaces as
multiple words, the system's shell will correctly parse the resulting
command.
This commit contains the following fixes:
- In ports/stm32/mboot/Makefile: Use relative paths to avoid spaces when
using built-in functions.
- In all other files: Use quotes to enclose paths when make is used to call
shell functions.
All changes have been tested with a directory containing spaces.
Signed-off-by: Iksas <iksas@mailbox.org>
This was originally needed because the .c --> .o rule is:
$(BUILD)/%.o: %.c
and because the generated frozen_content.c is inside build-FOO, it must
therefore generate build-FOO/build-FOO/frozen_content.o.
But 2eda513870 added a new build rule for
pins.c that can also be used for frozen_content.c.
Signed-off-by: iabdalkader <i.abdalkader@gmail.com>
When compiling with distcc, it does not understand the -MD flag on its own.
This fixes the interaction by explicitly adding the -MF option.
The error in distcc is described here under "Problems with gcc -MD":
https://www.distcc.org/faq.html
Signed-off-by: Peter Züger <zueger.peter@icloud.com>
MicroPython code may rely on the return value of sys.stdout.buffer.write()
to reflect the number of bytes actually written. While in most scenarios a
write() operation is successful, there are cases where it fails, leading to
data loss. This problem arises because, currently, write() merely returns
the number of bytes it was supposed to write, without indication of
failure.
One scenario where write() might fail, is where USB is used and the
receiving end doesn't read quickly enough to empty the receive buffer. In
that case, write() on the MicroPython side can timeout, resulting in the
loss of data without any indication, a behavior observed notably in
communication between a Pi Pico as a client and a Linux host using the ACM
driver.
A complex issue arises with mp_hal_stdout_tx_strn() when it involves
multiple outputs, such as USB, dupterm and hardware UART. The challenge is
in handling cases where writing to one output is successful, but another
fails, either fully or partially. This patch implements the following
solution:
mp_hal_stdout_tx_strn() attempts to write len bytes to all of the possible
destinations for that data, and returns the minimum successful write
length.
The implementation of this is complicated by several factors:
- multiple outputs may be enabled or disabled at compiled time
- multiple outputs may be enabled or disabled at runtime
- mp_os_dupterm_tx_strn() is one such output, optionally containing
multiple additional outputs
- each of these outputs may or may not be able to report success
- each of these outputs may or may not be able to report partial writes
As a result, there's no single strategy that fits all ports, necessitating
unique logic for each instance of mp_hal_stdout_tx_strn().
Note that addressing sys.stdout.write() is more complex due to its data
modification process ("cooked" output), and it remains unchanged in this
patch. Developers who are concerned about accurate return values from
write operations should use sys.stdout.buffer.write().
This patch might disrupt some existing code, but it's also expected to
resolve issues, considering that the peculiar return value behavior of
sys.stdout.buffer.write() is not well-documented and likely not widely
known. Therefore, it's improbable that much existing code relies on the
previous behavior.
Signed-off-by: Maarten van der Schrieck <maarten@thingsconnected.nl>
There are two main changes here to improve the calculation of the size of
the next heap area when automatically expanding the heap:
- Compute the existing total size by counting the total number of GC
blocks, and then using that to compute the corresponding number of bytes.
- Round the bytes value up to the nearest multiple of BYTES_PER_BLOCK.
This makes the calculation slightly simpler and more accurate, and makes
sure that, in the case of growing from one area to two areas, the number
of bytes allocated from the system for the second area is the same as the
first. For example on esp32 with an initial area size of 65536 bytes, the
subsequent allocation is also 65536 bytes. Previously it was a number that
was not even a multiple of 2.
Signed-off-by: Damien George <damien@micropython.org>
In "cat" mode, output was written to a file named "out", then moved to the
location of the real output file. There was no reason for this.
While makeqstrdefs.py does make an effort to not update the timestamp on an
existing output file that has not changed, the intermediate "out" file
isn't part of the that process.
Signed-off-by: Trent Piepho <tpiepho@gmail.com>
In "cat" mode a "$output_file.hash" file is checked to see if the hash of
the new output is the same as the existing, and if so the output file isn't
updated.
However, it's possible that the output file has been deleted but the hash
file has not. In this case the output file is not created.
Change the logic so that a hash file is considered stale if there is no
output file and still create the output.
Signed-off-by: Trent Piepho <tpiepho@gmail.com>
These are produced by the "cat" command to makeqstrdefs.py, to allow it to
not update unchanged files. cmake doesn't know about them and so they are
not removed on a "clean".
This triggered a bug in makeqstrdefs.py where it would not recreate a
deleted output file (which is removed by clean) if a stale hash file with a
valid hash still existed.
Listing them as byproducts will cause them to be deleted on clean.
Signed-off-by: Trent Piepho <tpiepho@gmail.com>
These are intended to replace MICROPY_EVENT_POLL_HOOK and
MICROPY_EVENT_POLL_HOOK_FAST, which are insufficient for tickless ports.
This implementation is along the lines suggested here:
https://github.com/micropython/micropython/issues/12925#issuecomment-1803038430
Currently any usage of these functions expands to use the existing hook
macros, but this can be switched over port by port.
This work was funded through GitHub Sponsors.
Signed-off-by: Angus Gratton <angus@redyak.com.au>
MICROPY_BEGIN_ATOMIC_SECTION/MICROPY_END_ATOMIC_SECTION belong more to the
MicroPython HAL rather than build configuration settings, so move their
default configuration to py/mphal.h, and require all users of these macros
to include py/mphal.h (here, py/objexcept.c and py/scheduler.c).
This helps ports separate configuration from their HAL implementations, and
can improve build times (because mpconfig.h is included everywhere, whereas
mphal.h is not).
Signed-off-by: Damien George <damien@micropython.org>
This allows e.g. a board (or make command line) to set
MICROPY_MANIFEST_MY_VARIABLE = path/to/somewhere
set(MICROPY_MANIFEST_MY_VARIABLE path/to/somewhere)
and then in the manifest.py they can query this, e.g. via
include("$(MY_VARIABLE)/path/manifest.py")
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Otherwise passing in a non-integer can lead to an invalid memory access.
Thanks to Junwha Hong and Wonil Jang @S2Lab, UNIST for finding the issue.
Fixes issue #13007.
Signed-off-by: Damien George <damien@micropython.org>
Add a .ico file with common icon image size, created from
vector-logo-2.png, and embed it into the resulting executable.
Signed-off-by: stijn <stijn@ignitron.net>
This handles the case where an empty bytes/bytearray/str could pass in
NULL as the str argument (with length zero). This would result in UB in
strncmp. Even though our bare-metal implementation of strncmp handles
this, best to avoid it for when we're using system strncmp.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This fixes the case where e.g.
struct foo_t {
mp_obj_t x;
uint16_t y;
char buf[];
};
will have `sizeof(struct foo_t)==8`, but `offsetof(struct foo_t, buf)==6`.
When computing the size to allocate for `m_new_obj_var` we need to use
offsetof to avoid over-allocating. This is important especially when it
might cause it to spill over into another GC block.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Set the position of new line tokens as the end of the preceding line
instead of the beginning of the next line. This is done by first moving
the pointer to the end of the current line to skip any whitespace, record
the position for the token, then finaly skip any other line and whitespace.
The previous behavior was to skip every new line and whitespace, including
the indent of the next line, before recording the token position.
(Note that both lex->emit_dent and lex->nested_bracket_level equal 0 if
had_physical_newline == true, which allows simplifying the if-logic for
MP_TOKEN_NEWLINE.)
And update the cmd_parsetree.py test expected output, because the position
of the new-line token has changed.
Fixes issue #12792.
Signed-off-by: Mathieu Serandour <mathieu.serandour@numworks.fr>
This prevents each port Makefile from having to add an explicit rule for
`build-BOARD/pins_BOARD.c`.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
ESP32-C3 is not Xtensa-based, so build settings are now tailored a bit
better following that fact. ESP-IDF 5.x already adds architecture-specific
modules by itself so there is no need to specify either the `xtensa` or the
`riscv` module in the build settings.
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This provides a significant performance boost for qstr_find_strn, which is
called a lot during parsing and loading of .mpy files, as well as interning
of string objects (which happens in most string methods that return new
strings).
Also adds comments to explain the "static" qstrs. These are part of the
.mpy ABI and avoid needing to duplicate string data for QSTRs known to
already be in the firmware. The static pool isn't currently sorted, but in
the future we could either split the static pool into the sorted regions,
or in the next .mpy version just sort them.
Based on initial work done by @amirgon in #6896.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
`ASM_MOV_REG_IMM_FIX_U16` and `ASM_MOV_REG_IMM_FIX_WORD` are no longer
used anywhere in the code.
See discussion in #12771.
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This provides a way to enable features and changes slated for MicroPython
2.x, by running `make MICROPY_PREVIEW_VERSION_2=1`. Also supported for
the cmake ports (except Zephyr).
This is an alternative to having a 2.x development branch (or equivalently,
keeping a 1.x release branch). Any feature or change that needs to be
"hidden" until 2.x can use this flag (either in the Makefile or the
preprocessor).
A good example is changing function arguments or other public API features,
in particular to aid in improving consistency between ports.
When `MICROPY_PREVIEW_VERSION_2` is enabled, the REPL banner is amended to
say "MicroPython (with v2.0 preview) vX.Y.Z", and sys.implementation gets a
new field `_v2` set to `True`.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
The main thread gets this because the thread state is in bss, but
subsequent threads need this field to be initialised.
Also added a note to mpstate.h to help avoid missing this in the future.
Fixes issue #12695.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This allows mp_get_buffer_raise() to be changed to a simple inline function
that in the majority of cases costs the same (in code size) to call as the
original mp_get_buffer_raise(), because the flags argument is a constant.
Signed-off-by: Damien George <damien@micropython.org>
This behaviour matches CPython. It's useful to be able to store bound
method objects in dicts/sets, and compare for equality, eg when storing
them in a list and using list.remove().
Signed-off-by: Daniël van de Giessen <daniel@dvdgiessen.nl>
If a non-string buffer was passed to execfile, then it would be passed
as a non-null-terminated char* to mp_lexer_new_from_file.
This changes mp_lexer_new_from_file to take a qstr instead (as in almost
all cases a qstr will be created from this input anyway to set the
`__file__` attribute on the module).
This now makes execfile require a string (not generic buffer) argument,
which is probably a good fix to make anyway.
Fixes issue #12522.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
"Raise SomeException() from None" is a common Python idiom to suppress
chained exceptions and thus shouldn't trigger a warning on a version of
Python that doesn't support them in the first place.
See https://github.com/micropython/micropython/issues/12127 for details.
Previously at the point when a release is made, we update mpconfig.h
and set a git tag. i.e. the version increments at the release.
Now the version increments immediately after the release. The workflow is:
1. Final commit in the cycle updates mpconfig.h to set (X, Y, 0, 0) (i.e.
clear the pre-release state).
2. This commit is tagged "vX.Y.0".
3. First commit for the new cycle updates mpconfig.h to set (X, Y+1, 0, 1)
(i.e. increment the minor version, set the pre-release state).
4. This commit is tagged "vX.Y+1.0-preview".
The idea is that a nightly build is actually a "preview" of the _next_
release. i.e. any documentation describing the current release may not
actually match the nightly build. So we use "preview" as our semver
pre-release identifier.
Changes in this commit:
- Add MICROPY_VERSION_PRERELEASE to mpconfig.h to allow indicating that
this is not a release version.
- Remove unused MICROPY_VERSION integer.
- Append "-preview" to MICROPY_VERSION_STRING when the pre-release state
is set.
- Update py/makeversionhdr.py to no longer generate MICROPY_GIT_HASH.
- Remove the one place MICROPY_GIT_HASH was used (it can use
MICROPY_GIT_TAG instead).
- Update py/makeversionhdr.py to also understand
MICROPY_VERSION_PRERELEASE in mpconfig.h.
- Update py/makeversionhdr.py to convert the git-describe output into
semver-compatible "X.Y.Z-preview.N.gHASH".
- Update autobuild.sh to generate filenames using the new scheme.
- Update remove_old_firmware.py to match new scheme.
- Update mpremote's pyproject.toml to handle the "-preview" suffix in the
tag. setuptools_scm maps to this "rc0" to match PEP440.
- Fix docs heading where it incorrectly said "vvX.Y.Z" for release docs.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This was missed in 692d36d779. Probably
never noticed because everything enables `MICROPY_GC_CONSERVATIVE_CLEAR`,
but found via ASAN thanks to @gwangmu & @chibinz.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This provides a way to build a non-DEBUG host binary that still has symbols
and debug information.
Document this for the unix port, and update a comment in the unix port
Makefile.
This work was funded through GitHub Sponsors.
Signed-off-by: Angus Gratton <angus@redyak.com.au>
Supported from GCC 8 and up, and Compiler Explorer suggests it works as
expected with Clang since 3.6 (2014).
- Fixes situation where building embedded MicroPython with -O0 and
MICROPY_NLR_X64 crashes at runtime (due to nlr_push pushing the
frame pointer register EBP). Closes#12421.
- Allows removing the macOS tweak to undo pushing EBP onto the stack
in the generated function prelude.
Signed-off-by: Angus Gratton <angus@redyak.com.au>
In CPython, `_thread.start_new_thread()` returns an ID that is the same ID
that is returned by `_thread.get_ident()`. The current MicroPython
implementation of `_thread.start_new_thread()` always returns `None`.
This modifies the required functions to return a value. The native thread
id is returned since this can be used for interop with other functions, for
example, `pthread_kill()` on *nix. `_thread.get_ident()` is also modified
to return the native thread id so that the values match and avoids the need
for a separate `native_id` attribute.
Fixes issue #12153.
Signed-off-by: David Lechner <david@pybricks.com>
Because mpy_ld.py doesn't know the target object representation, it emits
instances of `MP_OBJ_NEW_QSTR(MP_QSTR_Foo)` as const string objects, rather
than qstrs. However this doesn't work for map keys (e.g. for a locals dict)
because the map has all_keys_are_qstrs flag is set (and also auto-complete
requires the map keys to be qstrs).
Instead, emit them as regular qstrs, and make a functioning MP_OBJ_NEW_QSTR
function available (via `native_to_obj`, also used for e.g. making
integers).
Remove the code from mpy_ld.py to emit qstrs as constant strings, but leave
behind the scaffold to emit constant objects in case we want to do use this
in the future.
Strictly this should be a .mpy sub-version bump, even though the function
table isn't changing, it does lead to a change in behavior for a new .mpy
running against old MicroPython. `mp_native_to_obj` will incorrectly return
the qstr value directly as an `mp_obj_t`, leading to unexpected results.
But given that it's broken at the moment, it seems unlikely that anyone is
relying on this, so it's not work the other downsides of a sub-version bump
(i.e. breaking pure-Python modules that use @native). The opposite case of
running an old .mpy on new MicroPython is unchanged, and remains broken in
exactly the same way.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This adds support for the x format code in struct.pack and struct.unpack.
The primary use case for this is ignoring bytes while unpacking. When
interfacing with existing systems, it may often happen that you either have
fields in a struct that aren't properly specified or you simply don't care
about them. Being able to easily skip them is useful.
Signed-off-by: Daniël van de Giessen <daniel@dvdgiessen.nl>
Eliminate `TypeError` when format string contains no named conversions.
This matches CPython behavior.
Signed-off-by: mcskatkat <mc_skatkat@hotmail.com>
The rp2 port was enabling SSL and had finalizers enabled via the "extra
features" level, but missed explicitly enabling `MICROPY_PY_SSL_FINALISER`
(like esp32, stm32, and mimxrt did).
This commit makes `MICROPY_PY_SSL_FINALISER` default to enabled if
finalizers are enabled, and removes the explicit setting of this for
esp32, stm32, mimxrt (because they all use the "extra features" level).
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
The only reason that const had to be disabled was to make the test output
match CPython when const was involved. Instead, this commit fixes the test
to handle the lines where const is used.
Also:
- remove the special handling for MICROPY_PERSISTENT_CODE_SAVE in
unix/mpconfigport.h, and make this automatic.
- move the check for MICROPY_PERSISTENT_CODE_SAVE to where it's used (like
we do for other similar checks) and add a comment explaining it.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This allows the cc3200 port to be build with the standard autobuild script
rather than the custom build-cc3200-latest.sh (which is now removed).
This also fixes the path inside the zip file (by using the `-j` flag to
zip).
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This is difficult to implement on cmake-based ports, and having the list
of variants in mpconfigboard.{cmake,mk} duplicates information that's
already in board.json.
This removes the existing query-variants make target from stm32 & rp2
and the definition of BOARD_VARIANTS from the various board files.
Also renames the cmake variable to MICROPY_BOARD_VARIANT to match other
variables such as MICROPY_BOARD. The make variable stays as
BOARD_VARIANT.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
When set, the split heap is automatically extended with new areas on
demand, and shrunk if a heap area becomes empty during a GC pass or soft
reset.
To save code size the size allocation for a new heap block (including
metadata) is estimated at 103% of the failed allocation, rather than
working from the more complex algorithm in gc_try_add_heap(). This appears
to work well except in the extreme limit case when almost all RAM is
exhausted (~last few hundred bytes). However in this case some allocation
is likely to fail soon anyhow.
Currently there is no API to manually add a block of a given size to the
heap, although that could easily be added if necessary.
This work was funded through GitHub Sponsors.
Signed-off-by: Angus Gratton <angus@redyak.com.au>
This commit:
- Breaks up some long lines for readability.
- Fixes a potential macro argument expansion issue.
This work was funded through GitHub Sponsors.
Signed-off-by: Angus Gratton <angus@redyak.com.au>
A previous commit removed the unix-specific select module implementation
and made unix use the common one.
This commit adds an optimisation so that the system poll function is used
when polling objects that have a file descriptor. With this optimisation
enabled, if code registers both file-descriptor-based objects, and non-
file-descriptor-based objects with select.poll() then the following occurs:
- the system poll is called for all file-descriptor-based objects with a
timeout of 1ms
- then the bare-metal polling implementation is used for remaining objects,
which calls into their ioctl method (which can be in C or Python)
In the case where all objects have file descriptors, the system poll is
called with the full timeout requested by the caller. That makes it as
efficient as possible in the case everything has a file descriptor.
Benefits of this approach:
- all ports use the same select module implementation
- the unix port now supports polling of all objects and matches bare metal
implementations
- it's still efficient for existing cases where only files and sockets are
polled (on unix)
- the bare metal implementation does not change
- polling of SSL objects will now work on unix by calling in to the ioctl
method on SSL objects (this is required for asyncio ssl support)
Note that extmod/vfs_posix_file.c has poll disable when the optimisation is
enabled, because the code is not reachable when the optimisation is used.
Signed-off-by: Damien George <damien@micropython.org>
In applications that use little memory and run GC regularly, the cost of
the sweep phase quickly becomes prohibitives as the amount of RAM
increases.
On an ESP32-S3 with 2 MB of external SPIRAM, for example, a trivial GC
cycle takes a minimum of 40ms, virtually all of it in the sweep phase.
Similarly, on the UNIX port with 1 GB of heap, a trivial GC takes 47 ms,
again virtually all of it in the sweep phase.
This commit speeds up the sweep phase in the case most of the heap is empty
by keeping track of the ID of the highest block we allocated in an area
since the last GC.
The performance benchmark run on PYBV10 shows between +0 and +2%
improvement across the existing performance tests. These tests don't
really stress the GC, so they were also run with gc.threshold(30000) and
gc.threshold(10000). For the 30000 case, performance improved by up to
+10% with this commit. For the 10000 case, performance improved by at
least +10% on 6 tests, and up to +25%.
Signed-off-by: Damien George <damien@micropython.org>
Previously this was explicitly enabled on esp32/stm32/renesas/mimxrt/samd,
but didn't get a default feature level because it wasn't in py/mpconfig.h.
With this commit it's now enabled at the "extra features" level, which adds
rp2, unix-standard, windows, esp8266, webassembly, and some nrf boards.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
When MICROPY_SCHEDULER_STATIC_NODES is enabled, the logic is unchanged.
When MICROPY_SCHEDULER_STATIC_NODES is disable, sched_state is now always
initialised to MP_SCHED_IDLE when calling mp_init(). For example, the use
of mp_sched_vm_abort(), if it aborts a running scheduled function, can lead
to the scheduler starting off in a locked state when the runtime is
restarted, and then it stays locked. This commit fixes that case by
resetting sched_state.
Signed-off-by: Damien George <damien@micropython.org>
This provides similar functionality to the former zlib.DecompIO and
especially CPython's gzip.GzipFile for both compression and decompression.
This class can be used directly, and also can be used from Python to
implement (via io.BytesIO) zlib.decompress and zlib.compress, as well as
gzip.GzipFile.
Enable/disable this on all ports/boards that zlib was previously configured
for.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
There are enough places that implement __exit__ by forwarding directly to
mp_stream_close that this saves code size.
For the cases where __exit__ is a no-op, additionally make their
MP_STREAM_CLOSE ioctl handled as a no-op.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This will be replaced with a new deflate module providing the same
functionality, with an optional frozen Python wrapper providing a
replacement zlib module.
binascii.crc32 is temporarily disabled.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Follow-up to 24c02c4eb5 for when
MICROPY_ENABLE_EXTERNAL_IMPORT=0. It now needs to try both extensible and
non-extensible modules.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Prior to this fix, async for assumed the iterator expression was a simple
identifier, and used that identifier as a local to store the intermediate
iterator object. This is incorrect behaviour.
This commit fixes the issue by keeping the iterator object on the stack as
an anonymous local variable.
Fixes issue #11511.
Signed-off-by: Damien George <damien@micropython.org>
The asyncio module now has much better CPython compatibility and
deserves to be just called "asyncio".
This will avoid people having to write `from uasyncio import asyncio`.
Renames all files, and updates port manifests to use the new path. Also
renames the built-in _uasyncio to _asyncio.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
PEP-498 allows for conversion specifiers like !r and !s to convert the
expression declared in braces to be passed through repr() and str()
respectively.
This updates the logic that detects the end of the expression to also stop
when it sees "![rs]" that is either at the end of the f-string or before
the ":" indicating the start of the format specifier. The "![rs]" is now
retained in the format string, whereas previously it stayed on the end
of the expression leading to a syntax error.
Previously: `f"{x!y:z}"` --> `"{:z}".format(x!y)`
Now: `f"{x!y:z}"` --> `"{!y:z}".format(x)`
Note that "!a" is not supported by `str.format` as MicroPython has no
`ascii()`, but now this will raise the correct error.
Updated cpydiff and added tests.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
So that the delegation functions don't need to be put somewhere global,
like in mpconfigport.h. That would otherwise make it hard for extension
modules to use delegation.
Signed-off-by: Damien George <damien@micropython.org>
Starting with 2757acf6, the `top` variable in `nlr_jump()` in
`nlraarch64.c` was assigned to register `x19` by the compiler. However,
the assembly code writes over that register with
ldp x19, x20, [%0, #32]
since `%0` is now `x19`. This causes the next line
ldp lr, x9, [%0, #16]
to load the wrong values.
To fix the issue, we move the value of the `top` variable from an unknown
register to a known register at the beginning of the asm code then only use
known/hard-coded registers after that.
Fixes issue #11754.
Signed-off-by: David Lechner <david@pybricks.com>
Following how mkrules.cmake works. This makes it easy for a port to enable
frozen code, by defining FROZEN_MANIFEST in its Makefile.
Signed-off-by: Damien George <damien@micropython.org>
This is a MicroPython-specific module that existed to support the old
version of uasyncio. It's undocumented and not enabled on all ports and
takes up code size unnecessarily.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Previously sys.path could be modified by append/pop or slice assignment.
This allows `sys.path = [...]`, which can be simpler in many cases, but
also improves CPython compatibility.
It also allows sys.path to be set to a tuple which means that you can
clear sys.path (e.g. temporarily) with no allocations.
This also makes sys.path (and sys.argv for consistency) able to be disabled
via mpconfig. The unix port (and upytesthelper) require them, so they
explicitly verify that they're enabled.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Otherwise you can get into the confusing state where e.g. sys.ps1 is
enabled in config (via `MICROPY_PY_SYS_PS1_PS2`) but still doesn't actually
get enabled.
Also verify that the required delegation options are enabled in modsys.c.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
When compiling mpy-cross, there is no `sys` module, and so there will
be no entries in the `mp_builtin_module_delegation_table`.
MSVC doesn't like this, so instead pretend as if the feature isn't
enabled at all.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This replaces the previous QSTR_null entry in the globals dict which could
leak out to Python (e.g. via iteration of mod.__dict__) and could lead to
crashes.
It results in smaller code size at the expense of turning a lookup into a
loop, but the list it is looping over likely only contains one or two
elements.
To allow a module to register its custom attr function it can use the new
`MP_REGISTER_MODULE_DELEGATION` macro.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Updates any includes, and references from Makefiles/CMake.
This essentially reverts what was done long ago in commit
136b5cbd76
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
In order to keep "import umodule" working, the existing mechanism is
replaced with a simple fallback to drop the "u".
This makes importing of built-ins no longer touch the filesystem, which
makes a typical built-in import take ~0.15ms rather than 3-5ms.
(Weak links were added in c14a81662c)
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This renames the builtin-modules, such that help('modules') and printing
the module object will show "module" rather than "umodule".
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Prior to this commit, importing a module that exists but has a syntax error
or some other problem that happens at import time would result in a
potentially-incomplete module object getting added to sys.modules.
Subsequent imports would use that object, resulting in confusing error
messages that hide the root cause of the problem.
This commit fixes that issue by removing the failed module from sys.modules
using the new NLR callback mechanism.
Note that it is still important to add the module to sys.modules while the
import is happening so that we can support circular imports just like
CPython does.
Fixes issue #967.
Signed-off-by: David Grayson <davidegrayson@gmail.com>
The changed functions now use less stack, and don't have any issues with
local variables needing to be declared volatile.
Testing on a PYBv1.0, imports (of .py, .mpy and frozen code) now use 64
less bytes of C stack per import depth.
Signed-off-by: Damien George <damien@micropython.org>
NLR buffers are usually quite large (use lots of C stack) and expensive to
push and pop. Some of the time they are only needed to perform clean up if
an exception happens, and then they re-raise the exception.
This commit allows optimizing that scenario by introducing a linked-list of
NLR callbacks that are called automatically when an exception is raised.
They are essentially a light-weight NLR handler that can implement a
"finally" block, i.e. clean-up when an exception is raised, or (by passing
`true` to nlr_pop_jump_callback) when execution leaves the scope.
Signed-off-by: Damien George <damien@micropython.org>
This can lead to duplicate initialisations if a module can be imported
via multiple names, so the module must track this itself anyway.
This reduces code size (diff is -40 bytes), and avoids special treatment of
builtin-modules-with-init with respect to sys.modules. No other builtin
modules get put into sys.modules.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
To use this:
- Create a built-in module, and add the module object as a member of the
parent module's globals dict.
- The submodule can set its `__name__` to either `QSTR_foo_dot_bar` or
`QSTR_bar`. The former requires using qstrdefs(port).h to make the qstr.
Because `bar` is a member of `foo`'s globals, it is possible to write
`import foo` and then immediately use `foo.bar` without importing it
explicitly. This means that if `bar` has an `__init__`, it will not be
called in this situation, and for that reason, sub-modules should not have
`__init__` methods. If this is required, then all initalisation for
sub-modules should be done by the top-level module's (i.e. `foo`'s)
`__init__` method.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This makes it so that sub-packages are resolved relative to their parent's
`__path__`, rather than re-resolving each parent's filesystem path.
The previous behavior was that `import foo.bar` would first re-search
`sys.path` for `foo`, then use the resulting path to find `bar`.
For already-loaded and u-prefixed modules, because we no longer need to
build the path from level to level, we no longer unnecessarily search
the filesystem. This should improve startup time.
Explicitly makes the resolving process clear:
- Loaded modules are returned immediately without touching the filesystem.
- Exact-match of builtins are also returned immediately.
- Then the filesystem search happens.
- If that fails, then the weak-link handling is applied.
This maintains the existing behavior: if a user writes `import time` they
will get time.py if it exits, otherwise the built-in utime. Whereas `import
utime` will always return the built-in.
This also fixes a regression from a7fa18c203
where we search the filesystem for built-ins. It is now only possible to
override u-prefixed builtins. This will remove a lot of filesystem stats
at startup, as micropython-specific modules (e.g. `pyb`) will no longer
attempt to look at the filesystem.
Added several improvements to the comments and some minor renaming and
refactoring to make it clearer how the import mechanism works. Overall
code size diff is +56 bytes on STM32.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
If sys.path is enabled, but empty, this will now no longer search the
filesystem. Previously an empty sys.path was equivalent to having
`sys.path=[""]`. This is a breaking change, but this behavior now matches
CPython.
This also provides an alternative mechanism to the u-prefix to force an
import of a builtin module:
```
import sys
_path = sys.path[:]
sys.path.clear()
import foo # Forces the built-in foo.
sys.path.extend(_path)
del _path
```
Code size diff is -32 bytes on PYBV11.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This generalises and simplifies the code and follows CPython behaviour.
See similar change for floats in a07fc5b640.
Signed-off-by: Damien George <damien@micropython.org>
This is possible now that MP_UNARY_OP_INT_MAYBE exists.
As a consequence mp_obj_get_int now also supports user types, which was
previously possible with MP_UNARY_OP_INT but no tests existed for it.
Signed-off-by: Damien George <damien@micropython.org>
To be consistent with MP_UNARY_OP_INT_FLOAT and MP_UNARY_OP_INT_COMPLEX,
and allow int() to first check if a type supports __int__ before trying
other things (as per CPython).
Signed-off-by: Damien George <damien@micropython.org>
The code that handles inplace-operator to normal-binary-operator fallback
is moved in this commit from py/objtype.c to py/runtime.c, making it apply
to all types, not just user classes.
Signed-off-by: Damien George <damien@micropython.org>
So that user types can implement reverse operators and have them work with
str on the left-hand-side, eg `"a" + UserType()`.
Signed-off-by: Damien George <damien@micropython.org>
