Commit graph

4473 commits

Author SHA1 Message Date
Damien George
56e90cb60b py/mpconfig: Enable 2-argument built-in next() at basic feature level.
This is a pretty fundamental built-in and having CPython-compatible
behaviour is beneficial.  The code size increase is not much, and
ports/boards can still disable it if needed to save space.

Addresses issue #5384.

Signed-off-by: Damien George <damien@micropython.org>
2025-03-27 11:51:58 +11:00
Volodymyr Shymanskyy
51976110e2 tools/mpy_ld.py: Allow linking static libraries.
This commit introduces an additional symbol resolution mechanism to the
natmod linking process.  This allows the build scripts to look for required
symbols into selected libraries that are provided by the compiler
installation (libgcc and libm at the moment).

For example, using soft-float code in natmods, whilst technically possible,
was not an easy process and required some additional work to pull it off.
With this addition all the manual (and error-prone) operations have been
automated and folded into `tools/mpy_ld.py`.

Both newlib and picolibc toolchains are supported, albeit the latter may
require a bit of extra configuration depending on the environment the build
process runs on.  Picolibc's soft-float functions aren't in libm - in fact
the shipped libm is nothing but a stub - but they are inside libc.  This is
usually not a problem as these changes cater for that configuration quirk,
but on certain compilers the include paths used to find libraries in may
not be updated to take Picolibc's library directory into account.  The bare
metal RISC-V compiler shipped with the CI OS image (GCC 10.2.0 on Ubuntu
22.04LTS) happens to exhibit this very problem.

To work around that for CI builds, the Picolibc libraries' path is
hardcoded in the Makefile directives used by the linker, but this can be
changed by setting the PICOLIBC_ROOT environment library when building
natmods.

Signed-off-by: Volodymyr Shymanskyy <vshymanskyi@gmail.com>
Co-authored-by: Alessandro Gatti <a.gatti@frob.it>
2025-03-17 13:03:27 +11:00
Damien George
840b641024 py/runtime: Automatically mount ROMFS as part of mp_init.
This is put in `mp_init()` to make it consistent across all ports.

Signed-off-by: Damien George <damien@micropython.org>
2025-03-06 12:52:35 +11:00
Damien George
d4b8ca2ffc extmod/vfs: Add mp_vfs_mount_romfs_protected() helper.
This function will attempt to create a `VfsRom` instance and mount it at
location "/rom" in the filesystem.

Signed-off-by: Damien George <damien@micropython.org>
2025-03-06 12:52:35 +11:00
Damien George
89e6c58c80 extmod/modvfs: Add vfs.rom_ioctl function and its ioctl constants.
This is a generic interface to allow querying and modifying the read-only
memory area of a device, if it has such an area.

Signed-off-by: Damien George <damien@micropython.org>
2025-03-06 12:52:35 +11:00
Damien George
9dd4cef814 py/objarray: Add MP_DEFINE_MEMORYVIEW_OBJ convenience macro.
This allows defining a `memoryview` instance, either statically or on the
C stack.

Signed-off-by: Damien George <damien@micropython.org>
2025-03-06 12:52:35 +11:00
Volodymyr Shymanskyy
fc71f7832f py/makeqstrdata.py: Implement MicroPython compatibility.
This allows running `py/makeqstrdata.py` with MicroPython itself.

Signed-off-by: Volodymyr Shymanskyy <vshymanskyi@gmail.com>
Signed-off-by: Angus Gratton <angus@redyak.com.au>
2025-03-05 16:00:32 +11:00
Damien George
f5b4545761 py/modsys: Add sys.implementation._build entry.
For a given MicroPython firmware/executable it can be sometimes important
to know how it was built, which variant/board configuration it came from.

This commit adds a new field `sys.implementation._build` that can help
identify the configuration that MicroPython was built with.

For now it's either:
* <VARIANT> for unix, webassembly and windows ports
* <BOARD>-<VARIANT> for microcontroller ports (the variant is optional)

In the future additional elements may be added to this string, separated by
a hyphen.

Resolves issue #16498.

Signed-off-by: Damien George <damien@micropython.org>
2025-03-05 12:23:40 +11:00
Glenn Moloney
eb45d97898 py/objstr: Support tuples and start/end args in startswith and endswith.
This change allows tuples to be passed as the prefix/suffix argument to the
`str.startswith()` and `str.endswith()` methods.  The methods will return
`True` if the string starts/ends with any of the prefixes/suffixes in the
tuple.

Also adds full support for the `start` and `end` arguments to both methods
for compatibility with CPython.

Tests have been updated for the new behaviour.

Signed-off-by: Glenn Moloney <glenn.moloney@gmail.com>
2025-03-02 22:15:31 +11:00
Yoctopuce dev
5fdd249c55 py/parsenum: Reduce code footprint of mp_parse_num_float.
The mantissa parsing code uses a floating point variable to accumulate
digits.  Using an `mp_float_uint_t` variable instead and casting to
`mp_float_t` at the very end reduces code size.  In some cases, it also
improves the rounding behaviour as extra digits are taken into account
by the int-to-float conversion code.

An extra test case handles the special case where mantissa overflow occurs
while processing deferred trailing zeros.

Signed-off-by: Yoctopuce dev <dev@yoctopuce.com>
2025-02-28 13:35:12 +11:00
Alessandro Gatti
50fab08e6b py/emitinlinextensa: Simplify register name lookup.
This commit changes the Xtensa inline assembly parser to use a slightly
simpler (and probably a tiny bit more efficient) way to look up register
names when decoding instruction parameters.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2025-02-28 13:21:44 +11:00
Alessandro Gatti
8633abc082 py/emitinlinerv32: Reduce the footprint of compiled code.
This commit introduces a few changes aimed at reducing the amount of
space taken by the inline assembler once compiled:

* The register string table uses 2 bytes for each qstr rather than the
  usual 4
* The opcode table uses 2 bytes for each qstr rather than the usual 4
* Opcode masks are not embedded in each opcode entry but looked up via
  an additional smaller table, reducing the number of bytes taken by
  an opcode's masks from 12 to 2 (with a fixed overhead of 24 bytes for
  the the masks themselves stored elsewhere)
* Some error messages had a trailing period, now removed
* Error messages have been parameterised when possible, and the overall
  text length is smaller.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2025-02-27 18:42:28 +01:00
Alessandro Gatti
dc2c33b07f py/emitinlinerv32: Fix compilation with ESP-IDF v5.2 and later.
This commit fixes a compilation warning (turned error) about a
potentially uninitialised variable being used.  The warning can be
ignored as the variable in question is always written to, but the code
has been changed to silence that warning.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2025-02-27 12:06:09 +01:00
Christian Clauss
5e206fdeb5 all: Upgrade codespell to v2.4.1.
This commit upgrades from codespell==2.2.6 to the current codespell==2.4.1,
adding emac to the ignore-words-list.

Signed-off-by: Christian Clauss <cclauss@me.com>
2025-02-25 16:11:33 +11:00
Angus Gratton
516709be88 py/mkrules.cmake: Support passing CFLAGS_EXTRA in environment variable.
This works similarly to the existing support in "bare metal" make ports,
with the caveat that CMake will only set this value on a clean build and
will reuse the previous value otherwise.

This is slightly different to the CMake built-in support for CFLAGS,
as this variable is used when evaluating source files for qstr
generation, etc.

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
2025-02-18 12:09:23 +11:00
Damien George
c3a18d74eb extmod/modmarshal: Add new marshal module.
This commit implements a small subset of the CPython `marshal` module.  It
implements `marshal.dumps()` and `marshal.loads()`, but only supports
(un)marshalling code objects at this stage.  The semantics match CPython,
except that the actual marshalled bytes is not compatible with CPython's
marshalled bytes.

The module is enabled at the everything level (only on the unix coverage
build at this stage).

Signed-off-by: Damien George <damien@micropython.org>
2025-02-11 16:54:20 +11:00
Damien George
a11ba7775e py/persistentcode: Add mp_raw_code_save_fun_to_bytes.
Serialises a bytecode function/generator to a valid .mpy as bytes.

Signed-off-by: Damien George <damien@micropython.org>
2025-02-11 16:54:02 +11:00
Damien George
ceb8ba60b4 py/objfun: Implement function.__code__ and function constructor.
This allows retrieving the code object of a function using
`function.__code__`, and then reconstructing a function from a code object
using `FunctionType(code_object)`.

This feature is controlled by `MICROPY_PY_FUNCTION_ATTRS_CODE` and is
enabled at the full-features level.

Signed-off-by: Damien George <damien@micropython.org>
2025-02-11 16:51:50 +11:00
Damien George
62e821ccb8 py/objcode: Factor code object out into its own file.
The `mp_obj_code_t` and `mp_type_code` code object was defined internally
in both `py/builtinevex.c` and `py/profile.c`, with completely different
implementations (the former very minimal, the latter quite complete).

This commit factors these implementations into a new, separate source file,
and allows the code object to have four different modes, selected at
compile-time:

- MICROPY_PY_BUILTINS_CODE_NONE: code object not included in the build.

- MICROPY_PY_BUILTINS_CODE_MINIMUM: very simple code object that just holds
  a reference to the function that it represents.  This level is used when
  MICROPY_PY_BUILTINS_COMPILE is enabled.

- MICROPY_PY_BUILTINS_CODE_BASIC: simple code object that holds a reference
  to the proto-function and its constants.

- MICROPY_PY_BUILTINS_CODE_FULL: almost complete implementation of the code
  object.  This level is used when MICROPY_PY_SYS_SETTRACE is enabled.

Signed-off-by: Damien George <damien@micropython.org>
2025-02-11 16:42:14 +11:00
Alessandro Gatti
44a7731669 py/emitnative: Load and store words just once for Viper code.
This commit fixes two Xtensa sequences in order to terminate early when
loading and storing word values via an immediate index.

This was meant to be part of 55ca3fd675
but whilst it was part of the code being tested, it didn't end up in the
commit.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2025-02-07 17:45:10 +11:00
Alessandro Gatti
e37d498cc0 py/emitnative: Mark condition code tables as const.
This commit marks as const the condition code tables used when figuring
out which opcode sequence must be emitted depending on the requested
comparison type.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2025-02-07 17:44:38 +11:00
Andrew Leech
b603fa38b2 py/mkrules.mk: Reset USER_C_MODULES when building mpy-cross dependency.
When a port automatically compiles `mpy-cross`, if `USER_C_MODULES` is
provided by the user on the command line then it is also applied to the
`mpy-cross` build.  That can lead to build errors if the path is relative
and not found when building `mpy-cross`.

Fix that by explicitly resetting `USER_C_MODULES` when invoking the
`mpy-cross` build.

Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
2025-02-07 17:24:43 +11:00
Angus Gratton
990f50fbb8 py/gc: Reorder static functions for clarity.
- Renamed gc_sweep to gc_sweep_free_blocks.
- Call gc_sweep_run_finalisers from top level.
- Reordered the gc static functions to be in approximate
  runtime sequence (with forward declarations) rather than
  in declaration order.

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
2025-02-03 15:02:02 +11:00
Angus Gratton
4bcbe88e74 py: Add optional support for recursive mutexes, use for gc mutex.
Enabled by default if using threading and no GIL

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
2025-02-03 15:02:02 +11:00
Angus Gratton
40e1c111e1 py/gc: Allow gc_free from inside a gc_sweep finalizer.
Do this by tracking being inside gc collection with a
separate flag, GC_COLLECT_FLAG. In gc_free(),
ignore this flag when determining if the heap is locked.

* For finalisers calling gc_free() when heap is otherwise unlocked,
  this allows memory to be immediately freed (potentially
  avoiding a MemoryError).

* Hard IRQs still can't call gc_free(), as heap will be locked via
  gc_lock().

* If finalisers are disabled then all of this code can be compiled
  out to save some code size.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
2025-02-03 15:02:02 +11:00
Angus Gratton
8a2ff2ca73 py/gc: Split out running finalizers to a separate pass.
Currently a finalizer may run and access memory which has already been
freed. (This happens mostly during gc_sweep_all() but could happen during
any garbage collection pass.)

Includes some speed improvement tweaks to skip empty FTB blocks. These help
compensate for the inherent slowdown of having to walk the heap twice.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
2025-02-03 15:02:02 +11:00
bfb1bee6fe py/parsenumbase: Favor clarity of code over manual optimisation.
Follow up to 13b13d1fdd, based on some
testing on godbolt, the manual code optimisation seems unnecessary for code
size, at least on gcc x86_64 and ARM, and it's definitely not good for
clarity.

Signed-off-by: Jeff Epler <jepler@gmail.com>
2025-01-29 12:39:05 +11:00
Angus Gratton
22353e9e1e py/mkrules: Add GIT_SUBMODULES_FAIL_IF_EMPTY flag for CMake ports.
The way CMake gathers the submodule list, it can quietly be empty
if the previous step fails. This makes it an explicit error.

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
2025-01-29 11:41:32 +11:00
Alessandro Gatti
55ca3fd675 py/emitnative: Optimise Viper immediate offset load/stores on Xtensa.
This commit introduces the ability to emit optimised code paths on
Xtensa for load/store operations indexed via an immediate offset.

If an immediate offset for a load/store operation is within a certain
range that allows it to be embedded into an available opcode then said
opcode is emitted instead of the generic code sequence.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2025-01-26 23:42:36 +11:00
Alessandro Gatti
40585eaa8f py/emitnative: Emit shorter exception handler entry code on RV32.
This commit improves the RV32 code sequence that is emitted if a
function needs to set up an exception handler as its prologue.

The old code would clear a temporary register and then copy that value
to places that needed to be initialised with zero values.  On RV32
there's a dedicated register that's hardwired to be equal to zero, which
allows us to bypass the extra register clear and use the zero register
to initialise values.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2025-01-26 23:42:36 +11:00
Alessandro Gatti
bfc0d7b0b9 py/emitnative: Optimise Viper register offset load/stores on Xtensa.
This commit improves the emitted code sequences for address generation
in the Viper subsystem when loading/storing 16 and 32 bit values via a
register offset.

The Xtensa opcodes ADDX2 and ADDX4 are used to avoid performing the
extra shifts to align the final operation offset.  Those opcodes are
available on both xtensa and xtensawin MicroPython architectures.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2025-01-26 23:42:36 +11:00
13b13d1fdd py/parsenum: Throw an exception for invalid int literals like "01".
This includes making int("01") parse in base 10 like standard Python.
When a base of 0 is specified it means auto-detect based on the prefix, and
literals begining with 0 (except when the literal is all 0's) like "01" are
then invalid and now throw an exception.

The new error message is different from CPython. It says e.g.,
`SyntaxError: invalid syntax for integer with base 0: '09'`

Additional test cases were added to cover the changed & added code.

Co-authored-by: Damien George <damien@micropython.org>
Signed-off-by: Jeff Epler <jepler@gmail.com>
2025-01-26 22:54:58 +11:00
IhorNehrutsa
a4ab847688 py/persistentcode: Initialize prelude_ptr to prevent compiler warning.
The esp32 IDF toolchain can give a "may be used uninitialized" warning, at
least for ESP32-S3 with gcc 14.2.0.  Silence that warning by initializing
the variable with NULL.

Co-authored-by: Daniel van de Giessen <daniel@dvdgiessen.nl>
Signed-off-by: IhorNehrutsa <Ihor.Nehrutsa@gmail.com>
2025-01-24 23:17:05 +11:00
Damien George
6db29978ac py/mkrules.mk: Move comment about partial clones outside make rule.
Otherwise the comment is printed each time the rule is run.

Follow up to fdd606dd53.

Signed-off-by: Damien George <damien@micropython.org>
2025-01-17 16:58:54 +11:00
Alessandro Gatti
c610199f2d py/asmarm: Fix halfword loads with larger offsets.
This commit fixes code generation for loading halfwords using an offset
greater than 255.

The old code blindly encoded the offset into a `LDRH Rd, [Rn, #imm]`
opcode, but only the lowest 8 bits would be put into the opcode itself.
This commit instead generates a two-opcodes sequence, a constant load into
R8, and then `LDRH Rd, [Rn, R8]`.

This fixes `tests/extmod/vfs_rom.py` for the qemu/SABRELITE board.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2025-01-16 12:52:32 +11:00
Alessandro Gatti
928c71638c py/asmarm: Fix locals address loading code generation with large imm.
This commit fixes code generation for loading a local's address if its
index is greater than 63.

The old code blindly encoded the offset into an `ADD Rd, Rn, #imm` opcode,
but only the lowest 8 bits would be put into the opcode itself.  This
commit instead generates a two-opcodes sequence, a constant load into R8,
and then an `ADD Rd, Rn, R8` opcode.

This fixes `tests/float/math_domain.py` for the qemu/SABRELITE board.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2025-01-16 12:51:51 +11:00
Damien George
510e055c71 py/asmarm: Allow function state to be larger than 255.
Co-authored-by: Alessandro Gatti <a.gatti@frob.it>
Signed-off-by: Damien George <damien@micropython.org>
2025-01-06 17:55:11 +11:00
Damien George
6a90e513de py/asmarm: Fix asm_arm_ldrh_reg_reg_offset to emit correct machine code.
Prior to this fix, the assembler generated `LDRH Rd, [Rn, #imm]!`, so the
second `LDRH` from the same origin would load from the wrong base.

Co-authored-by: Alessandro Gatti <a.gatti@frob.it>
Signed-off-by: Damien George <damien@micropython.org>
2025-01-06 17:55:03 +11:00
Yoctopuce
5d12df51fc py/obj: Make literals unsigned in float get/new functions.
Fixes gcc warning when -Wsign-conversion is on.

Signed-off-by: Yoctopuce <dev@yoctopuce.com>
2025-01-02 13:36:33 +11:00
Yoctopuce
9bb6b50693 py/obj: Cast float literals to 64-bit to prevent overflow warning.
Fixes compilation warning C4307: '+': integral constant overflow.

Signed-off-by: Yoctopuce <dev@yoctopuce.com>
2025-01-02 13:33:39 +11:00
Alessandro Gatti
268acb714d py/emitinlinerv32: Add inline assembler support for RV32.
This commit adds support for writing inline assembler functions when
targeting a RV32IMC processor.

Given that this takes up a bit of rodata space due to its large
instruction decoding table and its extensive error messages, it is
enabled by default only on offline targets such as mpy-cross and the
qemu port.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2025-01-02 11:49:10 +11:00
Alessandro Gatti
3044233ea3 py/misc: Add a popcount(uint32_t) implementation.
This makes the existing popcount(uint32_t) implementation found in the
RV32 emitter available to the rest of the codebase.  This version of
popcount will use intrinsic or builtin implementations if they are
available, falling back to a generic implementation if that is not the
case.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2025-01-01 10:44:53 +01:00
Damien George
97518064cd py/persistentcode: Add support for loading .mpy files from a ROM reader.
This adds an optimisation for loading .mpy files from a reader that points
to ROM.  In such a case qstr, str and bytes data, along with bytecode, are
all referenced in-place in ROM.

Signed-off-by: Damien George <damien@micropython.org>
2024-12-23 13:04:55 +11:00
Damien George
f870e8d2d4 py/reader: Provide mp_reader_try_read_rom() function.
This allows accessing data directly in ROM if the reader supports it.

Signed-off-by: Damien George <damien@micropython.org>
2024-12-23 13:04:54 +11:00
Damien George
a1c32101ac py/qstr: Add qstr_from_strn_static() helper function.
Allows an interned string to reference static/ROM data, instead of
allocating it on the GC heap.

Signed-off-by: Damien George <damien@micropython.org>
2024-12-23 13:04:54 +11:00
Damien George
50637ff239 extmod/vfs_rom: Add VfsRom filesystem object.
This commit defines a new ROMFS filesystem for storing read-only files that
can be memory mapped, and a new VfsRom driver.  Files opened from this
filesystem support the buffer protocol.  This allows naturally getting the
memory-mapped address of the file using:
- memoryview(file)
- uctypes.addressof(file)

Furthermore, if these files are .mpy files then their content can be
referenced in-place when importing.  Such imports take up a lot less RAM
than importing from a normal filesystem.  This is essentially dynamically
frozen .mpy files, building on the revamped v6 .mpy file format.

Signed-off-by: Damien George <damien@micropython.org>
2024-12-23 13:04:54 +11:00
Alessandro Gatti
4bf087b272 py/dynruntime.mk: Delete compiled module file on clean.
This commit adds the compiled native module file to the list of files to
remove when `make clean` is issued in a native module source directory.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2024-12-23 10:03:02 +11:00
Alessandro Gatti
6760e00817 tools/mpy_ld.py: Add native modules support for RV32 code.
This commit adds support for RV32IMC native modules, as in embedding native
code into a self-contained MPY module and and make its exported functions
available to the MicroPython environment.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2024-12-23 10:02:20 +11:00
Damien George
8b6bd43eab extmod/vfs: Guard mutating fs functions with MICROPY_VFS_WRITABLE.
Enabled by default.  Useful for ports that need the VFS but don't have any
writable filesystems.

Signed-off-by: Damien George <damien@micropython.org>
2024-12-20 21:58:50 +11:00
Andrew Leech
fdd606dd53 py/mkrules.mk: Use partial clone for submodules if available.
MicroPython relies on a number of submodules for third party and chip
vendor libraries.  Users need to check these out before building their
desired ports and Github Actions CI here needs to clone them all multiple
times for every build.  Many of these are getting significantly larger over
time, slowing down usage and consuming more disk space.

Newer versions of git have features to avoid pulling all historic / blob
data which can have a significant impact of total data use.  This commit
uses a standard feature of git to do a partial clone, with automatic
fallback to previous behavior on error.

Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
2024-12-10 17:17:17 +11:00
Alessandro Gatti
2a8f6047ff py/emitglue: Fix clear cache builtin warning on Clang for AArch32.
This commit fixes a warning occurring on Clang when calling
`__builtin___clear_cache` with non-void pointers for its start and end
memory area locations.  The code now uses a char pointer for the end
location, and it still builds without warnings on GCC.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2024-12-10 12:04:14 +11:00
stijn
0f7d68043f py/misc: Fix msvc and C++ compatibility.
Use an explicit cast to suppress the implicit conversion which started
popping up in recent compiler versions (and wasn't there yet in 07bf3179).

Signed-off-by: stijn <stijn@ignitron.net>
2024-12-10 11:30:59 +11:00
Angus Gratton
160a4812cd esp32: Pass V=1 or BUILD_VERBOSE through to idf.py when building.
Allows verbose build to work the same on esp32 port as other ports.

To minimise copy/paste, split the BUILD_VERBOSE section of mkenv.mk
out to its own verbose.mk and include this in the port Makefile.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
2024-12-10 11:28:38 +11:00
Angus Gratton
3b3b48892f py/objfloat: Workaround non-constant NAN definition on Windows MSVC.
Recent MSVC versions have changed the definition of NAN to a non-constant
expression!  This is a bug, C standard says it should be a constant.

Good explanation and workaround at: https://stackoverflow.com/a/79199887

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
2024-11-28 23:11:24 +11:00
Andrew Leech
78d017fc4e py/usermod.cmake: If USER_C_MODULES is a folder add micropython.cmake.
This mirrors how it works when using a Makefile.

Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
2024-11-20 17:38:39 +11:00
Andrew Leech
dccd206f4c py/usermod.cmake: Add check that any specified USER_C_MODULES exists.
Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
2024-11-20 17:34:41 +11:00
Andrew Leech
b65e89107c py/py.mk: Add check that any specified USER_C_MODULES folder exists.
Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
2024-11-20 17:34:41 +11:00
Jan Sturm
7e1098befe py/objdeque: Fix buffer overflow in deque_subscr.
In `deque_subscr()`, if `index_val` equals `self->alloc`, the index
correction `index_val -= self->alloc` does not execute, leading to an
out-of-bounds access in `self->items[index_val]`.

The fix in this commit ensures that the index correction is applied
whenever `index_val >= self->alloc`, preventing access beyond the allocated
buffer size.

Signed-off-by: Jan Sturm <jansturm92@googlemail.com>
2024-11-04 11:21:56 +11:00
Alessandro Gatti
d34b15ac6f pic16bit: Make it build with recent XC16 versions.
The PIC16 port didn't catch up with the other ports, so it required a bit
of work to make it build with the latest version of XC16.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2024-10-30 17:21:30 +11:00
Damien George
44dc6eb30d all: Bump version to 1.25.0-preview.
Signed-off-by: Damien George <damien@micropython.org>
2024-10-28 11:08:24 +11:00
Damien George
f212bbe837 all: Bump version to 1.24.0.
Signed-off-by: Damien George <damien@micropython.org>
2024-10-26 01:43:23 +11:00
iabdalkader
2644f577f1 extmod/modtls_mbedtls: Add a thread-global ptr for current SSL context.
This is necessary for mbedTLS callbacks that do not carry any user state,
so those callbacks can be customised per SSL context.

Signed-off-by: iabdalkader <i.abdalkader@gmail.com>
2024-10-25 01:12:36 +11:00
Damien George
1b89c503db py/objtype: Don't delegate lookup of descriptor methods to __getattr__.
When descriptors are enabled, lookup of the `__get__`, `__set__` and
`__delete__` descriptor methods should not be delegated to `__getattr__`.
That follows CPython behaviour.

Signed-off-by: Damien George <damien@micropython.org>
2024-10-16 15:03:30 +11:00
Phil Howard
525fce7170 py/usermod.cmake: Check target exists in usermod_gather_sources.
Check a target exists before accessing properties.  Otherwise
usermod_gather_sources would recurse into garbage property names and break.

Signed-off-by: Phil Howard <phil@gadgetoid.com>
2024-10-15 11:51:56 +11:00
stijn
338df1ae35 py/objtype: Allow passing keyword arguments to native base __init__.
Allowing passing keyword arguments to a native base's __init__, i.e.
`make_new` in the C code.  Previously only positional arguments were
allowed.

The main trade-off in this commit is that every call to the native base's
`make_new` is now going to be preceded by a call to
`mp_map_init_fixed_table` even though most of what that does is unused and
instead it merely serves as a way to pass the number of keyword arguments.

Fixes issue #15465.

Signed-off-by: stijn <stijn@ignitron.net>
2024-10-07 11:37:52 +11:00
Alessandro Gatti
ca220b5fc5 py/nlrrv64: Add RISC-V RV64I NLR implementation.
Add custom NLR support for 64 bits RISC-V RV64I targets.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2024-10-03 12:06:19 +10:00
Alessandro Gatti
17d8234458 py/parse: Remove old esp32 compiler workaround.
The ESP32 port contains a workaround to avoid having a certain function
in `py/parse.c` being generated incorrectly.  The compiler in question
is not part of any currently supported version of ESP-IDF anymore, and the
problem inside the compiler (well, assembler in this case) has been
corrected a few years ago.

This commit removes all traces of that workaround from the source tree.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2024-09-27 00:03:16 +10:00
Alessandro Gatti
43b05afdf5 py/mpz: Skip separators when running out of digits to print.
This commit fixes the addition of a stray separator before the number
when printing an MPZ-backed integer and the first group is three digits
long.

This fixes #8984.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2024-09-26 23:39:06 +10:00
Damien George
5b22bde044 py/persistentcode: Explicitly track native BSS/rodata when needed.
Signed-off-by: Damien George <damien@micropython.org>
2024-09-26 22:15:28 +10:00
Andrew Leech
7e14680a83 py/objringio: Add micropython.RingIO() interface for general use.
This commit adds a new `RingIO` type which exposes the internal ring-buffer
code for general use in Python programs.  It has the stream interface
making it similar to `StringIO` and `BytesIO`, except `RingIO` has a fixed
buffer size and is automatically safe when reads and writes are in
different threads or an IRQ.

This new type is enabled at the "extra features" ROM level.

Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
2024-09-19 18:00:44 +10:00
Angus Gratton
52a593cdb1 py/scheduler: Only run callbacks on the main thread if GIL is disabled.
Otherwise it's very difficult to reason about thread safety in a
scheduler callback, as it can run at any time on any thread - including
racing against any bytecode operation on any thread.

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
2024-09-19 13:17:01 +10:00
Amirreza Hamzavi
cb7e99098e py/objint: Make byteorder argument optional in int.from_bytes() method.
This was made optional in CPython 3.11.

Signed-off-by: Amirreza Hamzavi <amirrezahamzavi2000@gmail.com>
2024-09-02 14:15:45 +10:00
Amirreza Hamzavi
0b432b3306 py/objint: Make length argument optional in int.to_bytes() method.
This was made optional in CPython 3.11.

Signed-off-by: Amirreza Hamzavi <amirrezahamzavi2000@gmail.com>
2024-09-02 14:14:36 +10:00
Amirreza Hamzavi
80c5e76483 py/objint: Make byteorder argument optional in int.to_bytes() method.
This was made optional in CPython 3.11.

Signed-off-by: Amirreza Hamzavi <amirrezahamzavi2000@gmail.com>
2024-09-02 14:14:22 +10:00
Christian Walther
0b7f6e1d3d py/mkrules.mk: Fix 'make submodules' when building out-of-tree.
When MicroPython is used as a submodule and built from the containing
project, e.g. for the embed port, `make submodules` fails because it goes
looking for the sub-sub-module paths in the outer repository instead of in
the micropython repository. Fix this by invoking git inside the micropython
submodule.

Signed-off-by: Christian Walther <cwalther@gmx.ch>
2024-08-26 12:30:24 +10:00
Alessandro Gatti
7d8b2d89cc py/asmrv32: Use REG_TEMP2 whenever possible.
The RV32 emitter used an additional temporary register, as certain code
sequences required extra storage.  This commit removes its usage in all
but one case, using REG_TEMP2 instead.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2024-08-19 15:53:50 +10:00
Alessandro Gatti
da0e027fa5 py/asmrv32: Emit C.LW opcodes only when necessary.
The RV32 emitter sometimes generated short load opcodes even when it
was not supposed to.  This commit fixes an off-by-one error in its
offset eligibility range calculation and corrects one case of offset
calculation, operating on the raw label index number rather than its
effective offset in the stack (C.LW assumes all loads are
word-aligned).

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2024-08-19 15:53:50 +10:00
Alessandro Gatti
326e1149ec py/asmrv32: Fix short/long jumps scheduling.
The RV32 emitter always scheduled short jumps even outside the emit
compiler pass.  Running the full test suite through the native emitter
instead of just the tests that depend on the emitter at runtime (as in,
`micropython/native_*` and `micropython/viper_* tests`) uncovered more
places where the invalid behaviour was still present.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2024-08-19 15:53:50 +10:00
Glenn Moloney
6367099f83 py/objstr: Skip whitespace in bytes.fromhex().
Skip whitespace characters between pairs of hex numbers.
This makes `bytes.fromhex()` compatible with cpython.

Includes simple test in `tests/basic/builtin_str_hex.py`.

Signed-off-by: Glenn Moloney <glenn.moloney@gmail.com>
2024-08-19 14:18:34 +10:00
Angus Gratton
86f2c285eb py: Add new cstack API for stack checking, with limit margin macro.
Currently the stack limit margin is hard-coded in each port's call to
`mp_stack_set_limit()`, but on threaded ports it's fiddlier and can lead to
bugs (such as incorrect thread stack margin on esp32).

This commit provides a new API to initialise the C Stack in one function
call, with a config macro to set the margin.  Where possible the new call
is inlined to reduce code size in thread-free ports.

Intended replacement for `MP_TASK_STACK_LIMIT_MARGIN` on esp32.

The previous `stackctrl.h` API is still present and unmodified apart from a
deprecation comment.  However it's not available when the
`MICROPY_PREVIEW_VERSION_2` macro is set.

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
2024-08-14 12:55:45 +10:00
Angus Gratton
6c870dc5ec py/obj: Remove the legacy object API for version 2.
These were changed in v1.11 (2019).  Prepare to remove the compatibility
macros as part of V2 changes.

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
2024-08-14 10:33:52 +10:00
Angus Gratton
b0c89377d0 py/modmath: Add option to work around -inf bug in a port's tgamma.
This is needed for a workaround on esp32 port (in child commit),
which produces incorrect results otherwise.

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
2024-08-07 14:10:46 +10:00
Damien George
afba3e0540 py/emitnative: Fix case of clobbered REG_TEMP0 when loading const obj.
The `emit_load_reg_with_object()` helper function will clobber `REG_TEMP0`.
This is currently OK on architectures where `REG_RET` and `REG_TEMP0` are
the same (all architectures except RV32), because all callers of
`emit_load_reg_with_object()` use either `REG_RET` or `REG_TEMP0` as the
destination register.  But on RV32 these registers are different and so
when `REG_RET` is the destination, `REG_TEMP0` is clobbered, leading to
incorrectly generated machine code.

This commit fixes the issue simply by using `REG_TEMP0` as the destination
register for all uses of `emit_load_reg_with_object()`, and adds a comment
to make sure the caller of this function is careful.

Signed-off-by: Damien George <damien@micropython.org>
2024-08-07 12:25:21 +10:00
George Hopkins
066243ea74 py/py.mk: Add SRC_USERMOD_LIB_ASM to include assembly files.
Introduce SRC_USERMOD_LIB_ASM to allow users to include assembly files as
part of their user modules.  It could be used to include optimized
functions or outputs of other programming languages.

Signed-off-by: George Hopkins <george-hopkins@null.net>
2024-08-01 12:01:18 +10:00
Damien George
233f5ce661 py/runtime: Fix self arg passed to classmethod when accessed via super.
Thanks to @AJMansfield for the original test case.

Signed-off-by: Damien George <damien@micropython.org>
2024-07-25 13:07:42 +10:00
stijn
07bf3179f6 py/misc: Fix msvc and C++ compatibility.
Use explicit casts to suppress warnings about implicit conversions, add a
workaround for constant expression conditional, and make functions static
inline (as is done in the rest of the codebase) to suppress 'warning C4505:
unreferenced function with internal linkage has been removed'.

(Follow up to fix commit 908ab1ceca)

Signed-off-by: stijn <stijn@ignitron.net>
2024-07-25 12:55:23 +10:00
stijn
093d0c0a17 py/objtype: Validate super() arguments.
This fixes various null dereferencing and out-of-bounds access because
super_attr assumes the held obj is effectively an object of the held type,
which is now verified.

Fixes issue #12830.

Signed-off-by: stijn <stijn@ignitron.net>
2024-07-25 12:27:33 +10:00
Laurens Valk
9ca668f881 py/objtype: Avoid crash on calling members of uninitialized native type.
When subclassing a native type, calling native members in `__init__` before
`super().__init__()` has been called could cause a crash.  In this
situation, `self` in `mp_convert_member_lookup` is the
`native_base_init_wrapper_obj`.  The check added in this commit ensures
that an `AttributeError` is raised before this happens, which is consistent
with other failed lookups.

Also fix a typo in a related comment.

Signed-off-by: Laurens Valk <laurens@pybricks.com>
2024-07-25 12:01:43 +10:00
Damien George
5f3ecc29f8 extmod/modmachine: Use sys.exit as implementation of machine.soft_reset.
It does the same thing, raising `SystemExit`.

Signed-off-by: Damien George <damien@micropython.org>
2024-07-20 12:18:07 +10:00
Damien George
5147dc5de5 py/gc: Remove commented-out functions.
These are old, unused, and most of them no longer compile.  The `gc_test()`
function is superseded by the test suite.

Signed-off-by: Damien George <damien@micropython.org>
2024-07-19 10:36:30 +10:00
Damien George
1548132979 py/sequence: Remove unused len argument from mp_seq_extract_slice.
Also put this function inside the `MICROPY_PY_BUILTINS_SLICE` guard,
because it's only usable when that option is enabled.

Signed-off-by: Damien George <damien@micropython.org>
2024-07-18 12:51:29 +10:00
Damien George
96007e7de5 py/lexer: Add static assert that token enum values all fit in a byte.
Signed-off-by: Damien George <damien@micropython.org>
2024-07-18 12:44:44 +10:00
Damien George
e00d80d9e2 py: Remove 5 TODOs in emitbc, objrange and repl.
These TODOs don't need to be done:

- Calling functions with keyword arguments is less common than without
  them, so adding an extra byte overhead to all calls regardless of whether
  they use keywords or not would overall increase generated bytecode size.

- Restricting `range` objects to machine-sized ints has been adequate for
  a long time now, so no need to change that and make it more complicated
  and slower.

- Printing spaces in tab completion does not need to be optimised.

Signed-off-by: Damien George <damien@micropython.org>
2024-07-18 12:35:42 +10:00
robert-hh
ee1036023e extmod/machine_spi: Support firstbit=LSB for machine.SoftSPI.
Being able to send data out in LSB format can be useful, and having support
in the low-level driver is much better than requiring Python code to
reorder the bits before sending them / after receiving them.  In particular
if the hardware does not support the LSB format (eg RP2040) then one needs
to use the SoftSPI in LSB mode.

For this change a default definition of `MICROPY_PY_MACHINE_SPI_MSB/_LSB`
was added to `py/mpconfig.h`, making them available to all ports.  The
identical defines in `esp32/mpconfigport.h` were deleted.

Resolves issues #5340, #11404.

Signed-off-by: robert-hh <robert@hammelrath.com>
2024-07-12 22:59:31 +10:00
robert-hh
2be45dd682 extmod/modmachine: Allow more than one argument to machine.freq().
The limit is set by a `MICROPY_PY_MACHINE_FREQ_NUM_ARGS_MAX` define, which
defaults to 1 and is set for stm32 to 4.

For stm32 this fixes a regression introduced in commit
e1ec6af654 where the maximum number of
arguments was changed from 4 to 1.

Signed-off-by: robert-hh <robert@hammelrath.com>
2024-07-11 23:38:07 +10:00
Jon Foster
92484d8822 all: Use new mp_obj_new_str_from_cstr() function.
Use new function mp_obj_new_str_from_cstr() where appropriate.  It
simplifies the code, and makes it smaller too.

Signed-off-by: Jon Foster <jon@jon-foster.co.uk>
2024-07-04 15:55:03 +10:00
Jon Foster
289b2dd879 py/objstr: Add new mp_obj_new_str_from_cstr() helper function.
There were lots of places where this pattern was duplicated, to convert a
standard C string to a MicroPython string:

    x = mp_obj_new_str(s, strlen(s));

This commit provides a simpler method that removes this code duplication:

    x = mp_obj_new_str_from_cstr(s);

This gives clearer, and probably smaller, code.

Signed-off-by: Jon Foster <jon@jon-foster.co.uk>
2024-07-04 15:52:47 +10:00
Alessandro Gatti
bb6a4669b2 py/asmrv32: Do not use binary literals.
As per discussion in #15347, non-standard binary literals have been
removed in favour of their hexadecimal counterparts.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2024-07-01 22:44:02 +02:00
Jim Mussared
557d31ed2c py/objint: Try to convert big-int back to small-int after binary op.
Before this change, long/mpz ints propagated into all future calculations,
even if their value could fit in a small-int object.  With this change, the
result of a big-int binary op will now be converted to a small-int object
if the value fits in a small-int.

For example, a relatively common operation like `x = a * b // c` where
a,b,c all small ints would always result in a long/mpz int, even if it
didn't need to, and then this would impact all future calculations with
x.

This adds +24 bytes on PYBV11 but avoids heap allocations and potential
surprises (e.g. `big-big` is now a small `0`, and can safely be accessed
with MP_OBJ_SMALL_INT_VALUE).

Performance tests are unchanged on PYBV10, except for `bm_pidigits.py`
which makes heavy use of big-ints and gains about 8% in speed.

Unix coverage tests have been updated to cover mpz code that is now
unreachable by normal Python code (removing the unreachable code would lead
to some surprising gaps in the internal C functions and the functionality
may be needed in the future, so it is kept because it has minimal
overhead).

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2024-07-01 13:52:59 +10:00
Alessandro Gatti
0600e4f273 py/asmrv32: Make some code sequences smaller.
This commit changes a few code sequences to use more compressed opcodes
where possible.  The sequences in question are the ones that show up the
most in the test suite and require the least amount of code changes, namely
short offset loads from memory to RET/ARG registers, indirect calls through
the function table, register-based jumps, locals' offset calculation,
reg-is-null jumps, and register comparisons.

There are no speed losses or gains from these changes, but there is an
average 15-20% generated code size reduction.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2024-07-01 11:36:03 +10:00