Commit graph

821 commits

Author SHA1 Message Date
Damien George
51b821ce82 tools/verifygitlog.py: Disallow a leading slash in commit subject line.
Signed-off-by: Damien George <damien@micropython.org>
2025-05-15 13:01:41 +10:00
Damien George
61eedbbd11 tools/verifygitlog.py: Apply stricter rules on git subject line.
There is a bit of ambiguity as to how the prefix of the git subject line
should look like.  Eg `py/vm: ...` vs `py/vm.c: ...` (whether the extension
should be there or not).

This commit makes the existing CI check of the git commit message stricter,
by applying extra rules to the prefix, the bit before the : in the subject
line.  It now checks that the subject prefix:
- doesn't start with unwanted bits: ., /, ports/
- doesn't have an extension: .c, .h, .cpp, .js, .rst or .md

Full error messages are given when a rule does not pass.

This helps to reduce maintainer burden by applying stricter rules, to keep
the git commit history consistent.

Signed-off-by: Damien George <damien@micropython.org>
2025-05-12 13:47:03 +10:00
Damien George
e53f262a85 tools/mpremote: For mip install, use hash to skip files that exist.
When using `mip install`, if a file that needs to be downloaded already
exists locally, then the hash of that local file will be computed and if it
matches the known hash of the remote file it will not be downloaded.

Hashes in mip are guaranteed unique, so this change should never leave
stale files on the filesystem.

This behaviour follows that of the `mip` package in `micropython-lib`.

Signed-off-by: Damien George <damien@micropython.org>
2025-05-02 11:44:32 +10:00
Damien George
4117a2d9b5 tools/ci.sh: Update URL for xtensa-lx106-elf-standalone.tar.gz.
The https://github.com/jepler/esp-open-sdk repository has been removed, so
use the file hosted at micropython.org (it's the same file).

Signed-off-by: Damien George <damien@micropython.org>
2025-05-02 11:39:49 +10:00
Maureen Helm
3c8d1b13f5 zephyr: Upgrade to Zephyr v4.0.0.
Updates the Zephyr port build instructions. The CI is updated to use
Zephyr docker image 0.27.4, SDK 0.17.0 and the latest Zephyr release
tag.

Tested on max32690fthr and frdm_k64f.

Signed-off-by: Maureen Helm <maureen.helm@analog.com>
Signed-off-by: Detlev Zundel <dzu@member.fsf.org>
2025-04-29 23:00:37 +10:00
Jos Verlinde
6406afb1f3 tools/mpremote: Prevent deletion of /remote files via rm -r.
Removes the risk of inadvertently deleting files on the host by preventing
the deletion of files via `rm -r` on the `/remote` vfs mount point.

Fixes issue #17147.

Signed-off-by: Jos Verlinde <jos_verlinde@hotmail.com>
2025-04-26 16:07:44 +10:00
Jos Verlinde
37fe3f66c3 tools/mpremote/tests: Add test for rm -r on /remote vfs.
Signed-off-by: Jos Verlinde <jos_verlinde@hotmail.com>
2025-04-26 16:07:40 +10:00
Anson Mansfield
dc46cf15c1 tools/mpremote: Fix possibly-missing EOPNOTSUPP errno name.
Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
2025-04-26 15:51:49 +10:00
Anson Mansfield
805fe083a3 tools/mpremote: Refactor error handling to apply generally to any errno.
This rewrites the code that previously manually emitted and caught various
OSError subclasses with equivalent code that uses the errno name dictionary
to do this generically, and updates the exception handler in do_filesystem
to catch them in a similarly-generic fashion using os.strerror to retrieve
an appropriate error message text equivalent to the current messages.

Note that in the CPython environments where mpremote runs, the call to the
OSError constructor already returns an instance of the corresponding mapped
exception subtype.

Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
2025-04-26 15:50:00 +10:00
Anson Mansfield
cee0419021 tools/mpremote/tests: Add tests for errno behavior.
Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
2025-04-26 15:49:48 +10:00
Damien George
dcca3ff602 tools/mpremote: Use zlib.compressobj instead of zlib.compress.
Because the `wbits` parameter was only added to `zlib.compress` in
CPython 3.11.  Using `zlib.compressobj` makes the code compatible with much
older CPython versions.

Fixes issue #17140.

Signed-off-by: Damien George <damien@micropython.org>
2025-04-24 22:33:05 +10:00
Angus Gratton
f5cb9eb974 top: Bump Ruff version to v0.11.6.
Brings it into sync with a matching change to micropython-lib (which was
much older).  Includes one small automatic fix.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
2025-04-24 21:42:21 +10:00
Jos Verlinde
ef8282c717 docs/reference/mpremote: Update docs for mpremote rm -r.
Signed-off-by: Jos Verlinde <Jos_Verlinde@hotmail.com>
2025-04-09 10:51:48 +10:00
Jos Verlinde
72d4c40941 tools/mpremote/tests: Add tests for mpremote rm -r.
Signed-off-by: Jos Verlinde <Jos_Verlinde@hotmail.com>
2025-04-09 10:51:45 +10:00
Jos Verlinde
1aa9b3d94b tools/mpremote: Add recursive remove functionality to filesystem cmds.
mpremote now supports `mpremote rm -r`.

Addresses #9802 and #16845.

Signed-off-by: Jos Verlinde <Jos_Verlinde@hotmail.com>
2025-04-09 10:44:45 +10:00
iabdalkader
547207ddc8 github/workflows: Add Alif port to CI.
Signed-off-by: iabdalkader <i.abdalkader@gmail.com>
2025-04-09 00:22:33 +10:00
iabdalkader
9e9be83fd6 tools/mpremote: Allow .img for ROMFS file and validate ROMFS image.
Currently the tool allows writing an invalid ROMFS image, with a bad header
or images smaller than minimum size, and only checks the image extension.

This commit allows deploying a ROMFS with either a ".img" or ".romfs"
extension (in the future support may be added for other extensions that
have different semantics, eg a manifest), and validates the image header
before writing.

Signed-off-by: iabdalkader <i.abdalkader@gmail.com>
2025-04-08 00:27:55 +10:00
Angus Gratton
74a5bf94c1 tools/gen-cpydiff.py: Fail CPython diff generation if output matches.
Previously this information was recorded in a "status" field of the result,
but nothing ever parsed this result which led to non-differences not being
removed.

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
2025-04-07 16:10:05 +10:00
Damien George
e34412f0f4 tools/ci.sh: Manually install picotool for rp2 builds.
If picotool is not installed, it's fetched and built when compiling each
rp2 board.  And the "develop" branch of picotool is used instead of a
release.  Installing it manually using the "master" branch means the latest
released version is used (instead of a possibly unstable development
version), and also makes building each rp2 board a little faster.

Signed-off-by: Damien George <damien@micropython.org>
2025-04-07 13:53:38 +10:00
Jon Nordby
3805e65ed3 tools/mpy_ld.py: Give better error for unsupported ARM absolute relocs.
This is a known limitation, so better to give a clear warning than a
catch-all AssertionError.  Happens for example when trying to use
soft-float on ARCH=armv6m

Also give more details on the assertion for unknown relocations, such that
one can see which symbol it affects etc, to aid in debugging.

References issue #14430.

Signed-off-by: Jon Nordby <jononor@gmail.com>
2025-04-03 16:37:38 +11:00
Damien George
2db0c0225f tools/mpremote: Make mip install skip /rom*/lib directories.
If a ROMFS is mounted then "/rom/lib" is usually in `sys.path` before the
writable filesystem's "lib" entry.  The ROMFS directory cannot be installed
to, so skip it if found.

Signed-off-by: Damien George <damien@micropython.org>
2025-03-27 17:10:00 +11:00
Alessandro Gatti
31a008c6e2 tools/ci.sh: Do not assume the Python interpreter is called "python".
This commit removes the assumption made by the CI scripts that the
system-provided python executable is simply named "python".  The scripts
will now look for a binary called "python3" first, and then fall back to
"python" if that is not found.

Whilst this is currently the case for the CI environment, there are no
guarantees for this going forward.  For example minimal CI environments
set up by some developers, using the same base OS, have their python
executable called "python3".

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2025-03-17 13:04:53 +11:00
Alessandro Gatti
1a7cbac879 tools/ci.sh: Build Xtensa natmods as part of the CI process.
This commit expands the CI tests by checking whether the example native
modules are able to be built for the Xtensa architecture.

This was made possible by the changes to mpy_ld that allow symbol
resolution across standard compiler-provided libraries.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2025-03-17 13:04:53 +11:00
Volodymyr Shymanskyy
51976110e2 tools/mpy_ld.py: Allow linking static libraries.
This commit introduces an additional symbol resolution mechanism to the
natmod linking process.  This allows the build scripts to look for required
symbols into selected libraries that are provided by the compiler
installation (libgcc and libm at the moment).

For example, using soft-float code in natmods, whilst technically possible,
was not an easy process and required some additional work to pull it off.
With this addition all the manual (and error-prone) operations have been
automated and folded into `tools/mpy_ld.py`.

Both newlib and picolibc toolchains are supported, albeit the latter may
require a bit of extra configuration depending on the environment the build
process runs on.  Picolibc's soft-float functions aren't in libm - in fact
the shipped libm is nothing but a stub - but they are inside libc.  This is
usually not a problem as these changes cater for that configuration quirk,
but on certain compilers the include paths used to find libraries in may
not be updated to take Picolibc's library directory into account.  The bare
metal RISC-V compiler shipped with the CI OS image (GCC 10.2.0 on Ubuntu
22.04LTS) happens to exhibit this very problem.

To work around that for CI builds, the Picolibc libraries' path is
hardcoded in the Makefile directives used by the linker, but this can be
changed by setting the PICOLIBC_ROOT environment library when building
natmods.

Signed-off-by: Volodymyr Shymanskyy <vshymanskyi@gmail.com>
Co-authored-by: Alessandro Gatti <a.gatti@frob.it>
2025-03-17 13:03:27 +11:00
Damien George
0c98c60b68 tools/mpremote: Add romfs query, build and deploy commands.
These commands use the `vfs.rom_ioctl()` function to manage the ROM
partitions on a device, and create and deploy ROMFS images.

Signed-off-by: Damien George <damien@micropython.org>
2025-03-06 12:52:35 +11:00
Volodymyr Shymanskyy
6fba1e406b tools/mpy-tool.py: Support calling main() from an external script.
Signed-off-by: Volodymyr Shymanskyy <vshymanskyi@gmail.com>
Signed-off-by: Angus Gratton <angus@redyak.com.au>
2025-03-05 16:02:08 +11:00
Volodymyr Shymanskyy
e1b2f2e078 tools/mpy-tool.py: Add support for self-hosting of mpy-tool.
This allows running mpy-tool using MicroPython itself.

An appropriate test is added to CI to make sure it continues to work.

Signed-off-by: Volodymyr Shymanskyy <vshymanskyi@gmail.com>
Signed-off-by: Angus Gratton <angus@redyak.com.au>
2025-03-05 16:01:12 +11:00
Damien George
71c7c03e41 tools/mpremote/tests: Add test for RemoteFile.readline.
Signed-off-by: Damien George <damien@micropython.org>
2025-02-28 14:14:58 +11:00
Andrew Leech
86526e9c2b tools/mpremote: Optimise readline support in mount.
This significantly speeds up readline on files opened directly from an
mpremote mount.

Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
2025-02-28 14:14:39 +11:00
Christian Clauss
dc2fcfcc55 all: Upgrade to ruff v0.9.6.
Signed-off-by: Christian Clauss <cclauss@me.com>
2025-02-25 13:02:49 +01:00
Angus Gratton
8b1ed4473d github/workflows: Include the Python version in the ESP-IDF cache key.
Changing runner OS can change Python version, and ESP-IDF installs are
keyed on ESP-IDF and Python version together.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
2025-02-25 15:13:13 +11:00
Damien George
7e9d19e06b github/workflows: Stop using ubuntu-20.04.
For GitHub Actions, ubuntu-20.04 is deprecated and will be removed by 1st
April 2025.  See announcement at
https://github.com/actions/runner-images/issues/11101

This commit changes actions that use ubuntu-20.04 to a newer image.

Signed-off-by: Damien George <damien@micropython.org>
2025-02-25 15:11:43 +11:00
Glenn Moloney
2992e34956 tools/mpremote: Add support for relative urls in package.json files.
URLs in `package.json` may now be specified relative to the base URL of the
`package.json` file.

Relative URLs wil work for `package.json` files installed from the web as
well as local file paths.

Docs: update `docs/reference/packages.rst` to add documentation for:

- Installing packages from local filesystems (PR #12476); and
- Using relative URLs in the `package.json` file (PR #12477);
- Update the packaging example to encourage relative URLs as the default
  in `package.json`.

Add `tools/mpremote/tests/test_mip_local_install.sh` to test the
installation of a package from local files using relative URLs in the
`package.json`.

Signed-off-by: Glenn Moloney <glenn.moloney@gmail.com>
2025-02-24 13:29:40 +11:00
Angus Gratton
1034b17558 tools/ci.sh: Build the W5100S_EVB_PICO board with no threads.
Serves as a build test for a config we don't otherwise support.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
2025-02-20 00:57:45 +11:00
rufusclark
1a67d720c7 tools/pyboard.py: Make get_time use machine.RTC instead of pyb.RTC.
The current code evaluates `pyb.RTC().datetime()` resulting in a remote
side exception, as `pyb` is not defined on most ports (only stm32).

The code should evaluate `machine.RTC().datetime()` and hence return the
current time.

Signed-off-by: rufusclark <50201718+rufusclark@users.noreply.github.com>
Signed-off-by: Damien George <damien@micropython.org>
2025-02-12 12:53:46 +11:00
Glenn Moloney
11c9656fad tools/mpremote: Support mip install from package.json on local fs.
Add support for `mpremote mip install package.json` where `package.json` is
a json file on the local filesystem.

Without this, package json files can only be loaded from http, https,
github or gitlab URLs.

This is useful for testing `package.json` files for pacages in development
and for constructing one's own `package.json` files for Python packages
which are not yet available for installation using mip.

Signed-off-by: Glenn Moloney <glenn.moloney@gmail.com>
2025-02-10 15:59:58 +11:00
iabdalkader
752c1672af tools/ci.sh: Build MIMXRT1060_EVK with MSC enabled as part of mimxrt CI.
Signed-off-by: iabdalkader <i.abdalkader@gmail.com>
2025-02-10 12:53:45 +11:00
robert-hh
e574f68820 mimxrt: Add support for a UF2 bootloader.
Allowing to use e.g. the Adafruit bootloaders with MicroPython.  The .uf2
file is created in addition to the .bin and .hex files allowing to use the
latter ones without the bootloader for debugging and testing.

Changes:

- Set the location of the ISR Vector and .text segment to 0x6000C000 and
  0x6000C400.
- Reserve an area at the start of ITCM for a copy of the interrupt vector
  table and copy the table on reset to this place.
- Extend `machine.bootloader()` by setting the magic number to enable the
  bootloader on reset.
- Create a .uf2 file which skips the segments below 0x6000C000.

The bootloader has to be installed as a preparation step using the board
specific methods, but then the firmware's .uf2 file version can be
installed using the bootloader.  The bootloader can be invoked with:

- double reset
- calling machine.bootloader()
- Using the touch1200 method

Double reset is hard to achieve on MIMXRT boards, since there is no clean
reset pin.  Some MIMXRT boards provide it by switching the power.

Some boards are excluded from the .uf2 build:

- MIMXRT1050_EVK: The uf2 bootloader is built for the QSPI version of the
  board.  MicroPython supports the Hyperflash version.
- MIMXRT1176_EVK: No support for this board yet, but it should be possible.

Signed-off-by: robert-hh <robert@hammelrath.com>
2025-02-10 11:28:39 +11:00
Alessandro Gatti
f594c6f66e tools/ci.sh: Add natmod tests for QEMU/Arm.
This commit adds the natmod tests for the MPS2_AN385 board running
inside QEMU to the CI pipeline.  Now natmod tests capabilities are equal
between the Arm and RV32 platforms for the QEMU port.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2025-02-07 17:37:04 +11:00
Hans Maerki
03fe9c55ea tools/mpremote: Introduce timeout_overall for read_until().
And use it in `enter_raw_repl()`.  This prevents waiting forever for a
serial device that does not respond to the Ctrl-C/Ctrl-D/etc commands and
is constantly outputting data.

Signed-off-by: Hans Maerki <buhtig.hans.maerki@ergoinfo.ch>
2025-01-29 10:45:08 +11:00
Hans Maerki
0d46e45a1f tools/mpremote: Avoid initial blocking read in read_until().
If the target does not return any data then `read_until()` will block
indefinitely.  Fix this by making the initial read part of the general read
look, which always checks `inWaiting() > 0` before reading from the serial
device.

Also added the UART timeout to the constructor.  This is not currently used
but may be used as an additional safeguard.

Signed-off-by: Hans Maerki <buhtig.hans.maerki@ergoinfo.ch>
2025-01-27 10:20:40 +11:00
Angus Gratton
d89e71e6c0 tools/autobuild,esp32: Template the generation of esp32 port deploy.md.
Allows two source files (ports/esp32/boards/deploy.md and
deploy_nativeusb.md for boards with only native USB) for all esp32
installation steps, with templated chip name and flash offset inserted via
string formatting.

The new files add more text to explain the esptool.py port auto-detection,
remove the unnecessary -z feature (already enabled by default), and add
a bit of troubleshooting and port detection info.

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
2025-01-17 16:03:45 +11:00
Alessandro Gatti
0950f65ac4 tools/pyboard.py: Wait a bit before accessing the PTY serial port.
Some PTY targets, namely `NETDUINO2` and `MICROBIT` under Qemu, take a bit
more time to present a REPL than usual.  The pyboard tool is a bit too
impatient and would bail out before any of those targets had a chance to
respond to the raw REPL request.

Co-authored-by: Damien George <damien@micropython.org>
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2025-01-16 12:10:28 +11:00
Damien George
d533c9067a tools/autobuild: Don't allow a board to change its ID.
All board IDs are now the board directory name.

Signed-off-by: Damien George <damien@micropython.org>
2025-01-15 16:48:10 +11:00
Damien George
495ce91cae tools/ci.sh: Run test_full for qemu port CI.
Signed-off-by: Damien George <damien@micropython.org>
2025-01-06 17:57:48 +11:00
Alessandro Gatti
4729a89504 tools/verifygitlog.py: Show invalid commit subjects in quotes.
If a commit subject line has any trailing whitespace it won't match the
repository validation rules, and the line will show up as part of the
relevant error message.  However, since there's no quotation marks
around the offending text, the trailing whitespace may go unnoticed, and
given that the commit message is then discarded when the commit
operation is retried this can get fairly annoying.

This commit simply modifies the error output for invalid subject lines
to add quotation marks around the offending text, so trailing whitespace
is much easier to see.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2024-12-23 12:55:20 +11:00
Alessandro Gatti
7ca6e5eb68 qemu: Add test_natmod target for RV32 and use as part of CI pipeline.
This commit brings the natmod tests in the CI build process for the RV32
platform.  Not all example natmods are tested at the moment, as
`features` requires soft-float support, and `btree` needs thread-local
storage support in `mpy_ld.py` when built with the CI's toolchain.

Co-authored-by: Damien George <damien@micropython.org>
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2024-12-23 10:04:19 +11:00
Alessandro Gatti
6760e00817 tools/mpy_ld.py: Add native modules support for RV32 code.
This commit adds support for RV32IMC native modules, as in embedding native
code into a self-contained MPY module and and make its exported functions
available to the MicroPython environment.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2024-12-23 10:02:20 +11:00
Angus Gratton
a9945fc528 ci: Add caching of ccache for Zephyr.
Similar to the ESP32 builds, but needs additional step to pass the
ccache directory through to the Zephyr container.

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
2024-12-18 15:11:01 +11:00
Angus Gratton
db4b095644 ci: Pull the Zephyr CI docker image from GitHub container reg.
This image is 15GB so in theory this may be faster, although
in testing the improvement is either non-existent or marginal.

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
2024-12-18 15:11:01 +11:00
Angus Gratton
92a5ea51b4 ci: Cache Zephyr workspace installation.
Can save several minutes downloading the Zephyr docker image and/or cloning
repo from GitHub. Cache keyed on the Zephyr version, which AFAIK is the
only determinant for the workspace contents.

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
2024-12-18 15:11:01 +11:00
Damien George
3203e950fc tools/boardgen.py: Provide macro defns for number of cpu/board pins.
So a port can use them if needed to exclude the Pin.cpu/Pin.board objects.

Signed-off-by: Damien George <damien@micropython.org>
2024-12-17 23:53:24 +11:00
Damien George
ee8d8b3448 tools/ci.sh: Re-enable vfs_posix tests on unix qemu MIPS CI.
These work now that _FILE_OFFSET_BITS=64.

Signed-off-by: Damien George <damien@micropython.org>
2024-12-16 11:58:43 +11:00
Angus Gratton
d49c1e836c tools/ci.sh: Remove explicit macOS pkg-config install.
Reasons to remove this:

- GitHub's macOS runners install this package by default nowadays.

- Brew renamed this package to 'pkgconf' so installing the old name on top
  of the new package name has started failing.

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
2024-11-30 11:24:36 +11:00
Damien George
4fd5b72a8b tools/mpremote: Support trailing slash on dest for non-recursive copy.
This fixes a regression in db59e55fe7: prior
to that commit `mpremote` supported trailing slashes on the destination of
a normal (non-recursive) copy.

Add back support for that, with the semantics that a trailing slash
requires the destination to be an existing directory.

Also add a test for this.

Signed-off-by: Damien George <damien@micropython.org>
2024-11-13 11:51:35 +11:00
Damien George
3b6024a699 tools/mpremote: Add test for forced copy.
Signed-off-by: Damien George <damien@micropython.org>
2024-11-13 11:51:35 +11:00
Damien George
76e6c6345c tools/mpremote: Make sure stdout and stderr output appear in order.
mpremote error messages now go to stderr, so make sure stdout is flushed
before printing them.

Also update the test runner to capture error messages.

Signed-off-by: Damien George <damien@micropython.org>
2024-11-13 11:51:35 +11:00
Damien George
8978102f35 tests/run-tests.py: Change --target/--device options to --test-instance.
Previously to this commit, running the test suite on a bare-metal board
required specifying the target (really platform) and device, eg:

    $ ./run-tests.py --target pyboard --device /dev/ttyACM1

That's quite a lot to type, and you also need to know what the target
platform is, when a lot of the time you either don't care or it doesn't
matter.

This commit makes it easier to run the tests by replacing both of these
options with a single `--test-instance` (`-t` for short) option.  That
option specifies the executable/port/device to test.  Then the target
platform is automatically detected.

The `--test-instance` can be passed:
- "unix" (the default) to use the unix version of MicroPython
- "webassembly" to test the webassembly port
- anything else is considered a port/device to pass to Pyboard

There are also some shortcuts to specify a port/device, following
`mpremote`:
- a<n> is short for /dev/ttyACM<n>
- u<n> is short for /dev/ttyUSB<n>
- c<n> is short for COM<n>

For example:

    $ ./run-tests.py -t a1

Note that the default test instance is "unix" and so this commit does not
change the standard way to run tests on the unix port, by just doing
`./run-tests.py`.

As part of this change, the platform (and it's native architecture if it
supports importing native .mpy files) is show at the start of the test run.

Signed-off-by: Damien George <damien@micropython.org>
2024-11-04 12:47:47 +11:00
Angus Gratton
787c424cfc tools/ci.sh: Fix reference commit for code size comparison.
Previously the code size comparison was between the merge base (i.e. where
the PR branched), and the generated merge commit into master.  If the PR
branch was older than current master, this meant the size comparison could
incorrectly include changes already merged on master but missing from the
PR branch.

This commit changes it to compare the generated merge commit against
current master, i.e. the size impact if this PR was to be merged.

This commit also disables running the code size check on "push", it now
only runs on pull_request events.

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
2024-10-31 23:19:55 +11:00
Angus Gratton
9591b0a53c tools/ci.sh: Fix commit msg checking when PR branch HEAD behind master.
Fixes the problem noted at
https://github.com/micropython/micropython/pull/15547#issuecomment-2434479702
which is that, because default CI HEAD for a PR is a (generated) merge
commit into the master branch's current HEAD, then if the PR branch isn't
fully rebased then the commit check runs against commits from master as
well!

Also drops running this check on push, the pull_request event is triggered
by default on open and update ("synchronized" event), which probably covers
the cases where this check should run.

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
2024-10-31 23:16:25 +11:00
Glenn Moloney
9ea8d2a031 tools/mpremote: Fix UnboundLocalError in Transport.fs_writefile().
The variable `written` was being used before it was defined in the
`fs_writefile()` method of the Transport class.  This was causing an
`UnboundLocalError` to be raised when the `progress_callback` was not
provided.

Fixes issue #16084.

Signed-off-by: Glenn Moloney <glenn.moloney@gmail.com>
2024-10-28 12:23:10 +11:00
Damien George
f9cebe676e tools/ci.sh: Add RPI_PICO2 to CI.
Signed-off-by: Damien George <damien@micropython.org>
2024-10-15 12:21:10 +11:00
Damien George
d92694c3e8 tools: Only issue a single Ctrl-C when entering raw REPL.
A long time ago when there was only the `stm` port, Ctrl-C would trigger a
preemptive NLR jump to break out of running code.  Then in commit
124df6f8d0 a more general approach to
asynchronous `KeyboardInterrupt` exceptions was implemented, and `stmhal`
supported both approaches, with the general (soft) interrupt taking
priority.

Then in commit bc1488a05f `pyboard.py` was
updated with a corresponding change to make it issue a double Ctrl-C to
break out of any existing code when entering the raw REPL (two Ctrl-C
characters were sent in order to more reliably trigger the preemptive NLR
jump).

No other port has preemptive NLR jumps and so a double Ctrl-C doesn't
really behave any differently to a single Ctrl-C: with USB CDC the double
Ctrl-C would most likely be in the same USB packet and so processed in the
same low-level USB callback, so it's just setting the keyboard interrupt
flag twice in a row.  The VM/runtime then just sees one keyboard interrupt
and acts as though only one Ctrl-C was sent.

This commit changes the double Ctrl-C to a single Ctrl-C in `pyboard.py`
and `mpremote`.  That keeps things as simple as they need to be.

Signed-off-by: Damien George <damien@micropython.org>
2024-10-15 11:40:08 +11:00
Damien George
a25b6b9c65 tools/mpremote: Add option to force copy.
This adds a -f/--force option to the "cp" command, which forces
unconditional copies, in particular does not check the hash.

Signed-off-by: Damien George <damien@micropython.org>
2024-10-09 16:39:06 +11:00
Jim Mussared
6461ffd9d1 tools/mpremote: Add initial regression tests for mpremote.
These tests are specifically for the command-line interface and cover:
 - resume/soft-reset/connect/disconnect
 - mount
 - fs cp,touch,mkdir,cat,sha256sum,rm,rmdir
 - eval/exec/run

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Signed-off-by: Damien George <damien@micropython.org>
2024-10-09 16:39:00 +11:00
Jim Mussared
dd6f78f014 tools/mpremote: Improve error output.
Makes the filesystem command give standard error messages rather than
just printing the exception from the device.

Makes the distinction between CommandError and TransportError clearer.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2024-10-09 16:38:08 +11:00
Jim Mussared
6f8157d880 tools/mpremote: Add hashing ability and use for recursive copy.
Changes in this commit:
- Adds transport API `fs_hashfile` to compute the hash of a file with given
  algorithm.
- Adds commands `mpremote <...>sum file` to compute and print hashes of
  various algorithms.
- Adds shortcut `mpremote sha256sum file`.
- Uses the hash computation to improve speed of recursive file copy to
  avoid copying a file where the target is identical.

For recursive copy, if possible it will use the board's support (e.g.
built-in hashlib or hashlib from micropython-lib), but will fall back to
downloading the file and using the local implementation.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Signed-off-by: Damien George <damien@micropython.org>
2024-10-09 16:35:46 +11:00
Jim Mussared
db59e55fe7 tools/mpremote: Make filesystem commands use transport API.
This introduces a Python filesystem API on `Transport` that is implemented
entirely with eval/exec provided by the underlying transport subclass.

Updates existing mpremote filesystem commands (and `edit) to use this API.

Also re-implements recursive `cp` to allow arbitrary source / destination.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Signed-off-by: Damien George <damien@micropython.org>
2024-10-09 16:12:30 +11:00
Jim Mussared
1091021995 tools/mpremote: Make eval parse by default.
This is a step towards making the transport expose a Python API rather than
functions that mostly print to stdout.

Most use cases of `transport.eval()` are to get some state back from the
device, so have it return as a value directly by default.

Updates uses of `transport.eval()` to remove the parse argument where it
now isn't needed, make the `rtc` command use eval/exec, and update the
`mip` command to use eval's parsing.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2024-10-09 15:56:17 +11:00
Maureen Helm
545d4efb55 zephyr: Upgrade to Zephyr v3.7.0.
Updates the Zephyr port build instructions and CI to use the latest
Zephyr release tag.

Tested on frdm_k64f.

Signed-off-by: Maureen Helm <maureen.helm@analog.com>
2024-10-02 07:49:53 +10:00
Maureen Helm
fa7b4c48bc tools/ci.sh: Upgrade Zephyr docker image 0.26.13 and SDK 0.16.8.
Upgrades CI to use the latest versions of the Zephyr docker image and
Zephyr SDK.

Signed-off-by: Maureen Helm <maureen.helm@analog.com>
2024-10-02 07:49:53 +10:00
Alessandro Gatti
3e085c3aa6 tools/ci.sh: Report code size for VIRT_RV32 as well.
This commit adds the `VIRT_RV32` board to the list of targets for
calculating code size changes as part of the CI pipeline.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2024-09-23 13:03:26 +10:00
Alessandro Gatti
58e7689142 tools/metrics.py: Add VIRT_RV32 to the code size metrics.
This commit adds the Qemu-based RISC-V 32 bits `VIRT_RV32` board to the
list of ports/boards to be built for measuring code size changes.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2024-09-23 13:03:26 +10:00
Alessandro Gatti
bb3c711c04 tools/ci.sh: Let RV64 use a source-built version of libffi.
This commit lets the RV64 port use the version of libffi that is bundled
as a submodule in the MicroPython source tree, as the packaged libffi
library coming from Ubuntu's RISC-V repository trashes foreign function
call results on exit.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2024-09-23 11:47:18 +10:00
Damien George
8b35f2c7fa tools/mpy_ld.py: Support jumping more than 2k on armv6m architectures.
Native .mpy files targetting armv6m (eg RP2040) cannot currently have more
than about 2kiB of native code (between the start of the file and the init
function).

This commit fixes that by using bigger jumps to jump to the init function.

Signed-off-by: Damien George <damien@micropython.org>
2024-09-19 14:52:58 +10:00
Alessandro Gatti
46d8db81d3 tools/ci.sh: Clean up the Unix port's Arm target.
The Unix port's Arm target CI steps have been updated to be more in
line with the other targets (the MicroPython binary doesn't need an
environment variable to be set in order to run now).

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2024-09-19 13:49:12 +10:00
Alessandro Gatti
52d5f39881 tools/ci.sh: Add missing FFI helper for CI RV64 Unix builds.
The FFI helper definition was accidentally omitted when committing the
necessary shell code for building RV64 Unix builds in the CI
environment.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2024-09-19 13:49:12 +10:00
Alessandro Gatti
7b5738ad86 tools/ci.sh: Clean up the Unix port's MIPS target.
The Unix port's MIPS target CI steps have been updated to be more in
line with the other targets (the MicroPython binary now runs as a
dynamic executable), and the test exceptions for ffi have been lifted.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2024-09-19 13:49:12 +10:00
Damien George
067ef81cd0 all: Remove tinytest component.
With the recent qemu (d9a0fdda9a and
0426934969) and zephyr
(05cad7b56f) changes to how their tests are
run, two things became unused:

- The tinytest framework, which embedded a set of tests and their expected
  output within firmware, so these tests could be run stand-alone.

- The `--write-exp` and `--list-tests` options to `tests/run-tests.py`,
  which were needed primarily to generated the expected test output for
  tinytest (also the associated `tests/run-tests-exp.py/.sh` scripts are
  now unused).

This commit removes the tinytest component and all its helper code.  This
eliminates a maintenance burden.

Signed-off-by: Damien George <damien@micropython.org>
2024-09-19 13:31:36 +10:00
Alessandro Gatti
a831c788f7 tools/mpy_ld.py: Ignore R_XTENSA_ASM_EXPAND relocation entries.
As reported in #14430 the Xtensa compiler can add R_XTENSA_ASM_EXPAND
relocation relaxation entries in object files, and they were not
supported by mpy_ld.

This commit adds handling for that entry, doing nothing with it, as it
is only of real use for an optimising linker.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2024-09-06 17:10:07 +10:00
Damien George
659113825d qemu: Rename qemu-arm port to qemu.
Because this port now supports multiple architectures.

Signed-off-by: Damien George <damien@micropython.org>
2024-09-06 11:08:39 +10:00
Damien George
0426934969 qemu-arm: Merge RISC-V 32-bit support into qemu-arm port.
Currently both the qemu-arm and qemu-riscv ports share a lot of code and
functionality.  This commit merges the qemu-riscv port into the qemu-arm
port.  The only real differences between the two are the toolchains used to
build the code, and the initialisation/startup framework.  Everything else
is pretty much the same, so this brings the following benefits:
- less code duplication
- less burden on maintenance
- generalised qemu port, could in the future support other architectures

A new board `VIRT_RV32` has been added to the qemu-arm port which is the
existing RISC-V board from the qemu-riscv port.  To build it:

    $ make BOARD=VIRT_RV32 repl

To cleanly separate the code for the different architectures, startup code
has been moved to ports/qemu-arm/mcu/<arch>/.

Signed-off-by: Damien George <damien@micropython.org>
2024-09-04 16:46:39 +10:00
Damien George
dc9ecd5860 qemu-arm: Factor board config to mk fragments.
Signed-off-by: Damien George <damien@micropython.org>
2024-09-04 16:46:39 +10:00
Damien George
9396572eee tools/mpy-tool.py: Support freezing rv32imc native code.
Signed-off-by: Damien George <damien@micropython.org>
2024-09-04 16:46:39 +10:00
Damien George
d9a0fdda9a qemu-arm: Rework to provide a REPL and run tests via a pty serial port.
Currently, the qemu-arm (and qemu-riscv) port has two build modes:
- a simple test that executes a Python string; and
- a full test that uses tinytest to embed all tests within the firmware,
  then executes that and captures the output.

This is very different to all the other ports.  A difficulty with using
tinytest is that with the large number of tests the firmware overflows its
virtual flash size.  It's also hard to run tests via .mpy files and with
the native emitter.  Being different to the other ports also means an extra
burden on maintenance.

This commit reworks the qemu-arm port so that it has a single build target
that creates a standard firmware which has a REPL.  When run under
qemu-system-arm, the REPL acts like any other bare-metal port, complete
with soft reset (use machine.reset() to turn it off and exit
qemu-system-arm).

This approach gives many benefits:
- allows playing with a REPL without hardware;
- allows running the test suite as it would on a bare-metal board, by
  making qemu-system-arm redirect the UART serial of the virtual device to
  a /dev/pts/xx file, and then running run-tests.py against that serial
  device;
- skipping tests is now done via the logic in `run-tests.py` and no longer
  needs multiple places to define which tests to skip
  (`tools/tinytest-codegen.py`, `ports/qemu-arm/tests_profile.txt` and also
  `tests/run-tests.py`);
- allows testing/using mpremote with the qemu-arm port.

Eventually the qemu-riscv port would have a similar change.

Prior to this commit the test results were:

    743 tests ok.  (121 skipped)

With this commit the test results are:

    753 tests performed (22673 individual testcases)
    753 tests passed
    138 tests skipped

More tests are skipped because more are included in the run. But overall
more tests pass.

Signed-off-by: Damien George <damien@micropython.org>
2024-08-28 11:52:08 +10:00
Damien George
c8838b5004 github/workflows: Add CI to run tests against zephyr with qemu.
With this change, the zephyr port is tested against the standard test suite
via the following scheme:
- the zephyr port is built with the `qemu_cortex_m3` board and the
  `prj_minimal.conf` configuration
- `qemu-system-arm` runs `zephyr.elf`
- the zephyr console is redirected to a pts/pty
- `tests/run-tests.py` is run in bare-metal mode against the pts/pty device

This allows testing the zephyr port as though it were a physical board
attached over a serial port.

Signed-off-by: Damien George <damien@micropython.org>
2024-08-28 11:23:11 +10:00
Damien George
b095c097e6 tools/pyboard.py: Capture stdout for pts line.
The pts line printed by qemu-system-arm goes to stdout, not stderr.

Redirect stderr to stdout in case other tools do print to stderr.

Signed-off-by: Damien George <damien@micropython.org>
2024-08-28 11:23:11 +10:00
Matt Trentini
b704ff66c3 esp32/boards: Remove all IDF3 variants.
IDF 3 builds are very old now (it seems like the last successful builds are
from 2021), and the current IDF 5 is stable.  So remove IDF 3 variants.

Signed-off-by: Matt Trentini <matt.trentini@gmail.com>
2024-08-20 12:49:12 +10:00
Angus Gratton
fbb02d3aee esp32: Add support for ESP-IDF v5.2.2.
Keeping older versions, however if the update goes well then these may be
dropped in the future.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
2024-08-14 15:58:12 +10:00
Alessandro Gatti
17f254df35 github/workflows: Add RISC-V 64 bits Unix port to CI.
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2024-07-25 16:45:04 +10:00
Damien George
abbce268af github/workflows: Use macos-latest for unix macos CI.
macos-11.0 is no longer available.

With this change in the macos version, some tests which previously failed
now pass, and some different tests now fail.  Exclude those that fail from
the CI until they can be fixed properly.

Signed-off-by: Damien George <damien@micropython.org>
2024-07-15 11:10:00 +10:00
Jos Verlinde
3af1425be7 tools/mpremote: Fix mpremote mip install with multiple lib in sys.path.
This is a fix for an algorithmic error in mpremote mip, that throws an
error due to a '\n' used in the concatenation and split when there is more
than one lib path in `sys.path`.

Signed-off-by: Jos Verlinde <Jos.Verlinde@microsoft.com>
2024-06-28 11:07:01 +10:00
Yoctopuce
b4213c9c92 tools/mpy-tool.py: Implement freezing of long-long ints.
Allow inclusion of large integer constants in frozen files using long-long
representation (mpy-cross option -mlongint-impl=longlong).

Signed-off-by: Yoctopuce <dev@yoctopuce.com>
2024-06-27 17:53:31 +10:00
Damien George
706a4b4477 tools/ci.sh: Build an stm32 board with -O2 enabled.
To test building with full optimisations.

Signed-off-by: Damien George <damien@micropython.org>
2024-06-27 12:00:20 +10:00
Damien George
cebc9b0ae2 tools/mpremote: Fix absolute path usage in remote mounted VFS.
Prior to this fix the current working path in the remote VFS would always
be prepended to the requested path to get the full path, even if the
requested path was already absolute, ie starting with "/".

So `os.chdir("/remote/dir1")` would set the working path to "/dir1/", and
a subsequent call with an absolute path like `os.listdir("/remote/dir2")`
would try to list the directory "/dir1/dir2/".

Fixes issue #15308.

Signed-off-by: Damien George <damien@micropython.org>
2024-06-21 16:46:49 +10:00
Alessandro Gatti
99f5659cf5 mpy-cross: Add RISC-V RV32IMC support in MPY files.
MPY files can now hold generated RV32IMC native code.  This can be
accomplished by passing the `-march=rv32imc` flag to mpy-cross.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2024-06-21 15:06:29 +10:00
Alessandro Gatti
1b10cb843c github/workflows: Add qemu-riscv port to CI.
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2024-06-17 12:06:09 +10:00
Alessandro Gatti
596f92bf77 tools/makemanifest.py: Generate working code for empty manifests.
When no usable manifest directives are found (as opposed to no manifest
being set in the makefile), non-compiling code was generated for the
empty frozen constants pool block.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2024-06-05 14:07:26 +02:00
Damien George
0e19286c94 tools/metrics.py: Change rp2 board selection to RPI_PICO_W.
This tool is used to compute size differences in the firmware (eg as part
of CI), but it doesn't currently check any firmware that has bare-metal
lwIP/networking, making it hard to see how firmware size changes when
networking related changes are made.

So, change the board selection for the rp2 port to RPI_PICO_W.  Changes in
size to standard RPI_PICO firmware will be very similar to other bare-metal
boards like PYBV10.

Signed-off-by: Damien George <damien@micropython.org>
2024-06-04 10:54:35 +10:00
Damien George
fbf811474a tools/ci.sh: Simplify selection of natmod tests to run.
The `run-natmodtests.py` script now properly excludes tests that don't use
the corresponding native module.

Signed-off-by: Damien George <damien@micropython.org>
2024-05-27 11:45:59 +10:00