When no usable manifest directives are found (as opposed to no manifest
being set in the makefile), non-compiling code was generated for the
empty frozen constants pool block.
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This tool is used to compute size differences in the firmware (eg as part
of CI), but it doesn't currently check any firmware that has bare-metal
lwIP/networking, making it hard to see how firmware size changes when
networking related changes are made.
So, change the board selection for the rp2 port to RPI_PICO_W. Changes in
size to standard RPI_PICO firmware will be very similar to other bare-metal
boards like PYBV10.
Signed-off-by: Damien George <damien@micropython.org>
The `run-natmodtests.py` script now properly excludes tests that don't use
the corresponding native module.
Signed-off-by: Damien George <damien@micropython.org>
Also define `mp_type_bytearray`. These all help to write native modules.
Signed-off-by: Brian Pugh <bnp117@gmail.com>
Signed-off-by: Damien George <damien@micropython.org>
This is required because the .mpy native ABI was changed by the
introduction of `mp_proto_fun_t`, see commits:
- 416465d81e
- 5e3006f117
- e2ff00e811
And three `mp_binary` functions were added to `mp_fun_table` in
commit d2276f0d41.
Signed-off-by: Damien George <damien@micropython.org>
These are needed to read/write array.array objects, which is useful in
native code to provide fast extensions that work with big arrays of data.
Signed-off-by: Damien George <damien@micropython.org>
It destroys a few manual alignments, but these seem minor compared to
the benefit of automated code style consistency.
Signed-off-by: Christian Walther <cwalther@gmx.ch>
The documentation for `freeze()` says that:
- If `script` is `None`, all files in `path` will be frozen.
- If `script` is an iterable then `freeze()` is called on all items of the
iterable.
This commit makes sure this behaviour is followed when an empty tuple/list
is passed in for `script` (previously an empty tuple/list froze all files).
Fixes issue #14125.
Signed-off-by: Damien George <damien@micropython.org>
The argument to MP_BC_MAKE_FUNCTION (raw code index) was being encoded as a
byte instead of a variable unsigned int. That meant that if there were
more than 128 merged mpy files the encoding would be invalid.
Fix that by using `mp_encode_uint(idx)` to encode the raw code index. And
also use `Opcode` constants for the opcode values to make it easier to
understand the code.
Signed-off-by: Damien George <damien@micropython.org>
The STATIC macro was introduced a very long time ago in commit
d5df6cd44a. The original reason for this was
to have the option to define it to nothing so that all static functions
become global functions and therefore visible to certain debug tools, so
one could do function size comparison and other things.
This STATIC feature is rarely (if ever) used. And with the use of LTO and
heavy inline optimisation, analysing the size of individual functions when
they are not static is not a good representation of the size of code when
fully optimised.
So the macro does not have much use and it's simpler to just remove it.
Then you know exactly what it's doing. For example, newcomers don't have
to learn what the STATIC macro is and why it exists. Reading the code is
also less "loud" with a lowercase static.
One other minor point in favour of removing it, is that it stops bugs with
`STATIC inline`, which should always be `static inline`.
Methodology for this commit was:
1) git ls-files | egrep '\.[ch]$' | \
xargs sed -Ei "s/(^| )STATIC($| )/\1static\2/"
2) Do some manual cleanup in the diff by searching for the word STATIC in
comments and changing those back.
3) "git-grep STATIC docs/", manually fixed those cases.
4) "rg -t python STATIC", manually fixed codegen lines that used STATIC.
This work was funded through GitHub Sponsors.
Signed-off-by: Angus Gratton <angus@redyak.com.au>
Follow up to 35dd959133, allows explicitly
adding the unix-ffi library path from the command line.
This option is needed when building unix-ffi manifests in micropython-lib
CI.
This work was funded through GitHub Sponsors.
Signed-off-by: Angus Gratton <angus@redyak.com.au>
Allows bytecode itself to be used instead of an mp_raw_code_t in the simple
and common cases of a bytecode function without any children.
This can be used to further reduce frozen code size, and has the potential
to optimise other areas like importing.
Signed-off-by: Damien George <damien@micropython.org>
To simplify their access and reduce code size.
The `scope_flags` member is only ever used to determine if a function is a
generator or not, so make it reflect that fact as a bool type.
Signed-off-by: Damien George <damien@micropython.org>
The `asm_n_pos_args` and `asm_type_sig` members of `mp_raw_code_t` are only
used for raw codes of type MP_CODE_NATIVE_ASM, which are rare, for example
in frozen code. So using a truncated `mp_raw_code_t` in these cases helps
to reduce frozen code size on targets that have MICROPY_EMIT_INLINE_ASM
enabled.
With this, change in firmware size of RPI_PICO builds is -648.
Signed-off-by: Damien George <damien@micropython.org>
The mp_raw_code_t struct has been reordered and some members resized. The
`n_pos_args` member is renamed to `asm_n_pos_args`, and `type_sig` renamed
to `asm_type_sig` to indicate that these are used only for the inline-asm
emitters. These two members are also grouped together in the struct.
The justifications for resizing the members are:
- `fun_data_len` can be 32-bits without issue
- `n_children` is already limited to 16-bits by
`mp_emit_common_t::ct_cur_child`
- `scope_flags` is already limited to 16-bits by `scope_t::scope_flags`
- `prelude_offset` is already limited to 16-bits by the argument to
`mp_emit_glue_assign_native()`
- it's reasonable to limit the maximim number of inline-asm arguments to 12
(24 bits for `asm_type_sig` divided by 2)
This change helps to reduce frozen code size (and in some cases RAM usage)
in the following cases:
- 64-bit targets
- builds with MICROPY_PY_SYS_SETTRACE enabled
- builds with MICROPY_EMIT_MACHINE_CODE enabled but MICROPY_EMIT_INLINE_ASM
disabled
With this change, unix 64-bit builds are -4080 bytes in size. Bare-metal
ports like rp2 are unchanged (because mp_raw_code_t is still 32 bytes on
those 32-bit targets).
Signed-off-by: Damien George <damien@micropython.org>
It's rare to freeze .mpy files without specifying a qstr header from a
firmware build, but it can be useful for testing, eg
`mpy-tool.py -f test.mpy`. Fix this case so static qstrs are properly
excluded from the frozen qstr list.
Signed-off-by: Damien George <damien@micropython.org>
This commit changes how library packages are searched for when a manifest
file is loaded: there is now simply a list of library paths that is
searched in order for the given package. This list defaults to the
main directories in micropython-lib, but can be added to -- either appended
or prepended -- by using `add_library()`.
In particular the way unix-ffi library packages are searched has changed,
because the `unix_ffi` argument to `require()` is now removed. Instead, if
a build wants to include packages from micropython-lib/unix-ffi, then it
must explicitly add this to the list of paths to search using:
add_library("unix-ffi", "$(MPY_LIB_DIR)/unix-ffi")
Work done in collaboration with Jim Mussared.
Signed-off-by: Damien George <damien@micropython.org>
Similar to ruff.yaml, it's simpler to run the codespell command directly
from a workflow file. And developers can run codespell directly from the
command line without the need for options, or just use pre-commit.
This commit also applies a specific version to codespell, same as
pre-commit (introduced in a166d805f4).
This work was funded through GitHub Sponsors.
Signed-off-by: Angus Gratton <angus@redyak.com.au>
This disables using qstr hashes altogether, which saves RAM and flash
(two bytes per interned string on a typical build) as well as code size.
On PYBV11 this is worth over 3k flash.
qstr comparison will now be done just by length then data. This affects
qstr_find_strn although this has a negligible performance impact as, for a
given comparison, the length and first character will ~usually be
different anyway.
String hashing (e.g. builtin `hash()` and map.c) now need to compute the
hash dynamically, and for the map case this does come at a performance
cost.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Required modifying the gen-cpydiff.py code to allow a "preamble" section to
be inserted at the top of any of the generated files.
This work was funded through GitHub Sponsors.
Signed-off-by: Angus Gratton <angus@redyak.com.au>
No longer require importlib_metadata on new Python versions as it is
included in the standard distribution.
Signed-off-by: Jochen Sprickerhof <git@jochen.sprickerhof.de>
This adds a `add_library(name, path)` method for use in manifest.py that
allows registering an external path (e.g. to another repo) by name.
This name can then be passed to `require("package", library="name")` to
reference packages in that repo/library rather than micropython-lib.
Within the external library, `require()` continues to work as normal
(referencing micropython-lib) by default, but they can also specify the
library name to require another package from that repo/library.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Also, IDF v5.1.2 is now supported, just not used by default.
IDF v5.0.2 still builds but we cannot guarantee continued support for this
version moving forward.
Signed-off-by: IhorNehrutsa <IhorNehrutsa@gmail.com>
CCaches are scoped per-job.
Uses https://github.com/hendrikmuhs/ccache-action to get desired behaviour
(updating the cache on each run).
Signed-off-by: Angus Gratton <angus@redyak.com.au>
Allows splitting the esp32 job into multiple parts without too much
boilerplate. The matrix is parameterised using the name of the function to
call in tools/ci.sh, to minimise the dependency on GitHub Actions.
This can get esp32 build times down around 3m if IDF is cached already.
If the cache is cold, the cache preparation step on each job can double up
against each other. However, restructuring the workflow to not do this
seems either complex or requires copy-pasting the entire cache step.
Signed-off-by: Angus Gratton <angus@redyak.com.au>
Cache is keyed on the ESP-IDF version used in CI, so there shouldn't be any
cache invalidation issues when ESP-IDF version changes.
Restoring from cache takes approx 15s, compared to 2-3m to perform these
steps (ESP-IDF tools install, ESP-IDF clone, ESP-IDF submodule clone) the
first time.
Cache size is approx 1.6GB, the git clone is tweaked as much as possible to
keep the size down.
Signed-off-by: Angus Gratton <angus@redyak.com.au>
Speeds up importing files from mounted filesystem.
Also fix the return code for invalid / unsupported ioctl requests.
Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Remove port-specific test directories and excluded tests from
tinytest-codegen, and let it read said information from an external file.
This way tinytest-codegen is not limited to always generate tests for the
`qemu-arm` target.
This allows having port-specific test directory and excluded tests for more
than one QEMU bare-metal target.
The `qemu-arm` port Makefile was modified to work with the generator
changes and a tests profile file was added to said port.
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
Requires additions to tools/boardgen.py for stm32 pin generation.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
For now, this implements the functionality required for esp32 and rp2,
including support for numeric pins, rp2 alternate functions, and rp2
extended pins.
This also updates the rp2 port to use the same structure for pins.h and
pins.csv as for esp32, and moves the pin definitions directly into the
table (rather than having a table of pointers), which is a small code size
improvement.
Support for "hidden" pins in pins.csv is added (matching the stm32
implementation).
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This updates a small number of files that change with ruff-format's (vs
black's) rules.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
- Add config for [tool.ruff.format] to pyproject.toml.
- Update pre-commit to run both ruff and ruff-format (and only check C
files when running codeformat.py)
- Update CI.
- Simplify codeformat.py to remove all the Python-specific logic (just run
"ruff format" directly).
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This port is largely unmaintained, has limited features (the only hardware
support is for GPIO and timer, and no machine module), only supports a
small number of Teensy boards, and can be confused with the mimxrt support
for Teensy 4.x.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Signed-off-by: Damien George <damien@micropython.org>
This provides a significant performance boost for qstr_find_strn, which is
called a lot during parsing and loading of .mpy files, as well as interning
of string objects (which happens in most string methods that return new
strings).
Also adds comments to explain the "static" qstrs. These are part of the
.mpy ABI and avoid needing to duplicate string data for QSTRs known to
already be in the firmware. The static pool isn't currently sorted, but in
the future we could either split the static pool into the sorted regions,
or in the next .mpy version just sort them.
Based on initial work done by @amirgon in #6896.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This provides a way to enable features and changes slated for MicroPython
2.x, by running `make MICROPY_PREVIEW_VERSION_2=1`. Also supported for
the cmake ports (except Zephyr).
This is an alternative to having a 2.x development branch (or equivalently,
keeping a 1.x release branch). Any feature or change that needs to be
"hidden" until 2.x can use this flag (either in the Makefile or the
preprocessor).
A good example is changing function arguments or other public API features,
in particular to aid in improving consistency between ports.
When `MICROPY_PREVIEW_VERSION_2` is enabled, the REPL banner is amended to
say "MicroPython (with v2.0 preview) vX.Y.Z", and sys.implementation gets a
new field `_v2` set to `True`.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This commit updates the ci script to automatically fetch all upstream if
the common commit hasn't been found; this should preserve the speed of CI
checks for most PR's, and use a reliable but slow fetch if needed for older
ones.
Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
See https://github.com/micropython/micropython/issues/12127 for details.
Previously at the point when a release is made, we update mpconfig.h
and set a git tag. i.e. the version increments at the release.
Now the version increments immediately after the release. The workflow is:
1. Final commit in the cycle updates mpconfig.h to set (X, Y, 0, 0) (i.e.
clear the pre-release state).
2. This commit is tagged "vX.Y.0".
3. First commit for the new cycle updates mpconfig.h to set (X, Y+1, 0, 1)
(i.e. increment the minor version, set the pre-release state).
4. This commit is tagged "vX.Y+1.0-preview".
The idea is that a nightly build is actually a "preview" of the _next_
release. i.e. any documentation describing the current release may not
actually match the nightly build. So we use "preview" as our semver
pre-release identifier.
Changes in this commit:
- Add MICROPY_VERSION_PRERELEASE to mpconfig.h to allow indicating that
this is not a release version.
- Remove unused MICROPY_VERSION integer.
- Append "-preview" to MICROPY_VERSION_STRING when the pre-release state
is set.
- Update py/makeversionhdr.py to no longer generate MICROPY_GIT_HASH.
- Remove the one place MICROPY_GIT_HASH was used (it can use
MICROPY_GIT_TAG instead).
- Update py/makeversionhdr.py to also understand
MICROPY_VERSION_PRERELEASE in mpconfig.h.
- Update py/makeversionhdr.py to convert the git-describe output into
semver-compatible "X.Y.Z-preview.N.gHASH".
- Update autobuild.sh to generate filenames using the new scheme.
- Update remove_old_firmware.py to match new scheme.
- Update mpremote's pyproject.toml to handle the "-preview" suffix in the
tag. setuptools_scm maps to this "rc0" to match PEP440.
- Fix docs heading where it incorrectly said "vvX.Y.Z" for release docs.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Defines the list of standard features and ensures that each board.json
only uses those ones. This list can be extended, but needs to be a
deliberate decision.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Also provide a basic README.md for dynamic native modules.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Because mpy_ld.py doesn't know the target object representation, it emits
instances of `MP_OBJ_NEW_QSTR(MP_QSTR_Foo)` as const string objects, rather
than qstrs. However this doesn't work for map keys (e.g. for a locals dict)
because the map has all_keys_are_qstrs flag is set (and also auto-complete
requires the map keys to be qstrs).
Instead, emit them as regular qstrs, and make a functioning MP_OBJ_NEW_QSTR
function available (via `native_to_obj`, also used for e.g. making
integers).
Remove the code from mpy_ld.py to emit qstrs as constant strings, but leave
behind the scaffold to emit constant objects in case we want to do use this
in the future.
Strictly this should be a .mpy sub-version bump, even though the function
table isn't changing, it does lead to a change in behavior for a new .mpy
running against old MicroPython. `mp_native_to_obj` will incorrectly return
the qstr value directly as an `mp_obj_t`, leading to unexpected results.
But given that it's broken at the moment, it seems unlikely that anyone is
relying on this, so it's not work the other downsides of a sub-version bump
(i.e. breaking pure-Python modules that use @native). The opposite case of
running an old .mpy on new MicroPython is unchanged, and remains broken in
exactly the same way.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Sections sometimes named .rodata.str1.1 etc, instead of just .rodata.
Avoid crashing in that case. Instead treat it like any other RO section.
Fix thanks to @phlash.
Fixes issue #8783.
Signed-off-by: Jon Nordby <jononor@gmail.com>
This allows the cc3200 port to be build with the standard autobuild script
rather than the custom build-cc3200-latest.sh (which is now removed).
This also fixes the path inside the zip file (by using the `-j` flag to
zip).
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
PICO might not always be a unique name across all ports, and the
convention generally for other boards is to do VENDOR_BOARD.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This merges the existing GENERIC, GENERIC_1M, and GENERIC_512k boards
into variants of the new ESP8266_GENERIC board (renamed from GENERIC so
as not to clash with other ports).
Also moves the generation of the "OTA" variant (previously generated by
autobuild/build-esp8266-latest.sh) into the variant.
Following the convention established for the WEACTSTUDIO rp2 board, the
names of the variants are FLASH_1M and FLASH_512K (but rename the .ld files
to use MiB and kiB).
Updates autobuild to build esp8266 firmware the same way as other ports.
This requires renaming the output from firmware-combined.bin to just
firmware.bin.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Board names need to be unique across ports, and GENERIC clashes with
the ESP8266 (which will be renamed to ESP8266_GENERIC).
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This is to support a future change to add the variant name to the build
directory and therefore should be the same style as the board name.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Spurious fix as the logic is structured such that these variables will be
set before dereferenced, but this keeps Ruff happy (no more F821
undefined-name).
Signed-off-by: Angus Gratton <angus@redyak.com.au>
Otherwise `mpremote soft-reset` will implicitly run the repl command.
Fixes issue #10871.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Ruff version 283 expanded E721 to fail when making direct comparison
against a built-in type. Change the code to use isinstance() as
suggested, these usages appear to have equivalent functionality.
Signed-off-by: Angus Gratton <angus@redyak.com.au>
On esp32, the build output consists of:
- micropython.elf
- micropython.map
- micropython.bin -- application only
- micropython.uf2 -- application only
- firmware.bin -- bootloader, partition table and application
Currently everything is available at the download page except
micropython.bin. This commit adds that file but with the extension changed
to .app-bin, to distinguish it from .bin (the full thing).
Signed-off-by: Damien George <damien@micropython.org>
This will be replaced with a new deflate module providing the same
functionality, with an optional frozen Python wrapper providing a
replacement zlib module.
binascii.crc32 is temporarily disabled.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This change enables the ULP (FSM) for all ESP32 variants rather than
requiring it to be enabled for each board specifically.
It also ensures the correct header file is included for each variant.
Lastly, it updates the IDF version we're builing against to v4.4.2, as that
version contains important fixes to make the ULP actually work on S2/S3
chips. See: https://github.com/espressif/esp-idf/commit/a0e3d48
Signed-off-by: Wilko Nienhaus <wilko.nienhaus@gmail.com>
The automatic code size build and GitHub comment is a really useful
feature. This commit adds a few more builds to it (mimxrt and samd).
Signed-off-by: Damien George <damien@micropython.org>
This is a MicroPython-specific module that existed to support the old
version of uasyncio. It's undocumented and not enabled on all ports and
takes up code size unnecessarily.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Applies to drivers/examples/extmod/port-modules/tools.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Updates any includes, and references from Makefiles/CMake.
This essentially reverts what was done long ago in commit
136b5cbd76
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This commit just takes the necessary parts of pyboard.py and merges them
with pyboardextended.py to make a new transport_serial.py, and updates the
rest of mpremote to use this instead.
It is difficult to continue to add features to mpremote (which usually
requires modification to pyboard.py) while also maintaining backwards
compatibility for pyboard.py.
The idea is that this provides a starting point for further refactoring of
mpremote to allow different transports (webrepl, BLE, etc).
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Changes in this commit:
- Add a extra detail to each of the commands.
- Add more about handling options and arguments.
- Include shortcut commands that behave like real commands to the command
list (e.g. bootloader, rtc).
- Add extra information and reword to address common misconceptions, in
particular how commands chain together.
- Add additional examples showing some more interesting combinations.
- Add descriptions to each of the examples.
- Add pipx installation instructions.
- Describe how user-configuration works.
This work was sponsored by Google Season of Docs.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Replaces the existing functionality provided by the `setrtc` alias to use
the current time, rather than a hard-coded date/time.
Use `rtc` to query the current time. Use `rtc --set` to set it.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
For example, the `reset` shortcut previously allowed an optional delay, but
the argument handling cannot handle `reset next-command` as `next-command`
will be interpreted as the delay argument. The fix in this commit allows
`reset + next-command`.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This allows the sequence to be paused (e.g. wait for device, etc).
Also removes the t_ms arg in reset/bootloader, because these arguments
don't really need to be changed, and keeping them would mean inconsistent
units used for time delays.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This commit adds the "--escape-non-printable" option to the repl command.
When specified the REPL console will escape non-printable characters,
printing them as their hex value in square brackets.
This escaping behaviour was previously the default and only behaviour, but
it is now opt-in.
As part of this change, the speed of echoing device data to the console is
improved by by reading and writing in chunks.
Signed-off-by: Damien George <damien@micropython.org>
Changes in this commit:
- Change MICROPY_HW_BOARD_NAME definition to match the product name.
- Rename board folder's name to match the product name style.
- Change related files like Makefile, document descriptions, test cases, CI
and tools.
Signed-off-by: Takeo Takahashi <takeo.takahashi.xv@renesas.com>
- Use HCI_TRACE macro consistently.
- Use the same colour formatting.
- Add a tool to convert to .pcap for Wireshark.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
The mpremote REPL can now be closed with either ctrl+] or ctrl+x, which
gives users a choice, useful if the ']' key is difficult to access.
Fixes issue #11197.
Signed-off-by: Jonas Scharpf <jonas@brainelectronics.de>
This adds a new MODE_PYPROJECT, which gives basic support to allow
packaging a small subset of micropython-lib packages to PyPI.
This change allows a package in micropython-lib to:
- Add a "pypi" name to its metadata indicating that it's based on a PyPI
package.
- Add "stdlib" to its metadata indicating that it's a micropython version
of a stdlib package.
- Add a "pypi_publish" name to its metadata to indicate that it can be
published to PyPI (this can be different to the package name, e.g. "foo"
might want to be published as "micropython-foo").
When a package requires() another one, if it's in MODE_PYPROJECT then if
the package is from pypi then it will record that as a pypi dependency
instead (or no dependency at all if it's from stdlib).
Also allows require() to explicitly specify the pypi name.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This is a follow up to d263438a6e, which
solved one problem (reset on disconnect) but introduced a second one (hang
in bootloader).
To solve both probles, False/False is needed for DTR/RTS for ESPxx, but
that would then block stm32 and others. Any unconditional combination of
DTR/RTS ends up blocking normal operation on some type of board or another.
A simple overview (for windows only):
DTR CTS ESP8266/ESP32 STM32/SAMD51/RP2040
unspecified unspecified Reset on disconnect OK
True False Hang in bootloader OK
False False OK No Repl
True True Reset on disconnect No Repl
False True Reset on disconnect No Repl
serial.manufacturer: wch.cn/Silicon Labs Microsoft
serial.description: USB-SERIAL CH340 / USB Serial Device
CP210x USB to UART
Bridge
The updated logic will only set the DTR/RTS signals for boards that do not
use standard Microsoft drivers (based on the manufacturer). It would also
be possible to check against a list of known driver manufactures (like
wch.cn or Silicon Labs) but this would require a list of known drivers for
all ports.
Signed-off-by: Jos Verlinde <jos_verlinde@hotmail.com>
Currently, certain mpremote filesystem operations can fail on Windows due
to a mixing of '/' and '\' for path separators. Eg if filesystem_command()
is called with a destination that ends in / then dest.endswith(os.path.sep)
will return False, which gives the wrong behaviour (it does end in a path
separator).
For similar reasons to 7e9a15966a, it's best
to use '/' everywhere in pyboard.py and mpremote, because the target device
understands only '/'. mpremote already does this, so the remaining place
to fix it is in pyboard.y, to convert all incoming paths to use '/' instead
of '\'.
This effectively reverts 57fd66b80f which
tried to fix the problem in a different way.
See also related 1f84440538.
Signed-off-by: Damien George <damien@micropython.org>
This allows the entire configuration to be defined in a single file,
including the logic for including pyboard.py and automatically versioning
based on the git tag.
Building the package works both via `python -m build` as well as
`hatch build`. `python -m build ` has the advantage of automatically
fetching all dependencies, you don't need to manually install any hatch
packages.
In order to make the versioning work, and also keep things simpler for end
users, mpremote releases will now be the same as MicroPython releases and
use the same tag. The version strings for mpremote will look like:
- X.Y.Z -- clean build at the tag
- X.Y.Z.postN+gHASH -- clean build, N revisions from the most recent tag
- X.Y.Z.postN+gHASH.dYYYYMMDD -- dirty build, N revisions from out
This commit extends on the idea from #8404.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Now that the code-size-check CI action gives a nice report (as a comment)
on the code size difference, it's possible to have a few more ports
reported there. In this commit, unix, stm32 and rp2 are added. Unix
represents non-MCU builds, and stm32 and rp2 represent ARM-based builds,
for ports that have lots of features enabled.
Signed-off-by: Damien George <damien@micropython.org>
These are for working with the filesystem when using pyboard.py as a
library, rather than at the command line.
- fs_listdir returns a list of tuples, in the same format as os.ilistdir().
- fs_readfile returns the contents of a file as a bytes object.
- fs_writefile allows writing a bytes object to a file.
- fs_stat returns an os.statresult.
All raise FileNotFoundError (or OSError(ENOENT) on Python 2) if the file is
not found (or PyboardError on other errors).
Updated fs_cp and fs_get to use fs_stat to compute file size.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This is useful when using pyboard.py as a library rather than at the
command line.
pyb.eval("1+1") --> b"2"
pyb.eval("{'a': '\x00'}") --> b"{'a': '\\x00'}"
Now you can also do
pyb.eval("1+1", parse=True) --> 2
pyb.eval("{'a': '\x00'}", parse=True) --> {'a': '\x00'}
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
On MacOS and Windows there are a few default serial devices that are
returned by `serial.tools.list_ports.comports()`. For example on MacOS:
```
{'description': 'n/a',
'device': '/dev/cu.Bluetooth-Incoming-Port',
'hwid': 'n/a',
'interface': None,
'location': None,
'manufacturer': None,
'name': 'cu.Bluetooth-Incoming-Port',
'pid': None,
'product': None,
'serial_number': None,
'vid': None}
{'description': 'n/a',
'device': '/dev/cu.wlan-debug',
'hwid': 'n/a',
'interface': None,
'location': None,
'manufacturer': None,
'name': 'cu.wlan-debug',
'pid': None,
'product': None,
'serial_number': None,
'vid': None}
```
Users of mpremote most likely do not want to connect to these ports. It
would be desirable if mpremote did not select this ports when using the
auto connect behavior. These serial ports do not have USB VID or PID
values and serial ports for Micropython boards with FTDI/serial-to-USB
adapter or native USB CDC/ACM support do.
Check for the presence of a USB VID / PID int value when selecting a
serial port to auto connect to. All serial ports will still be listed by
the `list` command and can still be selected by name when connecting.
Signed-off-by: Michael Mogenson <michael.mogenson@gmail.com>
The zephyr CI takes the most time out of all CI jobs, so remove the
standard qemu_x86 build to speed it up. The remaining builds should still
cover enough cases to catch errors.
Signed-off-by: Damien George <damien@micropython.org>
This used to be used to generate .rst docs from inline comments in the C
code (specifically for APIs) but is now unused.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
App the mp_ prefix to usbd_ symbols and files which are defined here and
not in TinyUSB.
rp2 only for now. This includes some groundwork for dynamic USB devices
(defined in Python).
This work was funded through GitHub Sponsors.
Signed-off-by: Angus Gratton <angus@redyak.com.au>
So that filesystems mounted with "mpremote mount" can have their files
iterated over, making them consistent with other files.
Signed-off-by: Damien George <damien@micropython.org>
The except handler for OSError didn't include the line that actually calls
os.listdir, so an invalid path wasn't handled correctly.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
In order for v1.19.1 to load a .mpy, the formerly-feature-flags which are
now used for the sub-version must be zero.
The sub-version is only used to indicate a native version change, so it
should be zero when emitting bytecode-only .mpy files.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This applies to nimble, btstack, axtls, mbedtls, lwip.
Rather than having the ports individually manage GIT_SUBMODULES for these
components, make extmod.mk append them when the relevant feature is
enabled.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This check used to just show the regular expression that failed to match,
but the rules are pretty subtle and hard to interpret from the regular
expression alone.
Add some basic checks for the main things that go wrong:
- Missing capitalisation.
- Missing full-stop.
- Missing path.
- Single-word subject.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
The "signed-off" check assumes that the Signed-off-by: line is the last,
but there may me many lines of comments after this.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Fixes in this commit are:
- Make --follow the default for "run" (accidentally changed in 68d094358).
- Add help strings for "repl": --capture --inject-file --inject-code
- Update help strings for "run".
- Fix encoding for --inject-code (accidentally broken in 68d094358).
- Remove ability to --no-follow for "eval". It was there previously because
it shared the same code path with "exec" and "run", but makes no sense
for "eval", so might as well remove.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Tweak the existing codeformat.py and verifygitlog.py to allow them to be
easily called by pre-commit.
(This turned out to be easier than using any existing pre-commit hooks,
without making subtle changes in the formatting.)
This work was funded through GitHub Sponsors.
Signed-off-by: Angus Gratton <angus@redyak.com.au>
Uncrustify versions are not mutually compatible:
1. Version 0.73 or newer produce slightly different formatting. It may be
possible to tweak these by adding more config items, but this will cause
older versions to error out with 'Unknown option'.
2. Version 0.75 prints a range of deprecation warnings due to config file
changes, and returns a non-zero exit code. These are actually fixable
as most are the default value, and pp_indent has changed from 'true' to '1'
which is backwards compatible. However issue 1 remains, so probably better
to have it fail explicitly.
This work was funded through GitHub Sponsors.
Signed-off-by: Angus Gratton <angus@redyak.com.au>
This supports the same package sources as the new `mip` tool.
- micropython-lib (by name)
- http(s) & github packages with json description
- directly downloading a .py/.mpy file
The version is specified with an optional `@version` on the end of the
package name. The target dir, index, and mpy/no-mpy can be set through
command line args.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
No functional change other than to allow slightly more flexibility in how
--foo arguments are specified.
This removes all custom handling for --foo args in all commands and
replaces it with per-command argparse configs.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
No functional change.
This makes each built-in command defined by just a handler method and
simplifies a lot of the logic around tracking the board state.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Updates all README.md and docs, and manifests to `require("mip")`.
Also extend and improve the documentation on freezing and packaging.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This uses the frozentest.mpy that is also used by ports/minimal.
Also fixes two bugs that these new tests picked up:
- File extension matching in manifestfile.py.
- Handling of freeze_mpy results in makemanifest.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
frozentest.mpy was previously duplicated in ports/minimal and
ports/powerpc.
This needs to be re-generated on every .mpy version increase, so might as
well just have a single copy of it.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
The intent is to allow us to make breaking changes to the native ABI (e.g.
changes to dynruntime.h) without needing the bytecode version to increment.
With this commit the two bits previously used for the feature flags (but
now unused as of .mpy version 6) encode a sub-version. A bytecode-only
.mpy file can be loaded as long as MPY_VERSION matches, but a native .mpy
(i.e. one with an arch set) must also match MPY_SUB_VERSION. This allows 3
additional updates to the native ABI per bytecode revision.
The sub-version is set to 1 because the previous commits that changed the
layout of mp_obj_type_t have changed the native ABI.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Signed-off-by: Damien George <damien@micropython.org>
This is a no-op for coverage and minimal.
The standard and dev variants have been merged and enable the same feature
set as a typical bare-metal board. And remove the CI for the dev build.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
When iterating over filesystem/folders with os.iterdir(), an open file
(directory) handle is used internally. Currently this file handle is only
closed once the iterator is completely drained, eg. once all entries have
been looped over / converted into list etc.
If a program opens an iterdir but does not loop over it, or starts to loop
over the iterator but breaks out of the loop, then the handle never gets
closed. In this state, when the iter object is cleaned up by the garbage
collector this open handle can cause corruption of the filesystem.
Fixes issues #6568 and #8506.
In case the version from pypi is installed or some other version is
available in sys.path, prepend `$(TOP)/mpy-cross` to sys.path instead.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Previous the build directory just used the board name, now make it use the
variant name too.
This shouldn't have any change because the existing directory should not
exist (all builds run by these scripts remove their build directory after
completion), but it makes debugging easier.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
The metadata can be version, description, and license.
After executing a manifest, the top-level metadata can be queried, and also
each file output from the manifest will have the metadata of the
containing manifest.
Use the version metadata to "tag" files before freezing such that they have
__version__ available.
By default, don't include micropython-lib/unix-ffi in the search.
If unix_ffi=True is passed to require(), then include unix-ffi and make it
take precedence over the other locations (e.g. python-stdlib).
This does two things:
- Prevents non-unix builds from using unix-only packages.
- Allows the unix build to optionally use a more full-featured (e.g. ffi)
based package, even with the same name as one from e.g. stdlib.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
If an include path is a directory, then it implicitly grabs the manifest.py
file inside that directory. This simplifies most manifest.py files.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This splits the manifest file loading logic from makemanifest.py and
updates makemanifest.py to use it.
This will allow non-freezing uses of manifests, such as defining packages
and dependencies in micropython-lib.
Also adds additional methods to the manifest "API":
- require() - to get a package from micropython-lib.
- module() - to define a single-file module
- package() - to define a multi-file package
module() and package() should replace most uses of freeze() and can also
be also used in non-freezing scenarios.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Having two separate manifests is confusing. It's simpler to have the daily
builds use the same configuration as the stable, release builds.
Signed-off-by: Damien George <damien@micropython.org>
It has been about 8 years since support for this chip was added. Reasons
to remove it are:
- It is no longer easy to obtain this part.
- There are now many other options for WiFi.
- It's not a good use of developer time to maintain it.
Signed-off-by: Damien George <damien@micropython.org>
Rather than having the autobuild know about the particular variants, have
the mpconfigboard.mk describe them and make autobuild discover them
automatically.
Adds a "query-variants" target to stm32/Makefile to allow the set of
possible variants to be queried.
Removes pybv3 from the autobuild as this isn't use by the downloads page.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
- Add lib/wiznet5k into the 'make submodules' step.
- Split the stm32 builds for wiznet5k and cc3k.
- Run 'make .... clean' after making the wiznet5k build.
This allows a remote file to be edited locally by copying it over, running
the local editor, then copying it back.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
The executable now lives in the build directory, and since the build
directory already contains the variant name there is no need to also add
it to the executable.
Signed-off-by: Damien George <damien@micropython.org>
Binaries built using the Make build system now no longer appear in the
working directory of the build, but rather in the build directory. Thus
some paths had to be adjusted.
Formerly, py/formatfloat would print whole numbers inaccurately with
nonzero digits beyond the decimal place. This resulted from its strategy
of successive scaling of the argument by 0.1 which cannot be exactly
represented in floating point. The change in this commit avoids scaling
until the value is smaller than 1, so all whole numbers print with zero
fractional part.
Fixes issue #4212.
Signed-off-by: Dan Ellis dan.ellis@gmail.com
The CI scripts were using a PPA to get a backported version of uncrustify
on Ubuntu 20.04. However, this causes CI to intermittently fail due to
connection issues to launchpad.net or the key server.
Ubuntu 22.04 has a newer version of uncrustify removing the need for the
PPA. Ubuntu 22.04 is now in beta on GitHub actions, so it can be used.
Signed-off-by: David Lechner <david@pybricks.com>
Updates the Zephyr port build instructions and CI to use the latest Zephyr
release tag.
Tested on frdm_k64f.
Signed-off-by: Maureen Helm <maureen.helm@intel.com>
This tests the build when -O2 is used, which can lead to additional
compiler analysis and warnings.
Signed-off-by: Damien George <damien@micropython.org>
Frozen identifiers now include their full name hierarchy, eg their class
name. This makes it easier to understand the generated code.
Signed-off-by: Damien George <damien@micropython.org>
