This is a fix for utf-8 decoding errors that are thrown when non-utf-8
content is received. For instance during a reboot of an ESP8266 module.
The fix is to handle conversion errors by replacing illegal characters.
Note that these illegal characters most often occur during an MCU reboot
sequence when the MCU is using baudrates different from 115200.
Signed-off-by: Jos Verlinde <Jos_Verlinde@hotmail.com>
Detection of ESP-XX devices was based on just the names of the USB driver
name, and did not account for the switch of the newer ESP-xx devices to
USB-CDC. On Windows this caused unwanted/unneeded resets as the DTR/RTS
signals are also used for automatic device reset over USB-CDC. See
https://github.com/micropython/micropython/issues/9659#issuecomment-3124704572
This commit uses the Espressif registered VID 0x303A to detect USB-CDC
ports, to enable the same DTR/RTS settings as used on the UART-USB
connection.
Also improved the robustness of the code using `getattr()`.
Signed-off-by: Jos Verlinde <Jos_Verlinde@hotmail.com>
Not all errors defined in stdlib errno are available on Windows.
Specifically, errno.ENOTBLK is not.
Fixes issue #17773.
Signed-off-by: Jos Verlinde <Jos_Verlinde@hotmail.com>
Commit dc2fcfcc55 seems to have accidentally
changed the ruff quote style to "preserve", instead of keeping it at the
default which is "double".
Put it back to the default and update relevant .py files with this rule.
Signed-off-by: Damien George <damien@micropython.org>
Targets without the `errno` module enabled will not render `OSError`s
with the name of the error. Instead they just print the numeric error
code.
Add support for such targets by explicitly recognising certain error codes.
Signed-off-by: Damien George <damien@micropython.org>
Changes in this commit:
- Handle SerialException on Windows when device disconnects.
- Print clean 'device disconnected' message instead of stack trace.
- Fix terminal formatting issues on Linux after disconnect.
- Return disconnected state after console cleanup to avoid terminal issues.
This ensures proper disconnect messages on both platforms without showing
confusing error traces to users.
Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
The existing `mpremote df` command is not very good, because it needs to
assume that all directories in the root directory are mount points, and
also doesn't correctly stat filesystems when the current directory is not
the root. This leads to wrong results
With the introduction of `vfs.mount()` to return a list of mounted
filesystems and their path, a much better df can be implemented, as done in
this commit.
The new df will also fall back to using the old approach of listing the
root directory if the no-arg `vfs.mount()` query is not supported.
Signed-off-by: Damien George <damien@micropython.org>
Add `mpremote fs tree` command to show a tree of the device's files. It:
- Shows a treeview from current path or specified path.
- Uses the graph chars ("├── ", "└── ") (not configurable).
- Has the options:
-v/--verbose adds the serial device name to the top of the tree
-s/--size add a size to the files
-h/--human add a human readable size to the files
Signed-off-by: Jos Verlinde <jos_verlinde@hotmail.com>
When using `mip install`, if a file that needs to be downloaded already
exists locally, then the hash of that local file will be computed and if it
matches the known hash of the remote file it will not be downloaded.
Hashes in mip are guaranteed unique, so this change should never leave
stale files on the filesystem.
This behaviour follows that of the `mip` package in `micropython-lib`.
Signed-off-by: Damien George <damien@micropython.org>
Removes the risk of inadvertently deleting files on the host by preventing
the deletion of files via `rm -r` on the `/remote` vfs mount point.
Fixes issue #17147.
Signed-off-by: Jos Verlinde <jos_verlinde@hotmail.com>
This rewrites the code that previously manually emitted and caught various
OSError subclasses with equivalent code that uses the errno name dictionary
to do this generically, and updates the exception handler in do_filesystem
to catch them in a similarly-generic fashion using os.strerror to retrieve
an appropriate error message text equivalent to the current messages.
Note that in the CPython environments where mpremote runs, the call to the
OSError constructor already returns an instance of the corresponding mapped
exception subtype.
Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
Because the `wbits` parameter was only added to `zlib.compress` in
CPython 3.11. Using `zlib.compressobj` makes the code compatible with much
older CPython versions.
Fixes issue #17140.
Signed-off-by: Damien George <damien@micropython.org>
Currently the tool allows writing an invalid ROMFS image, with a bad header
or images smaller than minimum size, and only checks the image extension.
This commit allows deploying a ROMFS with either a ".img" or ".romfs"
extension (in the future support may be added for other extensions that
have different semantics, eg a manifest), and validates the image header
before writing.
Signed-off-by: iabdalkader <i.abdalkader@gmail.com>
If a ROMFS is mounted then "/rom/lib" is usually in `sys.path` before the
writable filesystem's "lib" entry. The ROMFS directory cannot be installed
to, so skip it if found.
Signed-off-by: Damien George <damien@micropython.org>
These commands use the `vfs.rom_ioctl()` function to manage the ROM
partitions on a device, and create and deploy ROMFS images.
Signed-off-by: Damien George <damien@micropython.org>
This significantly speeds up readline on files opened directly from an
mpremote mount.
Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
URLs in `package.json` may now be specified relative to the base URL of the
`package.json` file.
Relative URLs wil work for `package.json` files installed from the web as
well as local file paths.
Docs: update `docs/reference/packages.rst` to add documentation for:
- Installing packages from local filesystems (PR #12476); and
- Using relative URLs in the `package.json` file (PR #12477);
- Update the packaging example to encourage relative URLs as the default
in `package.json`.
Add `tools/mpremote/tests/test_mip_local_install.sh` to test the
installation of a package from local files using relative URLs in the
`package.json`.
Signed-off-by: Glenn Moloney <glenn.moloney@gmail.com>
Add support for `mpremote mip install package.json` where `package.json` is
a json file on the local filesystem.
Without this, package json files can only be loaded from http, https,
github or gitlab URLs.
This is useful for testing `package.json` files for pacages in development
and for constructing one's own `package.json` files for Python packages
which are not yet available for installation using mip.
Signed-off-by: Glenn Moloney <glenn.moloney@gmail.com>
And use it in `enter_raw_repl()`. This prevents waiting forever for a
serial device that does not respond to the Ctrl-C/Ctrl-D/etc commands and
is constantly outputting data.
Signed-off-by: Hans Maerki <buhtig.hans.maerki@ergoinfo.ch>
If the target does not return any data then `read_until()` will block
indefinitely. Fix this by making the initial read part of the general read
look, which always checks `inWaiting() > 0` before reading from the serial
device.
Also added the UART timeout to the constructor. This is not currently used
but may be used as an additional safeguard.
Signed-off-by: Hans Maerki <buhtig.hans.maerki@ergoinfo.ch>
This fixes a regression in db59e55fe7: prior
to that commit `mpremote` supported trailing slashes on the destination of
a normal (non-recursive) copy.
Add back support for that, with the semantics that a trailing slash
requires the destination to be an existing directory.
Also add a test for this.
Signed-off-by: Damien George <damien@micropython.org>
mpremote error messages now go to stderr, so make sure stdout is flushed
before printing them.
Also update the test runner to capture error messages.
Signed-off-by: Damien George <damien@micropython.org>
The variable `written` was being used before it was defined in the
`fs_writefile()` method of the Transport class. This was causing an
`UnboundLocalError` to be raised when the `progress_callback` was not
provided.
Fixes issue #16084.
Signed-off-by: Glenn Moloney <glenn.moloney@gmail.com>
A long time ago when there was only the `stm` port, Ctrl-C would trigger a
preemptive NLR jump to break out of running code. Then in commit
124df6f8d0 a more general approach to
asynchronous `KeyboardInterrupt` exceptions was implemented, and `stmhal`
supported both approaches, with the general (soft) interrupt taking
priority.
Then in commit bc1488a05f `pyboard.py` was
updated with a corresponding change to make it issue a double Ctrl-C to
break out of any existing code when entering the raw REPL (two Ctrl-C
characters were sent in order to more reliably trigger the preemptive NLR
jump).
No other port has preemptive NLR jumps and so a double Ctrl-C doesn't
really behave any differently to a single Ctrl-C: with USB CDC the double
Ctrl-C would most likely be in the same USB packet and so processed in the
same low-level USB callback, so it's just setting the keyboard interrupt
flag twice in a row. The VM/runtime then just sees one keyboard interrupt
and acts as though only one Ctrl-C was sent.
This commit changes the double Ctrl-C to a single Ctrl-C in `pyboard.py`
and `mpremote`. That keeps things as simple as they need to be.
Signed-off-by: Damien George <damien@micropython.org>
This adds a -f/--force option to the "cp" command, which forces
unconditional copies, in particular does not check the hash.
Signed-off-by: Damien George <damien@micropython.org>
These tests are specifically for the command-line interface and cover:
- resume/soft-reset/connect/disconnect
- mount
- fs cp,touch,mkdir,cat,sha256sum,rm,rmdir
- eval/exec/run
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Signed-off-by: Damien George <damien@micropython.org>
Makes the filesystem command give standard error messages rather than
just printing the exception from the device.
Makes the distinction between CommandError and TransportError clearer.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Changes in this commit:
- Adds transport API `fs_hashfile` to compute the hash of a file with given
algorithm.
- Adds commands `mpremote <...>sum file` to compute and print hashes of
various algorithms.
- Adds shortcut `mpremote sha256sum file`.
- Uses the hash computation to improve speed of recursive file copy to
avoid copying a file where the target is identical.
For recursive copy, if possible it will use the board's support (e.g.
built-in hashlib or hashlib from micropython-lib), but will fall back to
downloading the file and using the local implementation.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Signed-off-by: Damien George <damien@micropython.org>
This introduces a Python filesystem API on `Transport` that is implemented
entirely with eval/exec provided by the underlying transport subclass.
Updates existing mpremote filesystem commands (and `edit) to use this API.
Also re-implements recursive `cp` to allow arbitrary source / destination.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Signed-off-by: Damien George <damien@micropython.org>
This is a step towards making the transport expose a Python API rather than
functions that mostly print to stdout.
Most use cases of `transport.eval()` are to get some state back from the
device, so have it return as a value directly by default.
Updates uses of `transport.eval()` to remove the parse argument where it
now isn't needed, make the `rtc` command use eval/exec, and update the
`mip` command to use eval's parsing.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This is a fix for an algorithmic error in mpremote mip, that throws an
error due to a '\n' used in the concatenation and split when there is more
than one lib path in `sys.path`.
Signed-off-by: Jos Verlinde <Jos.Verlinde@microsoft.com>
Prior to this fix the current working path in the remote VFS would always
be prepended to the requested path to get the full path, even if the
requested path was already absolute, ie starting with "/".
So `os.chdir("/remote/dir1")` would set the working path to "/dir1/", and
a subsequent call with an absolute path like `os.listdir("/remote/dir2")`
would try to list the directory "/dir1/dir2/".
Fixes issue #15308.
Signed-off-by: Damien George <damien@micropython.org>
No longer require importlib_metadata on new Python versions as it is
included in the standard distribution.
Signed-off-by: Jochen Sprickerhof <git@jochen.sprickerhof.de>
Speeds up importing files from mounted filesystem.
Also fix the return code for invalid / unsupported ioctl requests.
Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
This updates a small number of files that change with ruff-format's (vs
black's) rules.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
See https://github.com/micropython/micropython/issues/12127 for details.
Previously at the point when a release is made, we update mpconfig.h
and set a git tag. i.e. the version increments at the release.
Now the version increments immediately after the release. The workflow is:
1. Final commit in the cycle updates mpconfig.h to set (X, Y, 0, 0) (i.e.
clear the pre-release state).
2. This commit is tagged "vX.Y.0".
3. First commit for the new cycle updates mpconfig.h to set (X, Y+1, 0, 1)
(i.e. increment the minor version, set the pre-release state).
4. This commit is tagged "vX.Y+1.0-preview".
The idea is that a nightly build is actually a "preview" of the _next_
release. i.e. any documentation describing the current release may not
actually match the nightly build. So we use "preview" as our semver
pre-release identifier.
Changes in this commit:
- Add MICROPY_VERSION_PRERELEASE to mpconfig.h to allow indicating that
this is not a release version.
- Remove unused MICROPY_VERSION integer.
- Append "-preview" to MICROPY_VERSION_STRING when the pre-release state
is set.
- Update py/makeversionhdr.py to no longer generate MICROPY_GIT_HASH.
- Remove the one place MICROPY_GIT_HASH was used (it can use
MICROPY_GIT_TAG instead).
- Update py/makeversionhdr.py to also understand
MICROPY_VERSION_PRERELEASE in mpconfig.h.
- Update py/makeversionhdr.py to convert the git-describe output into
semver-compatible "X.Y.Z-preview.N.gHASH".
- Update autobuild.sh to generate filenames using the new scheme.
- Update remove_old_firmware.py to match new scheme.
- Update mpremote's pyproject.toml to handle the "-preview" suffix in the
tag. setuptools_scm maps to this "rc0" to match PEP440.
- Fix docs heading where it incorrectly said "vvX.Y.Z" for release docs.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
