When subclassing a native type, calling native members in `__init__` before
`super().__init__()` has been called could cause a crash. In this
situation, `self` in `mp_convert_member_lookup` is the
`native_base_init_wrapper_obj`. The check added in this commit ensures
that an `AttributeError` is raised before this happens, which is consistent
with other failed lookups.
Also fix a typo in a related comment.
Signed-off-by: Laurens Valk <laurens@pybricks.com>
This adds a separate `AdvancedTimer` class that demonstrates a few more
advanced concepts usch as custom handlers for printing and attributes.
Signed-off-by: Laurens Valk <laurens@pybricks.com>
Updates rp2 port to always resume from idle within 1ms max.
When rp2 port went tickless the behaviour of machine.idle() changed as
there is no longer a tick interrupt to wake it up every millisecond. On a
quiet system it would now block indefinitely. No other port does this.
See parent commit for justification of why this change is useful.
Also adds a test case that fails without this change.
This work was funded through GitHub Sponsors.
Signed-off-by: Angus Gratton <angus@redyak.com.au>
Before the fix in parent commit, some of these tests hung indefinitely.
After, they seem to consistently pass.
This work was funded through GitHub Sponsors.
Signed-off-by: Angus Gratton <angus@redyak.com.au>
Explicitly yield each time a thread mutex is unlocked.
Key to understanding this bug is that Python threads run at equal RTOS
priority, and although ESP-IDF FreeRTOS (and I think vanilla FreeRTOS)
scheduler will round-robin equal priority tasks in the ready state it does
not make a similar guarantee for tasks moving between ready and waiting.
The pathological case of this bug is when one Python thread task is busy
(i.e. never blocks) it will hog the CPU more than expected, sometimes for
an unbounded amount of time. This happens even though it periodically
unlocks the GIL to allow another task to run.
Assume T1 is busy and T2 is blocked waiting for the GIL. T1 is executing
and hits a condition to yield execution:
1. T1 calls MP_THREAD_GIL_EXIT
2. FreeRTOS sees T2 is waiting for the GIL and moves it to the Ready list
(but does not preempt, as T2 is same priority, so T1 keeps running).
3. T1 immediately calls MP_THREAD_GIL_ENTER and re-takes the GIL.
4. Pre-emptive context switch happens, T2 wakes up, sees GIL is not
available, and goes on the waiting list for the GIL again.
To break this cycle step 4 must happen before step 3, but this may be a
very narrow window of time so it may not happen regularly - and
quantisation of the timing of the tick interrupt to trigger a context
switch may mean it never happens.
Yielding at the end of step 2 maximises the chance for another task to run.
Adds a test that fails on esp32 before this fix and passes afterwards.
Fixes issue #15423.
This work was funded through GitHub Sponsors.
Signed-off-by: Angus Gratton <angus@redyak.com.au>
Fixes various null dereferencing, out-of-bounds memory accesses and
`assert(0)` failures in the case of an invalid `uctypes` descriptor.
By design `uctypes` can crash because it accesses arbitrary memory, but at
least describing the descriptor layout should be forced to be correct and
not crash.
Fixes issue #12702.
Signed-off-by: stijn <stijn@ignitron.net>
Fixes use-after-free when accessing the database after it is closed with
`btree_close`. `btree_close` always succeeds when called with an
already-closed database.
The new test checks that operations that access the underlying database
(get, set, flush, seq) fail with a `ValueError` when the btree is already
closed. It also checks that closing and printing the btree succeed when
the btree is already closed.
Fixes issue #12543.
Signed-off-by: Michael Vornovitsky <michaelvornovitskiy@outlook.com>
This commit makes it so that PyProxy objects are reused (on the JavaScript
side) when they correspond to an existing Python object that is the same
object.
For example, proxying the same Python function to JavaScript, the same
PyProxy instance is now used. This means that if `foo` is a Python
function then accessing it on the JavaScript side such as
`api.globals().get("foo")` has the property that:
api.globals().get("foo") === api.globals().get("foo")
Prior to this commit the above was not true because new PyProxy instances
were created each time `foo` was accessed.
Signed-off-by: Damien George <damien@micropython.org>
Follow up to 2e852522b1: instead of having
.exp files for the get_event_loop tests, tweak them so they are compatible
with CPython 3.12. This requires calling `asyncio.set_event_loop()` so
there is an active event loop and `asyncio.get_event_loop()` succeeds
without a warning.
Signed-off-by: Damien George <damien@micropython.org>
A target may have enough RAM to run the n=433 test but then run out of RAM
on the n=432 test. So allow the test to skip on the n=432 case before it
prints any output.
Signed-off-by: Damien George <damien@micropython.org>
Because the main thread executes `thread_entry()` it means there's an
additional one added to `count`, so the test must wait for the count to
reach `n_thread + 1`.
Signed-off-by: Damien George <damien@micropython.org>
Enable support for cipher suites like
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, as suggested in
https://github.com/micropython/micropython/issues/14204#issuecomment-2024366349
and https://github.com/micropython/micropython/issues/10485#issuecomment-1396426824
Tests have been run on the top 500 domains from moz.com. Without this
patch, 155 out of 500 fail to connect because of TLS issues. This patch
fixes them all. And it seems all existing mbedtls flags are needed to get
good coverage of those top 500 domains.
The `ssl_poll.py` test has the cipher bits increased from 512 to 1024 in
its test key/cert so that it can work with ECDHE-RSA which is now the
chosen cipher.
Signed-off-by: Sylvain Zimmer <sylvain@sylvainzimmer.com>
Signed-off-by: Damien George <damien@micropython.org>
Before this change, long/mpz ints propagated into all future calculations,
even if their value could fit in a small-int object. With this change, the
result of a big-int binary op will now be converted to a small-int object
if the value fits in a small-int.
For example, a relatively common operation like `x = a * b // c` where
a,b,c all small ints would always result in a long/mpz int, even if it
didn't need to, and then this would impact all future calculations with
x.
This adds +24 bytes on PYBV11 but avoids heap allocations and potential
surprises (e.g. `big-big` is now a small `0`, and can safely be accessed
with MP_OBJ_SMALL_INT_VALUE).
Performance tests are unchanged on PYBV10, except for `bm_pidigits.py`
which makes heavy use of big-ints and gains about 8% in speed.
Unix coverage tests have been updated to cover mpz code that is now
unreachable by normal Python code (removing the unreachable code would lead
to some surprising gaps in the internal C functions and the functionality
may be needed in the future, so it is kept because it has minimal
overhead).
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
In JavaScript when accessing an attribute such as `obj.attr` a value of
`undefined` is returned if the attribute does not exist. This is unlike
Python semantics where an `AttributeError` is raised. Furthermore, in some
cases in JavaScript (eg a Proxy instance) `attr in obj` can return false
yet `obj.attr` is still valid and returns something other than `undefined`.
So the source of truth for whether a JavaScript attribute exists is to just
right away attempt `obj.attr`.
To more closely match these JavaScript semantics when proxying a JavaScript
object through to Python, change the attribute lookup logic on a `JsProxy`
so that it immediately attempts `obj.attr` instead of first testing if the
attribute exists via `attr in obj`.
This allows JavaScript objects which dynamically create attributes to work
correctly on the Python side, with both `obj.attr` and `obj["attr"]`. Note
that `obj["attr"]` already works in all cases because it immediately does
the subscript access without first testing if the attribute exists.
As a benefit, this new behaviour matches the Pyodide behaviour.
Signed-off-by: Damien George <damien@micropython.org>
Follow-up to a84c7a0ed9, this commit works most of the time but has an
intermittent bug where USB doesn't resume as expected after waking from
light sleep.
Turns out waking calls clocks_init() which will re-initialise the USB PLL.
Most of the time this is OK but occasionally it seems like the clock
glitches the USB peripheral and it stops working until the next hard reset.
Adds a machine.lightsleep() test that consistently hangs in the first
two dozen iterations on rp2 without this fix. Passed over 100 times in a
row with this fix.
The test is currently rp2-only as it seems similar lightsleep USB issues
exist on other ports (both pyboard and ESP32-S3 native USB don't send any
data to the host after waking, until they receive something from the host
first.)
This work was funded through GitHub Sponsors.
Signed-off-by: Angus Gratton <angus@redyak.com.au>
mp_thread_begin_atomic_section() is expected to be recursive (i.e. for
nested machine.disable_irq() calls, or if Python code calls disable_irq()
and then the Python runtime calls mp_handle_pending() which also enters an
atomic section to check the scheduler state).
On rp2 when not using core1 the atomic sections are recursive.
However when core1 was active (i.e. _thread) then there was a bug that
caused the core to live-lock if an atomic section recursed.
Adds a test case specifically for mutual exclusion and recursive atomic
sections when using two threads. Without this fix the test immediately
hangs on rp2.
This work was funded through GitHub Sponsors.
Signed-off-by: Angus Gratton <angus@redyak.com.au>
Fixes and improvements to `int.to_bytes()` are:
- No longer overflows if byte size is 0 (closes#13041).
- Raises OverflowError in any case where number won't fit into byte length
(now matches CPython, previously MicroPython would return a truncated
bytes object).
- Document that `micropython int.to_bytes()` doesn't implement the optional
signed kwarg, but will behave as if `signed=True` when the integer is
negative (this is the current behaviour). Add tests for this also.
Requires changes for small ints, MPZ large ints, and "long long" large
ints.
Adds a new set of unit tests for ints between 32 and 64 bits to increase
coverage of "long long" large ints, which are otherwise untested.
Tested on unix port (64 bit small ints, MPZ long ints) and Zephyr STM32WB
board (32 bit small ints, long long large ints).
This work was funded through GitHub Sponsors.
Signed-off-by: Angus Gratton <angus@redyak.com.au>
The code generating the entry to the finally handler of an async-with
statement was simply wrong for the case of the native emitter. Among other
things the layout of the stack was incorrect.
This is fixed by this commit. The setup of the async-with finally handler
is now put in a dedicated emit function, for both the bytecode and native
emitters to implement in their own way (the bytecode emitter is unchanged,
just factored to a function).
With this fix all of the async-with tests now work when using the native
emitter.
Signed-off-by: Damien George <damien@micropython.org>
A value thrown/injected into a native generator needs to be stored in a
dedicated variable outside `nlr_buf_t`, following the `inject_exc` variable
in `py/vm.c`.
Signed-off-by: Damien George <damien@micropython.org>
Otherwise GC stays disabled (not re-enabled by soft reset) and later test
runs fail with MemoryError.
Signed-off-by: Angus Gratton <angus@redyak.com.au>
This allows increasing the Python recursion depth if needed.
Also increase the default to 2k words. There is enough RAM in the
browser/node context for this to be increased, and having a larger pystack
allows more complex code to run without hitting the limit.
Signed-off-by: Damien George <damien@micropython.org>
In the webassembly port there is no asyncio run loop running at the top
level. Instead the Python asyncio run loop is scheduled through setTimeout
and run by the outer JavaScript event loop. Because tasks can become
runable from an external (to Python) event (eg a JavaScript callback), the
run loop must be scheduled whenever a task is pushed to the asyncio task
queue, otherwise tasks may be waiting forever on the queue.
Signed-off-by: Damien George <damien@micropython.org>
This change allows doing a top-level await on an asyncio primitive like
Task and Event.
This feature enables a better interaction and synchronisation between
JavaScript and Python, because `api.runPythonAsync` can now be used (called
from JavaScript) to await on the completion of asyncio primitives.
Signed-off-by: Damien George <damien@micropython.org>
This adds a QEMU-based bare metal RISC-V 32 bits port. For the time being
only QEMU's "virt" 32 bits board is supported, using the ilp32 ABI and the
RV32IMC architecture.
The top-level README and the run-tests.py files are updated for this new
port.
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
If the socket timeout is 0 then a failed socket.connect() raises
EINPROGRESS (which is what the lwIP bindings already did), but if the
socket timeout is non-zero then a failed socket.connect() should raise
ETIMEDOUT. The latter is fixed in this commit.
A test is added for these timeout cases.
Signed-off-by: Damien George <damien@micropython.org>
Support for raw str/bytes already exists, and extending that to raw
f-strings is easy. It also reduces code size because it eliminates an
error message.
Signed-off-by: Damien George <damien@micropython.org>
This is quite a simple and small change to support concatenation of
adjacent f-strings, and improve compatibility with CPython.
Signed-off-by: Damien George <damien@micropython.org>
In CPython 3.12 these invalid str/bytes/fstring escapes will issue a
SyntaxWarning, and so differ to MicroPython.
Signed-off-by: Damien George <damien@micropython.org>
Otherwise CPython gives a deprecation warning.
This test is not actually testing inversion of bools, rather that bit of
the test is used to compute the pass/fail result.
Signed-off-by: Damien George <damien@micropython.org>
The `cert` argument passed to the verify callback is actually a memoryview.
And the `depth` argument seems to start at 1 for the tested URL.
Signed-off-by: Damien George <damien@micropython.org>
The three-argument form of `.throw()` is deprecated since CPython 3.12. So
split out into separate tests (with .exp files) the parts of the generator
tests that test more than one argument.
Signed-off-by: Damien George <damien@micropython.org>
And use `asyncio.new_event_loop()` where possible. This change is needed
because CPython 3.12 deprecated the `get_event_loop()` function.
Signed-off-by: Damien George <damien@micropython.org>
Also define `mp_type_bytearray`. These all help to write native modules.
Signed-off-by: Brian Pugh <bnp117@gmail.com>
Signed-off-by: Damien George <damien@micropython.org>
Instead of raising KeyError. These semantics match JavaScript behaviour
and make it much more seamless to pass Python dicts through to JavaScript
as though they were JavaScript {} objects.
Signed-off-by: Damien George <damien@micropython.org>
This adds a new undefined singleton to Python, that corresponds directly to
JavaScript `undefined`. It's accessible via `js.undefined`.
Signed-off-by: Damien George <damien@micropython.org>
This reverts part of commit fa23e4b093, to
make it so that Python `None` converts to JavaScript `null` (and JavaScript
`null` already converts to Python `None`). That's consistent with how the
`json` module converts these values back and forth.
Signed-off-by: Damien George <damien@micropython.org>
And change Py None conversion so it converts to JS undefined.
The semantics for conversion of these objects are then:
- Python None -> JavaScript undefined
- JavaScript undefined -> Python None
- JavaScript null -> Python None
This follows Pyodide:
https://pyodide.org/en/stable/usage/type-conversions.html
Signed-off-by: Damien George <damien@micropython.org>
This commit defines a new `JsException` exception type which is used on the
Python side to wrap JavaScript errors. That's then used when a JavaScript
Promise is rejected, and the reason is then converted to a `JsException`
for the Python side to handle.
This new exception is exposed as `jsffi.JsException`.
Signed-off-by: Damien George <damien@micropython.org>
JavaScript semantics are such that the caller of an async function does not
need to await that function for it to run to completion. This commit makes
that behaviour also apply to top-level async Python code run via
`runPythonAsync()`.
Signed-off-by: Damien George <damien@micropython.org>
The `reason` in a rejected promise should be an instance of `Error`. That
leads to better error messages on the JavaScript side.
Signed-off-by: Damien George <damien@micropython.org>
This commit adds a significant portion of the existing MicroPython asyncio
module to the webassembly port, using parts of the existing asyncio code
and some custom JavaScript parts.
The key difference to the standard asyncio is that this version uses the
JavaScript runtime to do the actual scheduling and waiting on events, eg
Promise fulfillment, timeouts, fetching URLs.
This implementation does not include asyncio.run(). Instead one just uses
asyncio.create_task(..) to start tasks and then returns to the JavaScript.
Then JavaScript will run the tasks.
The implementation here tries to reuse as much existing asyncio code as
possible, and gets all the semantics correct for things like cancellation
and asyncio.wait_for. An alternative approach would reimplement Task,
Event, etc using JavaScript Promise's. That approach is very difficult to
get right when trying to implement cancellation (because it's not possible
to cancel a JavaScript Promise).
Signed-off-by: Damien George <damien@micropython.org>
Two cases, one assigning to a slice.
Closes https://github.com/micropython/micropython/issues/13283
Second is extending a slice from itself, similar logic.
In both cases the problem occurs when m_renew causes realloc to move the
buffer, leaving a dangling pointer behind.
There are more complex and hard to fix cases when either argument is a
memoryview into the buffer, currently resizing to a new address breaks
memoryviews into that object.
Reproducing this bug and confirming the fix was done by running the unix
port under valgrind with GC-aware extensions.
Note in default configurations with GIL this bug exists but has no impact
(the free buffer won't be reused while the function is still executing, and
is no longer referenced after it returns).
Signed-off-by: Angus Gratton <angus@redyak.com.au>
This optimises the case where a Python function is, for example, stored to
a JavaScript attribute and then later retrieved from Python. The Python
function no longer needs to be a proxy with double proxying needed for the
call from Python -> JavaScript -> Python.
Signed-off-by: Damien George <damien@micropython.org>
Ports that use axtls cannot run the `test_tls_sites.py` test because the
sites it connects to use advanced ciphers. So skip this test on such
ports, and add a new, simpler test that doesn't require certificate
verification and works with axtls.
Signed-off-by: Damien George <damien@micropython.org>
This is required because the .mpy native ABI was changed by the
introduction of `mp_proto_fun_t`, see commits:
- 416465d81e
- 5e3006f117
- e2ff00e811
And three `mp_binary` functions were added to `mp_fun_table` in
commit d2276f0d41.
Signed-off-by: Damien George <damien@micropython.org>
This allows running tests with a .js/.mjs suffix, and also .py tests using
node and the webassembly port.
Signed-off-by: Damien George <damien@micropython.org>
This commit implements the 'e' half-float format: 10-bit mantissa, 5-bit
exponent. It uses native _Float16 if supported by the compiler, otherwise
uses custom bitshifting encoding/decoding routines.
Signed-off-by: Matthias Urlichs <matthias@urlichs.de>
Signed-off-by: Damien George <damien@micropython.org>
So that the MicroPython-specific behaviour can be isolated, and the CPython
compatible test don't need a .exp file.
Signed-off-by: Damien George <damien@micropython.org>
Add `pop()`, `appendleft()`, and `extend()` methods, support iteration
and indexing, and initializing from an existing sequence.
Iteration and indexing (subscription) have independent configuration flags
to enable them. They are enabled by default at the same level that
collections.deque is enabled (the extra features level).
Also add tests for checking new behavior.
Signed-off-by: Damien George <damien@micropython.org>
This new DMA API corrects possible cache coherency issues on chips with
D-Cache, when working with buffers at arbitrary memory locations (i.e.
supplied by Python code).
The API is used by SPI to fix an issue with corrupt data when reading from
SPI using DMA in certain cases. A regression test is included (it depends
on external hardware connection).
Explanation:
1) It's necessary to invalidate D-Cache after a DMA RX operation completes
in case the CPU reads (or speculatively reads) from the DMA RX region
during the operation. This seems to have been the root cause of issue
#13471 (only when src==dest for this case).
2) More generally, it is also necessary to temporarily mark the first and
last cache lines of a DMA RX operation as "uncached", in case the DMA
buffer shares this cache line with unrelated data. The CPU could
otherwise write the other data at any time during the DMA operation (for
example from an interrupt handler), creating a dirty cache line that's
inconsistent with the DMA result.
Fixes issue #13471.
This work was funded through GitHub Sponsors.
Signed-off-by: Angus Gratton <angus@redyak.com.au>
This test cannot run on boards that have a native USB REPL, so rename it to
indicate that its "special". This makes it easier to run a subset of
tests, for example:
./run-multitests.py multi_bluetooth/ble*.py
./run-multitests.py multi_bluetooth/perf*.py
./run-multitests.py multi_bluetooth/stress*.py
Signed-off-by: Damien George <damien@micropython.org>
When the websocket closes currently, it does not send a proper
"close"-frame, but rather encodes the 0x8800-sequence inside a binary
packet, which is wrong. The close packet is a different kind of websocket
frame, according to https://www.rfc-editor.org/rfc/rfc6455.
This change resolves an error in Firefox when the websocket closes.
Signed-off-by: Felix Dörre <felix@dogcraft.de>
This is now easy to support, since the first machine-word of a native
function tells how to find the prelude, from which the function name can be
extracted in the same way as for bytecode.
Signed-off-by: Damien George <damien@micropython.org>
Similar to the previous commit but for MP_BLUETOOTH_IRQ_GATTC_READ_DONE:
the pending_value_handle needs to be reset before calling
mp_bluetooth_gattc_on_read_write_status(), which will call the Python IRQ
handler, which may in turn call back into BTstack to perform an action like
a write. In that case the pending_value_handle will need to be available
for the write/read/etc to proceed.
Fixes issue #13634.
Signed-off-by: Damien George <damien@micropython.org>
The pending_value_handle needs to be freed and reset before calling
mp_bluetooth_gattc_on_read_write_status(), which will call the Python IRQ
handler, which may in turn call back into BTstack to perform an action like
a write. In that case the pending_value_handle will need to be available
for the write/read/etc to proceed.
Fixes issue #13611.
Signed-off-by: Damien George <damien@micropython.org>
The current `ssl` module has quite a few differences to the CPython
implementation. This change moves the MicroPython variant to a new `tls`
module and provides a wrapper module for `ssl` (in micropython-lib).
Users who only rely on implemented comparible behavior can continue to use
`ssl`, while users that rely on non-compatible behavior should switch to
`tls`. Then we can make the facade in `ssl` more strictly adhere to
CPython.
Signed-off-by: Felix Dörre <felix@dogcraft.de>
By moving to GitHub actions, all MicroPython CI builds are now on GitHub
actions. This allows faster parallel builds and saves time by not building
when no relevant files changed.
This reveals a few failing tests, so those are temporarily disabled until
they can be fixed.
Signed-off-by: David Lechner <david@pybricks.com>
Signed-off-by: Damien George <damien@micropython.org>
If a return is executed within the try block of a try-finally then the
return value is stored on the top of the Python stack during the execution
of the finally block. In this case the Python stack is one larger than it
normally would be in the finally block.
Prior to this commit, the compiler was not taking this case into account
and could have a Python stack overflow if the Python stack used by the
finally block was more than that used elsewhere in the function. In such
a scenario the last argument of the function would be clobbered by the
top-most temporary value used in the deepest Python expression/statement.
This commit fixes that case by making sure enough Python stack is allocated
to the function.
Fixes issue #13562.
Signed-off-by: Damien George <damien@micropython.org>
The timing of the DMA transfer can vary a bit, so tweak the allowed values.
Also test the return value of `rp2.DMA.irq.flags()` to make sure the IRQ is
correctly signalled.
Signed-off-by: Damien George <damien@micropython.org>
Changes include:
- Some mbedtls source files renamed or deprecated.
- Our `mbedtls_config.h` files are renamed to `mbedtls_config_port.h`, so
they don't clash with mbedtls's new default configuration file named
`mbedtls_config.h`.
- MBEDTLS_TLS_DEFAULT_ALLOW_SHA1_IN_KEY_EXCHANGE is deprecated.
- MBEDTLS_HAVE_TIME now requires an `mbedtls_ms_time` function to be
defined but it's only used for TLSv1.3 (currently not enabled in
MicroPython so there is a lazy implementation, i.e. seconds * 1000).
- `tests/multi_net/ssl_data.py` is removed (due to deprecation of
MBEDTLS_TLS_DEFAULT_ALLOW_SHA1_IN_KEY_EXCHANGE), there are the existing
`ssl_cert_rsa.py` and `sslcontext_server_client.py` tests which do very
similar, simple SSL data transfer.
- Tests now use an EC key by default (they are smaller and faster), and the
RSA key has been regenerated due to the old PKCS encoding used by openssl
rsa command, see
https://stackoverflow.com/questions/40822328/openssl-rsa-key-pem-and-der-conversion-does-not-match
(and `tests/README.md` has been updated accordingly).
Signed-off-by: Carlos Gil <carlosgilglez@gmail.com>
Prior to this commit it would skip every second cipher returned from
mbedtls.
The corresponding test is also updated and now passes on esp32, rp2, stm32
and unix.
Signed-off-by: Damien George <damien@micropython.org>
These were added back in commit c4935f3049
because the tests required CPython 3.8, which was quite new at the time.
But CPython 3.8 was released over 4 years ago (October 2019) and the CI
test runners, and developers, have this (or a more recent) CPython version.
Removing the .exp files also helps keep MicroPython semantics the same as
CPython.
The asyncio_fair.py test it adjusted slightly to have more deterministic
timing and output.
Signed-off-by: Damien George <damien@micropython.org>
Adds support to asyncio.gather() for the case that one or more (or all)
sub-tasks finish and/or raise an exception before the gather starts.
Signed-off-by: Damien George <damien@micropython.org>
Compare the full absolute path instead of relying on the path form
passed by the user.
For instance, this will make
python3 run-tests.py -d basics
python3 run-tests.py -d ./basics
python3 run-tests.py -d ../tests/basics
python3 run-tests.py -d /full/path/to/basics
all behave the same by correctly treating the bytes_compare3 and
builtin_help tests as special, whereas previously only the first
invocation would do that and hence result in these tests to fail
when called with a different path form.
Signed-off-by: stijn <stijn@ignitron.net>
Codespell doesn't pick up "re-used" or "re-uses", and ignores the tests/
directory, so fix these manually.
Signed-off-by: Damien George <damien@micropython.org>
Implement the typical 're-run the failed tests' most test runners have, for
convenience. Accessible via the new --run-failures argument, and
implemented using a json file containing a list of the failed tests.
Signed-off-by: stijn <stijn@ignitron.net>
Otherwise running the tests can take a long time when the server is a slow
target (RP2040 takes 23 seconds for a handshake when using 4096-bit RSA).
Also add instructions on how to generate elliptic curve key/certs.
Signed-off-by: Damien George <damien@micropython.org>
This matches the behaviour of run-tests.py, which sets cwd to the directory
containing the test script, which helps to isolate the filesystem.
It means that the SSL tests no longer need to know the name of their
containing directory to find the certificate files, and helps to run these
tests on bare-metal.
Signed-off-by: Damien George <damien@micropython.org>
The existing thread_sleep1.py test only tests execution, not accuracy, of
time.sleep. Also the existing test only tests sleep(0) on targets like rp2
that can only create a single thread.
The new test in this commit checks for timing accuracy on the main thread
and one other thread when they run at the same time.
Signed-off-by: Damien George <damien@micropython.org>
The aim of this commit is to make it so that the existing thread tests can
be used to test the _thread module on the rp2 port. The rp2 port only
allows up to one thread to be created at a time, and does not have the GIL
enabled.
The following changes have been made:
- run-tests.py skips mutation tests on rp2, because there's no GIL.
- run-tests.py skips other tests on rp2 that require more than one thread.
- The tests stop trying to start a new thread after there is an OSError,
which indicates that the system cannot create more threads.
- Some of these tests also now run the test function on the main thread,
not just the spawned threads.
- In some tests the output printing is adjusted so it's the same regardless
of how many threads were spawned.
- Some time.sleep(1) are replaced with time.sleep(0) to make the tests run
a little faster (finish sooner when the work is done).
For the most part the tests are unchanged for existing platforms like esp32
and unix.
Signed-off-by: Damien George <damien@micropython.org>
The poll_obj_t instances have their pollfd field point into this
allocation. So if re-allocating results in a move, we need to update the
existing poll_obj_t's.
Update the test to cover this case.
Fixes issue #12887.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This adds asyncio ssl support with SSLContext and the corresponding
tests in `tests/net_inet` and `tests/multi_net`.
Note that not doing the handshake on connect will delegate the handshake to
the following `mbedtls_ssl_read/write` calls. However if the handshake
fails when a client certificate is required and not presented by the peer,
it needs to be notified of this handshake error (otherwise it will hang
until timeout if any). Finally at MicroPython side raise the proper
mbedtls error code and message.
Signed-off-by: Carlos Gil <carlosgilglez@gmail.com>
Changes are:
- use ssl.SSLContext.wrap_socket instead of ssl.wrap_socket
- disable check_hostname and call load_default_certs() where appropriate,
to get CPython to run the tests correctly
- pass socket.AF_INET to getaddrinfo and socket.socket(), to force IPv4
- change tests to use github.com instead of google.com, because certificate
validation was failing with google.com
Signed-off-by: Damien George <damien@micropython.org>
And only enable this method when the relevant feature is available in
mbedtls. Otherwise, if mbedtls doesn't support getting the peer
certificate, this method always returns None and it's confusing why it does
that. It's better to remove the method altogether, so the error trying to
use it is more obvious.
Signed-off-by: Damien George <damien@micropython.org>
This commit adds:
1) Methods to SSLContext class that match CPython signature:
- `SSLContext.load_cert_chain(certfile, keyfile)`
- `SSLContext.load_verify_locations(cafile=, cadata=)`
- `SSLContext.get_ciphers()` --> ["CIPHERSUITE"]
- `SSLContext.set_ciphers(["CIPHERSUITE"])`
2) `sslsocket.cipher()` to get current ciphersuite and protocol
version.
3) `ssl.MBEDTLS_VERSION` string constant.
4) Certificate verification errors info instead of
`MBEDTLS_ERR_X509_CERT_VERIFY_FAILED`.
5) Tests in `net_inet` and `multi_net` to test these new methods.
`SSLContext.load_cert_chain` method allows loading key and cert from disk
passing a filepath in `certfile` or `keyfile` options.
`SSLContext.load_verify_locations`'s `cafile` option enables the same
functionality for ca files.
Signed-off-by: Carlos Gil <carlosgilglez@gmail.com>
Otherwise passing in a non-integer can lead to an invalid memory access.
Thanks to Junwha Hong and Wonil Jang @S2Lab, UNIST for finding the issue.
Fixes issue #13007.
Signed-off-by: Damien George <damien@micropython.org>
Set the position of new line tokens as the end of the preceding line
instead of the beginning of the next line. This is done by first moving
the pointer to the end of the current line to skip any whitespace, record
the position for the token, then finaly skip any other line and whitespace.
The previous behavior was to skip every new line and whitespace, including
the indent of the next line, before recording the token position.
(Note that both lex->emit_dent and lex->nested_bracket_level equal 0 if
had_physical_newline == true, which allows simplifying the if-logic for
MP_TOKEN_NEWLINE.)
And update the cmd_parsetree.py test expected output, because the position
of the new-line token has changed.
Fixes issue #12792.
Signed-off-by: Mathieu Serandour <mathieu.serandour@numworks.fr>
This updates a small number of files that change with ruff-format's (vs
black's) rules.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This test depends on the order in which qstrs are stored in ROM, which
affects the order in which `dir()` will probe the object to see what it
supports. Because of the lazy-loading in asyncio/__init__.py, if it
tries to do e.g. `wait_for_ms` before `funcs` then it will import funcs,
making `funcs` later succeed. But in the other way around, `funcs` will
initially not be found.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
These tests are designed to measure changes in performance relating to:
- string interning / searching for existing strings
- map lookup
- string operations
- string hashing
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
The unwritten API contract expected of a VFS.getcwd() by mp_vfs_getcwd()
is that its return value should be either "" or "/" when the CWD is at
the root of the VFS and otherwise start with a slash and not end with a
slash. This was not correctly implemented in VfsPosix for instances with
a non-empty root - the required leading slash, if any, was cut off
because the root length includes a trailing slash. This would result in
missing slashes in the middle of the return value of os.getcwd() or in
uninitialized garbage from beyond a string's null terminator when the
CWD was at the VFS root.
Signed-off-by: Christian Walther <cwalther@gmx.ch>
The unwritten API contract expected of a VFS by mp_vfs_lookup_path() is
that paths passed in are relative to the root of the VFS if they start
with '/' and relative to the current directory of the VFS otherwise.
This was not correctly implemented in VfsPosix for instances with a
non-empty root - all paths were interpreted relative to the root. Fix
that. Since VfsPosix tracks its CWD using the "external" CWD of the Unix
process, the correct handling for relative paths is to pass them through
unmodified.
Also, when concatenating absolute paths, fix an off-by-one resulting in
a harmless double slash (the root path already has a trailing slash).
Signed-off-by: Christian Walther <cwalther@gmx.ch>
These tests test an unrealistic situation and only pass by accident due
to a bug. The upcoming fix for the bug would make them fail.
The unrealistic situation is that VfsPosix methods are called with
relative paths while the current working directory is somewhere outside
of the root of the VFS. In the intended use of VFS objects via
os.mount() (as opposed to calling methods directly as the tests do),
this never happens, as mp_vfs_lookup_path() directs incoming calls to
the VFS that contains the CWD.
Make the testing situation realistic by changing the working directory
to the root of the VFS before calling methods on it, as the subsequent
relative path accesses expect.
Thanks to the preceding commit, the tests still pass, but still for the
wrong reason. The following commit "Fix relative paths on non-root VFS"
will make them pass for the correct reason.
Signed-off-by: Christian Walther <cwalther@gmx.ch>
A VfsPosix created with a relative root path would get confused when
chdir() was called on it and become unable to properly resolve absolute
paths, because changing directories effectively shifted its root. The
simplest fix for that would be to say "don't do that", but since the
unit tests themselves do it, fix it by making a relative path absolute
before storing it.
Signed-off-by: Christian Walther <cwalther@gmx.ch>
This wasn't correctly accounting for the bits-per-pixel and was returning a
bufinfo struct with the incorrect length. Instead, just forward directly
to the underlying buffer object.
Fixes issue #12563.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This ensures that the buffer is large enough for the specified width,
height, bits-per-pixel, and stride.
Also makes the legacy FrameBuffer1 constructor re-use the FrameBuffer
make_new to save some code size.
Fixes issue #12562.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This commit adds tests for bound method comparison and hashing to support
the changes in the previous commit.
Signed-off-by: Ned Konz <ned@productcreationstudio.com>
If a non-string buffer was passed to execfile, then it would be passed
as a non-null-terminated char* to mp_lexer_new_from_file.
This changes mp_lexer_new_from_file to take a qstr instead (as in almost
all cases a qstr will be created from this input anyway to set the
`__file__` attribute on the module).
This now makes execfile require a string (not generic buffer) argument,
which is probably a good fix to make anyway.
Fixes issue #12522.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
"Raise SomeException() from None" is a common Python idiom to suppress
chained exceptions and thus shouldn't trigger a warning on a version of
Python that doesn't support them in the first place.
The following code:
server = await asyncio.start_server(...)
async with server:
... code that raises ...
would lose the original exception because the server's task would not have
had a chance to be scheduled yet, and so awaiting the task in wait_closed
would raise the cancellation instead of the original exception.
Additionally, ensures that explicitly cancelling the parent task delivers
the cancellation correctly (previously was masked by the server loop), now
this only happens if the server was closed, not when the task was
cancelled.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
1. Remove the skip for detecting support for polling user-defined objects
as this is always possible now on all ports.
2. Don't print when the scheduled task runs as the ordering of this
relative to the other prints is dependent on other factors (e.g. if
using the native emitter).
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Update expired certificate, increase time validity period to five years and
fix command arguments typos in commentaries.
Signed-off-by: Carlos Gil <carlosgilglez@gmail.com>
The PYBD_SF2 is right on the limit of being able to run this test and so
it succeeds the first two cases and fails the next two with MemoryError.
This causes it to SKIP, but that only works if it's the first thing
printed. So reverse the order of the tests so it fails on the biggest
one first.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Increase asyncio tests timeouts to account for different WiFi modules and
CPU clocks on different boards.
Signed-off-by: iabdalkader <i.abdalkader@gmail.com>
In CPython, `_thread.start_new_thread()` returns an ID that is the same ID
that is returned by `_thread.get_ident()`. The current MicroPython
implementation of `_thread.start_new_thread()` always returns `None`.
This modifies the required functions to return a value. The native thread
id is returned since this can be used for interop with other functions, for
example, `pthread_kill()` on *nix. `_thread.get_ident()` is also modified
to return the native thread id so that the values match and avoids the need
for a separate `native_id` attribute.
Fixes issue #12153.
Signed-off-by: David Lechner <david@pybricks.com>
Add the buffer attribute to sys.stdin, sys.stdout and sys.stderr. This
provides raw access to underlying stdio streams for the unix port (and
others that use VfsPosix).
Signed-off-by: stephanelsmith <stephane.smith@titansensor.com>
Because mpy_ld.py doesn't know the target object representation, it emits
instances of `MP_OBJ_NEW_QSTR(MP_QSTR_Foo)` as const string objects, rather
than qstrs. However this doesn't work for map keys (e.g. for a locals dict)
because the map has all_keys_are_qstrs flag is set (and also auto-complete
requires the map keys to be qstrs).
Instead, emit them as regular qstrs, and make a functioning MP_OBJ_NEW_QSTR
function available (via `native_to_obj`, also used for e.g. making
integers).
Remove the code from mpy_ld.py to emit qstrs as constant strings, but leave
behind the scaffold to emit constant objects in case we want to do use this
in the future.
Strictly this should be a .mpy sub-version bump, even though the function
table isn't changing, it does lead to a change in behavior for a new .mpy
running against old MicroPython. `mp_native_to_obj` will incorrectly return
the qstr value directly as an `mp_obj_t`, leading to unexpected results.
But given that it's broken at the moment, it seems unlikely that anyone is
relying on this, so it's not work the other downsides of a sub-version bump
(i.e. breaking pure-Python modules that use @native). The opposite case of
running an old .mpy on new MicroPython is unchanged, and remains broken in
exactly the same way.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This adds support for the x format code in struct.pack and struct.unpack.
The primary use case for this is ignoring bytes while unpacking. When
interfacing with existing systems, it may often happen that you either have
fields in a struct that aren't properly specified or you simply don't care
about them. Being able to easily skip them is useful.
Signed-off-by: Daniël van de Giessen <daniel@dvdgiessen.nl>
Eliminate `TypeError` when format string contains no named conversions.
This matches CPython behavior.
Signed-off-by: mcskatkat <mc_skatkat@hotmail.com>
The "true" command by default is unavailable on windows so use
an equivalent which works on both unix and windows.
Signed-off-by: stijn <stijn@ignitron.net>
The only reason that const had to be disabled was to make the test output
match CPython when const was involved. Instead, this commit fixes the test
to handle the lines where const is used.
Also:
- remove the special handling for MICROPY_PERSISTENT_CODE_SAVE in
unix/mpconfigport.h, and make this automatic.
- move the check for MICROPY_PERSISTENT_CODE_SAVE to where it's used (like
we do for other similar checks) and add a comment explaining it.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This test was failing on CPython 3.11 as it now emits `0` as the line
number for the "call" event corresponding to import, where as in 3.6 it had
`1` as the line number.
We maintain the old behavior, but in order to make this test pass on both
CPython versions, the trace handler now converts the `0` to a `1`.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
An SSL stream can only handle CLOSE and POLL ioctls. Other ones do not
make sense, or at least it doesn't make sense to pass the ioctl request
directly down to the underlying stream.
In particular MP_STREAM_GET_FILENO should not be passed to the underlying
stream because the SSL stream is not directly related to a file descriptor,
and the SSL stream must handle the polling itself.
Signed-off-by: Damien George <damien@micropython.org>
Some targets (eg PYBV10) have the socket module but are unable to create
UDP sockets without a registered NIC. So skip UDP tests on these targets.
Signed-off-by: Damien George <damien@micropython.org>
Previously this was explicitly enabled on esp32/stm32/renesas/mimxrt/samd,
but didn't get a default feature level because it wasn't in py/mpconfig.h.
With this commit it's now enabled at the "extra features" level, which adds
rp2, unix-standard, windows, esp8266, webassembly, and some nrf boards.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This provides similar functionality to the former zlib.DecompIO and
especially CPython's gzip.GzipFile for both compression and decompression.
This class can be used directly, and also can be used from Python to
implement (via io.BytesIO) zlib.decompress and zlib.compress, as well as
gzip.GzipFile.
Enable/disable this on all ports/boards that zlib was previously configured
for.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This will be replaced with a new deflate module providing the same
functionality, with an optional frozen Python wrapper providing a
replacement zlib module.
binascii.crc32 is temporarily disabled.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Prior to this fix, async for assumed the iterator expression was a simple
identifier, and used that identifier as a local to store the intermediate
iterator object. This is incorrect behaviour.
This commit fixes the issue by keeping the iterator object on the stack as
an anonymous local variable.
Fixes issue #11511.
Signed-off-by: Damien George <damien@micropython.org>
This commit adds the SSLContext class to the ssl module, and retains the
existing ssl.wrap_socket() function to maintain backwards compatibility.
CPython deprecated the ssl.wrap_socket() function since CPython 3.7 and
instead one should use ssl.SSLContext().wrap_socket(). This commit makes
that possible.
For the axtls implementation:
- ssl.SSLContext is added, although it doesn't hold much state because
axtls requires calling ssl_ctx_new() for each new socket
- ssl.SSLContext.wrap_socket() is added
- ssl.PROTOCOL_TLS_CLIENT and ssl.PROTOCOL_TLS_SERVER are added
For the mbedtls implementation:
- ssl.SSLContext is added, and holds most of the mbedtls state
- ssl.verify_mode is added (getter and setter)
- ssl.SSLContext.wrap_socket() is added
- ssl.PROTOCOL_TLS_CLIENT and ssl.PROTOCOL_TLS_SERVER are added
The signatures match CPython:
- SSLContext(protocol)
- SSLContext.wrap_socket(sock, *, server_side=False,
do_handshake_on_connect=True, server_hostname=None)
The existing ssl.wrap_socket() functions retain their existing signature.
Signed-off-by: Damien George <damien@micropython.org>
Tests framebuf1 and framebuf2 do not take the need for byte-aligned
strides into consideration when calculating buffer lengths.
Accordingly, the buffers allocated are slightly too small. Fixed
buffer length calculations.
Signed-off-by: Duncan Lowther <Duncan.Lowther@glasgow.ac.uk>
Structure descriptor in test extmod/uctypes_array_assign_le
is 6 bytes long, due to member "arr3" having length 4
(2 * UINT16) and offset 2, but only 5 bytes are allocated.
Increased buffer length to 6 bytes.
Signed-off-by: Duncan Lowther <Duncan.Lowther@glasgow.ac.uk>
This allows existing code that does `import uasyncio` or
`import uasyncio as asyncio` to continue working.
It uses the same lazy-loading as asyncio to prevent loading of unused
features.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Instead of having a special set of arguments to test for each math-module
function, just test all functions with all sets of arguments. This gives
improved test cases to prevent regressions.
Signed-off-by: Damien George <damien@micropython.org>
PEP-498 allows for conversion specifiers like !r and !s to convert the
expression declared in braces to be passed through repr() and str()
respectively.
This updates the logic that detects the end of the expression to also stop
when it sees "![rs]" that is either at the end of the f-string or before
the ":" indicating the start of the format specifier. The "![rs]" is now
retained in the format string, whereas previously it stayed on the end
of the expression leading to a syntax error.
Previously: `f"{x!y:z}"` --> `"{:z}".format(x!y)`
Now: `f"{x!y:z}"` --> `"{!y:z}".format(x)`
Note that "!a" is not supported by `str.format` as MicroPython has no
`ascii()`, but now this will raise the correct error.
Updated cpydiff and added tests.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This is a MicroPython-specific module that existed to support the old
version of uasyncio. It's undocumented and not enabled on all ports and
takes up code size unnecessarily.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Make tests run in an isolated environment (i.e. `import io` would
otherwise get the `tests/io` directory).
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Make tests run in an isolated environment (i.e. `import io` would
otherwise get the `tests/io` directory).
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Make tests run in an isolated environment (i.e. `import io` would
otherwise get the `tests/io` directory).
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Prior to this commit, importing a module that exists but has a syntax error
or some other problem that happens at import time would result in a
potentially-incomplete module object getting added to sys.modules.
Subsequent imports would use that object, resulting in confusing error
messages that hide the root cause of the problem.
This commit fixes that issue by removing the failed module from sys.modules
using the new NLR callback mechanism.
Note that it is still important to add the module to sys.modules while the
import is happening so that we can support circular imports just like
CPython does.
Fixes issue #967.
Signed-off-by: David Grayson <davidegrayson@gmail.com>
When foo.bar is imported, bar is added as an attribute to foo. Previously
this happened on every import, but should only happen on first import.
This verifies the behavior for relative imports and overriding.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This verifies the behavior:
- Exact matches of built-ins bypass filesystem.
- u-prefix modules can be overridden from the filesystem.
- Builtin import can be forced using either u-prefix or sys.path=[].
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This demonstrates how to add a sub-package in a user c module, as well
as how to define the necessary qstrs and enable the feature in the build.
This is used by the unix coverage build to test this feature.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This generalises and simplifies the code and follows CPython behaviour.
See similar change for floats in a07fc5b640.
Signed-off-by: Damien George <damien@micropython.org>
This is possible now that MP_UNARY_OP_INT_MAYBE exists.
As a consequence mp_obj_get_int now also supports user types, which was
previously possible with MP_UNARY_OP_INT but no tests existed for it.
Signed-off-by: Damien George <damien@micropython.org>
To be consistent with MP_UNARY_OP_INT_FLOAT and MP_UNARY_OP_INT_COMPLEX,
and allow int() to first check if a type supports __int__ before trying
other things (as per CPython).
Signed-off-by: Damien George <damien@micropython.org>
MicroPython does not support these special methods, and they may get in the
way of other tests (eg indexing with __int__).
Signed-off-by: Damien George <damien@micropython.org>
The code that handles inplace-operator to normal-binary-operator fallback
is moved in this commit from py/objtype.c to py/runtime.c, making it apply
to all types, not just user classes.
Signed-off-by: Damien George <damien@micropython.org>
So that user types can implement reverse operators and have them work with
str on the left-hand-side, eg `"a" + UserType()`.
Signed-off-by: Damien George <damien@micropython.org>
This adds a unary_op implementation for the dict_view type that makes
the implementation of `hash()` for these types compatible with CPython.
Signed-off-by: David Lechner <david@pybricks.com>
As per https://bugs.python.org/issue408326, the slice object should not be
hashable. Since MicroPython has an implicit fallback when the unary_op
slot is empty, we need to fill this slot.
Signed-off-by: David Lechner <david@pybricks.com>
Previously when using --via-mpy, the file was compiled to tests/<tmp>.mpy
and then run using `micropython -m <tmp>` in the current cwd
(usually tests/). This meant that an import in the test would be resolved
relative to tests/.
This is different to regular (non-via-mpy) tests, where we run (for
example) `micropython basics/test.py` which means that an import would be
resolved relative to basics/.
Now --via-mpy matches the .py behavior. This is important because:
a) It makes it so import tests do the right thing.
b) There are directory names in tests/ that match built-in module names.
Furthermore, it always ensures the cwd (for both micropython and cpython)
is the test directory (e.g. basics/) rather than being left unset. This
also makes it clearer inside the test that e.g. file access is relative to
the Python file.
Updated tests with file paths to match.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
When a tuple is the condition of an if statement, it's only possible to
optimise that tuple away when it is a constant tuple (ie all its elements
are constants), because if it's not constant then the elements must be
evaluated in case they have side effects (even though the resulting tuple
will always be "true").
The code before this change handled the empty tuple OK (because it doesn't
need to be evaluated), but it discarded non-empty tuples without evaluating
them, which is incorrect behaviour (as show by the updated test).
This optimisation is anyway rarely applied because it's not common Python
coding practice to write things like `if (): ...` and `if (1, 2): ...`, so
removing this optimisation completely won't affect much code, if any.
Furthermore, when MICROPY_COMP_CONST_TUPLE is enabled, constant tuples are
already optimised by the parser, so expression with constant tuples like
`if (): ...` and `if (1, 2): ...` will continue to be optimised properly
(and so when this option is enabled the code that's deleted in this commit
is actually unreachable when the if condition is a constant tuple).
Signed-off-by: Damien George <damien@micropython.org>
ESP-NOW is a proprietary wireless communication protocol which supports
connectionless communication between ESP32 and ESP8266 devices, using
vendor specific WiFi frames. This commit adds support for this protocol
through a new `espnow` module.
This commit builds on original work done by @nickzoic, @shawwwn and with
contributions from @zoland. Features include:
- Use of (extended) ring buffers in py/ringbuf.[ch] for robust IO.
- Signal strength (RSSI) monitoring.
- Core support in `_espnow` C module, extended by `espnow.py` module.
- Asyncio support via `aioespnow.py` module (separate to this commit).
- Docs provided at `docs/library/espnow.rst`.
Methods available in espnow.ESPNow class are:
- active(True/False)
- config(): set rx buffer size, read timeout and tx rate
- recv()/irecv()/recvinto() to read incoming messages from peers
- send() to send messages to peer devices
- any() to test if a message is ready to read
- irq() to set callback for received messages
- stats() returns transfer stats:
(tx_pkts, tx_pkt_responses, tx_failures, rx_pkts, lost_rx_pkts)
- add_peer(mac, ...) registers a peer before sending messages
- get_peer(mac) returns peer info: (mac, lmk, channel, ifidx, encrypt)
- mod_peer(mac, ...) changes peer info parameters
- get_peers() returns all peer info tuples
- peers_table supports RSSI signal monitoring for received messages:
{peer1: [rssi, time_ms], peer2: [rssi, time_ms], ...}
ESP8266 is a pared down version of the ESP32 ESPNow support due to code
size restrictions and differences in the low-level API. See docs for
details.
Also included is a test suite in tests/multi_espnow. This tests basic
espnow data transfer, multiple transfers, various message sizes, encrypted
messages (pmk and lmk), and asyncio support.
Initial work is from https://github.com/micropython/micropython/pull/4115.
Initial import of code is from:
https://github.com/nickzoic/micropython/tree/espnow-4115.
Changes in this commit:
- Change MICROPY_HW_BOARD_NAME definition to match the product name.
- Rename board folder's name to match the product name style.
- Change related files like Makefile, document descriptions, test cases, CI
and tools.
Signed-off-by: Takeo Takahashi <takeo.takahashi.xv@renesas.com>
btstack only supports central-initiated, so this allows us to have a test
that works on both (ble_mtu.py), and then another one for just the NimBLE
supported behavior (ble_mtu_peripheral.py).
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
On unix, time.sleep is implemented as select(timeout=<time>) which means
that it does not run the poll hook during sleeping.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Following other vfs_fat tests, so the test works on ports like stm32 that
only support 512-byte block size.
Signed-off-by: Damien George <damien@micropython.org>
If a multitest calls `multitest.output_metric(...)` then that output will
be collected separately, not considered as part of the test verification
output, and instead be printed at the end. This is useful for tests that
want to output performance/timing metrics that may change from one run to
the next.
Signed-off-by: Damien George <damien@micropython.org>
When iterating over os.ilistdir(), the special directories '.' and '..'
are filtered from the results. But the code inadvertently also filtered
any file/directory which happened to match '..*'. This change fixes the
filter.
Fixes issue #11032.
Signed-off-by: Jeremy Rand <jeremy@rand-family.com>
When := is used in a comprehension the target variable is bound to the
parent scope, so it's either a global or a nonlocal. Prior to this commit
that was handled by simply using the parent scope's id_info for the
target variable. That's completely wrong because it uses the slot number
for the parent's Python stack to store the variable, rather than the slot
number for the comprehension. This will in most cases lead to incorrect
behaviour or memory faults.
This commit fixes the scoping of the target variable by explicitly
declaring it a global or nonlocal, depending on whether the parent is the
global scope or not. Then the id_info of the comprehension can be used to
access the target variable. This fixes a lot of cases of using := in a
comprehension.
Code size change for this commit:
bare-arm: +0 +0.000%
minimal x86: +0 +0.000%
unix x64: +152 +0.019% standard
stm32: +96 +0.024% PYBV10
cc3200: +96 +0.052%
esp8266: +196 +0.028% GENERIC
esp32: +156 +0.010% GENERIC[incl +8(data)]
mimxrt: +96 +0.027% TEENSY40
renesas-ra: +88 +0.014% RA6M2_EK
nrf: +88 +0.048% pca10040
rp2: +104 +0.020% PICO
samd: +88 +0.033% ADAFRUIT_ITSYBITSY_M4_EXPRESS
Fixes issue #10895.
Signed-off-by: Damien George <damien@micropython.org>
The device-under-test should use `multitest.expect_reboot()` to indicate
that it will reboot.
Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Prior to this fix, pow(1.5, inf) and pow(0.5, -inf) (among other things)
would incorrectly raise a ValueError, because the result is inf with the
first argument being finite. This commit fixes this by allowing the result
to be infinite if the first or second (or both) argument is infinite.
This fix doesn't affect the other three math functions that have two
arguments:
- atan2 never returns inf, so always fails isinf(ans)
- copysign returns inf only if the first argument x is inf, so will never
reach the isinf(y) check
- fmod never returns inf, so always fails isinf(ans)
Signed-off-by: Damien George <damien@micropython.org>
So it can run on targets with low memory, eg esp8266.
Also enable the viper_4args() sub-test, which is now supported.
Signed-off-by: Damien George <damien@micropython.org>
This is important for literal tuples, e.g.
f"{a,b,}, {c}" --> "{}".format((a,b), (c),)
which would otherwise result in either a syntax error or the wrong result.
Fixes issue #9635.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
32-bit platforms only support a slice offset start of 24 bit max due to the
limited size of the mp_obj_array_t.free member. Similarly on 64-bit
platforms the limit is 56 bits.
This commit adds an OverflowError if the user attempts to slice a
memoryview beyond this limit.
Signed-off-by: Damien George <damien@micropython.org>
In @micropython.native code the types of variables and expressions are
always Python objects, so they can be initialised as such. This prevents
problems with compiling optimised code like while-loops where a local may
be referenced before it is assigned to.
Signed-off-by: Damien George <damien@micropython.org>
During the initial handshake or subsequent renegotiation, the protocol
might need to read in order to write (or conversely to write in order
to read). It might be blocked from doing so by the state of the
underlying socket (i.e. there is no data to read, or there is no space
to write).
The library indicates this condition by returning one of the errors
`MBEDTLS_ERR_SSL_WANT_READ` or `MBEDTLS_ERR_SSL_WANT_WRITE`. When that
happens, we need to enforce that the next poll operation only considers
the direction that the library indicated.
In addition, mbedtls does its own read buffering that we need to take
into account while polling, and we need to save the last error between
read()/write() and ioctl().
The assertion that is added here (to gc.c) fails when running this new test
if ALLOC_TABLE_GAP_BYTE is set to 0.
Signed-off-by: Jeff Epler <jepler@gmail.com>
Signed-off-by: Damien George <damien@micropython.org>
The code was already checking for duplicate kwargs for named parameters but
if `**kwargs` was given as a parameter, it did not check for multiples of
the same argument name.
This fixes the issue by adding an addition test to catch duplicates and
adds a test to exercise the code.
Fixes issue #10083.
Signed-off-by: David Lechner <david@pybricks.com>
This test could occasionally fail because some operations take longer
than expected. This relaxes the timing constraints and defers printing
until the very end.
Signed-off-by: Laurens Valk <laurens@pybricks.com>
Now that the Timer class has been merged in a separate pull request,
this can be added to the module test too.
Signed-off-by: Laurens Valk <laurens@pybricks.com>
Implements dictionary union according to PEP 584's specifications, minus
the fact that dictionary entries are not guaranteed to be in insertion
order. This feature is enabled with MICROPY_CPYTHON_COMPAT.
Includes a new test.
With the assistance of Fangrui Qin <qinf@purdue.edu>
Signed-off-by: Rayane Chatrieux <rayane.chatrieux@gmail.com>
Signed-off-by: Damien George <damien@micropython.org>
This shows how ports can add their own custom types/classes.
It is part of the unix coverage build, so we can use it for tests too.
Signed-off-by: Laurens Valk <laurens@pybricks.com>
This will make mpy-cross auto-detect. Allow overriding for non-default
configurations (e.g. using 32-bit build of the unix port).
Also use armv7m by default for qemu-arm (the default qemu target is
Cortex-M3).
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This prevents a very subtle bug caused by writing e.g. `bytearray('\xfd')`
which gives you `(0xc3, 0xbd)`.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Gives the absolute path to the unix micropython binary.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Signed-off-by: Damien George <damien@micropython.org>
`b'\xaa \xaa'.count(b'\xaa')` now (correctly) returns 2 instead of 1.
Fixes issue #9404.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Allows optimisation of cases like:
import micropython
_DEBUG = micropython.const(False)
if _DEBUG:
print('Debugging info')
Previously the 'if' statement was only optimised out if the type of the
const() argument was integer.
The change is implemented in a way that makes the compiler slightly smaller
(-16 bytes on PYBV11) but compilation will also be very slightly slower.
As a bonus, if const support is enabled then the compiler can now optimise
const truthy/falsey expressions of other types, like:
while "something":
pass
... unclear if that is useful, but perhaps it could be.
Signed-off-by: Angus Gratton <angus@redyak.com.au>
This uses the frozentest.mpy that is also used by ports/minimal.
Also fixes two bugs that these new tests picked up:
- File extension matching in manifestfile.py.
- Handling of freeze_mpy results in makemanifest.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
frozentest.mpy was previously duplicated in ports/minimal and
ports/powerpc.
This needs to be re-generated on every .mpy version increase, so might as
well just have a single copy of it.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Prevent handle leaks when file objects aren't closed explicitly and
fix some MICROPY_CPYTHON_COMPAT issues: this wasn't properly adhered
to because #ifdef was used so it was always on, and closing files
multiple times should be avoided unconditionally.
This matches class `__dict__`, and is similarly gated on
MICROPY_CPYTHON_COMPAT. Unlike class though, because modules's globals are
actually dict instances, the result is a mutable dictionary.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
The intent is to allow us to make breaking changes to the native ABI (e.g.
changes to dynruntime.h) without needing the bytecode version to increment.
With this commit the two bits previously used for the feature flags (but
now unused as of .mpy version 6) encode a sub-version. A bytecode-only
.mpy file can be loaded as long as MPY_VERSION matches, but a native .mpy
(i.e. one with an arch set) must also match MPY_SUB_VERSION. This allows 3
additional updates to the native ABI per bytecode revision.
The sub-version is set to 1 because the previous commits that changed the
layout of mp_obj_type_t have changed the native ABI.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Signed-off-by: Damien George <damien@micropython.org>
The goal here is to remove a slot (making way to turn make_new into a slot)
as well as reduce code size by the ~40 references to mp_identity_getiter
and mp_stream_unbuffered_iter.
This introduces two new type flags:
- MP_TYPE_FLAG_ITER_IS_ITERNEXT: This means that the "iter" slot in the
type is "iternext", and should use the identity getiter.
- MP_TYPE_FLAG_ITER_IS_CUSTOM: This means that the "iter" slot is a pointer
to a mp_getiter_iternext_custom_t instance, which then defines both
getiter and iternext.
And a third flag that is the OR of both, MP_TYPE_FLAG_ITER_IS_STREAM: This
means that the type should use the identity getiter, and
mp_stream_unbuffered_iter as iternext.
Finally, MP_TYPE_FLAG_ITER_IS_GETITER is defined as a no-op flag to give
the default case where "iter" is "getiter".
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This is a latent issue that wasn't caught by CI because there was no
configuration that had both stackless+uasyncio.
The previous check to skip with stackless builds only worked when the
bytecode emitter was used by default. Force the check to use the bytecode
emitter.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
When iterating over filesystem/folders with os.iterdir(), an open file
(directory) handle is used internally. Currently this file handle is only
closed once the iterator is completely drained, eg. once all entries have
been looped over / converted into list etc.
If a program opens an iterdir but does not loop over it, or starts to loop
over the iterator but breaks out of the loop, then the handle never gets
closed. In this state, when the iter object is cleaned up by the garbage
collector this open handle can cause corruption of the filesystem.
Fixes issues #6568 and #8506.
Rather than drawing the entire boundary to catch missing pixels, just
detect the cases where boundary pixels are skipped during node calculation
and pre-emptively draw them then.
This adds 72 bytes on PYBV11, but makes filled poly() 20% faster.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Add method for drawing polygons.
For non-filled polygons, uses the existing line-drawing code to render
arbitrary polygons using the given coords list, at the given x,y position,
in the given colour.
For filled polygons, arbitrary closed polygons are rendered using a fast
point-in-polygon algorithm to determine where the edges of the polygon lie
on each pixel row.
Tests and documentation updates are also included.
Signed-off-by: Mat Booth <mat.booth@gmail.com>
Rework the conversion of floats to decimal strings so it aligns precisely
with the conversion of strings to floats in parsenum.c. This is to avoid
rendering 1eX as 9.99999eX-1 etc. This is achieved by removing the power-
of-10 tables and using pow() to compute the exponent directly, and that's
done efficiently by first estimating the power-of-10 exponent from the
power-of-2 exponent in the floating-point representation.
Code size is reduced by roughly 100 to 200 bytes by this commit.
Signed-off-by: Dan Ellis <dan.ellis@gmail.com>
This is useful in situations where the ThreadSafeFlag is reused and needs
to be cleared of any previous, unwanted event.
For example, clear the flag at the start of an operation, trigger the
operation (eg an I2C write), then (a)wait for an external event to set the
flag (eg a pin IRQ). Further events may trigger the flag again but these
are unwanted and should be cleared before the next cycle starts.
This commit adds the bytes methods to bytearray, matching CPython. The
existing implementations of these methods for str/bytes are reused for
bytearray with minor updates to match CPython return types.
For details on the CPython behaviour see
https://docs.python.org/3/library/stdtypes.html#bytes-and-bytearray-operations
The work to merge locals tables for str/bytes/bytearray/array was done by
@jimmo. Because of this merging of locals the change in code size for this
commit is mostly negative:
bare-arm: +0 +0.000%
minimal x86: +29 +0.018%
unix x64: -792 -0.128% standard[incl -448(data)]
unix nanbox: -436 -0.078% nanbox[incl -448(data)]
stm32: -40 -0.010% PYBV10
cc3200: -32 -0.017%
esp8266: -28 -0.004% GENERIC
esp32: -72 -0.005% GENERIC[incl -200(data)]
mimxrt: -40 -0.011% TEENSY40
renesas-ra: -40 -0.006% RA6M2_EK
nrf: -16 -0.009% pca10040
rp2: -64 -0.013% PICO
samd: +148 +0.105% ADAFRUIT_ITSYBITSY_M4_EXPRESS
The executable now lives in the build directory, and since the build
directory already contains the variant name there is no need to also add
it to the executable.
Signed-off-by: Damien George <damien@micropython.org>
Binaries built using the Make build system now no longer appear in the
working directory of the build, but rather in the build directory. Thus
some paths had to be adjusted.
The reallocation trigger for unpacking star args with unknown length
did not take into account the number of fixed args remaining. So it was
possible that the unpacked iterators could take up exactly the memory
allocated then nothing would be left for fixed args after the star args.
This causes a segfault crash.
This is fixed by taking into account the remaining number of fixed args
in the check to decide whether to realloc yet or not.
Signed-off-by: David Lechner <david@pybricks.com>
Formerly, py/formatfloat would print whole numbers inaccurately with
nonzero digits beyond the decimal place. This resulted from its strategy
of successive scaling of the argument by 0.1 which cannot be exactly
represented in floating point. The change in this commit avoids scaling
until the value is smaller than 1, so all whole numbers print with zero
fractional part.
Fixes issue #4212.
Signed-off-by: Dan Ellis dan.ellis@gmail.com
On ports with more than one filesystem, the type will be wrong, for example
if using LFS but FAT enabled, then the type will be FAT. So it's not
possible to use these classes to identify a file object type.
Furthermore, constructing an io.FileIO currently crashes on FAT, and
make_new isn't supported on LFS.
And the io.TextIOWrapper class does not match CPython at all.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This commit simplifies mp_obj_get_complex_maybe() by first calling
mp_obj_get_float_maybe() to handle the cases corresponding to floats.
Only if that fails does it attempt to extra a full complex number.
This reduces code size and also means that mp_obj_get_complex_maybe() now
supports user-defined classes defining __float__; in particular this allows
user-defined classes to be used as arguments to cmath-module function.
Furthermore, complex_make_new() can now be simplified to directly call
mp_obj_get_complex(), instead of mp_obj_get_complex_maybe() followed by
mp_obj_get_float(). This also improves error messages from complex with
an invalid argument, it now raises "can't convert <type> to complex" rather
than "can't convert <type> to float".
Signed-off-by: Damien George <damien@micropython.org>
Add cert_reqs and cadata keyword-args to ssl.wrap_socket() and
ssl.CERT_NONE, ssl.CERT_OPTIONAL, ssl.CERT_REQUIRED constants to allow
certificate validation.
CPython doesn't accept cadata in ssl.wrap_socket(), but it does in
SSLContext.load_verify_locations(), so we use this name to at least match
the same name in load_verify_locations().
Add docs for these new arguments, as well as docs for the existing
server_hostname argument which is important for certificate validation.
Tests are added as well.
Signed-off-by: Carlos Gil <carlosgilglez@gmail.com>
The empty tuple is usually a constant object, but named tuples must be
allocated to allow modification. Added explicit allocation to fix this.
Also added a regression test to verify creating an empty named tuple works.
Fixes issue #7870.
Signed-off-by: Lars Haulin <lars.haulin@gmail.com>
For STM32L072 and similar, very low end targets.
The other perf_bench tests run out of memory, crash, or fail on
prerequisite features.
Signed-off-by: Angus Gratton <gus@projectgus.com>
Prior to this commit, complex("j") would return 0j, and complex("nanj")
would return nan+0j. This commit makes sure "j" is tested for after
parsing the number (nan, inf or a decimal), and also supports the case of
"j" on its own.
Signed-off-by: Damien George <damien@micropython.org>
