Compare commits

..

1 commit
master ... py3

Author SHA1 Message Date
Michal Moskal
2a64bf669b Start on Python3 port 2018-06-27 13:50:10 -07:00
8 changed files with 76 additions and 1468 deletions

View file

@ -5,7 +5,8 @@ UF2 is a file format, developed by Microsoft for [PXT](https://github.com/Micros
flashing microcontrollers over MSC (Mass Storage Class; aka removable flash drive).
For a more friendly explanation, check out [this blog post](https://makecode.com/blog/one-chip-to-flash-them-all).
Also, take a look at the list of implementations at the bottom of this document.
For source code, see the implementation of [UF2 in a SAMD21 bootloader](https://github.com/Microsoft/uf2-samd21)
## Overview
@ -66,7 +67,7 @@ struct UF2_Block {
### Flags
Currently, there are three flags defined:
Currently, there are two flags is defined:
* `0x00000001` - **not main flash** - this block should be skipped when writing the
device flash; it can be used to store "comments" in the file, typically
@ -74,7 +75,6 @@ Currently, there are three flags defined:
* `0x00001000` - **file container** - see below
* `0x00002000` - **familyID present** - when set, the `fileSize/familyID` holds a value
identifying the board family (usually corresponds to an MCU)
* `0x00004000` - **MD5 checksum present** - see below
### Family ID
@ -85,37 +85,21 @@ If you're developing your own bootloader, and your
board family isn't listed here, pick a new family ID at random. It's good
to also send a PR here, so your family can be listed.
If the `familyID` doesn't match, the bootloader should disregard the
entire block, including `blockNo` and `numBlocks` fields.
In particular, writing a full UF2 file with non-matching `familyID`
should not reset the board.
This also allows for several files with different `familyID` to be
simply concatenated together, and the whole resulting file to be copied
to the device with only one actually being written to flash.
#### Picking numbers at random
The reason to pick numbers at random is to minimize risk of collisions
in the wild. Do not pick random numbers by banging on keyboard, or by using
`0xdeadf00d`, `0x42424242` etc. A good way is to use the following
shell command: `printf "0x%04x%04x\n" $RANDOM $RANDOM`
Another good way is the link at the bottom of https://microsoft.github.io/uf2/patcher/
This procedure was unfortunately not used for the SAMD51 and NRF52840 below.
#### Family list
* Microchip (Atmel) SAMD21 - 0x68ed2b88
* Microchip (Atmel) SAML21 - 0x1851780a
* Microchip (Atmel) SAMD51 - 0x55114460
* Nordic NRF52840 - 0xada52840
* ST STM32F103 - 0x5ee21072
* ST STM32F401 - 0x57755a57
* ST STM32F407 - 0x6d0922fa
* ST STM32F407VG - 0x8fb060fe
* Microchip (Atmel) ATmega32 - 0x16573617
* Cypress FX2 - 0x5a18069b
* ESP32 - 0x1c5f21b0
* NXP i.MX RT10XX - 0x4fb2d5bd
* SAMD21 - 0x68ed2b88
* SAMD51 - 0x55114460
* NRF52 - 0x1b57745f
* STM32F1 - 0x5ee21072
* STM32F4 - 0x57755a57
* ATmega32 - 0x16573617
### Rationale
@ -265,44 +249,20 @@ Typical writing procedure is as follows:
The fields `blockNo` and `numBlocks` refer to the entire UF2 file, not the current
file.
## MD5 checksum
When the `0x4000` flag is set, the last 24 bytes of `data[]` hold the following structure:
| Offset | Size | Value |
|--------|------|---------------------------------------------------|
| 0 | 4 | Start address of region |
| 4 | 4 | Length of region in bytes |
| 8 | 16 | MD5 checksum in binary format |
The flashing program should compute the MD5 sum of the specified region.
If the region checksum matches, flashing of the current block can be skipped.
Typically, many blocks in sequence will have the same region specified,
and can all be skipped, if the matching succeeded.
The position of the current block will typically be inside of the region.
The position and size of the region should be multiple of page erase size
(4k or 64k on typical SPI flash).
This is currently only used on ESP32, which is also why MD5 checksum is used.
## Implementations
### Bootloaders
* [Microchip ATSAMD21 and ATSAMD51](https://github.com/Microsoft/uf2-samdx1)
* [SAMD21](https://github.com/Microsoft/uf2-samd21)
* [Arduino UNO](https://github.com/mmoskal/uf2-uno)
* [STM32F103](https://github.com/mmoskal/uf2-stm32)
* [STM32F4](https://github.com/mmoskal/uf2-stm32f)
* [Nordic NRF52840](https://github.com/adafruit/Adafruit_nRF52840_Bootloader)
* [STM32](https://github.com/mmoskal/uf2-stm32f)
* [NRF52840](https://github.com/adafruit/Adafruit_nRF52840_Bootloader)
* [Linux (RPi Zero)](https://github.com/microsoft/uf2-linux)
* [Cypress FX2](https://github.com/whitequark/libfx2/tree/master/firmware/boot-uf2)
There's an ongoing effort to implement UF2 in [Codal](https://github.com/lancaster-university/codal-core).
There's an ongoing effort to implement UF2 in [Codal](https://github.com/lancaster-university/codal-core), see `msc` branch.
### Editors
* https://arcade.makecode.com
* https://makecode.adafruit.com
* https://makecode.seeedstudio.com
* https://maker.makecode.com

View file

@ -1 +0,0 @@
theme: jekyll-theme-cayman

126
cf2.md
View file

@ -1,126 +0,0 @@
# Binary Bootloader-embedded Config (CF2)
## Motivation
The goal of CF2 is to run the same binary on a number of devices with different
peripheral configuration, but the same MCU.
Typically, this has been achieved by embedding identification components in hardware
(eg. connecting certain MCU pins to GND or 3V) and having the firmware dynamically
detect these.
While simple, this only allows a few bits of configuration data to be stored on device.
Instead, we use the bootloader area for storing device specific configuration data.
Such data is not altered when the device is programmed with new firmware,
and the said firmware can refer to the configuration data in the bootloader area to
alter its behavior.
For example, MakeCode Arcade requires a device with 7 buttons (directions, A, B, and MENU),
a screen from a specific family, and one of the few supported accelerometers.
The exact configuration depends on the manufacturer, but the binary produced by MakeCode
will work on any device with the right MCU, provided it has the right configuration
data in the bootloader area.
Thus, the user only needs to select the type of MCU (which is color coded,
and is assigned a simple user-facing name), and the resulting binary will work
on their device.
Moreover, they can drag the UF2 file between devices, provided they run the same MCU.
Once bootloaders are updated with networking support (wired or wireless), one device with
a given MCU will be able to flash another.
## Configuration data format
Data comes as pairs of 32 bit unsigned integers (in machine byte order).
* first pair is `0x1e9e10f1, 0x20227a79`
* second pair consists of the number of configuration entries, and a zero (reserved)
* then configuration entries follow, where the first element is the configuration
entry key number, and the second is the value (these entries are sorted by key number)
* finally, a number of zeroes follows (typically at least a few hundred),
to store any additional configuration data in future
The file `configkeys.h` with definitions of key names can be downloaded from the
patcher website (see below).
If you require your own key names, please use
[random large key names](https://github.com/Microsoft/uf2#picking-numbers-at-random).
## The Patcher
At https://microsoft.github.io/uf2/patcher/ we provide a tool, which given a UF2 or BIN file
and a configuration definition, binary-patches the UF2/BIN file and lets the user download it.
The tool also parses existing configuration information from the UF2/BIN file to show it.
Such tool is to be used by makers of devices.
For example, for MakeCode Arcade, this includes both factories and users who take an existing
multi-purpose board and connect keys and screen.
Such user would then download generic MakeCode Arcade bootloader (either a UF2 file,
which upgrades the bootloader, or a .BIN file with the bootloader) for the given MCU,
and binary patch it with their configuration.
Then, they would update the bootloader, and have a Arcade-compatible device.
## Configuration file syntax
Example:
```bash
# This is comment, which is ignored
// this is also a comment
# Configuration values for display registers and size
DISPLAY_CFG0 = 0x80
DISPLAY_CFG1 = 0x603
DISPLAY_CFG2 = 0x16
DISPLAY_HEIGHT = 128
DISPLAY_WIDTH = 160
PINS_PORT_SIZE = PA_32 # see below
# pin headers (if any)
PIN_A0 = PA02
PIN_A1 = PA05
PIN_A2 = PB08
PIN_D2 = PA07
PIN_D3 = PB22
PIN_SCK = PA01
PIN_MISO = PB23
PIN_MOSI = PA00
# pin functions
PIN_BTN_LEFT = PIN_D2 # use pin header name
PIN_BTN_UP = PB13 # use pin directly
PIN_DISPLAY_CS = 18 # can even just use a number for the pin
PIN_DISPLAY_SCK = PIN_SCK
PIN_DISPLAY_MOSI = PIN_MOSI
# custom configuration keys
_679732427 = 123
_815320287 = 0x80192
# remove an existing config entry
PIN_BTN_MENU2 = null
```
The keys are either key names, or underscore followed by a decimal number.
The values are numbers in either decimal or hexadecimal,
references to other keys, or pin numbers.
The way pin numbers are parsed depends on the `PINS_PORT_SIZE`:
* `PA_16` - pins are `PA00`-`PA15`, `PB00-PB15`, ..., used on STM32
* `PA_32` - pins are `PA00`-`PA31`, ... - used on Microchip ATSAMD
* `P0_16` - pins are `P0_0`-`P0_15`, `P1_0-P1_15`, ... - not used?
* `P0_32` - pins are `P0_0-P0_31`, ... - used on Nordic NRF
* `0` or missing - pins are `P_0`, `P_1`, ..., `P_1000`
Certain keys (like `PINS_PORT_SIZE`) have a number of pre-defined values,
which can be used instead of integers.
Because you're usually use this syntax to patch an existing configuration,
you sometimes may want to remove an entry that's already there.
To do that, use `null` as the value of the key.
For list of keys and predefined values, see `configkeys.h` which can be downloaded
from the patcher website.
## Running from node.js
The patcher tool can be also run from command line.
Download [patcher.js](https://raw.githubusercontent.com/Microsoft/uf2/master/patcher/patcher.js)
and run it with node. It will print out help.

5
hf2.md
View file

@ -14,8 +14,8 @@ ones, but be less efficient.
In particular, it is suitable for running over USB HID (Human Interface Device),
which is widely supported in various operating systems without the need for kernel-space
drivers. It is also possible to run the protocol over a WebUSB link with either a single
interrupt endpoint or two bulk endpoints, as well as using just the control pipe,
allowing direct access from supported browsers.
interrupt endpoint or two bulk endpoints, allowing direct access from supported
browsers.
## Raw message format
@ -142,7 +142,6 @@ struct HF2_BININFO_Result {
uint32_t flash_page_size;
uint32_t flash_num_pages;
uint32_t max_message_size;
uint32_t family_id; // optional
};
```

View file

@ -1,83 +0,0 @@
<!DOCTYPE html>
<html lang="en-US">
<head>
<meta charset="UTF-8">
<title>CF2 binary patcher</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="theme-color" content="#157878">
<link rel="stylesheet" href="https://microsoft.github.io/uf2/assets/css/style.css">
<script src="patcher.js"></script>
<script src="web.js"></script>
<style>
#patch {
font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace;
font-size: 0.9rem;
width: 100%;
height: 12em;
}
.page-header {
padding: 2rem 2rem !important;
}
.btn-dl,
.btn-dl:hover {
background-color: green;
float: right;
color: white;
}
footer {
text-align: center;
margin-top: 5em;
}
footer a {
margin-right: 2em;
}
</style>
</head>
<body ondrop="dropHandler(event);" ondragover="dragOverHandler(event);">
<section class="page-header">
<h1 class="project-name">CF2 patcher</h1>
</section>
<section class="main-content">
<h2>Your patch</h2>
<textarea id="patch" spellcheck=false onchange="savePatch(event);"></textarea>
<a href="#" id="apply" class="btn btn-dl">Apply my patch</a>
<h2>Information</h2>
<pre><code id="currconfig">Drop UF2 or BIN file above to see its config.</code></pre>
<p>
<a href="https://github.com/Microsoft/uf2/blob/master/cf2.md">Learn more</a>
or
<a href="#" onclick="defines()">download configkeys.h</a>.
</p>
<p>
<a href="#" id="rnd">Generate random number</a>
<span id="rnd-res"></span>
</p>
<footer>
<a href="https://makecode.com/privacy" target="_blank" rel="noopener">Privacy &amp; Cookies</a>
<a href="https://makecode.com/termsofuse" target="_blank" rel="noopener"> Terms Of Use</a>
<a href="https://makecode.com/trademarks" target="_blank" rel="noopener">Trademarks</a>
<span>© 2019 Microsoft</span>
</footer>
</section>
<script>
restorePatch()
</script>
</body>
</html>

File diff suppressed because it is too large Load diff

View file

@ -1,109 +0,0 @@
"use strict";
function savePatch(ev) {
let text = document.getElementById("patch")
localStorage["UF2_PATCH"] = text.value
}
function genRnd(ev) {
ev.preventDefault()
let vals = new Uint32Array(1)
window.crypto.getRandomValues(vals)
document.getElementById("rnd-res").textContent = "Random number: 0x" +
("000000000" + vals[0].toString(16)).slice(-8)
}
function restorePatch() {
let text = document.getElementById("patch")
text.value = localStorage["UF2_PATCH"] || ""
document.getElementById("apply").onclick = applyPatch
document.getElementById("rnd").onclick = genRnd
}
function download(buf, name) {
let blob = new Blob([buf], {
type: "application/x-uf2"
});
let url = URL.createObjectURL(blob);
let a = document.createElement("a");
document.body.appendChild(a);
a.style = "display: none";
a.href = url;
a.download = name;
a.click();
window.URL.revokeObjectURL(url);
}
let currUF2 = null
let currUF2Name = ""
function showMSG() {
if (infoMsg)
document.getElementById("currconfig").textContent = infoMsg
}
function wrap(f) {
try {
infoMsg = ""
f()
showMSG()
} catch (e) {
log("Exception: " + e.message)
showMSG()
}
}
function defines() {
download(configkeysH(), "configkeys.h")
}
function applyPatch() {
wrap(() => {
let text = document.getElementById("patch")
let newcfg = text.value.trim()
if (!currUF2)
log("You have to drop a UF2 or BIN file with bootloader above before applying patches.")
else if (!newcfg)
log("You didn't give any patch to apply.")
else {
let buf = currUF2.slice()
let r = patchConfig(buf, newcfg)
if (!r.changes) {
log("No changes.")
} else {
log("\nChanges:\n" + r.changes)
}
log("Downloading " + currUF2Name)
download(r.patched, currUF2Name)
}
})
}
function dropHandler(ev) {
ev.preventDefault();
for (let i = 0; i < ev.dataTransfer.items.length; i++) {
if (ev.dataTransfer.items[i].kind === 'file') {
let file = ev.dataTransfer.items[i].getAsFile();
let reader = new FileReader();
infoMsg = ""
reader.onload = e => {
wrap(() => {
let buf = new Uint8Array(reader.result)
let cfg = readConfig(buf)
currUF2 = buf
infoMsg += "\n" + cfg
currUF2Name = file.name
})
}
reader.readAsArrayBuffer(file);
break
}
}
}
function dragOverHandler(ev) {
ev.preventDefault();
ev.dataTransfer.dropEffect = 'copy';
}

135
utils/uf2conv.py Normal file → Executable file
View file

@ -1,4 +1,5 @@
#!/usr/bin/env python3
#!/usr/bin/python
import sys
import struct
import subprocess
@ -7,20 +8,17 @@ import os
import os.path
import argparse
UF2_MAGIC_START0 = 0x0A324655 # "UF2\n"
UF2_MAGIC_START1 = 0x9E5D5157 # Randomly selected
UF2_MAGIC_END = 0x0AB16F30 # Ditto
families = {
'SAMD21': 0x68ed2b88,
'SAML21': 0x1851780a,
'SAMD51': 0x55114460,
'NRF52': 0x1b57745f,
'STM32F1': 0x5ee21072,
'STM32F4': 0x57755a57,
'ATMEGA32': 0x16573617,
'MIMXRT10XX': 0x4FB2D5BD
}
INFO_FILE = "/INFO_UF2.TXT"
@ -28,29 +26,25 @@ INFO_FILE = "/INFO_UF2.TXT"
appstartaddr = 0x2000
familyid = 0x0
def is_uf2(buf):
def isUF2(buf):
w = struct.unpack("<II", buf[0:8])
return w[0] == UF2_MAGIC_START0 and w[1] == UF2_MAGIC_START1
def is_hex(buf):
try:
w = buf[0:30].decode("utf-8")
except UnicodeDecodeError:
return False
if w[0] == ':' and re.match(b"^[:0-9a-fA-F\r\n]+$", buf):
def isHEX(buf):
w = buf[0:30]
if w[0] == ':' and re.match("^[:0-9a-fA-F\r\n]+$", buf):
return True
return False
def convert_from_uf2(buf):
def convertFromUF2(buf):
global appstartaddr
numblocks = len(buf) // 512
numblocks = len(buf) >> 9
curraddr = None
outp = b""
for blockno in range(numblocks):
outp = ""
for blockno in range(0, numblocks):
ptr = blockno * 512
block = buf[ptr:ptr + 512]
hd = struct.unpack(b"<IIIIIIII", block[0:32])
hd = struct.unpack("<IIIIIIII", block[0:32])
if hd[0] != UF2_MAGIC_START0 or hd[1] != UF2_MAGIC_START1:
print("Skipping block at " + ptr + "; bad magic")
continue
@ -78,34 +72,25 @@ def convert_from_uf2(buf):
curraddr = newaddr + datalen
return outp
def convert_to_carray(file_content):
outp = "const unsigned char bindata[] __attribute__((aligned(16))) = {"
for i in range(len(file_content)):
if i % 16 == 0:
outp += "\n"
outp += "0x%02x, " % ord(file_content[i])
outp += "\n};\n"
return outp
def convert_to_uf2(file_content):
def convertToUF2(fileContent):
global familyid
datapadding = b""
while len(datapadding) < 512 - 256 - 32 - 4:
datapadding += b"\x00\x00\x00\x00"
numblocks = (len(file_content) + 255) // 256
numblocks = (len(fileContent) + 255) >> 8
outp = b""
for blockno in range(numblocks):
for blockno in range(0, numblocks):
ptr = 256 * blockno
chunk = file_content[ptr:ptr + 256]
chunk = fileContent[ptr:ptr + 256]
flags = 0x0
if familyid:
flags |= 0x2000
hd = struct.pack(b"<IIIIIIII",
hd = struct.pack("<IIIIIIII",
UF2_MAGIC_START0, UF2_MAGIC_START1,
flags, ptr + appstartaddr, 256, blockno, numblocks, familyid)
while len(chunk) < 256:
chunk += b"\x00"
block = hd + chunk + datapadding + struct.pack(b"<I", UF2_MAGIC_END)
block = hd + chunk + datapadding + struct.pack("<I", UF2_MAGIC_END)
assert len(block) == 512
outp += block
return outp
@ -113,7 +98,9 @@ def convert_to_uf2(file_content):
class Block:
def __init__(self, addr):
self.addr = addr
self.bytes = bytearray(256)
self.bytes = []
for i in range(0, 256):
self.bytes.append(0)
def encode(self, blockno, numblocks):
global familyid
@ -123,13 +110,14 @@ class Block:
hd = struct.pack("<IIIIIIII",
UF2_MAGIC_START0, UF2_MAGIC_START1,
flags, self.addr, 256, blockno, numblocks, familyid)
hd += self.bytes[0:256]
for i in range(0, 256):
hd += chr(self.bytes[i])
while len(hd) < 512 - 4:
hd += b"\x00"
hd += struct.pack("<I", UF2_MAGIC_END)
return hd
def convert_from_hex_to_uf2(buf):
def convertFromHexToUF2(buf):
global appstartaddr
appstartaddr = None
upper = 0
@ -164,21 +152,17 @@ def convert_from_hex_to_uf2(buf):
addr += 1
i += 1
numblocks = len(blocks)
resfile = b""
resfile = ""
for i in range(0, numblocks):
resfile += blocks[i].encode(i, numblocks)
return resfile
def to_str(b):
return b.decode("utf-8")
def get_drives():
def getdrives():
drives = []
if sys.platform == "win32":
r = subprocess.check_output(["wmic", "PATH", "Win32_LogicalDisk",
"get", "DeviceID,", "VolumeName,",
"FileSystem,", "DriveType"])
for line in to_str(r).split('\n'):
r = subprocess.check_output(["wmic", "PATH", "Win32_LogicalDisk", "get", "DeviceID,", "VolumeName,", "FileSystem,", "DriveType"])
for line in r.split('\n'):
words = re.split('\s+', line)
if len(words) >= 3 and words[1] == "2" and words[2] == "FAT":
drives.append(words[0])
@ -193,33 +177,28 @@ def get_drives():
for d in os.listdir(rootpath):
drives.append(os.path.join(rootpath, d))
def has_info(d):
def hasInfo(d):
try:
return os.path.isfile(d + INFO_FILE)
except:
return False
return list(filter(has_info, drives))
return filter(hasInfo, drives)
def board_id(path):
def boardID(path):
with open(path + INFO_FILE, mode='r') as file:
file_content = file.read()
return re.search("Board-ID: ([^\r\n]*)", file_content).group(1)
fileContent = file.read()
return re.search("Board-ID: ([^\r\n]*)", fileContent).group(1)
def listdrives():
for d in getdrives():
print(d, boardID(d))
def list_drives():
for d in get_drives():
print(d, board_id(d))
def write_file(name, buf):
def writeFile(name, buf):
with open(name, "wb") as f:
f.write(buf)
print("Wrote %d bytes to %s." % (len(buf), name))
def main():
global appstartaddr, familyid
def error(msg):
@ -239,13 +218,9 @@ def main():
help='list connected devices')
parser.add_argument('-c' , '--convert', action='store_true',
help='do not flash, just convert')
parser.add_argument('-D' , '--deploy', action='store_true',
help='just flash, do not convert')
parser.add_argument('-f' , '--family', dest='family', type=str,
default="0x0",
help='specify familyID - number or name (default: 0x0)')
parser.add_argument('-C' , '--carray', action='store_true',
help='convert binary file to a C array, not UF2')
args = parser.parse_args()
appstartaddr = int(args.base, 0)
@ -258,44 +233,38 @@ def main():
error("Family ID needs to be a number or one of: " + ", ".join(families.keys()))
if args.list:
list_drives()
listdrives()
else:
if not args.input:
error("Need input file")
with open(args.input, mode='rb') as f:
inpbuf = f.read()
from_uf2 = is_uf2(inpbuf)
with open(args.input, mode='rb') as file:
inpbuf = file.read()
fromUF2 = isUF2(inpbuf)
ext = "uf2"
if args.deploy:
outbuf = inpbuf
elif from_uf2:
outbuf = convert_from_uf2(inpbuf)
if fromUF2:
outbuf = convertFromUF2(inpbuf)
ext = "bin"
elif is_hex(inpbuf):
outbuf = convert_from_hex_to_uf2(inpbuf.decode("utf-8"))
elif args.carray:
outbuf = convert_to_carray(inpbuf)
ext = "h"
elif isHEX(inpbuf):
outbuf = convertFromHexToUF2(inpbuf)
else:
outbuf = convert_to_uf2(inpbuf)
print("Converting to %s, output size: %d, start address: 0x%x" %
(ext, len(outbuf), appstartaddr))
if args.convert or ext != "uf2":
outbuf = convertToUF2(inpbuf)
print("Converting to %s, output size: %d, start address: 0x%x" % (ext, len(outbuf), appstartaddr))
if args.convert:
drives = []
if args.output == None:
args.output = "flash." + ext
else:
drives = get_drives()
drives = getdrives()
if args.output:
write_file(args.output, outbuf)
writeFile(args.output, outbuf)
else:
if len(drives) == 0:
error("No drive to deploy.")
for d in drives:
print("Flashing %s (%s)" % (d, board_id(d)))
write_file(d + "/NEW.UF2", outbuf)
print("Flashing %s (%s)" % (d, boardID(d)))
writeFile(outbuf, d + "/NEW.UF2")
if __name__ == "__main__":
main()