Release/v5.1 (#115)

* Rework the lib-builder for ESP-IDF v5.1

* Update package json with tolls matching the ESP-IDF version

* fix: rainmaker examples crashing on s3 due to low stack memory. (#106) (#107)

* Update scripts with the latest requirements

* Update configs + SR Support

* Add esp-elf-gdp to the list of packages

* Fix RainMaker builds and new sr models path

* Temporary force arduino branch for CI to work

* fix target branch

* Delete esp-dl component manifest for requiring IDF 4.4.x

* Temporary changes to allow Cron CI to run

* Support builds based on ESP-IDF tag

* Push to esp32-arduino-libs

* Update repository_dispatch.sh

* Rework scripts to allow build when either dst needs it

* Github complains when pushing to the libs repo

* Authenticate to the libs repo

* Attempt at splitting SDK from Arduino

* Archive only the result and reorder deploy commands

* Update cron.sh

* Fix script and zip paths

* Fix download URL and json merger

* Change sdk folder structure and fix json generation

* Switch output folder from sdk to esp32-arduino-libs

* arduino_tinyusb: compile support for DFU mode (#116)

* Update PlatformIO build script templates (#118)

Adds support for new package with precompiled SDK libraries

* Autogenerate PlatformIO manifest file for prebuilt SDK libs (#119)

* Autogenerate PlatformIO manifest file for prebuilt SDK libs

- Add a special Python script that generates "package.json" with IDF revision from the "version.txt" according to SemVer

* Tidy up

* Refactor manifest generator

Now IDF version and commit hash are passed directly from Git client instead of reading from a pregenerated "version.txt" file

* Move IDF definitions to be available with any build

* Use more components from registry and add mp3 decoder

* esp-sr component requires clearing before each build

* revert ESP_SR from component manager

* Build ESP_SR only for ESP32-S3 for now

* [TinyUSB] Update esp32sx dcd and add dwc2 option

* Workaround for recent change in ESP-Insights

* Add initial support for ESP32-C6

* Run build tests on ESP32-C6

* Remove -mlongcalls for non-xtensa chips

* Temp fix for ESP-Insights on C6

* Add support for ESP32H2

* Added tflite-micro component (#128)

* Update build badge in README.md

* Added tflite-micro component

---------

Co-authored-by: Me No Dev <me-no-dev@users.noreply.github.com>

* Make cron rebuild the libs if they need to be pushed to Arduino

For when we change something in the lib-builder, but no new updates are available in ESP-IDF

* Update build actions

* Fix permissions

* Do not build for obsolete Flash modes

* Try separate detect for cron builds

* Add permissions to github api

* Try more basic commit detection

* another try to pass vars and get commit

* Update push.yml

* Update config.sh

* Enable builds again

* Update build.sh

* Combine the artifacts from all jobs

* fix and test deploy check

* Update push.yml

* Disable Memprot to allow loading external elfs

* Fix archive name

* Disable coredump to flash

* Enable SPI ETH KSZ8851SNL

* Add temporary support for Arduino SPI Ethernet

* Add a temporary fix for relative include in BT lib

* Enable Classic BT HID Host and Device for ESP32

* Revert "Enable Classic BT HID Host and Device for ESP32"

This reverts commit aa0040ad271d00ac507fd2b478ee143b6c118615.

* C6 was added to ESP-SR

* Update Ethernet and remove SR workaround

* Pin RainMaker version

* Update target branch

* Add back cron.sh

---------

Co-authored-by: Sanket Wadekar <67091512+sanketwadekar@users.noreply.github.com>
Co-authored-by: Luca Burelli <pil@iol.it>
Co-authored-by: Valerii Koval <valeros@users.noreply.github.com>

.github/workflows/cron.yml

@@ -13,6 +13,10 @@ on:
 #             * * * * *
     - cron:  '0 */6 * * *'
 
+defaults:
+  run:
+    shell: bash
+
 jobs:
   run:
     name: Build with IDF ${{ matrix.idf_branch }}
@@ -22,7 +26,7 @@ jobs:
       matrix:
         idf_branch: [release/v5.1, release/v4.4] #, release/v3.3]
     steps:
-    - uses: actions/checkout@v1
+    - uses: actions/checkout@v3
     - name: Install dependencies
       run: bash ./tools/prepare-ci.sh
     - name: Build
@@ -31,9 +35,101 @@ jobs:
         GIT_AUTHOR_EMAIL: ${{ secrets.PUSH_EMAIL }}
         GIT_COMMITTER_EMAIL: ${{ secrets.PUSH_EMAIL }}
         IDF_BRANCH: ${{ matrix.idf_branch }}
-      run: bash ./tools/cron.sh
+      run: |
+        git checkout ${{ matrix.idf_branch }}
+        bash ./tools/cron.sh
     - name: Upload archive
-      uses: actions/upload-artifact@v1
+      uses: actions/upload-artifact@v3
       with:
         name: artifacts
         path: dist
+
+
+  # check:
+  #   name: Check if result should be deployed
+  #   runs-on: ubuntu-latest
+  #   strategy:
+  #     matrix:
+  #       branch: [release/v5.1, release/v4.4] #, release/v3.3]
+  #   outputs:
+  #     idf_branch: ${{ steps.check.outputs.idf_branch }}
+  #     idf_commit: ${{ steps.check.outputs.idf_commit }}
+  #     ar_branch: ${{ steps.check.outputs.ar_branch }}
+  #     ar_new_commit_message: ${{ steps.check.outputs.ar_new_commit_message }}
+  #     ar_new_branch_name: ${{ steps.check.outputs.ar_new_branch_name }}
+  #     ar_new_pr_title: ${{ steps.check.outputs.ar_new_pr_title }}
+  #     ar_has_commit: ${{ steps.check.outputs.ar_has_commit }}
+  #     ar_has_branch: ${{ steps.check.outputs.ar_has_branch }}
+  #     ar_has_pr: ${{ steps.check.outputs.ar_has_pr }}
+  #     libs_version: ${{ steps.check.outputs.libs_version }}
+  #     libs_has_commit: ${{ steps.check.outputs.libs_has_commit }}
+  #     libs_has_branch: ${{ steps.check.outputs.libs_has_branch }}
+  #   steps:
+  #   - uses: actions/checkout@v3
+  #   - id: check
+  #     env:
+  #       GITHUB_TOKEN: ${{ secrets.PUSH_TOKEN }}
+  #       GIT_AUTHOR_EMAIL: ${{ secrets.PUSH_EMAIL }}
+  #       GIT_COMMITTER_EMAIL: ${{ secrets.PUSH_EMAIL }}
+  #       IDF_BRANCH: ${{ matrix.idf_branch }}
+  #     run: bash ./tools/check-deploy-needed.sh
+
+  # build:
+  #   name: Build Libs for ${{ matrix.target }}
+  #   runs-on: ubuntu-latest
+  #   needs: check
+  #   if: needs.check.outputs.libs_has_commit == '0' || needs.check.outputs.ar_has_commit == '0'
+  #   strategy:
+  #     matrix:
+  #       target: [esp32, esp32s2, esp32s3, esp32c3, esp32c6, esp32h2]
+  #     fail-fast: false
+  #   steps:
+  #   - uses: actions/checkout@v3
+  #   # - name: Install dependencies
+  #   #   run: bash ./tools/prepare-ci.sh
+  #   - shell: bash
+  #     name: Build Libs for ${{ matrix.target }}
+  #     run: echo ${{ matrix.target }}
+  #   #   run: bash ./build.sh -t ${{ matrix.target }}
+  #   # - name: Upload archive
+  #   #   uses: actions/upload-artifact@v3
+  #   #   with:
+  #   #     name: artifacts
+  #   #     path: dist
+
+  # deploy:
+  #   name: Deploy build
+  #   runs-on: ubuntu-latest
+  #   needs: [check, build]
+  #   steps:
+  #   - uses: actions/checkout@v3
+  #   - shell: bash
+  #     env:
+  #       GITHUB_TOKEN: ${{ secrets.PUSH_TOKEN }}
+  #       GIT_AUTHOR_EMAIL: ${{ secrets.PUSH_EMAIL }}
+  #       GIT_COMMITTER_EMAIL: ${{ secrets.PUSH_EMAIL }}
+  #       IDF_BRANCH: ${{ needs.check.outputs.idf_branch }}
+  #       IDF_COMMIT: ${{ needs.check.outputs.idf_commit }}
+  #       AR_BRANCH: ${{ needs.check.outputs.ar_branch }}
+  #       AR_NEW_COMMIT_MESSAGE: ${{ needs.check.outputs.ar_new_commit_message }}
+  #       AR_NEW_BRANCH_NAME: ${{ needs.check.outputs.ar_new_branch_name }}
+  #       AR_NEW_PR_TITLE: ${{ needs.check.outputs.ar_new_pr_title }}
+  #       AR_HAS_COMMIT: ${{ needs.check.outputs.ar_has_commit }}
+  #       AR_HAS_BRANCH: ${{ needs.check.outputs.ar_has_branch }}
+  #       AR_HAS_PR: ${{ needs.check.outputs.ar_has_pr }}
+  #       LIBS_VERSION: ${{ needs.check.outputs.libs_version }}
+  #       LIBS_HAS_COMMIT: ${{ needs.check.outputs.libs_has_commit }}
+  #       LIBS_HAS_BRANCH: ${{ needs.check.outputs.libs_has_branch }}
+  #     run: |
+  #       echo "IDF_COMMIT: $IDF_COMMIT"
+  #       echo "AR_BRANCH: $AR_BRANCH"
+  #       echo "AR_NEW_COMMIT_MESSAGE: $AR_NEW_COMMIT_MESSAGE"
+  #       echo "AR_NEW_BRANCH_NAME: $AR_NEW_BRANCH_NAME"
+  #       echo "AR_NEW_PR_TITLE: $AR_NEW_PR_TITLE"
+  #       echo "AR_HAS_COMMIT: $AR_HAS_COMMIT"
+  #       echo "AR_HAS_BRANCH: $AR_HAS_BRANCH"
+  #       echo "AR_HAS_PR: $AR_HAS_PR"
+  #       echo "LIBS_VERSION: $LIBS_VERSION"
+  #       echo "LIBS_HAS_COMMIT: $LIBS_HAS_COMMIT"
+  #       echo "LIBS_HAS_BRANCH: $LIBS_HAS_BRANCH"
+

.github/workflows/push.yml

@@ -11,21 +11,50 @@ concurrency:
   cancel-in-progress: true
 
 jobs:
+
   build-libs:
     name: Build Libs for ${{ matrix.target }}
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        target: [esp32, esp32s2, esp32s3, esp32c3]
+        target: [esp32, esp32s2, esp32s3, esp32c3, esp32c6, esp32h2]
       fail-fast: false
     steps:
     - uses: actions/checkout@v3
     - name: Install dependencies
       run: bash ./tools/prepare-ci.sh
     - name: Build Libs for ${{ matrix.target }}
-      run: bash ./build.sh -t ${{ matrix.target }}
+      run: bash ./build.sh -e -t ${{ matrix.target }}
     - name: Upload archive
-      uses: actions/upload-artifact@v1
+      uses: actions/upload-artifact@v3
       with:
-        name: artifacts-${{ matrix.target }}
+        name: artifacts
         path: dist
+
+  combine-artifacts:
+    name: Combine artifacts
+    needs: build-libs
+    runs-on: ubuntu-latest
+    steps:
+      - name: Download artifacts
+        uses: actions/download-artifact@v3
+        with:
+          name: artifacts
+          path: dist
+      - shell: bash
+        run: |
+          mkdir -p out
+          find dist -name 'arduino-esp32-libs-esp*.tar.gz' -exec tar zxvf {} -C out \;
+          cd out/tools/esp32-arduino-libs && tar zcf ../../../dist/esp32-arduino-libs.tar.gz * && cd ../../..
+          cp out/package_esp32_index.template.json dist/package_esp32_index.template.json
+      - name: Upload full esp32-arduino-libs archive
+        uses: actions/upload-artifact@v3
+        with:
+          name: esp32-arduino-libs
+          path: dist/esp32-arduino-libs.tar.gz
+      - name: Upload package_esp32_index.template.json
+        uses: actions/upload-artifact@v3
+        with:
+          name: package-esp32-index-json
+          path: dist/package_esp32_index.template.json
+

.github/workflows/repository_dispatch.yml

@@ -7,7 +7,7 @@ jobs:
     name: Dispatch Event
     runs-on: ubuntu-latest
     steps:
-    - uses: actions/checkout@v1
+    - uses: actions/checkout@v3
     - name: Install dependencies
       run: bash ./tools/prepare-ci.sh
     - name: Handle Event
@@ -17,7 +17,7 @@ jobs:
         GIT_COMMITTER_EMAIL: ${{ secrets.PUSH_EMAIL }}
       run: bash ./tools/repository_dispatch.sh
     - name: Upload archive
-      uses: actions/upload-artifact@v1
+      uses: actions/upload-artifact@v3
       with:
         name: artifacts
         path: dist

.gitignore

@@ -10,6 +10,7 @@ components/esp-rainmaker/
 components/espressif__esp-dsp/
 components/esp-insights/
 components/arduino_tinyusb/tinyusb/
+components/tflite-micro/
 esp-idf/
 out/
 build/
@@ -19,3 +20,5 @@ sdkconfig
 sdkconfig.old
 version.txt
 dependencies.lock
+managed_components/
+target/

CMakeLists.txt

@@ -3,7 +3,10 @@
 cmake_minimum_required(VERSION 3.5)
 
 set(RMAKER_PATH ${CMAKE_SOURCE_DIR}/components/esp-rainmaker)
-set(EXTRA_COMPONENT_DIRS ${RMAKER_PATH}/components/esp-insights/components ${RMAKER_PATH}/components ${CMAKE_SOURCE_DIR}/components/esp-insights/components)
+set(INSIGHTS_PATH ${RMAKER_PATH}/components/esp-insights)
+set(TFLITE_PATH ${CMAKE_SOURCE_DIR}/components/tflite-micro)
+
+set(EXTRA_COMPONENT_DIRS ${INSIGHTS_PATH}/components ${RMAKER_PATH}/components ${TFLITE_PATH}/components)
 
 include($ENV{IDF_PATH}/tools/cmake/project.cmake)
 project(arduino-lib-builder)
@@ -13,7 +16,7 @@ idf_build_get_property(elf EXECUTABLE GENERATOR_EXPRESSION)
 add_custom_command(
 	OUTPUT "idf_libs" 
 	COMMAND ${CMAKE_SOURCE_DIR}/tools/copy-libs.sh ${IDF_TARGET} "${CONFIG_LIB_BUILDER_FLASHMODE}" "${CONFIG_SPIRAM_MODE_OCT}" "${CONFIG_IDF_TARGET_ARCH_XTENSA}"
-	DEPENDS ${elf} 
+	DEPENDS all 
 	WORKING_DIRECTORY ${CMAKE_SOURCE_DIR} 
 	VERBATIM
 )

build.sh

@@ -14,14 +14,18 @@ TARGET="all"
 BUILD_TYPE="all"
 SKIP_ENV=0
 COPY_OUT=0
-DEPLOY_OUT=0
+ARCHIVE_OUT=0
+if [ -z $DEPLOY_OUT ]; then
+    DEPLOY_OUT=0
+fi
 
 function print_help() {
-    echo "Usage: build.sh [-s] [-A <arduino_branch>] [-I <idf_branch>] [-i <idf_commit>] [-c <path>] [-t <target>] [-b <build|menuconfig|idf_libs|copy_bootloader|mem_variant>] [config ...]"
+    echo "Usage: build.sh [-s] [-A <arduino_branch>] [-I <idf_branch>] [-i <idf_commit>] [-c <path>] [-t <target>] [-b <build|menuconfig|reconfigure|idf_libs|copy_bootloader|mem_variant>] [config ...]"
     echo "       -s     Skip installing/updating of ESP-IDF and all components"
     echo "       -A     Set which branch of arduino-esp32 to be used for compilation"
     echo "       -I     Set which branch of ESP-IDF to be used for compilation"
     echo "       -i     Set which commit of ESP-IDF to be used for compilation"
+    echo "       -e     Archive the build to dist"
     echo "       -d     Deploy the build to github arduino-esp32"
     echo "       -c     Set the arduino-esp32 folder to copy the result to. ex. '$HOME/Arduino/hardware/espressif/esp32'"
     echo "       -t     Set the build target(chip). ex. 'esp32s3'"
@@ -30,7 +34,7 @@ function print_help() {
     exit 1
 }
 
-while getopts ":A:I:i:c:t:b:sd" opt; do
+while getopts ":A:I:i:c:t:b:sde" opt; do
     case ${opt} in
         s )
             SKIP_ENV=1
@@ -38,6 +42,9 @@ while getopts ":A:I:i:c:t:b:sd" opt; do
         d )
             DEPLOY_OUT=1
             ;;
+        e )
+            ARCHIVE_OUT=1
+            ;;
         c )
             export ESP32_ARDUINO="$OPTARG"
             COPY_OUT=1
@@ -58,6 +65,7 @@ while getopts ":A:I:i:c:t:b:sd" opt; do
             b=$OPTARG
             if [ "$b" != "build" ] && 
                [ "$b" != "menuconfig" ] && 
+               [ "$b" != "reconfigure" ] && 
                [ "$b" != "idf_libs" ] && 
                [ "$b" != "copy_bootloader" ] && 
                [ "$b" != "mem_variant" ]; then
@@ -78,19 +86,31 @@ done
 shift $((OPTIND -1))
 CONFIGS=$@
 
+mkdir -p dist
+
 if [ $SKIP_ENV -eq 0 ]; then
     echo "* Installing/Updating ESP-IDF and all components..."
     # update components from git
     ./tools/update-components.sh
     if [ $? -ne 0 ]; then exit 1; fi
 
+    # install arduino component
+    ./tools/install-arduino.sh
+    if [ $? -ne 0 ]; then exit 1; fi
+
     # install esp-idf
     source ./tools/install-esp-idf.sh
     if [ $? -ne 0 ]; then exit 1; fi
 else
+    # $IDF_PATH/install.sh
+    # source $IDF_PATH/export.sh
     source ./tools/config.sh
 fi
 
+if [ -f "./managed_components/espressif__esp-sr/.component_hash" ]; then
+    rm -rf ./managed_components/espressif__esp-sr/.component_hash
+fi
+
 if [ "$BUILD_TYPE" != "all" ]; then
     if [ "$TARGET" = "all" ]; then
         echo "ERROR: You need to specify target for non-default builds"
@@ -123,17 +143,17 @@ fi
 rm -rf build sdkconfig out
 
 # Add components version info
-mkdir -p "$AR_TOOLS/sdk" && rm -rf version.txt && rm -rf "$AR_TOOLS/sdk/versions.txt"
+mkdir -p "$AR_TOOLS/esp32-arduino-libs" && rm -rf version.txt && rm -rf "$AR_TOOLS/esp32-arduino-libs/versions.txt"
 component_version="esp-idf: "$(git -C "$IDF_PATH" symbolic-ref --short HEAD || git -C "$IDF_PATH" tag --points-at HEAD)" "$(git -C "$IDF_PATH" rev-parse --short HEAD)
-echo $component_version >> version.txt && echo $component_version >> "$AR_TOOLS/sdk/versions.txt"
+echo $component_version >> version.txt && echo $component_version >> "$AR_TOOLS/esp32-arduino-libs/versions.txt"
 for component in `ls "$AR_COMPS"`; do
     if [ -d "$AR_COMPS/$component/.git" ] || [ -d "$AR_COMPS/$component/.github" ]; then
         component_version="$component: "$(git -C "$AR_COMPS/$component" symbolic-ref --short HEAD || git -C "$AR_COMPS/$component" tag --points-at HEAD)" "$(git -C "$AR_COMPS/$component" rev-parse --short HEAD)
-        echo $component_version >> version.txt && echo $component_version >> "$AR_TOOLS/sdk/versions.txt"
+        echo $component_version >> version.txt && echo $component_version >> "$AR_TOOLS/esp32-arduino-libs/versions.txt"
     fi
 done
 component_version="tinyusb: "$(git -C "$AR_COMPS/arduino_tinyusb/tinyusb" symbolic-ref --short HEAD || git -C "$AR_COMPS/arduino_tinyusb/tinyusb" tag --points-at HEAD)" "$(git -C "$AR_COMPS/arduino_tinyusb/tinyusb" rev-parse --short HEAD)
-echo $component_version >> version.txt && echo $component_version >> "$AR_TOOLS/sdk/versions.txt"
+echo $component_version >> version.txt && echo $component_version >> "$AR_TOOLS/esp32-arduino-libs/versions.txt"
 
 #targets_count=`jq -c '.targets[] | length' configs/builds.json`
 for target_json in `jq -c '.targets[]' configs/builds.json`; do
@@ -157,6 +177,11 @@ for target_json in `jq -c '.targets[]' configs/builds.json`; do
     for defconf in `echo "$target_json" | jq -c '.idf_libs[]' | tr -d '"'`; do
         idf_libs_configs="$idf_libs_configs;configs/defconfig.$defconf"
     done
+
+    if [ -f "./managed_components/espressif__esp-sr/.component_hash" ]; then
+        rm -rf ./managed_components/espressif__esp-sr/.component_hash
+    fi
+
     echo "* Build IDF-Libs: $idf_libs_configs"
     rm -rf build sdkconfig
     idf.py -DIDF_TARGET="$target" -DSDKCONFIG_DEFAULTS="$idf_libs_configs" idf_libs
@@ -168,6 +193,11 @@ for target_json in `jq -c '.targets[]' configs/builds.json`; do
         for defconf in `echo "$boot_conf" | jq -c '.[]' | tr -d '"'`; do
             bootloader_configs="$bootloader_configs;configs/defconfig.$defconf";
         done
+
+        if [ -f "./managed_components/espressif__esp-sr/.component_hash" ]; then
+            rm -rf ./managed_components/espressif__esp-sr/.component_hash
+        fi
+
         echo "* Build BootLoader: $bootloader_configs"
         rm -rf build sdkconfig
         idf.py -DIDF_TARGET="$target" -DSDKCONFIG_DEFAULTS="$bootloader_configs" copy_bootloader
@@ -180,6 +210,11 @@ for target_json in `jq -c '.targets[]' configs/builds.json`; do
         for defconf in `echo "$mem_conf" | jq -c '.[]' | tr -d '"'`; do
             mem_configs="$mem_configs;configs/defconfig.$defconf";
         done
+
+        if [ -f "./managed_components/espressif__esp-sr/.component_hash" ]; then
+            rm -rf ./managed_components/espressif__esp-sr/.component_hash
+        fi
+
         echo "* Build Memory Variant: $mem_configs"
         rm -rf build sdkconfig
         idf.py -DIDF_TARGET="$target" -DSDKCONFIG_DEFAULTS="$mem_configs" mem_variant
@@ -190,20 +225,30 @@ done
 # update package_esp32_index.template.json
 if [ "$BUILD_TYPE" = "all" ]; then
     python3 ./tools/gen_tools_json.py -i "$IDF_PATH" -j "$AR_COMPS/arduino/package/package_esp32_index.template.json" -o "$AR_OUT/"
+    python3 ./tools/gen_tools_json.py -i "$IDF_PATH" -o "$TOOLS_JSON_OUT/"
     if [ $? -ne 0 ]; then exit 1; fi
 fi
 
-# archive the build
+# Generate PlatformIO manifest file
 if [ "$BUILD_TYPE" = "all" ]; then
-    ./tools/archive-build.sh
+    python3 ./tools/gen_platformio_manifest.py -o "$TOOLS_JSON_OUT/" -s $(git -C "$IDF_PATH" symbolic-ref --short HEAD || git -C "$IDF_PATH" tag --points-at HEAD) -c $(git -C "$IDF_PATH" rev-parse --short HEAD)
     if [ $? -ne 0 ]; then exit 1; fi
 fi
 
 # copy everything to arduino-esp32 installation
 if [ $COPY_OUT -eq 1 ] && [ -d "$ESP32_ARDUINO" ]; then
     ./tools/copy-to-arduino.sh
+    if [ $? -ne 0 ]; then exit 1; fi
 fi
 
+# push changes to esp32-arduino-libs and create pull request into arduino-esp32
 if [ $DEPLOY_OUT -eq 1 ]; then
     ./tools/push-to-arduino.sh
+    if [ $? -ne 0 ]; then exit 1; fi
+fi
+
+# archive the build
+if [ $ARCHIVE_OUT -eq 1 ]; then
+    ./tools/archive-build.sh "$TARGET"
+    if [ $? -ne 0 ]; then exit 1; fi
 fi

components/arduino_tinyusb/CMakeLists.txt

@@ -1,62 +1,74 @@
-idf_component_register(REQUIRES esp_rom freertos soc PRIV_REQUIRES arduino main)
-
 if(CONFIG_TINYUSB_ENABLED)
 
   ### variables ###
   #################
-  # if(IDF_TARGET STREQUAL "esp32s2")
+
+  if(IDF_TARGET STREQUAL "esp32s2")
     set(compile_options
         "-DCFG_TUSB_MCU=OPT_MCU_ESP32S2"
         "-DCFG_TUSB_DEBUG=${CONFIG_TINYUSB_DEBUG_LEVEL}"
         "-Wno-type-limits" # needed for the vanila tinyusb with turned off classes
     )
-  # elseif(IDF_TARGET STREQUAL "esp32s3")
+  elseif(IDF_TARGET STREQUAL "esp32s3")
-  #   set(compile_options
+    set(compile_options
-  #       "-DCFG_TUSB_MCU=OPT_MCU_ESP32S2"
+        "-DCFG_TUSB_MCU=OPT_MCU_ESP32S3"
-  #       "-DCFG_TUSB_DEBUG=${CONFIG_TINYUSB_DEBUG_LEVEL}"
+        "-DCFG_TUSB_DEBUG=${CONFIG_TINYUSB_DEBUG_LEVEL}"
-  #       "-Wno-type-limits" # needed for the vanila tinyusb with turned off classes
+        "-Wno-type-limits" # needed for the vanila tinyusb with turned off classes
-  #   )
+    )
-  # endif()
+  endif()
-  idf_component_get_property(FREERTOS_ORIG_INCLUDE_PATH freertos
-                             ORIG_INCLUDE_PATH)
-  set(includes_private
-      # tusb:
-      "${COMPONENT_DIR}/tinyusb/hw/bsp/"
-      "${COMPONENT_DIR}/tinyusb/src/"
-      "${COMPONENT_DIR}/tinyusb/src/device"
-      )
 
-  set(includes_public
-      # tusb:
-      "${FREERTOS_ORIG_INCLUDE_PATH}"
-      "${COMPONENT_DIR}/tinyusb/src/"
-      # espressif:
-      "${COMPONENT_DIR}/include")
   set(srcs
       # espressif:
       "${COMPONENT_DIR}/src/dcd_esp32sx.c"
+      #"${COMPONENT_DIR}/src/dcd_dwc2.c"
       # tusb:
       #"${COMPONENT_DIR}/tinyusb/src/portable/espressif/esp32sx/dcd_esp32sx.c"
+      #"{COMPONENT_DIR}/tinyusb/src/portable/synopsys/dwc2/dcd_dwc2.c"
       "${COMPONENT_DIR}/tinyusb/src/class/cdc/cdc_device.c"
       "${COMPONENT_DIR}/tinyusb/src/class/hid/hid_device.c"
       "${COMPONENT_DIR}/tinyusb/src/class/midi/midi_device.c"
       "${COMPONENT_DIR}/tinyusb/src/class/msc/msc_device.c"
       "${COMPONENT_DIR}/tinyusb/src/class/video/video_device.c"
       "${COMPONENT_DIR}/tinyusb/src/class/dfu/dfu_rt_device.c"
+      "${COMPONENT_DIR}/tinyusb/src/class/dfu/dfu_device.c"
       "${COMPONENT_DIR}/tinyusb/src/class/vendor/vendor_device.c"
       "${COMPONENT_DIR}/tinyusb/src/common/tusb_fifo.c"
       "${COMPONENT_DIR}/tinyusb/src/device/usbd_control.c"
       "${COMPONENT_DIR}/tinyusb/src/device/usbd.c"
       "${COMPONENT_DIR}/tinyusb/src/tusb.c")
 
+  set(includes_private
+      # tusb:
+      "${COMPONENT_DIR}/tinyusb/hw/bsp/"
+      "${COMPONENT_DIR}/tinyusb/src/"
+      "${COMPONENT_DIR}/tinyusb/src/device"
+      "${COMPONENT_DIR}/tinyusb/src/portable/synopsys/dwc2"
+      )
+
+  idf_component_get_property(FREERTOS_ORIG_INCLUDE_PATH freertos
+                             ORIG_INCLUDE_PATH)
+  set(includes_public
+      # tusb:
+      "${FREERTOS_ORIG_INCLUDE_PATH}"
+      "${COMPONENT_DIR}/tinyusb/src/"
+      # espressif:
+      "${COMPONENT_DIR}/include")
+
+  set(requires esp_rom freertos soc)
+  set(priv_requires arduino main)
   ### tinyusb lib ###
   ###################
-  add_library(arduino_tinyusb STATIC ${srcs})
+  idf_component_register(INCLUDE_DIRS ${includes_public} PRIV_INCLUDE_DIRS ${includes_private} SRCS ${srcs} REQUIRES ${requires} PRIV_REQUIRES ${priv_requires})
-  target_include_directories(
+  # add_library(${COMPONENT_TARGET} STATIC ${srcs})
-    arduino_tinyusb
+  # target_include_directories(
-    PUBLIC ${includes_public}
+  #   ${COMPONENT_TARGET}
-    PRIVATE ${includes_private})
+  #   PUBLIC ${includes_public}
-  target_compile_options(arduino_tinyusb PRIVATE ${compile_options})
+  #   PRIVATE ${includes_private})
-  target_link_libraries(${COMPONENT_TARGET} INTERFACE arduino_tinyusb)
+  target_compile_options(${COMPONENT_TARGET} PRIVATE ${compile_options})
+  #target_link_libraries(${COMPONENT_TARGET} INTERFACE ${COMPONENT_TARGET})
+
+else()
+
+  idf_component_register()
 
 endif()

components/arduino_tinyusb/Kconfig.projbuild

@@ -175,6 +175,31 @@ menu "Arduino TinyUSB"
 
     endmenu
 
+    menu "DFU driver"
+        depends on TINYUSB_ENABLED
+
+        config TINYUSB_DFU_ENABLED
+            bool "Enable USB DFU TinyUSB driver"
+            default y
+            help
+                Enable USB DFU TinyUSB driver.
+
+        config TINYUSB_DESC_DFU_STRING
+            string "DFU Device String"
+            default "Espressif DFU Device"
+            depends on TINYUSB_DFU_ENABLED
+            help
+                Specify name of the DFU device
+
+        config TINYUSB_DFU_BUFSIZE
+            int "DFU buffer size"
+            default 4096
+            depends on TINYUSB_DFU_ENABLED
+            help
+                DFU buffer size
+
+    endmenu
+
     menu "VENDOR driver"
         depends on TINYUSB_ENABLED
 

components/arduino_tinyusb/include/tusb_config.h

@@ -64,6 +64,10 @@ extern "C" {
 #   define CONFIG_TINYUSB_DFU_RT_ENABLED 0
 #endif
 
+#ifndef CONFIG_TINYUSB_DFU_ENABLED
+#   define CONFIG_TINYUSB_DFU_ENABLED 0
+#endif
+
 #ifndef CONFIG_TINYUSB_VENDOR_ENABLED
 #   define CONFIG_TINYUSB_VENDOR_ENABLED 0
 #endif
@@ -106,6 +110,7 @@ extern "C" {
 #define CFG_TUD_VIDEO               CONFIG_TINYUSB_VIDEO_ENABLED
 #define CFG_TUD_CUSTOM_CLASS 		CONFIG_TINYUSB_CUSTOM_CLASS_ENABLED
 #define CFG_TUD_DFU_RUNTIME			CONFIG_TINYUSB_DFU_RT_ENABLED
+#define CFG_TUD_DFU					CONFIG_TINYUSB_DFU_ENABLED
 #define CFG_TUD_VENDOR 				CONFIG_TINYUSB_VENDOR_ENABLED
 
 // CDC FIFO size of TX and RX
@@ -126,6 +131,9 @@ extern "C" {
 #define CFG_TUD_VIDEO_STREAMING     CONFIG_TINYUSB_VIDEO_STREAMING_IFS
 #define CFG_TUD_VIDEO_STREAMING_EP_BUFSIZE  CONFIG_TINYUSB_VIDEO_STREAMING_BUFSIZE
 
+// DFU buffer size
+#define CFG_TUD_DFU_XFER_BUFSIZE	CONFIG_TINYUSB_DFU_BUFSIZE
+
 // VENDOR FIFO size of TX and RX
 #define CFG_TUD_VENDOR_RX_BUFSIZE 	CONFIG_TINYUSB_VENDOR_RX_BUFSIZE
 #define CFG_TUD_VENDOR_TX_BUFSIZE 	CONFIG_TINYUSB_VENDOR_TX_BUFSIZE

components/arduino_tinyusb/src/dcd_esp32sx.c

@@ -28,17 +28,16 @@
 
 #include "tusb_option.h"
 
-#if (((CFG_TUSB_MCU == OPT_MCU_ESP32S2) ||  (CFG_TUSB_MCU == OPT_MCU_ESP32S3)) && TUSB_OPT_DEVICE_ENABLED)
+#if (((CFG_TUSB_MCU == OPT_MCU_ESP32S2) ||  (CFG_TUSB_MCU == OPT_MCU_ESP32S3)) && CFG_TUD_ENABLED)
 
 // Espressif
-#include "driver/periph_ctrl.h"
 #include "freertos/xtensa_api.h"
 #include "esp_intr_alloc.h"
 #include "esp_log.h"
-#include "driver/gpio.h"
 #include "soc/dport_reg.h"
 #include "soc/gpio_sig_map.h"
 #include "soc/usb_periph.h"
+#include "soc/periph_defs.h" // for interrupt source
 
 #include "device/dcd.h"
 
@@ -60,6 +59,7 @@ typedef struct {
     uint16_t queued_len;
     uint16_t max_size;
     bool short_packet;
+    uint8_t interval;
 } xfer_ctl_t;
 
 static const char *TAG = "TUSB:DCD";
@@ -284,6 +284,14 @@ void dcd_disconnect(uint8_t rhport)
   USB0.dctl |= USB_SFTDISCON_M;
 }
 
+void dcd_sof_enable(uint8_t rhport, bool en)
+{
+  (void) rhport;
+  (void) en;
+
+  // TODO implement later
+}
+
 /*------------------------------------------------------------------*/
 /* DCD Endpoint port
  *------------------------------------------------------------------*/
@@ -303,6 +311,7 @@ bool dcd_edpt_open(uint8_t rhport, tusb_desc_endpoint_t const *desc_edpt)
 
   xfer_ctl_t *xfer = XFER_CTL_BASE(epnum, dir);
   xfer->max_size = tu_edpt_packet_size(desc_edpt);
+  xfer->interval = desc_edpt->bInterval;
 
   if (dir == TUSB_DIR_OUT) {
     out_ep[epnum].doepctl &= ~(USB_D_EPTYPE0_M | USB_D_MPS0_M);
@@ -423,6 +432,13 @@ bool dcd_edpt_xfer(uint8_t rhport, uint8_t ep_addr, uint8_t *buffer, uint16_t to
     USB0.in_ep_reg[epnum].dieptsiz = (num_packets << USB_D_PKTCNT0_S) | total_bytes;
     USB0.in_ep_reg[epnum].diepctl |= USB_D_EPENA1_M | USB_D_CNAK1_M; // Enable | CNAK
 
+    // For ISO endpoint with interval=1 set correct DATA0/DATA1 bit for next frame
+    if ((USB0.in_ep_reg[epnum].diepctl & USB_D_EPTYPE0_M) == (1 << USB_D_EPTYPE1_S) && xfer->interval == 1) {
+      // Take odd/even bit from frame counter.
+      uint32_t const odd_frame_now = (USB0.dsts & (1u << USB_SOFFN_S));
+      USB0.in_ep_reg[epnum].diepctl |= (odd_frame_now ? USB_DI_SETD0PID1 : USB_DI_SETD1PID1);
+    }
+
     // Enable fifo empty interrupt only if there are something to put in the fifo.
     if(total_bytes != 0) {
       USB0.dtknqr4_fifoemptymsk |= (1 << epnum);
@@ -431,6 +447,13 @@ bool dcd_edpt_xfer(uint8_t rhport, uint8_t ep_addr, uint8_t *buffer, uint16_t to
     // Each complete packet for OUT xfers triggers XFRC.
     USB0.out_ep_reg[epnum].doeptsiz |= USB_PKTCNT0_M | ((xfer->max_size & USB_XFERSIZE0_V) << USB_XFERSIZE0_S);
     USB0.out_ep_reg[epnum].doepctl  |= USB_EPENA0_M | USB_CNAK0_M;
+
+    // For ISO endpoint with interval=1 set correct DATA0/DATA1 bit for next frame
+    if ((USB0.out_ep_reg[epnum].doepctl & USB_D_EPTYPE0_M) == (1 << USB_D_EPTYPE1_S) && xfer->interval == 1) {
+      // Take odd/even bit from frame counter.
+      uint32_t const odd_frame_now = (USB0.dsts & (1u << USB_SOFFN_S));
+      USB0.out_ep_reg[epnum].doepctl |= (odd_frame_now ? USB_DO_SETD0PID1 : USB_DO_SETD1PID1);
+    }
   }
   return true;
 }
@@ -459,7 +482,8 @@ void dcd_edpt_stall(uint8_t rhport, uint8_t ep_addr)
     } else {
       // Stop transmitting packets and NAK IN xfers.
       in_ep[epnum].diepctl |= USB_DI_SNAK1_M;
-      while ((in_ep[epnum].diepint & USB_DI_SNAK1_M) == 0) ;
+      // while ((in_ep[epnum].diepint & USB_DI_SNAK1_M) == 0) ;
+      while ((in_ep[epnum].diepint & USB_D_INEPNAKEFF1_M) == 0) ;
 
       // Disable the endpoint. Note that both SNAK and STALL are set here.
       in_ep[epnum].diepctl |= (USB_DI_SNAK1_M | USB_D_STALL1_M | USB_D_EPDIS1_M);
@@ -469,9 +493,16 @@ void dcd_edpt_stall(uint8_t rhport, uint8_t ep_addr)
 
     // Flush the FIFO, and wait until we have confirmed it cleared.
     uint8_t const fifo_num = ((in_ep[epnum].diepctl >> USB_D_TXFNUM1_S) & USB_D_TXFNUM1_V);
-    USB0.grstctl |= (fifo_num << USB_TXFNUM_S);
+    // USB0.grstctl |= (fifo_num << USB_TXFNUM_S);
-    USB0.grstctl |= USB_TXFFLSH_M;
+    // USB0.grstctl |= USB_TXFFLSH_M;
+    // while ((USB0.grstctl & USB_TXFFLSH_M) != 0) ;
+    uint32_t rstctl_last = USB0.grstctl;
+    uint32_t rstctl = USB_TXFFLSH_M;
+    rstctl |= (fifo_num << USB_TXFNUM_S);
+    USB0.grstctl = rstctl;
     while ((USB0.grstctl & USB_TXFFLSH_M) != 0) ;
+    USB0.grstctl = rstctl_last;
+    // TODO: Clear grstctl::fifo_num after fifo flsh
   } else {
     // Only disable currently enabled non-control endpoint
     if ((epnum == 0) || !(out_ep[epnum].doepctl & USB_EPENA0_M)) {

configs/builds.json

@@ -4,7 +4,13 @@
 			"file":"libspi_flash.a",
 			"src":"build/esp-idf/spi_flash/libspi_flash.a",
 			"out":"lib/libspi_flash.a",
-			"targets":["esp32","esp32c3","esp32s2","esp32s3"]
+			"targets":["esp32","esp32c3","esp32s2","esp32s3","esp32c6","esp32h2"]
+		},
+		{
+			"file":"libesp_psram.a",
+			"src":"build/esp-idf/esp_psram/libesp_psram.a",
+			"out":"lib/libesp_psram.a",
+			"targets":["esp32s3"]
 		},
 		{
 			"file":"libesp_system.a",
@@ -39,8 +45,78 @@
 	],
 	"targets":[
 		{
-			"target": "esp32s3",
+			"target": "esp32h2",
 			"features":[],
+			"idf_libs":["qio","64m"],
+			"bootloaders":[
+				["qio","64m"],
+				["dio","64m"],
+				["qio","16m"],
+				["dio","16m"]
+			],
+			"mem_variants":[
+				["dio","64m"]
+			]
+		},
+		{
+			"target": "esp32c6",
+			"features":[],
+			"idf_libs":["qio","80m"],
+			"bootloaders":[
+				["qio","80m"],
+				["dio","80m"],
+				["qio","40m"],
+				["dio","40m"]
+			],
+			"mem_variants":[
+				["dio","80m"]
+			]
+		},
+		{
+			"target": "esp32c3",
+			"features":[],
+			"idf_libs":["qio","80m"],
+			"bootloaders":[
+				["qio","80m"],
+				["dio","80m"],
+				["qio","40m"],
+				["dio","40m"]
+			],
+			"mem_variants":[
+				["dio","80m"]
+			]
+		},
+		{
+			"target": "esp32",
+			"features":["qio_ram"],
+			"idf_libs":["qio","80m"],
+			"bootloaders":[
+				["qio","80m"],
+				["dio","80m"],
+				["qio","40m"],
+				["dio","40m"]
+			],
+			"mem_variants":[
+				["dio","80m"]
+			]
+		},
+		{
+			"target": "esp32s2",
+			"features":["qio_ram"],
+			"idf_libs":["qio","80m"],
+			"bootloaders":[
+				["qio","80m"],
+				["dio","80m"],
+				["qio","40m"],
+				["dio","40m"]
+			],
+			"mem_variants":[
+				["dio","80m"]
+			]
+		},
+		{
+			"target": "esp32s3",
+			"features":["esp_sr"],
 			"idf_libs":["qio","80m","qio_ram"],
 			"bootloaders":[
 				["qio","120m","qio_ram"],
@@ -55,66 +131,6 @@
 				["opi","80m","opi_ram"],
 				["opi","80m","qio_ram"]
 			]
-		},
-		{
-			"target": "esp32s2",
-			"features":["qio_ram"],
-			"idf_libs":["qio","80m"],
-			"bootloaders":[
-				["qio","80m"],
-				["qout","80m"],
-				["dio","80m"],
-				["dout","80m"],
-				["qio","40m"],
-				["qout","40m"],
-				["dio","40m"],
-				["dout","40m"]
-			],
-			"mem_variants":[
-				["qout","80m"],
-				["dio","80m"],
-				["dout","80m"]
-			]
-		},
-		{
-			"target": "esp32c3",
-			"features":[],
-			"idf_libs":["qio","80m"],
-			"bootloaders":[
-				["qio","80m"],
-				["qout","80m"],
-				["dio","80m"],
-				["dout","80m"],
-				["qio","40m"],
-				["qout","40m"],
-				["dio","40m"],
-				["dout","40m"]
-			],
-			"mem_variants":[
-				["qout","80m"],
-				["dio","80m"],
-				["dout","80m"]
-			]
-		},
-		{
-			"target": "esp32",
-			"features":["qio_ram"],
-			"idf_libs":["qio","80m"],
-			"bootloaders":[
-				["qio","80m"],
-				["qout","80m"],
-				["dio","80m"],
-				["dout","80m"],
-				["qio","40m"],
-				["qout","40m"],
-				["dio","40m"],
-				["dout","40m"]
-			],
-			"mem_variants":[
-				["qout","80m"],
-				["dio","80m"],
-				["dout","80m"]
-			]
 		}
 	]
 }

configs/defconfig.16m

@@ -0,0 +1 @@
+CONFIG_ESPTOOLPY_FLASHFREQ_16M=y

configs/defconfig.64m

@@ -0,0 +1 @@
+CONFIG_ESPTOOLPY_FLASHFREQ_64M=y

configs/defconfig.common

@@ -21,15 +21,16 @@ CONFIG_ESP_TASK_WDT_PANIC=y
 CONFIG_ESP_TIMER_TASK_STACK_SIZE=4096
 CONFIG_ESPTOOLPY_FLASHSIZE_4MB=y
 CONFIG_ESP_WIFI_FTM_ENABLE=y
-CONFIG_ESP32_WIFI_STATIC_RX_BUFFER_NUM=8
+CONFIG_ESP_WIFI_STATIC_RX_BUFFER_NUM=8
-CONFIG_ESP32_WIFI_STATIC_TX_BUFFER_NUM=8
+CONFIG_ESP_WIFI_STATIC_TX_BUFFER_NUM=8
-CONFIG_ESP32_WIFI_CACHE_TX_BUFFER_NUM=16
+CONFIG_ESP_WIFI_CACHE_TX_BUFFER_NUM=16
-CONFIG_ESP32_WIFI_CSI_ENABLED=y
+CONFIG_ESP_WIFI_CSI_ENABLED=y
-CONFIG_ESP32_WIFI_ENABLE_WPA3_SAE=y
+CONFIG_ESP_WIFI_ENABLE_WPA3_SAE=y
-# CONFIG_ESP32_WIFI_IRAM_OPT is not set
+# CONFIG_ESP_WIFI_IRAM_OPT is not set
-# CONFIG_ESP32_WIFI_RX_IRAM_OPT is not set
+# CONFIG_ESP_WIFI_RX_IRAM_OPT is not set
 CONFIG_ETH_SPI_ETHERNET_DM9051=y
 CONFIG_ETH_SPI_ETHERNET_W5500=y
+CONFIG_ETH_SPI_ETHERNET_KSZ8851SNL=y
 CONFIG_FATFS_CODEPAGE_850=y
 CONFIG_FATFS_LFN_STACK=y
 # CONFIG_FATFS_API_ENCODING_ANSI_OEM is not set
@@ -37,6 +38,7 @@ CONFIG_FATFS_API_ENCODING_UTF_8=y
 # CONFIG_FMB_CONTROLLER_SLAVE_ID_SUPPORT is not set
 CONFIG_FMB_TIMER_PORT_ENABLED=y
 CONFIG_FREERTOS_HZ=1000
+CONFIG_FREERTOS_ENABLE_BACKWARD_COMPATIBILITY=y
 # CONFIG_FREERTOS_ASSERT_ON_UNTESTED_FUNCTION is not set
 CONFIG_FREERTOS_IDLE_TASK_STACKSIZE=1024
 CONFIG_HEAP_POISONING_LIGHT=y
@@ -75,7 +77,7 @@ CONFIG_LWIP_IPV6_AUTOCONFIG=y
 CONFIG_ESP_RMAKER_SKIP_VERSION_CHECK=y
 CONFIG_ESP_RMAKER_USER_ID_CHECK=y
 CONFIG_ESP_INSIGHTS_ENABLED=y
-CONFIG_ESP_INSIGHTS_COREDUMP_ENABLE=y
+CONFIG_ESP_INSIGHTS_COREDUMP_ENABLE=n
 CONFIG_ESP_INSIGHTS_TRANSPORT_HTTPS=y
 CONFIG_DIAG_LOG_DROP_WIFI_LOGS=y
 CONFIG_DIAG_ENABLE_METRICS=y
@@ -84,7 +86,7 @@ CONFIG_DIAG_ENABLE_WIFI_METRICS=y
 CONFIG_DIAG_ENABLE_VARIABLES=y
 CONFIG_DIAG_ENABLE_NETWORK_VARIABLES=y
 CONFIG_ESP_COREDUMP_ENABLE=y
-CONFIG_ESP_COREDUMP_ENABLE_TO_FLASH=y
+CONFIG_ESP_COREDUMP_ENABLE_TO_FLASH=n
 CONFIG_ESP_COREDUMP_DATA_FORMAT_ELF=y
 CONFIG_ESP_COREDUMP_CHECKSUM_CRC32=y
 CONFIG_ESP_COREDUMP_MAX_TASKS_NUM=64

configs/defconfig.esp32

@@ -8,10 +8,9 @@ CONFIG_BT_SPP_ENABLED=y
 CONFIG_BT_HFP_ENABLE=y
 CONFIG_BT_STACK_NO_LOG=y
 CONFIG_BT_BLE_DYNAMIC_ENV_MEMORY=y
-CONFIG_ESP32_SPIRAM_SUPPORT=y
+CONFIG_SPIRAM=y
 CONFIG_SPIRAM_OCCUPY_HSPI_HOST=y
-CONFIG_ESP32_ULP_COPROC_ENABLED=y
+CONFIG_ULP_COPROC_ENABLED=y
-CONFIG_ESP32_XTAL_FREQ_AUTO=y
 # CONFIG_ESP_TASK_WDT_CHECK_IDLE_TASK_CPU1 is not set
 CONFIG_FREERTOS_FPU_IN_ISR=y
 # CONFIG_USE_WAKENET is not set

configs/defconfig.esp32c3

@@ -1,4 +1,4 @@
 CONFIG_BT_BLE_BLUFI_ENABLE=y
-CONFIG_ESP32C3_RTC_CLK_CAL_CYCLES=576
+CONFIG_RTC_CLK_CAL_CYCLES=576
 # CONFIG_ESP_TASK_WDT_CHECK_IDLE_TASK_CPU0 is not set
 CONFIG_FREERTOS_IDLE_TASK_STACKSIZE=2304

configs/defconfig.esp32c6

@@ -0,0 +1,4 @@
+CONFIG_BT_BLE_BLUFI_ENABLE=y
+CONFIG_RTC_CLK_CAL_CYCLES=576
+# CONFIG_ESP_TASK_WDT_CHECK_IDLE_TASK_CPU0 is not set
+CONFIG_FREERTOS_IDLE_TASK_STACKSIZE=2304

configs/defconfig.esp32h2

@@ -0,0 +1,4 @@
+CONFIG_BT_BLE_BLUFI_ENABLE=y
+CONFIG_RTC_CLK_CAL_CYCLES=576
+# CONFIG_ESP_TASK_WDT_CHECK_IDLE_TASK_CPU0 is not set
+CONFIG_FREERTOS_IDLE_TASK_STACKSIZE=2304

configs/defconfig.esp32s2

@@ -1,8 +1,9 @@
 CONFIG_SPIRAM_TRY_ALLOCATE_WIFI_LWIP=y
-CONFIG_ESP32S2_DEFAULT_CPU_FREQ_240=y
+CONFIG_ESP_DEFAULT_CPU_FREQ_MHZ_240=y
-CONFIG_ESP32S2_SPIRAM_SUPPORT=y
+CONFIG_SPIRAM=y
 CONFIG_ESP32S2_KEEP_USB_ALIVE=y
 # CONFIG_ESP_TASK_WDT_CHECK_IDLE_TASK_CPU0 is not set
 # CONFIG_USE_WAKENET is not set
 # CONFIG_USE_MULTINET is not set
 CONFIG_FREERTOS_WATCHPOINT_END_OF_STACK=y
+CONFIG_ESP_SYSTEM_MEMPROT_FEATURE=n

configs/defconfig.esp32s3

@@ -1,13 +1,10 @@
 CONFIG_SPIRAM_TRY_ALLOCATE_WIFI_LWIP=y
-CONFIG_ESP32S3_DEFAULT_CPU_FREQ_240=y
+CONFIG_ESP_DEFAULT_CPU_FREQ_MHZ_240=y
-CONFIG_ESP32S3_SPIRAM_SUPPORT=y
+CONFIG_SPIRAM=y
-CONFIG_ESP32S3_RTC_CLK_CAL_CYCLES=576
+CONFIG_RTC_CLK_CAL_CYCLES=576
 CONFIG_ESP32S3_UNIVERSAL_MAC_ADDRESSES_TWO=y
 # CONFIG_ESP_SLEEP_GPIO_RESET_WORKAROUND is not set
 # CONFIG_ESP_TASK_WDT_CHECK_IDLE_TASK_CPU1 is not set
-CONFIG_SR_WN_MODEL_WN8_QUANT=y
-CONFIG_SR_WN_WN8_HIESP=y
-CONFIG_SR_MN_ENGLISH=y
-CONFIG_SR_MN_EN_MULTINET5_SINGLE_RECOGNITION_QUANT8=y
 CONFIG_FREERTOS_WATCHPOINT_END_OF_STACK=y
 CONFIG_FREERTOS_TIMER_TASK_STACK_DEPTH=3120
+CONFIG_ESP_SYSTEM_MEMPROT_FEATURE=n

configs/defconfig.esp_sr

@@ -0,0 +1,37 @@
+CONFIG_ESPTOOLPY_FLASHSIZE_16MB=y
+CONFIG_PARTITION_TABLE_CUSTOM=y
+CONFIG_SR_WN_WN9_HIESP=y
+CONFIG_SR_MN_CN_NONE=y
+CONFIG_SR_MN_EN_MULTINET5_SINGLE_RECOGNITION_QUANT8=y
+CONFIG_EN_SPEECH_COMMAND_ID0=""
+CONFIG_EN_SPEECH_COMMAND_ID1=""
+CONFIG_EN_SPEECH_COMMAND_ID2=""
+CONFIG_EN_SPEECH_COMMAND_ID3=""
+CONFIG_EN_SPEECH_COMMAND_ID4=""
+CONFIG_EN_SPEECH_COMMAND_ID5=""
+CONFIG_EN_SPEECH_COMMAND_ID6=""
+CONFIG_EN_SPEECH_COMMAND_ID7=""
+CONFIG_EN_SPEECH_COMMAND_ID8=""
+CONFIG_EN_SPEECH_COMMAND_ID9=""
+CONFIG_EN_SPEECH_COMMAND_ID10=""
+CONFIG_EN_SPEECH_COMMAND_ID11=""
+CONFIG_EN_SPEECH_COMMAND_ID12=""
+CONFIG_EN_SPEECH_COMMAND_ID13=""
+CONFIG_EN_SPEECH_COMMAND_ID14=""
+CONFIG_EN_SPEECH_COMMAND_ID15=""
+CONFIG_EN_SPEECH_COMMAND_ID16=""
+CONFIG_EN_SPEECH_COMMAND_ID17=""
+CONFIG_EN_SPEECH_COMMAND_ID18=""
+CONFIG_EN_SPEECH_COMMAND_ID19=""
+CONFIG_EN_SPEECH_COMMAND_ID20=""
+CONFIG_EN_SPEECH_COMMAND_ID21=""
+CONFIG_EN_SPEECH_COMMAND_ID22=""
+CONFIG_EN_SPEECH_COMMAND_ID23=""
+CONFIG_EN_SPEECH_COMMAND_ID24=""
+CONFIG_EN_SPEECH_COMMAND_ID25=""
+CONFIG_EN_SPEECH_COMMAND_ID26=""
+CONFIG_EN_SPEECH_COMMAND_ID27=""
+CONFIG_EN_SPEECH_COMMAND_ID28=""
+CONFIG_EN_SPEECH_COMMAND_ID29=""
+CONFIG_EN_SPEECH_COMMAND_ID30=""
+CONFIG_EN_SPEECH_COMMAND_ID31=""

configs/pio_end.txt

@@ -0,0 +1,10 @@
+        "ARDUINO_ARCH_ESP32",
+        "ESP32",
+        ("F_CPU", "$BOARD_F_CPU"),
+        ("ARDUINO", 10812),
+        ("ARDUINO_VARIANT", '\\"%s\\"' % board_config.get("build.variant").replace('"', "")),
+        ("ARDUINO_BOARD", '\\"%s\\"' % board_config.get("name").replace('"', "")),
+        "ARDUINO_PARTITION_%s" % basename(board_config.get(
+            "build.partitions", "default.csv")).replace(".csv", "").replace("-", "_")
+    ]
+)

configs/pio_start.txt

@@ -0,0 +1,40 @@
+# Copyright 2014-present PlatformIO <contact@platformio.org>
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Arduino
+
+Arduino Wiring-based Framework allows writing cross-platform software to
+control devices attached to a wide range of Arduino boards to create all
+kinds of creative coding, interactive objects, spaces or physical experiences.
+
+http://arduino.cc/en/Reference/HomePage
+"""
+
+# Extends: https://github.com/platformio/platform-espressif32/blob/develop/builder/main.py
+
+from os.path import basename, join
+
+from SCons.Script import DefaultEnvironment
+
+env = DefaultEnvironment()
+
+FRAMEWORK_DIR = env.PioPlatform().get_package_dir("framework-arduinoespressif32")
+FRAMEWORK_SDK_DIR = env.PioPlatform().get_package_dir(
+    "framework-arduinoespressif32-libs"
+)
+
+board_config = env.BoardConfig()
+
+env.Append(

main/Kconfig.projbuild

@@ -10,9 +10,12 @@ config LIB_BUILDER_FLASHFREQ
     string
     default "120m" if ESPTOOLPY_FLASHFREQ_120M
     default "80m" if ESPTOOLPY_FLASHFREQ_80M
+    default "64m" if ESPTOOLPY_FLASHFREQ_64M
     default "40m" if ESPTOOLPY_FLASHFREQ_40M
+    default "32m" if ESPTOOLPY_FLASHFREQ_32M
     default "26m" if ESPTOOLPY_FLASHFREQ_26M
     default "20m" if ESPTOOLPY_FLASHFREQ_20M
+    default "16m" if ESPTOOLPY_FLASHFREQ_16M
 
 config LIB_BUILDER_COMPILE
     bool

main/idf_component.yml

@@ -0,0 +1,48 @@
+dependencies:
+  # Required IDF version
+  idf: ">=5.1"
+
+  mdns: "^1.1.0"
+  chmorgan/esp-libhelix-mp3: "1.0.3"
+  esp-dsp: "^1.3.4"
+
+  # esp-sr: "^1.3.1"
+  # esp32-camera: "^2.0.4"
+  # esp-dl:
+  #   git: https://github.com/espressif/esp-dl.git
+  # espressif/esp_rainmaker:
+  #   path: components/esp_rainmaker
+  #   git: https://github.com/espressif/esp-rainmaker.git
+
+  # # Defining a dependency from the registry:
+  # # https://components.espressif.com/component/example/cmp
+  # example/cmp: "^3.3.3" # Automatically update minor releases
+  #
+  # # Other ways to define dependencies
+  #
+  # # For components maintained by Espressif only name can be used.
+  # # Same as `espressif/cmp`
+  # component: "~1.0.0" # Automatically update bugfix releases
+  #
+  # # Or in a longer form with extra parameters
+  # component2:
+  #   version: ">=2.0.0"
+  #
+  #   # For transient dependencies `public` flag can be set.
+  #   # `public` flag doesn't have an effect for the `main` component.
+  #   # All dependencies of `main` are public by default.
+  #   public: true
+  #
+  #   # For components hosted on non-default registry:
+  #   service_url: "https://componentregistry.company.com"
+  #
+  # # For components in git repository:
+  # test_component:
+  #   path: test_component
+  #   git: ssh://git@gitlab.com/user/components.git
+  #
+  # # For test projects during component development
+  # # components can be used from a local directory
+  # # with relative or absolute path
+  # some_local_component:
+  #   path: ../../projects/component

partitions.csv

@@ -0,0 +1,8 @@
+# Name,   Type, SubType, Offset,  Size, Flags
+nvs,      data, nvs,       0x9000,   0x5000,
+otadata,  data, ota,       0xe000,   0x2000,
+app0,     app,  ota_0,    0x10000, 0x300000,
+app1,     app,  ota_1,   0x310000, 0x300000,
+spiffs,   data, spiffs,  0x610000, 0x700000,
+model,    data, spiffs,  0xD10000, 0x2E0000,
+coredump, data, coredump,0xFF0000,  0x10000,

tools/add_sdk_json.py

@@ -0,0 +1,121 @@
+#!/usr/bin/env python
+
+from __future__ import print_function
+
+__author__ = "Hristo Gochkov"
+__version__ = "2023"
+
+import os
+import shutil
+import errno
+import os.path
+import json
+import platform
+import sys
+import stat
+import argparse
+
+if sys.version_info[0] == 3:
+    unicode = lambda s: str(s)
+
+def add_system(systems, host, url, filename, sha, size):
+    system = {
+        "host": host,
+        "url": url,
+        "archiveFileName": filename,
+        "checksum": "SHA-256:"+sha,
+        "size": str(size)
+    }
+    systems.append(system)
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser(
+        prog = 'add_sdk_json',
+        description = 'Update SDK in Arduino package index')
+    parser.add_argument('-j', '--pkg-json', dest='arduino_json',  required=True, help='path to package json')
+    parser.add_argument('-n', '--name',     dest='tool_name',     required=True, help='name of the SDK package')
+    parser.add_argument('-v', '--version',  dest='tool_version',  required=True, help='version of the new SDK')
+    parser.add_argument('-u', '--url',      dest='tool_url',      required=True, help='url to the zip of the new SDK')
+    parser.add_argument('-f', '--filename', dest='tool_filename', required=True, help='filename of the zip of the new SDK')
+    parser.add_argument('-s', '--size',     dest='tool_size',     required=True, help='size of the zip of the new SDK')
+    parser.add_argument('-c', '--sha',      dest='tool_sha',      required=True, help='sha256 of the zip of the new SDK')
+    args = parser.parse_args()
+
+    print('Destination   : {0}.'.format(args.arduino_json))
+    print('Tool Name     : {0}.'.format(args.tool_name))
+    print('Tool Version  : {0}.'.format(args.tool_version))
+    print('Tool URL      : {0}.'.format(args.tool_url))
+    print('Tool File Name: {0}.'.format(args.tool_filename))
+    print('Tool Size     : {0}.'.format(args.tool_size))
+    print('Tool SHA256   : {0}.'.format(args.tool_sha))
+
+    arduino_json = args.arduino_json;
+    tool_name = args.tool_name;
+    tool_version = args.tool_version;
+    tool_url = args.tool_url;
+    tool_filename = args.tool_filename;
+    tool_size = args.tool_size;
+    tool_sha = args.tool_sha;
+
+    # code start
+    farray = {"packages":[{"platforms":[{"toolsDependencies":[]}],"tools":[]}]}
+    if os.path.isfile(arduino_json) == True:
+        farray = json.load(open(arduino_json))
+
+    dep_found = False
+    dep_skip = False
+    for dep in farray['packages'][0]['platforms'][0]['toolsDependencies']:
+        if dep['name'] == tool_name:
+            if dep['version'] == tool_version:
+                print('Skipping {0}. Same version {1}'.format(tool_name, tool_version))
+                dep_skip = True
+                break
+            print('Updating dependency version of {0} from {1} to {2}'.format(tool_name, dep['version'], tool_version))
+            dep['version'] = tool_version
+            dep_found = True
+            break
+
+    if dep_skip == False:
+        if dep_found == False:
+            print('Adding new dependency: {0} version {1}'.format(tool_name, tool_version))
+            deps = {
+                "packager": "esp32",
+                "name": tool_name,
+                "version": tool_version
+            }
+            farray['packages'][0]['platforms'][0]['toolsDependencies'].append(deps)
+
+        systems = []
+        add_system(systems, "i686-mingw32", tool_url, tool_filename, tool_sha, tool_size)
+        add_system(systems, "x86_64-mingw32", tool_url, tool_filename, tool_sha, tool_size)
+        add_system(systems, "arm64-apple-darwin", tool_url, tool_filename, tool_sha, tool_size)
+        add_system(systems, "x86_64-apple-darwin", tool_url, tool_filename, tool_sha, tool_size)
+        add_system(systems, "x86_64-pc-linux-gnu", tool_url, tool_filename, tool_sha, tool_size)
+        add_system(systems, "i686-pc-linux-gnu", tool_url, tool_filename, tool_sha, tool_size)
+        add_system(systems, "aarch64-linux-gnu", tool_url, tool_filename, tool_sha, tool_size)
+        add_system(systems, "arm-linux-gnueabihf", tool_url, tool_filename, tool_sha, tool_size)
+
+        tool_found = False
+        for t in farray['packages'][0]['tools']:
+            if t['name'] == tool_name:
+                t['version'] = tool_version
+                t['systems'] = systems
+                tool_found = True
+                print('Updating systems of {0} to version {1}'.format(tool_name, tool_version))
+                break
+
+        if tool_found == False:
+            print('Adding new tool: {0} version {1}'.format(tool_name, tool_version))
+            tools = {
+                "name": tool_name,
+                "version": tool_version,
+                "systems": systems
+            }
+            farray['packages'][0]['tools'].append(tools)
+
+        json_str = json.dumps(farray, indent=2)
+        with open(arduino_json, "w") as f:
+            f.write(json_str+"\n")
+            f.close()
+        # print(json_str)
+        print('{0} generated'.format(arduino_json))

tools/archive-build.sh

@@ -2,15 +2,11 @@
 
 IDF_COMMIT=$(git -C "$IDF_PATH" rev-parse --short HEAD || echo "")
 IDF_BRANCH=$(git -C "$IDF_PATH" symbolic-ref --short HEAD || git -C "$IDF_PATH" tag --points-at HEAD || echo "")
-
 idf_version_string=${IDF_BRANCH//\//_}"-$IDF_COMMIT"
-archive_path="dist/arduino-esp32-libs-$idf_version_string.tar.gz"
-build_archive_path="dist/arduino-esp32-build-$idf_version_string.tar.gz"
 
-mkdir -p dist && rm -rf "$archive_path" "$build_archive_path"
+archive_path="dist/arduino-esp32-libs-$1-$idf_version_string.tar.gz"
+
+mkdir -p dist && rm -rf "$archive_path"
 if [ -d "out" ]; then
 	cd out && tar zcf "../$archive_path" * && cd ..
 fi
-if [ -d "build" ]; then
-	cd build && tar zcf "../$build_archive_path" * && cd ..
-fi

tools/check-deploy-needed.sh

@@ -0,0 +1,85 @@
+#/bin/bash
+
+source ./tools/config.sh
+
+IDF_COMMIT=`github_last_commit "$IDF_REPO" "$IDF_BRANCH"`
+
+if [ -z $GITHUB_HEAD_REF ]; then
+	current_branch=`git branch --show-current`
+else
+	current_branch="$GITHUB_HEAD_REF"
+fi
+
+AR_BRANCH="master"
+if [[ "$current_branch" != "master" && `github_branch_exists "$AR_REPO" "$current_branch"` == "1" ]]; then
+	AR_BRANCH="$current_branch"
+else
+	AR_BRANCH_NAME="idf-$IDF_BRANCH"
+	has_ar_branch=`github_branch_exists "$AR_REPO" "$AR_BRANCH_NAME"`
+	if [ "$has_ar_branch" == "1" ]; then
+		AR_BRANCH="$AR_BRANCH_NAME"
+	else
+		has_ar_branch=`github_branch_exists "$AR_REPO" "$AR_PR_TARGET_BRANCH"`
+		if [ "$has_ar_branch" == "1" ]; then
+			AR_BRANCH="$AR_PR_TARGET_BRANCH"
+		fi
+	fi
+fi
+
+# format new branch name and pr title
+AR_NEW_BRANCH_NAME="idf-$IDF_BRANCH"
+AR_NEW_COMMIT_MESSAGE="IDF $IDF_BRANCH $IDF_COMMIT"
+AR_NEW_PR_TITLE="IDF $IDF_BRANCH"
+
+LIBS_VERSION="idf-"${IDF_BRANCH//\//_}"-$IDF_COMMIT"
+
+AR_HAS_BRANCH=`github_branch_exists "$AR_REPO" "$AR_NEW_BRANCH_NAME"`
+if [ "$AR_HAS_BRANCH" == "1" ]; then
+	AR_HAS_COMMIT=`github_commit_exists "$AR_REPO" "$AR_NEW_BRANCH_NAME" "$IDF_COMMIT"`
+else
+	AR_HAS_COMMIT=`github_commit_exists "$AR_REPO" "$AR_BRANCH" "$IDF_COMMIT"`
+fi
+AR_HAS_PR=`github_pr_exists "$AR_REPO" "$AR_NEW_BRANCH_NAME"`
+
+LIBS_HAS_BRANCH=`github_branch_exists "$AR_LIBS_REPO" "$AR_NEW_BRANCH_NAME"`
+LIBS_HAS_COMMIT=`github_commit_exists "$AR_LIBS_REPO" "$AR_NEW_BRANCH_NAME" "$IDF_COMMIT"`
+
+export IDF_COMMIT
+
+export AR_NEW_BRANCH_NAME
+export AR_NEW_COMMIT_MESSAGE
+export AR_NEW_PR_TITLE
+
+export AR_HAS_COMMIT
+export AR_HAS_BRANCH
+export AR_HAS_PR
+
+export LIBS_VERSION
+export LIBS_HAS_COMMIT
+export LIBS_HAS_BRANCH
+
+echo "IDF_COMMIT: $IDF_COMMIT"
+echo "AR_BRANCH: $AR_BRANCH"
+echo "AR_NEW_COMMIT_MESSAGE: $AR_NEW_COMMIT_MESSAGE"
+echo "AR_NEW_BRANCH_NAME: $AR_NEW_BRANCH_NAME"
+echo "AR_NEW_PR_TITLE: $AR_NEW_PR_TITLE"
+echo "AR_HAS_COMMIT: $AR_HAS_COMMIT"
+echo "AR_HAS_BRANCH: $AR_HAS_BRANCH"
+echo "AR_HAS_PR: $AR_HAS_PR"
+echo "LIBS_VERSION: $LIBS_VERSION"
+echo "LIBS_HAS_COMMIT: $LIBS_HAS_COMMIT"
+echo "LIBS_HAS_BRANCH: $LIBS_HAS_BRANCH"
+
+if [ ! -x $GITHUB_OUTPUT ]; then
+	echo "idf_commit=$IDF_COMMIT" >> "$GITHUB_OUTPUT"
+	echo "ar_branch=$AR_BRANCH" >> "$GITHUB_OUTPUT"
+	echo "ar_new_commit_message=$AR_NEW_COMMIT_MESSAGE" >> "$GITHUB_OUTPUT"
+	echo "ar_new_branch_name=$AR_NEW_BRANCH_NAME" >> "$GITHUB_OUTPUT"
+	echo "ar_new_pr_title=$AR_NEW_PR_TITLE" >> "$GITHUB_OUTPUT"
+	echo "ar_has_commit=$AR_HAS_COMMIT" >> "$GITHUB_OUTPUT"
+	echo "ar_has_branch=$AR_HAS_BRANCH" >> "$GITHUB_OUTPUT"
+	echo "ar_has_pr=$AR_HAS_PR" >> "$GITHUB_OUTPUT"
+	echo "libs_version=$LIBS_VERSION" >> "$GITHUB_OUTPUT"
+	echo "libs_has_commit=$LIBS_HAS_COMMIT" >> "$GITHUB_OUTPUT"
+	echo "libs_has_branch=$LIBS_HAS_BRANCH" >> "$GITHUB_OUTPUT"
+fi

tools/config.sh

@@ -2,40 +2,42 @@
 
 
 if [ -z $IDF_PATH ]; then
-	export IDF_PATH="$PWD/esp-idf"
+    export IDF_PATH="$PWD/esp-idf"
 fi
 
 if [ -z $IDF_BRANCH ]; then
-	IDF_BRANCH="release/v4.4"
+    IDF_BRANCH="release/v5.1"
 fi
 
 if [ -z $AR_PR_TARGET_BRANCH ]; then
-	AR_PR_TARGET_BRANCH="release/v2.x"
+    AR_PR_TARGET_BRANCH="master"
 fi
 
 if [ -z $IDF_TARGET ]; then
-	if [ -f sdkconfig ]; then
+    if [ -f sdkconfig ]; then
-		IDF_TARGET=`cat sdkconfig | grep CONFIG_IDF_TARGET= | cut -d'"' -f2`
+        IDF_TARGET=`cat sdkconfig | grep CONFIG_IDF_TARGET= | cut -d'"' -f2`
-		if [ "$IDF_TARGET" = "" ]; then
+        if [ "$IDF_TARGET" = "" ]; then
-			IDF_TARGET="esp32"
+            IDF_TARGET="esp32"
-		fi
+        fi
-	else
+    else
-		IDF_TARGET="esp32"
+        IDF_TARGET="esp32"
-	fi
+    fi
 fi
 
-IDF_COMPS="$IDF_PATH/components"
-IDF_TOOLCHAIN="xtensa-$IDF_TARGET-elf"
-
 # Owner of the target ESP32 Arduino repository
 AR_USER="espressif"
 
 # The full name of the repository
 AR_REPO="$AR_USER/arduino-esp32"
+IDF_REPO="$AR_USER/esp-idf"
+AR_LIBS_REPO="$AR_USER/esp32-arduino-libs"
 
 AR_REPO_URL="https://github.com/$AR_REPO.git"
+IDF_REPO_URL="https://github.com/$IDF_REPO.git"
+AR_LIBS_REPO_URL="https://github.com/$AR_LIBS_REPO.git"
 if [ -n $GITHUB_TOKEN ]; then
-	AR_REPO_URL="https://$GITHUB_TOKEN@github.com/$AR_REPO.git"
+    AR_REPO_URL="https://$GITHUB_TOKEN@github.com/$AR_REPO.git"
+    AR_LIBS_REPO_URL="https://$GITHUB_TOKEN@github.com/$AR_LIBS_REPO.git"
 fi
 
 AR_ROOT="$PWD"
@@ -44,30 +46,38 @@ AR_OUT="$AR_ROOT/out"
 AR_TOOLS="$AR_OUT/tools"
 AR_PLATFORM_TXT="$AR_OUT/platform.txt"
 AR_GEN_PART_PY="$AR_TOOLS/gen_esp32part.py"
-AR_SDK="$AR_TOOLS/sdk/$IDF_TARGET"
+AR_SDK="$AR_TOOLS/esp32-arduino-libs/$IDF_TARGET"
+PIO_SDK="FRAMEWORK_SDK_DIR, \"$IDF_TARGET\""
+TOOLS_JSON_OUT="$AR_TOOLS/esp32-arduino-libs"
+IDF_LIBS_DIR="$AR_ROOT/../esp32-arduino-libs"
+
+if [ -d "$IDF_PATH" ]; then
+    export IDF_COMMIT=$(git -C "$IDF_PATH" rev-parse --short HEAD)
+    export IDF_BRANCH=$(git -C "$IDF_PATH" symbolic-ref --short HEAD || git -C "$IDF_PATH" tag --points-at HEAD)
+fi
 
 function get_os(){
-  	OSBITS=`arch`
+    OSBITS=`arch`
-  	if [[ "$OSTYPE" == "linux"* ]]; then
+    if [[ "$OSTYPE" == "linux"* ]]; then
         if [[ "$OSBITS" == "i686" ]]; then
-        	echo "linux32"
+            echo "linux32"
         elif [[ "$OSBITS" == "x86_64" ]]; then
-        	echo "linux64"
+            echo "linux64"
         elif [[ "$OSBITS" == "armv7l" ]]; then
-        	echo "linux-armel"
+            echo "linux-armel"
         else
-        	echo "unknown"
+            echo "unknown"
-	    	return 1
+            return 1
         fi
-	elif [[ "$OSTYPE" == "darwin"* ]]; then
+    elif [[ "$OSTYPE" == "darwin"* ]]; then
-	    echo "macos"
+        echo "macos"
-	elif [[ "$OSTYPE" == "cygwin" ]] || [[ "$OSTYPE" == "msys" ]] || [[ "$OSTYPE" == "win32" ]]; then
+    elif [[ "$OSTYPE" == "cygwin" ]] || [[ "$OSTYPE" == "msys" ]] || [[ "$OSTYPE" == "win32" ]]; then
-	    echo "win32"
+        echo "win32"
-	else
+    else
-	    echo "$OSTYPE"
+        echo "$OSTYPE"
-	    return 1
+        return 1
-	fi
+    fi
-	return 0
+    return 0
 }
 
 AR_OS=`get_os`
@@ -76,54 +86,84 @@ export SED="sed"
 export SSTAT="stat -c %s"
 
 if [[ "$AR_OS" == "macos" ]]; then
-	if ! [ -x "$(command -v gsed)" ]; then
+    if ! [ -x "$(command -v gsed)" ]; then
-		echo "ERROR: gsed is not installed! Please install gsed first. ex. brew install gsed"
+        echo "ERROR: gsed is not installed! Please install gsed first. ex. brew install gsed"
-		exit 1
+        exit 1
-	fi
+    fi
-	if ! [ -x "$(command -v gawk)" ]; then
+    if ! [ -x "$(command -v gawk)" ]; then
-		echo "ERROR: gawk is not installed! Please install gawk first. ex. brew install gawk"
+        echo "ERROR: gawk is not installed! Please install gawk first. ex. brew install gawk"
-		exit 1
+        exit 1
-	fi
+    fi
-	export SED="gsed"
+    export SED="gsed"
-	export SSTAT="stat -f %z"
+    export SSTAT="stat -f %z"
 fi
 
-function git_commit_exists(){ #git_commit_exists <repo-path> <commit-message>
+function github_commit_exists(){ #github_commit_exists <repo-path> <branch-name> <commit-message>
-	local repo_path="$1"
+    local repo_path="$1"
-	local commit_message="$2"
+    local branch_name="$2"
-	local commits_found=`git -C "$repo_path" log --all --grep="$commit_message" | grep commit`
+    local commit_message="$3"
-	if [ -n "$commits_found" ]; then echo 1; else echo 0; fi
+    local commits_found=`curl -s -k -H "Authorization: token $GITHUB_TOKEN" -H "Accept: application/vnd.github.v3.raw+json" "https://api.github.com/repos/$repo_path/commits?sha=$branch_name" | jq -r '.[].commit.message' | grep "$commit_message" | wc -l`
+    if [ ! "$commits_found" == "" ] && [ ! "$commits_found" == "null" ] && [ ! "$commits_found" == "0" ]; then echo $commits_found; else echo 0; fi
 }
 
+function github_last_commit(){ # github_last_commit <repo-path> <branch-name>
+    local repo_path="$1"
+    local branch_name="$2"
+    local commit=`curl -s -k -H "Authorization: token $GITHUB_TOKEN" -H "Accept: application/vnd.github.v3.raw+json" "https://api.github.com/repos/$repo_path/commits/heads/$branch_name" | jq -r '.sha'`
+    if [ ! "$commit" == "" ] && [ ! "$commit" == "null" ]; then
+        echo ${commit:0:8}
+    else
+        echo ""
+    fi
+}
+
+function github_branch_exists(){ # github_branch_exists <repo-path> <branch-name>
+    local repo_path="$1"
+    local branch_name="$2"
+    local branch=`curl -s -k -H "Authorization: token $GITHUB_TOKEN" -H "Accept: application/vnd.github.v3.raw+json" "https://api.github.com/repos/$repo_path/branches/$branch_name" | jq -r '.name'`
+    if [ "$branch" == "$branch_name" ]; then echo 1; else echo 0; fi
+}
+
+function github_pr_exists(){ # github_pr_exists <repo-path> <branch-name>
+    local repo_path="$1"
+    local branch_name="$2"
+    local pr_num=`curl -s -k -H "Authorization: token $GITHUB_TOKEN" -H "Accept: application/vnd.github.v3.raw+json" "https://api.github.com/repos/$repo_path/pulls?head=$AR_USER:$branch_name&state=open" | jq -r '.[].number'`
+    if [ ! "$pr_num" == "" ] && [ ! "$pr_num" == "null" ]; then echo 1; else echo 0; fi
+}
+
+
+
 function git_branch_exists(){ # git_branch_exists <repo-path> <branch-name>
-	local repo_path="$1"
+    local repo_path="$1"
-	local branch_name="$2"
+    local branch_name="$2"
-	local branch_found=`git -C "$repo_path" ls-remote --heads origin "$branch_name"`
+    local branch_found=`git -C "$repo_path" ls-remote --heads origin "$branch_name"`
-	if [ -n "$branch_found" ]; then echo 1; else echo 0; fi
+    if [ -n "$branch_found" ]; then echo 1; else echo 0; fi
 }
 
-function git_pr_exists(){ # git_pr_exists <branch-name>
+function git_commit_exists(){ #git_commit_exists <repo-path> <commit-message>
-	local pr_num=`curl -s -k -H "Authorization: token $GITHUB_TOKEN" -H "Accept: application/vnd.github.v3.raw+json" "https://api.github.com/repos/$AR_REPO/pulls?head=$AR_USER:$1&state=open" | jq -r '.[].number'`
+    local repo_path="$1"
-	if [ ! "$pr_num" == "" ] && [ ! "$pr_num" == "null" ]; then echo 1; else echo 0; fi
+    local commit_message="$2"
+    local commits_found=`git -C "$repo_path" log --all --grep="$commit_message" | grep commit`
+    if [ -n "$commits_found" ]; then echo 1; else echo 0; fi
 }
 
 function git_create_pr(){ # git_create_pr <branch> <title>
-	local pr_branch="$1"
+    local pr_branch="$1"
-	local pr_title="$2"
+    local pr_title="$2"
-	local pr_target="$3"
+    local pr_target="$3"
-	local pr_body=""
+    local pr_body=""
-	pr_body+="esp-idf: "$(git -C "$IDF_PATH" symbolic-ref --short HEAD || git -C "$IDF_PATH" tag --points-at HEAD)" "$(git -C "$IDF_PATH" rev-parse --short HEAD)"\r\n"
+    pr_body+="esp-idf: "$(git -C "$IDF_PATH" symbolic-ref --short HEAD || git -C "$IDF_PATH" tag --points-at HEAD)" "$(git -C "$IDF_PATH" rev-parse --short HEAD)"\r\n"
-	for component in `ls "$AR_COMPS"`; do
+    for component in `ls "$AR_COMPS"`; do
-		if [ ! $component == "arduino" ]; then
+        if [ ! $component == "arduino" ]; then
-			if [ -d "$AR_COMPS/$component/.git" ] || [ -d "$AR_COMPS/$component/.github" ]; then
+            if [ -d "$AR_COMPS/$component/.git" ] || [ -d "$AR_COMPS/$component/.github" ]; then
-				pr_body+="$component: "$(git -C "$AR_COMPS/$component" symbolic-ref --short HEAD || git -C "$AR_COMPS/$component" tag --points-at HEAD)" "$(git -C "$AR_COMPS/$component" rev-parse --short HEAD)"\r\n"
+                pr_body+="$component: "$(git -C "$AR_COMPS/$component" symbolic-ref --short HEAD || git -C "$AR_COMPS/$component" tag --points-at HEAD)" "$(git -C "$AR_COMPS/$component" rev-parse --short HEAD)"\r\n"
-			fi
+            fi
-		fi
+        fi
-	done
+    done
-	pr_body+="tinyusb: "$(git -C "$AR_COMPS/arduino_tinyusb/tinyusb" symbolic-ref --short HEAD || git -C "$AR_COMPS/arduino_tinyusb/tinyusb" tag --points-at HEAD)" "$(git -C "$AR_COMPS/arduino_tinyusb/tinyusb" rev-parse --short HEAD)"\r\n"
+    pr_body+="tinyusb: "$(git -C "$AR_COMPS/arduino_tinyusb/tinyusb" symbolic-ref --short HEAD || git -C "$AR_COMPS/arduino_tinyusb/tinyusb" tag --points-at HEAD)" "$(git -C "$AR_COMPS/arduino_tinyusb/tinyusb" rev-parse --short HEAD)"\r\n"
-	local pr_data="{\"title\": \"$pr_title\", \"body\": \"$pr_body\", \"head\": \"$AR_USER:$pr_branch\", \"base\": \"$pr_target\"}"
+    local pr_data="{\"title\": \"$pr_title\", \"body\": \"$pr_body\", \"head\": \"$AR_USER:$pr_branch\", \"base\": \"$pr_target\"}"
-	git_create_pr_res=`echo "$pr_data" | curl -k -H "Authorization: token $GITHUB_TOKEN" -H "Accept: application/vnd.github.v3.raw+json" --data @- "https://api.github.com/repos/$AR_REPO/pulls"`
+    git_create_pr_res=`echo "$pr_data" | curl -k -H "Authorization: token $GITHUB_TOKEN" -H "Accept: application/vnd.github.v3.raw+json" --data @- "https://api.github.com/repos/$AR_REPO/pulls"`
-	local done_pr=`echo "$git_create_pr_res" | jq -r '.title'`
+    local done_pr=`echo "$git_create_pr_res" | jq -r '.title'`
-	if [ ! "$done_pr" == "" ] && [ ! "$done_pr" == "null" ]; then echo 1; else echo 0; fi
+    if [ ! "$done_pr" == "" ] && [ ! "$done_pr" == "null" ]; then echo 1; else echo 0; fi
 }
 

tools/copy-libs.sh

@@ -30,9 +30,15 @@ fi
 if [ -e "$AR_SDK/include" ]; then
 	rm -rf "$AR_SDK/include"
 fi
+if [ -e "$AR_SDK/flags" ]; then
+	rm -rf "$AR_SDK/flags"
+fi
 if [ -e "$AR_SDK/$MEMCONF" ]; then
 	rm -rf "$AR_SDK/$MEMCONF"
 fi
+if [ -e "$AR_SDK/platformio-build.py" ]; then
+	rm -rf "$AR_SDK/platformio-build.py"
+fi
 mkdir -p "$AR_SDK"
 
 function get_actual_path(){
@@ -95,7 +101,7 @@ for item in "${@:2:${#@}-5}"; do
 	elif [ "$prefix" = "-O" ]; then
 		PIO_CC_FLAGS+="$item "
 	elif [[ "$item" != "-Wall" && "$item" != "-Werror=all"  && "$item" != "-Wextra" ]]; then
-		if [[ "${item:0:23}" != "-mfix-esp32-psram-cache" && "${item:0:18}" != "-fmacro-prefix-map" ]]; then
+		if [[ "${item:0:23}" != "-mfix-esp32-psram-cache" && "${item:0:18}" != "-fmacro-prefix-map" && "${item:0:20}" != "-fdiagnostics-color=" ]]; then
 			C_FLAGS+="$item "
 		fi
 	fi
@@ -109,7 +115,7 @@ set -- $str
 for item in "${@:2:${#@}-5}"; do
 	prefix="${item:0:2}"
 	if [[ "$prefix" != "-I" && "$prefix" != "-D" && "$item" != "-Wall" && "$item" != "-Werror=all"  && "$item" != "-Wextra" && "$prefix" != "-O" ]]; then
-		if [[ "${item:0:23}" != "-mfix-esp32-psram-cache" && "${item:0:18}" != "-fmacro-prefix-map" ]]; then
+		if [[ "${item:0:23}" != "-mfix-esp32-psram-cache" && "${item:0:18}" != "-fmacro-prefix-map" && "${item:0:20}" != "-fdiagnostics-color=" ]]; then
 			AS_FLAGS+="$item "
 			if [[ $C_FLAGS == *"$item"* ]]; then
 				PIO_CC_FLAGS+="$item "
@@ -128,7 +134,7 @@ set -- $str
 for item in "${@:2:${#@}-5}"; do
 	prefix="${item:0:2}"
 	if [[ "$prefix" != "-I" && "$prefix" != "-D" && "$item" != "-Wall" && "$item" != "-Werror=all"  && "$item" != "-Wextra" && "$prefix" != "-O" ]]; then
-		if [[ "${item:0:23}" != "-mfix-esp32-psram-cache" && "${item:0:18}" != "-fmacro-prefix-map" ]]; then
+		if [[ "${item:0:23}" != "-mfix-esp32-psram-cache" && "${item:0:18}" != "-fmacro-prefix-map" && "${item:0:20}" != "-fdiagnostics-color=" ]]; then
 			CPP_FLAGS+="$item "
 			if [[ $PIO_CC_FLAGS != *"$item"* ]]; then
 				PIO_CXX_FLAGS+="$item "
@@ -155,13 +161,15 @@ else
 	libs="${libs:19:${#libs}-1}"
 	flags=`cat build/build.ninja | grep LINK_FLAGS`
 	flags="${flags:15:${#flags}-1}"
+	paths=`cat build/build.ninja | grep LINK_PATH`
+	paths="${paths:14:${#paths}-1}"
 	if [ "$IDF_TARGET" = "esp32" ]; then
 		flags="-Wno-frame-address $flags"
 	fi
-	if [ "$IDF_TARGET" != "esp32c3" ]; then
+	if [ "$IS_XTENSA" = "y" ]; then
 		flags="-mlongcalls $flags"
 	fi
-	str="$flags $libs"
+	str="$flags $libs $paths"
 fi
 if [ "$IDF_TARGET" = "esp32" ]; then
 	LD_SCRIPTS+="-T esp32.rom.redefined.ld "
@@ -271,13 +279,11 @@ done
 # END OF DATA EXTRACTION FROM CMAKE
 #
 
-AR_PLATFORMIO_PY="$AR_TOOLS/platformio-build-$IDF_TARGET.py"
+mkdir -p "$AR_SDK"
 
 # start generation of platformio-build.py
-awk "/ASFLAGS=\[/{n++}{print>n\"pio_start.txt\"}" $AR_COMPS/arduino/tools/platformio-build-$IDF_TARGET.py
+AR_PLATFORMIO_PY="$AR_SDK/platformio-build.py"
-awk "/\"ARDUINO_ARCH_ESP32\"/{n++}{print>n\"pio_end.txt\"}" 1pio_start.txt
+cat configs/pio_start.txt > "$AR_PLATFORMIO_PY"
-cat pio_start.txt > "$AR_PLATFORMIO_PY"
-rm pio_end.txt 1pio_start.txt pio_start.txt
 
 echo "    ASFLAGS=[" >> "$AR_PLATFORMIO_PY"
 if [ "$IS_XTENSA" = "y" ]; then
@@ -348,8 +354,8 @@ echo "        '-Wl,-Map=\"%s\"' % join(\"\${BUILD_DIR}\", \"\${PROGNAME}.map\")"
 echo "    ]," >> "$AR_PLATFORMIO_PY"
 echo "" >> "$AR_PLATFORMIO_PY"
 
-# # include dirs
+# include dirs
-AR_INC=""
+REL_INC=""
 echo "    CPPPATH=[" >> "$AR_PLATFORMIO_PY"
 
 set -- $INCLUDES
@@ -376,13 +382,13 @@ for item; do
 
 		out_sub="${item#*$ipath}"
 		out_cpath="$AR_SDK/include/$fname$out_sub"
-		AR_INC+=" \"-I{compiler.sdk.path}/include/$fname$out_sub\""
+		REL_INC+="-iwithprefixbefore $fname$out_sub "
 		if [ "$out_sub" = "" ]; then
-			echo "        join(FRAMEWORK_DIR, \"tools\", \"sdk\", \"$IDF_TARGET\", \"include\", \"$fname\")," >> "$AR_PLATFORMIO_PY"
+			echo "        join($PIO_SDK, \"include\", \"$fname\")," >> "$AR_PLATFORMIO_PY"
 		else
 			pio_sub="${out_sub:1}"
 			pio_sub=`echo $pio_sub | sed 's/\//\\", \\"/g'`
-			echo "        join(FRAMEWORK_DIR, \"tools\", \"sdk\", \"$IDF_TARGET\", \"include\", \"$fname\", \"$pio_sub\")," >> "$AR_PLATFORMIO_PY"
+			echo "        join($PIO_SDK, \"include\", \"$fname\", \"$pio_sub\")," >> "$AR_PLATFORMIO_PY"
 		fi
 		for f in `find "$item" -name '*.h'`; do
 			rel_f=${f#*$item}
@@ -396,10 +402,15 @@ for item; do
 			mkdir -p "$out_cpath$rel_p"
 			cp -n $f "$out_cpath$rel_p/"
 		done
+		# Temporary measure to fix issues caused by https://github.com/espressif/esp-idf/commit/dc4731101dd567cc74bbe4d0f03afe52b7db9afb#diff-1d2ce0d3989a80830fdf230bcaafb3117f32046d16cf46616ac3d55b4df2a988R17
+		if [[ "$fname" == "bt" && "$out_sub" == "/include/$IDF_TARGET/include" && -f "$ipath/controller/$IDF_TARGET/esp_bt_cfg.h" ]]; then
+			mkdir -p "$AR_SDK/include/$fname/controller/$IDF_TARGET"
+			cp -n "$ipath/controller/$IDF_TARGET/esp_bt_cfg.h" "$AR_SDK/include/$fname/controller/$IDF_TARGET/esp_bt_cfg.h"
+		fi
 	fi
 done
-echo "        join(FRAMEWORK_DIR, \"tools\", \"sdk\", \"$IDF_TARGET\", env.BoardConfig().get(\"build.arduino.memory_type\", (env.BoardConfig().get(\"build.flash_mode\", \"dio\") + \"_$OCT_PSRAM\")), \"include\")," >> "$AR_PLATFORMIO_PY"
+echo "        join($PIO_SDK, board_config.get(\"build.arduino.memory_type\", (board_config.get(\"build.flash_mode\", \"dio\") + \"_$OCT_PSRAM\")), \"include\")," >> "$AR_PLATFORMIO_PY"
-echo "        join(FRAMEWORK_DIR, \"cores\", env.BoardConfig().get(\"build.core\"))" >> "$AR_PLATFORMIO_PY"
+echo "        join(FRAMEWORK_DIR, \"cores\", board_config.get(\"build.core\"))" >> "$AR_PLATFORMIO_PY"
 echo "    ]," >> "$AR_PLATFORMIO_PY"
 echo "" >> "$AR_PLATFORMIO_PY"
 
@@ -421,9 +432,9 @@ for item; do
 done
 
 echo "    LIBPATH=[" >> "$AR_PLATFORMIO_PY"
-echo "        join(FRAMEWORK_DIR, \"tools\", \"sdk\", \"$IDF_TARGET\", \"lib\")," >> "$AR_PLATFORMIO_PY"
+echo "        join($PIO_SDK, \"lib\")," >> "$AR_PLATFORMIO_PY"
-echo "        join(FRAMEWORK_DIR, \"tools\", \"sdk\", \"$IDF_TARGET\", \"ld\")," >> "$AR_PLATFORMIO_PY"
+echo "        join($PIO_SDK, \"ld\")," >> "$AR_PLATFORMIO_PY"
-echo "        join(FRAMEWORK_DIR, \"tools\", \"sdk\", \"$IDF_TARGET\", env.BoardConfig().get(\"build.arduino.memory_type\", (env.BoardConfig().get(\"build.flash_mode\", \"dio\") + \"_$OCT_PSRAM\")))" >> "$AR_PLATFORMIO_PY"
+echo "        join($PIO_SDK, board_config.get(\"build.arduino.memory_type\", (board_config.get(\"build.flash_mode\", \"dio\") + \"_$OCT_PSRAM\")))" >> "$AR_PLATFORMIO_PY"
 echo "    ]," >> "$AR_PLATFORMIO_PY"
 echo "" >> "$AR_PLATFORMIO_PY"
 
@@ -449,40 +460,36 @@ for item; do
 	fi
 done
 
-# remove backslashes for Arduino
-DEFINES=`echo "$DEFINES" | tr -d '\\'`
-
-
 # end generation of platformio-build.py
-cat 1pio_end.txt >> "$AR_PLATFORMIO_PY"
+cat configs/pio_end.txt >> "$AR_PLATFORMIO_PY"
-rm 1pio_end.txt
 
-# arduino platform.txt
+# replace double backslashes with single one
-platform_file="$AR_COMPS/arduino/platform.txt"
+DEFINES=`echo "$DEFINES" | tr -s '\'`
-if [ -f "$AR_PLATFORM_TXT" ]; then
+
-	# use the file we have already compiled for other chips
+# target flags files
-	platform_file="$AR_PLATFORM_TXT"
+FLAGS_DIR="$AR_SDK/flags"
+mkdir -p "$FLAGS_DIR"
+echo -n "$DEFINES" > "$FLAGS_DIR/defines"
+echo -n "$REL_INC" > "$FLAGS_DIR/includes"
+echo -n "$C_FLAGS" > "$FLAGS_DIR/c_flags"
+echo -n "$CPP_FLAGS" > "$FLAGS_DIR/cpp_flags"
+echo -n "$AS_FLAGS" > "$FLAGS_DIR/S_flags"
+echo -n "$LD_FLAGS" > "$FLAGS_DIR/ld_flags"
+echo -n "$LD_SCRIPTS" > "$FLAGS_DIR/ld_scripts"
+echo -n "$AR_LIBS" > "$FLAGS_DIR/ld_libs"
+
+# sr model.bin
+if [ -f "build/srmodels/srmodels.bin" ]; then
+	mkdir -p "$AR_SDK/esp_sr"
+	cp -f "build/srmodels/srmodels.bin" "$AR_SDK/esp_sr/"
+	cp -f "partitions.csv" "$AR_SDK/esp_sr/"
 fi
-awk "/compiler.cpreprocessor.flags.$IDF_TARGET=/{n++}{print>n\"platform_start.txt\"}" "$platform_file"
-$SED -i "/compiler.cpreprocessor.flags.$IDF_TARGET\=/d" 1platform_start.txt
-awk "/compiler.ar.flags.$IDF_TARGET=/{n++}{print>n\"platform_mid.txt\"}" 1platform_start.txt
-rm -rf 1platform_start.txt
-
-cat platform_start.txt > "$AR_PLATFORM_TXT"
-echo "compiler.cpreprocessor.flags.$IDF_TARGET=$DEFINES $AR_INC" >> "$AR_PLATFORM_TXT"
-echo "compiler.c.elf.libs.$IDF_TARGET=$AR_LIBS" >> "$AR_PLATFORM_TXT"
-echo "compiler.c.flags.$IDF_TARGET=$C_FLAGS -MMD -c" >> "$AR_PLATFORM_TXT"
-echo "compiler.cpp.flags.$IDF_TARGET=$CPP_FLAGS -MMD -c" >> "$AR_PLATFORM_TXT"
-echo "compiler.S.flags.$IDF_TARGET=$AS_FLAGS -x assembler-with-cpp -MMD -c" >> "$AR_PLATFORM_TXT"
-echo "compiler.c.elf.flags.$IDF_TARGET=$LD_SCRIPTS $LD_FLAGS" >> "$AR_PLATFORM_TXT"
-cat 1platform_mid.txt >> "$AR_PLATFORM_TXT"
-rm -rf platform_start.txt platform_mid.txt 1platform_mid.txt
 
 # sdkconfig
 cp -f "sdkconfig" "$AR_SDK/sdkconfig"
 
 # gen_esp32part.py
-cp "$IDF_COMPS/partition_table/gen_esp32part.py" "$AR_GEN_PART_PY"
+# cp "$IDF_PATH/components/partition_table/gen_esp32part.py" "$AR_GEN_PART_PY"
 
 # copy precompiled libs (if we need them)
 function copy_precompiled_lib(){

tools/copy-to-arduino.sh

@@ -16,10 +16,8 @@ fi
 
 echo "Installing new libraries to $ESP32_ARDUINO"
 
-rm -rf $ESP32_ARDUINO/tools/sdk $ESP32_ARDUINO/tools/gen_esp32part.py $ESP32_ARDUINO/tools/platformio-build-*.py $ESP32_ARDUINO/platform.txt
+rm -rf $ESP32_ARDUINO/package/package_esp32_index.template.json && \
-
-cp -f $AR_OUT/platform.txt $ESP32_ARDUINO/
 cp -f $AR_OUT/package_esp32_index.template.json $ESP32_ARDUINO/package/package_esp32_index.template.json
-cp -Rf $AR_TOOLS/sdk $ESP32_ARDUINO/tools/
+
-cp -f $AR_TOOLS/gen_esp32part.py $ESP32_ARDUINO/tools/
+rm -rf $ESP32_ARDUINO/tools/esp32-arduino-libs && \
-cp -f $AR_TOOLS/platformio-build-*.py $ESP32_ARDUINO/tools/
+cp -Rf $AR_TOOLS/esp32-arduino-libs $ESP32_ARDUINO/tools/

tools/cron.sh

@@ -5,7 +5,4 @@ if [ ! "$GITHUB_EVENT_NAME" == "schedule" ]; then
     exit 1
 fi
 
-git checkout "$IDF_BRANCH" #local branches should match what the matrix wants to build
+bash ./build.sh -d
-DEPLOY_OUT=1
-source ./build.sh
-# bash ./tools/push-to-arduino.sh

tools/gen_platformio_manifest.py

@@ -0,0 +1,86 @@
+import argparse
+import json
+import os
+import re
+import sys
+
+MANIFEST_DATA = {
+    "name": "framework-arduinoespressif32-libs",
+    "description": "Precompiled libraries for Arduino Wiring-based Framework for the Espressif ESP32 series of SoCs",
+    "keywords": ["framework", "arduino", "espressif", "esp32"],
+    "license": "LGPL-2.1-or-later",
+    "repository": {
+        "type": "git",
+        "url": "https://github.com/espressif/esp32-arduino-libs",
+    },
+}
+
+
+def convert_version(version_string):
+    """A helper function that converts a custom IDF version string
+    extracted from a Git repository to a suitable SemVer alternative. For example:
+    'release/v5.1' becomes '5.1.0',
+    'v7.7.7' becomes '7.7.7'
+    """
+
+    regex_pattern = (
+        r"v(?P<MAJOR>0|[1-9]\d*)\.(?P<MINOR>0|[1-9]\d*)\.*(?P<PATCH>0|[1-9]\d*)*"
+    )
+    match = re.search(regex_pattern, version_string)
+    if not match:
+        sys.stderr.write(
+            f"Failed to find a regex match for '{regex_pattern}' in '{version_string}'\n"
+        )
+        return ""
+
+    major, minor, patch = match.groups()
+    if not patch:
+        patch = "0"
+
+    return ".".join((major, minor, patch))
+
+
+def main(dst_dir, version_string, commit_hash):
+
+    converted_version = convert_version(version_string)
+    if not converted_version:
+        sys.stderr.write(f"Failed to convert version '{version_string}'\n")
+        return -1
+
+    manifest_file_path = os.path.join(dst_dir, "package.json")
+    with open(manifest_file_path, "w", encoding="utf8") as fp:
+        MANIFEST_DATA["version"] = f"{converted_version}+sha.{commit_hash}"
+        json.dump(MANIFEST_DATA, fp, indent=2)
+
+    print(
+        f"Generated PlatformIO manifest file '{manifest_file_path}' with '{converted_version}' version"
+    )
+    return 0
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+    parser.add_argument(
+        "-o",
+        "--dst-dir",
+        dest="dst_dir",
+        required=True,
+        help="Destination folder where the 'package.json' manifest will be located",
+    )
+    parser.add_argument(
+        "-s",
+        "--version-string",
+        dest="version_string",
+        required=True,
+        help="ESP-IDF version string used for compiling libraries",
+    )
+    parser.add_argument(
+        "-c",
+        "--commit-hash",
+        dest="commit_hash",
+        required=True,
+        help="ESP-IDF revision in form of a commit hash",
+    )
+    args = parser.parse_args()
+
+    sys.exit(main(args.dst_dir, args.version_string, args.commit_hash))

tools/gen_tools_json.py

@@ -25,10 +25,17 @@ if __name__ == '__main__':
         prog = 'gen_tools_json',
         description = 'Update Arduino package index with the tolls found in ESP-IDF')
     parser.add_argument('-i', '--esp-idf', dest='idf_path', required=True, help='Path to ESP-IDF')
-    parser.add_argument('-j', '--pkg-json', dest='arduino_json', required=True, help='path to Arduino package json')
+    parser.add_argument('-j', '--pkg-json', dest='arduino_json', required=False, help='path to Arduino package json')
     parser.add_argument('-o', '--out-path', dest='out_path', required=True, help='Output path to store the update package json')
     args = parser.parse_args()
 
+    simple_output = False
+    if args.arduino_json == None:
+        print('Source was not selected')
+        simple_output = True
+    else:
+        print('Source {0}.'.format(args.arduino_json))
+
     idf_path = args.idf_path;
     arduino_json = args.arduino_json;
     out_path = args.out_path;
@@ -37,7 +44,9 @@ if __name__ == '__main__':
     arduino_tools = ["xtensa-esp32-elf","xtensa-esp32s2-elf","xtensa-esp32s3-elf","xtensa-esp-elf-gdb","riscv32-esp-elf","riscv32-esp-elf-gdb","openocd-esp32"]
 
     # code start
-    farray = json.load(open(arduino_json))
+    farray = {"packages":[{"platforms":[{"toolsDependencies":[]}],"tools":[]}]}
+    if simple_output == False:
+        farray = json.load(open(arduino_json))
 
     idf_tools = json.load(open(idf_path + '/tools/tools.json'))
     for tool in idf_tools['tools']:
@@ -51,21 +60,30 @@ if __name__ == '__main__':
             tool_name += '-gcc'
         print('Found {0}, version: {1}'.format(tool_name, tool_version))
         
-        dep_found = False
+        if simple_output == False:
-        dep_skip = False
+            dep_found = False
-        for dep in farray['packages'][0]['platforms'][0]['toolsDependencies']:
+            dep_skip = False
-            if dep['name'] == tool_name:
+            for dep in farray['packages'][0]['platforms'][0]['toolsDependencies']:
-                if dep['version'] == tool_version:
+                if dep['name'] == tool_name:
-                    print('Skipping {0}. Same version {1}'.format(tool_name, tool_version))
+                    if dep['version'] == tool_version:
-                    dep_skip = True
+                        print('Skipping {0}. Same version {1}'.format(tool_name, tool_version))
-                    break
+                        dep_skip = True
-                print('Updating dependency version of {0} from {1} to {2}'.format(tool_name, dep['version'], tool_version))
+                        break
-                dep['version'] = tool_version
+                    print('Updating dependency version of {0} from {1} to {2}'.format(tool_name, dep['version'], tool_version))
-                dep_found = True
+                    dep['version'] = tool_version
-        if dep_skip == True:
+                    dep_found = True
-            continue
+            if dep_skip == True:
-        if dep_found == False:
+                continue
-            print('Adding new dependency: {0} version {1}'.format(tool_name, tool_version))
+            if dep_found == False:
+                print('Adding new dependency: {0} version {1}'.format(tool_name, tool_version))
+                deps = {
+                    "packager": "esp32",
+                    "name": tool_name,
+                    "version": tool_version
+                }
+                farray['packages'][0]['platforms'][0]['toolsDependencies'].append(deps)
+        else:
+            print('Adding dependency: {0} version {1}'.format(tool_name, tool_version))
             deps = {
                 "packager": "esp32",
                 "name": tool_name,
@@ -84,7 +102,7 @@ if __name__ == '__main__':
                 "url": tool_data['url'],
                 "archiveFileName": os.path.basename(tool_data['url']),
                 "checksum": "SHA-256:"+tool_data['sha256'],
-                "size": tool_data['size']
+                "size": str(tool_data['size'])
             }
 
             if arch == "win32":
@@ -111,15 +129,24 @@ if __name__ == '__main__':
 
             systems.append(system)
 
-        tool_found = False
+        if simple_output == False:
-        for t in farray['packages'][0]['tools']:
+            tool_found = False
-            if t['name'] == tool_name:
+            for t in farray['packages'][0]['tools']:
-                t['version'] = tool_version
+                if t['name'] == tool_name:
-                t['systems'] = systems
+                    t['version'] = tool_version
-                tool_found = True
+                    t['systems'] = systems
-                print('Updating binaries of {0} to version {1}'.format(tool_name, tool_version))
+                    tool_found = True
-        if tool_found == False:
+                    print('Updating binaries of {0} to version {1}'.format(tool_name, tool_version))
-            print('Adding new tool: {0} version {1}'.format(tool_name, tool_version))
+            if tool_found == False:
+                print('Adding new tool: {0} version {1}'.format(tool_name, tool_version))
+                tools = {
+                    "name": tool_name,
+                    "version": tool_version,
+                    "systems": systems
+                }
+                farray['packages'][0]['tools'].append(tools)
+        else:
+            print('Adding tool: {0} version {1}'.format(tool_name, tool_version))
             tools = {
                 "name": tool_name,
                 "version": tool_version,
@@ -128,7 +155,10 @@ if __name__ == '__main__':
             farray['packages'][0]['tools'].append(tools)
 
     json_str = json.dumps(farray, indent=2)
-    out_file = out_path + os.path.basename(arduino_json)
+    out_file = out_path + "tools.json"
+    if simple_output == False:
+        out_file = out_path + os.path.basename(arduino_json)
+
     with open(out_file, "w") as f:
         f.write(json_str+"\n")
         f.close()

tools/get_projbuild_gitconfig.py

@@ -0,0 +1,124 @@
+# This file is expected to be present in ${COMPONENT_DIR}
+# accessed from components/esp_insights/CMakeLists.txt
+# Used in:
+# 1. Project ESP Insights build package tar file
+
+#from __future__ import unicode_literals
+import os
+import sys
+import json
+import subprocess
+from builtins import range, str
+from io import open
+
+# Input project directory from CMakeLists.txt
+PROJ_DIR=sys.argv[1]
+# Input project name
+PROJ_NAME=sys.argv[2]
+# Input project version
+PROJ_VER=sys.argv[3]
+# Input custom config filename from CMakeLists.txt
+FILENAME=sys.argv[4]
+# Input IDF_PATH from CMakeLists.txt
+IDF_PATH=sys.argv[5]
+# Input target
+TARGET=sys.argv[6]
+
+NEWLINE = "\n"
+
+CONFIG = {}
+
+# Set Config
+
+# Set current directory i.e Set ${COMPONENT_DIR} as current directory
+CURR_DIR = os.getcwd()
+
+def _change_dir(dirname):
+    # Change directory
+    os.chdir(dirname)
+
+
+def _set_submodule_cfg(submodules, repo_name):
+    # Set config for submodules
+    CFG_TITLE = "submodules"
+    NAME_STR = "name"
+    VERSION_STR = "version"
+    CONFIG[repo_name][CFG_TITLE] = []
+    
+    if submodules:
+        # Get the submodule name and version
+        submodules_list = submodules.strip().split(NEWLINE)
+        for i in range(0, len(submodules_list), 2):
+            name = submodules_list[i].split('\'')[1]
+            version = submodules_list[i+1]
+            submodule_json = { NAME_STR: name, VERSION_STR: version }
+            CONFIG[repo_name][CFG_TITLE].append(submodule_json)
+
+
+def run_cmd(command, get_basename=False):
+    try:
+        resp = subprocess.check_output(command, shell=True).strip().decode('utf-8')
+        if get_basename:
+            resp = os.path.basename(resp)
+        return resp
+    except subprocess.CalledProcessError:
+        raise Exception("ERROR: Please check command : {}".format(command))
+
+def set_cfg(config_name):
+    # Set config for ESP-IDF Repo
+    if config_name  == "esp-idf":
+        # Get repo name (for IDF repo)
+        REPO_CMD='git rev-parse --show-toplevel'
+        repo_name = run_cmd(REPO_CMD, get_basename=True)
+        CONFIG[repo_name] = {}
+
+        # Get commit HEAD
+        GITHEAD_STR = "HEAD"
+        HEAD='git describe --always --tags --dirty'
+        head_ver = run_cmd(HEAD)
+        CONFIG[repo_name][GITHEAD_STR] = head_ver
+
+        # Get submodule latest refs
+        SUBMODULE = 'git submodule foreach git describe --always --tags --dirty'
+        submodules = run_cmd(SUBMODULE)
+        _set_submodule_cfg(submodules, repo_name)
+    elif config_name == "toolchain":
+        # Set config for Toolchain Version
+        arch_target = "xtensa-" + TARGET
+        if TARGET == "esp32c3" or TARGET == "esp32c2" or TARGET == "esp32h2" or TARGET == "esp32c6":
+            arch_target = "riscv32-esp"
+        # Get toolchain version
+        TOOLCHAIN_STR = "toolchain"
+        TOOLCHAIN = arch_target + '-elf-gcc --version'
+        toolchain = run_cmd(TOOLCHAIN)
+        CONFIG[TOOLCHAIN_STR] = toolchain.strip().split(NEWLINE)[0]
+
+# Set project details - name and version
+def set_project_details():
+    # Set project name and version
+    CONFIG['project'] = {}
+    CONFIG['project']['name'] = PROJ_NAME
+    CONFIG['project']['version'] = PROJ_VER
+
+try:
+    with open(FILENAME, "w+", encoding="utf-8") as output_file:
+        # ESP-IDF REPO CONFIG
+        # Change to ESP-IDF Directory
+        _change_dir(IDF_PATH)
+        set_cfg("esp-idf")
+
+        # Change back to ${COMPONENT_DIR}
+        _change_dir(CURR_DIR)
+
+        # Set project name and version
+        set_project_details()
+
+        # GET TOOLCHAIN VERSION
+        set_cfg("toolchain")
+
+        output_file.write(str(json.dumps(CONFIG, indent=4, sort_keys=True)))
+
+except Exception as e:
+    # Remove config file created if error occurs
+    os.system("rm " + FILENAME)
+    sys.exit(e)

tools/install-arduino.sh

@@ -0,0 +1,62 @@
+#/bin/bash
+
+source ./tools/config.sh
+
+#
+# CLONE/UPDATE ARDUINO
+#
+echo "Updating ESP32 Arduino..."
+if [ ! -d "$AR_COMPS/arduino" ]; then
+	git clone $AR_REPO_URL "$AR_COMPS/arduino"
+fi
+
+if [ -z $AR_BRANCH ]; then
+	if [ -z $GITHUB_HEAD_REF ]; then
+		current_branch=`git branch --show-current`
+	else
+		current_branch="$GITHUB_HEAD_REF"
+	fi
+	echo "Current Branch: $current_branch"
+	if [[ "$current_branch" != "master" && `git_branch_exists "$AR_COMPS/arduino" "$current_branch"` == "1" ]]; then
+		export AR_BRANCH="$current_branch"
+	else
+		if [ "$IDF_TAG" ]; then #tag was specified at build time
+			AR_BRANCH_NAME="idf-$IDF_TAG"
+		elif [ "$IDF_COMMIT" ]; then #commit was specified at build time
+			AR_BRANCH_NAME="idf-$IDF_COMMIT"
+		else
+			AR_BRANCH_NAME="idf-$IDF_BRANCH"
+		fi
+		has_ar_branch=`git_branch_exists "$AR_COMPS/arduino" "$AR_BRANCH_NAME"`
+		if [ "$has_ar_branch" == "1" ]; then
+			export AR_BRANCH="$AR_BRANCH_NAME"
+		else
+			has_ar_branch=`git_branch_exists "$AR_COMPS/arduino" "$AR_PR_TARGET_BRANCH"`
+			if [ "$has_ar_branch" == "1" ]; then
+				export AR_BRANCH="$AR_PR_TARGET_BRANCH"
+			fi
+		fi
+	fi
+fi
+
+if [ "$AR_BRANCH" ]; then
+	echo "AR_BRANCH='$AR_BRANCH'"
+	git -C "$AR_COMPS/arduino" checkout "$AR_BRANCH" && \
+	git -C "$AR_COMPS/arduino" fetch && \
+	git -C "$AR_COMPS/arduino" pull --ff-only
+fi
+if [ $? -ne 0 ]; then exit 1; fi
+
+#
+# CLONE/UPDATE ESP32-ARDUINO-LIBS
+#
+if [ ! -d "$IDF_LIBS_DIR" ]; then
+	echo "Cloning esp32-arduino-libs..."
+	git clone "$AR_LIBS_REPO_URL" "$IDF_LIBS_DIR"
+else
+	echo "Updating esp32-arduino-libs..."
+	git -C "$IDF_LIBS_DIR" fetch && \
+	git -C "$IDF_LIBS_DIR" pull --ff-only
+fi
+if [ $? -ne 0 ]; then exit 1; fi
+

tools/install-esp-idf.sh

@@ -11,14 +11,16 @@ fi
 # CLONE ESP-IDF
 #
 
-IDF_REPO_URL="https://github.com/espressif/esp-idf.git"
 if [ ! -d "$IDF_PATH" ]; then
 	echo "ESP-IDF is not installed! Installing local copy"
 	git clone $IDF_REPO_URL -b $IDF_BRANCH
 	idf_was_installed="1"
 fi
 
-if [ "$IDF_COMMIT" ]; then
+if [ "$IDF_TAG" ]; then
+    git -C "$IDF_PATH" checkout "tags/$IDF_TAG"
+    idf_was_installed="1"
+elif [ "$IDF_COMMIT" ]; then
     git -C "$IDF_PATH" checkout "$IDF_COMMIT"
     commit_predefined="1"
 fi
@@ -30,6 +32,13 @@ fi
 if [ ! -x $idf_was_installed ] || [ ! -x $commit_predefined ]; then
 	git -C $IDF_PATH submodule update --init --recursive
 	$IDF_PATH/install.sh
+	export IDF_COMMIT=$(git -C "$IDF_PATH" rev-parse --short HEAD)
+	export IDF_BRANCH=$(git -C "$IDF_PATH" symbolic-ref --short HEAD || git -C "$IDF_PATH" tag --points-at HEAD)
+
+	# Temporarily patch the ETH driver to support custom SPI
+	cd $IDF_PATH
+	patch -p1 -i ../patches/spi_eth.diff
+	cd -
 fi
 
 #
@@ -37,8 +46,6 @@ fi
 #
 
 source $IDF_PATH/export.sh
-export IDF_COMMIT=$(git -C "$IDF_PATH" rev-parse --short HEAD)
-export IDF_BRANCH=$(git -C "$IDF_PATH" symbolic-ref --short HEAD || git -C "$IDF_PATH" tag --points-at HEAD)
 
 #
 # SETUP ARDUINO DEPLOY
@@ -55,43 +62,27 @@ if [ "$GITHUB_EVENT_NAME" == "schedule" ] || [ "$GITHUB_EVENT_NAME" == "reposito
 		AR_NEW_COMMIT_MESSAGE="IDF $IDF_COMMIT"
 		AR_NEW_PR_TITLE="$AR_NEW_COMMIT_MESSAGE"
 	fi
+	LIBS_VERSION="idf-"${IDF_BRANCH//\//_}"-$IDF_COMMIT"
 
 	AR_HAS_COMMIT=`git_commit_exists "$AR_COMPS/arduino" "$AR_NEW_COMMIT_MESSAGE"`
 	AR_HAS_BRANCH=`git_branch_exists "$AR_COMPS/arduino" "$AR_NEW_BRANCH_NAME"`
-	AR_HAS_PR=`git_pr_exists "$AR_NEW_BRANCH_NAME"`
+	AR_HAS_PR=`github_pr_exists "$AR_REPO" "$AR_NEW_BRANCH_NAME"`
+
+	LIBS_HAS_COMMIT=`git_commit_exists "$IDF_LIBS_DIR" "$AR_NEW_COMMIT_MESSAGE"`
+	LIBS_HAS_BRANCH=`git_branch_exists "$IDF_LIBS_DIR" "$AR_NEW_BRANCH_NAME"`
+
+	if [ "$LIBS_HAS_COMMIT" == "1" ]; then
+		echo "Commit '$AR_NEW_COMMIT_MESSAGE' Already Exists in esp32-arduino-libs"
+	fi
 
 	if [ "$AR_HAS_COMMIT" == "1" ]; then
-		echo "Commit '$AR_NEW_COMMIT_MESSAGE' Already Exists"
+		echo "Commit '$AR_NEW_COMMIT_MESSAGE' Already Exists in arduino-esp32"
-		mkdir -p dist && echo "Commit '$AR_NEW_COMMIT_MESSAGE' Already Exists" > dist/log.txt
+	fi
+
+	if [ "$LIBS_HAS_COMMIT" == "1" ] && [ "$AR_HAS_COMMIT" == "1" ]; then
 		exit 0
 	fi
 
-	if [ "$AR_HAS_BRANCH" == "1" ]; then
-		echo "Branch '$AR_NEW_BRANCH_NAME' Already Exists"
-	fi
-
-	if [ "$AR_HAS_PR" == "1" ]; then
-		echo "PR '$AR_NEW_PR_TITLE' Already Exists"
-	fi
-
-	# setup git for pushing
-	git config --global github.user "$GITHUB_ACTOR"
-	git config --global user.name "$GITHUB_ACTOR"
-	git config --global user.email "$GITHUB_ACTOR@github.com"
-
-	# create or checkout the branch
-	if [ ! $AR_HAS_BRANCH == "0" ]; then
-		echo "Switching to arduino branch '$AR_NEW_BRANCH_NAME'..."
-		git -C "$AR_COMPS/arduino" checkout $AR_NEW_BRANCH_NAME
-	else
-		echo "Creating arduino branch '$AR_NEW_BRANCH_NAME'..."
-		git -C "$AR_COMPS/arduino" checkout -b $AR_NEW_BRANCH_NAME
-	fi
-	if [ $? -ne 0 ]; then
-	    echo "ERROR: Checkout of branch '$AR_NEW_BRANCH_NAME' failed"
-		exit 1
-	fi
-
 	export AR_NEW_BRANCH_NAME
 	export AR_NEW_COMMIT_MESSAGE
 	export AR_NEW_PR_TITLE
@@ -99,4 +90,8 @@ if [ "$GITHUB_EVENT_NAME" == "schedule" ] || [ "$GITHUB_EVENT_NAME" == "reposito
 	export AR_HAS_COMMIT
 	export AR_HAS_BRANCH
 	export AR_HAS_PR
+
+	export LIBS_VERSION
+	export LIBS_HAS_COMMIT
+	export LIBS_HAS_BRANCH
 fi

tools/push-to-arduino.sh

@@ -11,14 +11,117 @@ if ! [ -d "$AR_COMPS/arduino" ]; then
 	exit 1
 fi
 
+# setup git for pushing
+git config --global github.user "$GITHUB_ACTOR"
+git config --global user.name "$GITHUB_ACTOR"
+git config --global user.email "$GITHUB_ACTOR@github.com"
+
 #
 # UPDATE FILES
 #
 
+#
+# esp32-arduino-libs
+#
+
+if [ $LIBS_HAS_COMMIT == "0" ] || [ $AR_HAS_COMMIT == "0" ]; then
+	cd "$AR_ROOT"
+	# create branch if necessary
+	if [ "$LIBS_HAS_BRANCH" == "1" ]; then
+		echo "Branch '$AR_NEW_BRANCH_NAME' Already Exists"
+		echo "Switching to esp32-arduino-libs branch '$AR_NEW_BRANCH_NAME'..."
+		git -C "$IDF_LIBS_DIR" checkout $AR_NEW_BRANCH_NAME
+	else
+		echo "Creating esp32-arduino-libs branch '$AR_NEW_BRANCH_NAME'..."
+		git -C "$IDF_LIBS_DIR" checkout -b $AR_NEW_BRANCH_NAME
+	fi
+	if [ $? -ne 0 ]; then
+		echo "ERROR: Checkout of branch '$AR_NEW_BRANCH_NAME' failed"
+		exit 1
+	fi
+
+	# make changes to the files
+	echo "Patching files in esp32-arduino-libs branch '$AR_NEW_BRANCH_NAME'..."
+	rm -rf $IDF_LIBS_DIR/* && cp -Rf $AR_TOOLS/esp32-arduino-libs/* $IDF_LIBS_DIR/
+	
+	cd $IDF_LIBS_DIR
+	if [ -f "README.md" ]; then
+		rm -rf "README.md"
+	fi
+
+	# did any of the files change?
+	if [ -n "$(git status --porcelain)" ]; then
+		echo "Pushing changes to esp32-arduino-libs branch '$AR_NEW_BRANCH_NAME'..."
+	    git add . && git commit --message "$AR_NEW_COMMIT_MESSAGE" && git push -u origin $AR_NEW_BRANCH_NAME
+		if [ $? -ne 0 ]; then
+		    echo "ERROR: Pushing to branch '$AR_NEW_BRANCH_NAME' failed"
+			exit 1
+		fi
+		IDF_LIBS_COMMIT=`git rev-parse --verify HEAD`
+		IDF_LIBS_DL_URL="https://codeload.github.com/espressif/esp32-arduino-libs/zip/$IDF_LIBS_COMMIT"
+		# ToDo: this URL needs to get into Arduino's package.json
+
+		# Download the file
+		filename="esp32-arduino-libs-$IDF_LIBS_COMMIT.zip"
+		curl -s -o "$filename" "$IDF_LIBS_DL_URL"
+
+		# Check if the download was successful
+		if [ $? -ne 0 ]; then
+		  echo "Error downloading file from $IDF_LIBS_DL_URL"
+		  exit 1
+		fi
+
+		# Calculate the size in bytes and SHA-256 sum
+		size=$(stat -c%s "$filename")
+		sha256sum=$(sha256sum "$filename" | awk '{print $1}')
+
+		# Clean up the downloaded file
+		rm "$filename"
+
+		# Print the results
+		echo "Tool: esp32-arduino-libs"
+		echo "Version: $LIBS_VERSION"
+		echo "URL: $IDF_LIBS_DL_URL"
+		echo "File: $filename"
+		echo "Size: $size bytes"
+		echo "SHA-256: $sha256sum"
+		echo "JSON: $AR_OUT/package_esp32_index.template.json"
+		cd "$AR_ROOT"
+		python3 tools/add_sdk_json.py -j "$AR_OUT/package_esp32_index.template.json" -n "esp32-arduino-libs" -v "$LIBS_VERSION" -u "$IDF_LIBS_DL_URL" -f "$filename" -s "$size" -c "$sha256sum"
+		if [ $? -ne 0 ]; then exit 1; fi
+
+	else
+	    echo "No changes in esp32-arduino-libs branch '$AR_NEW_BRANCH_NAME'"
+	    if [ $LIBS_HAS_BRANCH == "0" ]; then
+	    	echo "Delete created branch '$AR_NEW_BRANCH_NAME'"
+	    	git branch -d $AR_NEW_BRANCH_NAME
+	    fi
+	    exit 0
+	fi
+fi
+
+#
+# esp32-arduino
+#
+
 if [ $AR_HAS_COMMIT == "0" ]; then
+	cd "$AR_ROOT"
+	# create or checkout the branch
+	if [ ! $AR_HAS_BRANCH == "0" ]; then
+		echo "Switching to arduino branch '$AR_NEW_BRANCH_NAME'..."
+		git -C "$AR_COMPS/arduino" checkout $AR_NEW_BRANCH_NAME
+	else
+		echo "Creating arduino branch '$AR_NEW_BRANCH_NAME'..."
+		git -C "$AR_COMPS/arduino" checkout -b $AR_NEW_BRANCH_NAME
+	fi
+	if [ $? -ne 0 ]; then
+	    echo "ERROR: Checkout of branch '$AR_NEW_BRANCH_NAME' failed"
+		exit 1
+	fi
+
 	# make changes to the files
 	echo "Patching files in branch '$AR_NEW_BRANCH_NAME'..."
-	ESP32_ARDUINO="$AR_COMPS/arduino" ./tools/copy-to-arduino.sh
+	rm -rf "$AR_COMPS/arduino/package/package_esp32_index.template.json" && cp -f "$AR_OUT/package_esp32_index.template.json" "$AR_COMPS/arduino/package/package_esp32_index.template.json"
 	
 	cd $AR_COMPS/arduino
 
@@ -38,17 +141,18 @@ if [ $AR_HAS_COMMIT == "0" ]; then
 	    fi
 	    exit 0
 	fi
-fi
 
-#
+	# CREATE PULL REQUEST
-# CREATE PULL REQUEST
+	if [ "$AR_HAS_PR" == "0" ]; then
-#
+		echo "Creating PR '$AR_NEW_PR_TITLE'..."
-
+		pr_created=`git_create_pr "$AR_NEW_BRANCH_NAME" "$AR_NEW_PR_TITLE" "$AR_PR_TARGET_BRANCH"`
-if [ "$AR_HAS_PR" == "0" ]; then
+		if [ $pr_created == "0" ]; then
-	pr_created=`git_create_pr "$AR_NEW_BRANCH_NAME" "$AR_NEW_PR_TITLE" "$AR_PR_TARGET_BRANCH"`
+			echo "ERROR: Failed to create PR '$AR_NEW_PR_TITLE': "`echo "$git_create_pr_res" | jq -r '.message'`": "`echo "$git_create_pr_res" | jq -r '.errors[].message'`
-	if [ $pr_created == "0" ]; then
+			exit 1
-		echo "ERROR: Failed to create PR '$AR_NEW_PR_TITLE': "`echo "$git_create_pr_res" | jq -r '.message'`": "`echo "$git_create_pr_res" | jq -r '.errors[].message'`
+		fi
-		exit 1
+	else
+		echo "PR '$AR_NEW_PR_TITLE' Already Exists"
 	fi
 fi
+
 exit 0

tools/update-components.sh

@@ -6,52 +6,9 @@ CAMERA_REPO_URL="https://github.com/espressif/esp32-camera.git"
 DL_REPO_URL="https://github.com/espressif/esp-dl.git"
 SR_REPO_URL="https://github.com/espressif/esp-sr.git"
 RMAKER_REPO_URL="https://github.com/espressif/esp-rainmaker.git"
-INSIGHTS_REPO_URL="https://github.com/espressif/esp-insights.git"
-DSP_REPO_URL="https://github.com/espressif/esp-dsp.git"
 LITTLEFS_REPO_URL="https://github.com/joltwallet/esp_littlefs.git"
 TINYUSB_REPO_URL="https://github.com/hathach/tinyusb.git"
-
+TFLITE_REPO_URL="https://github.com/espressif/tflite-micro-esp-examples.git"
-#
-# CLONE/UPDATE ARDUINO
-#
-echo "Updating ESP32 Arduino..."
-if [ ! -d "$AR_COMPS/arduino" ]; then
-	git clone $AR_REPO_URL "$AR_COMPS/arduino"
-fi
-
-if [ -z $AR_BRANCH ]; then
-	if [ -z $GITHUB_HEAD_REF ]; then
-		current_branch=`git branch --show-current`
-	else
-		current_branch="$GITHUB_HEAD_REF"
-	fi
-	echo "Current Branch: $current_branch"
-	if [[ "$current_branch" != "master" && `git_branch_exists "$AR_COMPS/arduino" "$current_branch"` == "1" ]]; then
-		export AR_BRANCH="$current_branch"
-	else
-		if [ -z "$IDF_COMMIT" ]; then #commit was not specified at build time
-			AR_BRANCH_NAME="idf-$IDF_BRANCH"
-		else
-			AR_BRANCH_NAME="idf-$IDF_COMMIT"
-		fi
-		has_ar_branch=`git_branch_exists "$AR_COMPS/arduino" "$AR_BRANCH_NAME"`
-		if [ "$has_ar_branch" == "1" ]; then
-			export AR_BRANCH="$AR_BRANCH_NAME"
-		else
-			has_ar_branch=`git_branch_exists "$AR_COMPS/arduino" "$AR_PR_TARGET_BRANCH"`
-			if [ "$has_ar_branch" == "1" ]; then
-				export AR_BRANCH="$AR_PR_TARGET_BRANCH"
-			fi
-		fi
-	fi
-fi
-
-if [ "$AR_BRANCH" ]; then
-	git -C "$AR_COMPS/arduino" checkout "$AR_BRANCH" && \
-	git -C "$AR_COMPS/arduino" fetch && \
-	git -C "$AR_COMPS/arduino" pull --ff-only
-fi
-if [ $? -ne 0 ]; then exit 1; fi
 
 #
 # CLONE/UPDATE ESP32-CAMERA
@@ -63,10 +20,6 @@ else
 	git -C "$AR_COMPS/esp32-camera" fetch && \
 	git -C "$AR_COMPS/esp32-camera" pull --ff-only
 fi
-#this is a temp measure to fix build issue
-# if [ -f "$AR_COMPS/esp32-camera/idf_component.yml" ]; then
-# 	rm -rf "$AR_COMPS/esp32-camera/idf_component.yml"
-# fi
 if [ $? -ne 0 ]; then exit 1; fi
 
 #
@@ -75,11 +28,22 @@ if [ $? -ne 0 ]; then exit 1; fi
 echo "Updating ESP-DL..."
 if [ ! -d "$AR_COMPS/esp-dl" ]; then
 	git clone $DL_REPO_URL "$AR_COMPS/esp-dl"
+	#this is a temp measure to fix build issue
+	mv "$AR_COMPS/esp-dl/CMakeLists.txt" "$AR_COMPS/esp-dl/CMakeListsOld.txt"
+	echo "idf_build_get_property(target IDF_TARGET)" > "$AR_COMPS/esp-dl/CMakeLists.txt"
+	echo "if(NOT \${IDF_TARGET} STREQUAL \"esp32c6\" AND NOT \${IDF_TARGET} STREQUAL \"esp32h2\")" >> "$AR_COMPS/esp-dl/CMakeLists.txt"
+	cat "$AR_COMPS/esp-dl/CMakeListsOld.txt" >> "$AR_COMPS/esp-dl/CMakeLists.txt"
+	echo "endif()" >> "$AR_COMPS/esp-dl/CMakeLists.txt"
+	rm -rf "$AR_COMPS/esp-dl/CMakeListsOld.txt"
 else
 	git -C "$AR_COMPS/esp-dl" fetch && \
 	git -C "$AR_COMPS/esp-dl" pull --ff-only
 fi
 if [ $? -ne 0 ]; then exit 1; fi
+#this is a temp measure to fix build issue
+if [ -f "$AR_COMPS/esp-dl/idf_component.yml" ]; then
+	rm -rf "$AR_COMPS/esp-dl/idf_component.yml"
+fi
 
 #
 # CLONE/UPDATE ESP-SR
@@ -91,12 +55,29 @@ else
 	git -C "$AR_COMPS/esp-sr" fetch && \
 	git -C "$AR_COMPS/esp-sr" pull --ff-only
 fi
-#this is a temp measure to fix build issue
+if [ $? -ne 0 ]; then exit 1; fi
-if [ -f "$AR_COMPS/esp-sr/idf_component.yml" ]; then
+
-	rm -rf "$AR_COMPS/esp-sr/idf_component.yml"
+#
+# CLONE/UPDATE ESP-RAINMAKER
+#
+echo "Updating ESP-RainMaker..."
+if [ ! -d "$AR_COMPS/esp-rainmaker" ]; then
+    git clone $RMAKER_REPO_URL "$AR_COMPS/esp-rainmaker" && \
+	git -C "$AR_COMPS/esp-rainmaker" reset --hard d8e93454f495bd8a414829ec5e86842b373ff555 && \
+    git -C "$AR_COMPS/esp-rainmaker" submodule update --init --recursive
+# else
+# 	git -C "$AR_COMPS/esp-rainmaker" fetch && \
+# 	git -C "$AR_COMPS/esp-rainmaker" pull --ff-only && \
+#     git -C "$AR_COMPS/esp-rainmaker" submodule update --init --recursive
 fi
 if [ $? -ne 0 ]; then exit 1; fi
 
+#this is a temp measure to fix build issue
+if [ -f "$AR_COMPS/esp-rainmaker/components/esp-insights/components/esp_insights/scripts/get_projbuild_gitconfig.py" ] && [ `cat "$AR_COMPS/esp-rainmaker/components/esp-insights/components/esp_insights/scripts/get_projbuild_gitconfig.py" | grep esp32c6 | wc -l` == "0" ]; then
+	echo "Overwriting 'get_projbuild_gitconfig.py'"
+	cp -f "tools/get_projbuild_gitconfig.py" "$AR_COMPS/esp-rainmaker/components/esp-insights/components/esp_insights/scripts/get_projbuild_gitconfig.py"
+fi
+
 #
 # CLONE/UPDATE ESP-LITTLEFS
 #
@@ -111,46 +92,6 @@ else
 fi
 if [ $? -ne 0 ]; then exit 1; fi
 
-#
-# CLONE/UPDATE ESP-RAINMAKER
-#
-echo "Updating ESP-RainMaker..."
-if [ ! -d "$AR_COMPS/esp-rainmaker" ]; then
-    git clone $RMAKER_REPO_URL "$AR_COMPS/esp-rainmaker" && \
-    git -C "$AR_COMPS/esp-rainmaker" submodule update --init --recursive
-else
-	git -C "$AR_COMPS/esp-rainmaker" fetch && \
-	git -C "$AR_COMPS/esp-rainmaker" pull --ff-only && \
-    git -C "$AR_COMPS/esp-rainmaker" submodule update --init --recursive
-fi
-if [ $? -ne 0 ]; then exit 1; fi
-
-#
-# CLONE/UPDATE ESP-INSIGHTS
-#
-echo "Updating ESP-Insights..."
-if [ ! -d "$AR_COMPS/esp-insights" ]; then
-    git clone $INSIGHTS_REPO_URL "$AR_COMPS/esp-insights" && \
-    git -C "$AR_COMPS/esp-insights" submodule update --init --recursive
-else
-	git -C "$AR_COMPS/esp-insights" fetch && \
-	git -C "$AR_COMPS/esp-insights" pull --ff-only && \
-    git -C "$AR_COMPS/esp-insights" submodule update --init --recursive
-fi
-if [ $? -ne 0 ]; then exit 1; fi
-
-#
-# CLONE/UPDATE ESP-DSP
-#
-echo "Updating ESP-DSP..."
-if [ ! -d "$AR_COMPS/espressif__esp-dsp" ]; then
-	git clone $DSP_REPO_URL "$AR_COMPS/espressif__esp-dsp"
-else
-	git -C "$AR_COMPS/espressif__esp-dsp" fetch && \
-	git -C "$AR_COMPS/espressif__esp-dsp" pull --ff-only
-fi
-if [ $? -ne 0 ]; then exit 1; fi
-
 #
 # CLONE/UPDATE TINYUSB
 #
@@ -163,3 +104,16 @@ else
 fi
 if [ $? -ne 0 ]; then exit 1; fi
 
+#
+# CLONE/UPDATE TFLITE MICRO
+#
+echo "Updating TFLite Micro..."
+if [ ! -d "$AR_COMPS/tflite-micro" ]; then
+	git clone $TFLITE_REPO_URL "$AR_COMPS/tflite-micro"
+	git -C "$AR_COMPS/tflite-micro" submodule update --init --recursive
+else
+	git -C "$AR_COMPS/tflite-micro" fetch && \
+	git -C "$AR_COMPS/tflite-micro" pull --ff-only
+	git -C "$AR_COMPS/tflite-micro" submodule update --init --recursive
+fi
+if [ $? -ne 0 ]; then exit 1; fi