The last beta deploy did not go well. Parsing of LongFi packets was broken in concentrate so the selected hotspots were not able to engage in Proof of Coverage with each other as planned.
Rolling back the hotspot firmware update resulted in a hard reboot of the beta hotspots. This hard
reboot resulted in minor file system corruption which triggered reformatting of the
mounted on /var because of an incorrect return code check from fsck.
WiFi credentials were lost so hotspots that were connected to the internet over WiFi fell offline.
Older pre-production units also lost their
swarm_keys which reside on the SD card rather than the
hardware security module in production units.
/var partition was lost on the affected beta hotspots, the blockchain cache was also
lost resulting in any production hotspots in the beta group having to resync the blockchain over
about 24 hours.
To remedy the inadvertent reformatting of hotspots persistent file system we have fast-tracked a PR to correct the behavior of the file system check and repair script that runs on start up.
To minimize risk all the recent LongFi related commits were reverted from the master branch of the hotspot firmware so that shut down behavior remains unchanged.
The beta group will be updated with Hotspot firmware version
2019.10.25.0 which includes:
- Revert concentrate upgrade for LongFi (not in
- Do not treat all non-zero fsck return codes as unrecoverable errors
- Revert miner upgrade for LongFi (not in
- Revert CMake upgrade (not in
We plan to let this beta run for at least a couple of hours. If that proves to be stable we will tag
the branch as
2019.10.25.0 and confirm that the GA release OTA updates successfully before making
it available to all hotspots.