The last beta deploy did not go well. Parsing of LongFi packets was broken in concentrate so the selected hotspots were not able to engage in Proof of Coverage with each other as planned.
Rolling back the hotspot firmware update resulted in a hard reboot of the beta hotspots. This hard
reboot resulted in minor file system corruption which triggered reformatting of the /var partition
mounted on /var because of an incorrect return code check from fsck.
WiFi credentials were lost so hotspots that were connected to the internet over WiFi fell offline.
Older pre-production units also lost their swarm_keys which reside on the SD card rather than the
hardware security module in production units.
Since the /var partition was lost on the affected beta hotspots, the blockchain cache was also
lost resulting in any production hotspots in the beta group having to resync the blockchain over
about 24 hours.
To remedy the inadvertent reformatting of hotspots persistent file system we have fast-tracked a PR to correct the behavior of the file system check and repair script that runs on start up.
To minimize risk all the recent LongFi related commits were reverted from the master branch of the hotspot firmware so that shut down behavior remains unchanged.
Content
The beta group will be updated with Hotspot firmware version 2019.10.25.0 which includes:
- Revert concentrate upgrade for LongFi (not in
2019.10.21.0GA release) - Do not treat all non-zero fsck return codes as unrecoverable errors
- Revert miner upgrade for LongFi (not in
2019.10.21.0GA release) - Revert CMake upgrade (not in
2019.10.21.0GA release)
Deployment Plan
We plan to let this beta run for at least a couple of hours. If that proves to be stable we will tag
the branch as 2019.10.25.0 and confirm that the GA release OTA updates successfully before making
it available to all hotspots.