Basic Miner Operation
Basic Miner Operation
As you can see below, the Miner is central in routing data across the Helium Network. It is one of three pieces:
- Packet Forwarder: this is a utility that interacts with the radio front-end and sends and receives raw radio packets with the Helium Miner
- Miner: the Helium Blockchain comes into the picture here; the Miner is responsible for routing packets to the appropriate Router and entering into micro-transactions brokered via libp2p
- Router: a Helium compatible LoRaWAN Network Server, basically; this component is interested in receiving the packets relating to its devices and handles downlink messages when appropriate
In addition to packet routing, the Miner also participates in PoC and submits PoC packets as witness events. When challenge, the Miner sends packets to the packet forwarder for transmission.
The Miner connects to other Miners over libp2p where, amongst other things, it is gossiping and saving blocks, while maintaining a ledger of the blockchain. Constantly syncing the blockchain and downloading blocks uses significant bandwidth and CPU.
In this guide, we will explain how to get a Docker image of the Helium Miner running on Ubuntu 20.04 LTS, and finally some tips on how to interact with a running Miner.
When preparing the Miner for a Hotspot product, you will want to go a little further and follow the steps outlined here, where you will learn how to integrate it with security modules and BLE.
If you are interested in contributing to Miner development and code-review, please visit the Miner's repository on Github.
Setting Up Ubuntu for Docker
Ubuntu is a widely available Linux distribution. Notably, it has an easy-to-use image available for Raspberry Pi 3 and 4, so we use it as an example system. That being said, any ARM64 or AMD64 (X86-64) based OS that can run Docker is suitable.
For Raspberry Pi, we recommend running the latest 64-bit RaspiOS image. We currently do not have Docker image support for 32-bit systems, so please double-check that you're using a 64-bit image. Once you have followed the instructions and are logged into the system, you are ready for the rest of this guide.
For most cloud service providers, launching an instance with Ubuntu 20.04 LTS should be fairly straightforward. With AWS for example, create an EC2 instance running Ubuntu 20.04. A t2.small will run the miner well once the initial blockchain sync has completed. Once that's launched and you're connected, you are ready for the rest of this guide.
First, update the package manager registry:
sudo apt-get update
Then, install Docker:
sudo apt-get install docker.io
To avoid needing to user docker with sudo privileges, add your user to the docker group,
replacing $USER with your username:
sudo usermod -aG docker $USER
Log in and out of your account to apply these changes. You are now ready to use Docker!
Port Forwarding
Before launching the Miner, you will want to configure ports on your network to forward two ports:
- 44158/TCP: the Miner communicates to other Miners over this port. The networking logic knows how to get around a lack of forwarding here, but you will get better performance by forwarding the port
- 1680/UDP: the radio connects to the Miner over this port. You will not be able to forward packets or participate in Proof of Coverage without this
"Forwarding" on the second port is less relevant if you are running a radio packet forwarder on the same system at the miner (both on a Raspberry Pi, for example). But it is essential if you are running a Miner on the cloud for example.
For AWS, for example, you will want to configure the "Security Group" of your EC2 as so:
Run a Docker Container
Miner releases are available as amd64 and arm64 images on at quay.io. We do not currently provide 32-bit support.
Note: on amd64 systems, AVX support is required. Verify that it exists on your host system:
grep avx /proc/cpuinfo
grep avx2 /proc/cpuinfo
If nothing is returned from these commands, your host system does not have AVX support and your Miner may not be stable.
Start Container
Before running the container for the first time, it is advisable to pick a 'system mount point`, i.e. a directory in your host system; some long-term miner data is stored there. This allows you to easily maintain your miner's blockchain identity (i.e. swarm keys) and blockchain state through miner image updates.
If you are using a Linux system, you can just create a directory in your user's home directory:
mkdir ~/miner_data
If you are using Ubuntu as user ubuntu, this path would now be /home/ubuntu/miner_data. This will
be used later.
Now you can try the run command to start your container for the first time:
docker run -d --init \
--ulimit nofile=64000:64000 \
--env REGION_OVERRIDE=US915 \
--restart always \
--publish 1680:1680/udp \
--publish 44158:44158/tcp \
--name miner \
--mount type=bind,source=/home/ubuntu/miner_data,target=/var/data \
quay.io/team-helium/miner:miner-xxxNN_YYYY.MM.DD
Replace xxxNN with the architecture used, i.e. amd64 or arm64, and with the release date of the image.
The -d option runs in detached mode, which makes the command return or not; you may want to omit
if you have a daemon manager running the docker for you.
The -env REGION_OVERRIDE=US915 tells your miner that you are connecting to a packet forwarder
configured for the US915 region. You will want to change this to your region i.e.
US915 | EU868 | EU433 | CN470 | CN779 | AU915 | AS923 | KR920 | IN865
Note: REGION_OVERRIDE may be completely omitted once your Miner has asserted location and is fully synced, but leaving it there (as long as the region is properly configured) is not harmful
The --restart always option asks Docker to keep the image running, starting the image on boot and
restarting the image if it crashes. Depending on how you installed Docker in your system, it'll
start on boot. In the AWS AMI above, we use systemd (systemctl status docker to check).
The --publish 1680:1680/udp binds your system port 1680 to the container's port 1680, where the
Miner is hosting a packet forwarder UDP server; this is necessary if you want to do any radio
interactions with your miner.
The --name miner names the container, which makes interacting with the docker easier, but feel
free to name the container whatever you want.
The --mount with the parameters above will mount the container's /var/data/ directory to the
systems directory /home/ec2-user/miner_data.
Configure AWS Instance for Sync
Amazon EC2 instances have CPU usage credits that will easily be depleted during the initial sync of the blockchain. To avoid having your instance throttled, you can temporarily uncap your instance by setting CPU credit usage to unlimited. Once your miner has reached full block height, a t2.small instance is sufficient to keep your miner running.
Interact with the Miner within the Container
You may want to interrogate the Miner or interact with it it as described in the
using a miner section. Docker's exec command enables this e.g.
docker exec miner miner info height
In other words, prepend docker exec miner to any of the commands documented in the
using a miner section). Or create an alias such as:
alias miner="docker exec miner miner"
Updating the Docker Image
the releases here. Depending on whether you are running a miner for fun, to route packets, or to participate in PoC, keeping it updated may be more or less urgent. Each release tagged on the Github will be on the quay repository. Simply remove the current image:
docker stop miner && docker rm miner
And start the container again as described above, but with the new release tag! Thanks to the
--mount option, the blockchain data and the miner keys are preserved through updates.
Using the Miner
These commands will assume you are running in Docker and they have the same prefix to get you
executing a command within the docker: docker exec miner . If you want to make it easier, you can
always create an an alias such as: alias miner="docker exec miner miner".
Checking the Logs
This is always helpful to get some idea of what's going on:
docker exec miner tail -F /var/data/log/console.log
Also, if you are particularly interested in radio traffic, it can be helpful to filter for lora as
so:
docker exec miner tail -f /var/data/log/console.log | grep lora
Checking the P2P Network
This is the first health check you can do to see how your Miner is doing. Is it finding other Helium miners over libp2p properly?
The Helium blockchain uses an Erlang implementation of libp2p. Because we expect Hotspot hardware to be installed in a wide variety of networking environments erlang-libp2p includes a number of additions to the core specification that provides support for NAT detection, proxying and relaying.
The first order of business once the Miner is running is to see if you're connected to any peers, whether your NAT type has been correctly identified, and that you have some listen addresses:
docker exec miner miner peer book -s
You will see an output roughly like the following:
+--------------------------------------------------------+------------------------------+------------+-----------+---------+------------+
| address | name |listen_addrs|connections| nat |last_updated|
+--------------------------------------------------------+------------------------------+------------+-----------+---------+------------+
|/p2p/11dwT67atkEe1Ru6xhDqPhSXKXmNhWf3ZHxX5S4SXhcdmhw3Y1t|{ok,"genuine-steel-crocodile"}| 2 | 4 |symmetric| 3.148s |
+--------------------------------------------------------+------------------------------+------------+-----------+---------+------------+
+----------------------------------------------------------------------------------------------------------------------------+
| listen_addrs (prioritized) |
+----------------------------------------------------------------------------------------------------------------------------+
|/p2p/11apmNb8phR7WXMx8Pm65ycjVY16rjWw3PvhSeMFkviWAUu9KRD/p2p-circuit/p2p/11dwT67atkEe1Ru6xhDqPhSXKXmNhWf3ZHxX5S4SXhcdmhw3Y1t|
| /ip4/192.168.3.6/tcp/36397 |
+----------------------------------------------------------------------------------------------------------------------------+
+--------------------------+-----------------------------+---------------------------------------------------------+------------------------------+
| local | remote | p2p | name |
+--------------------------+-----------------------------+---------------------------------------------------------+------------------------------+
|/ip4/192.168.3.6/tcp/36397|/ip4/104.248.122.141/tcp/2154|/p2p/112GZJvJ4yUc7wubREyBHZ4BVYkWxQdY849LC2GGmwAnv73i5Ufy|{ok,"atomic-parchment-snail"} |
|/ip4/192.168.3.6/tcp/36397| /ip4/73.15.36.201/tcp/13984 |/p2p/112MtP4Um2UXo8FtDHeme1U5A91M6Jj3TZ3i2XTJ9vNUMawqoPVW| {ok,"fancy-glossy-rat"} |
|/ip4/192.168.3.6/tcp/36397| /ip4/24.5.52.135/tcp/41103 |/p2p/11AUHAqBatgrs2v6j3j75UQ73NyEYZoH41CdJ56P1SzeqqYjZ4o | {ok,"skinny-fuchsia-mink"} |
|/ip4/192.168.3.6/tcp/46059| /ip4/34.222.64.221/tcp/2154 |/p2p/11LBadhdCmwHFnTzCTucn6aSPieDajw4ri3kpgAoikgnEA62Dc6 | {ok,"skinny-lilac-mustang"} |
+--------------------------+-----------------------------+---------------------------------------------------------+------------------------------+
As long as you have an address listed in listen_addrs and some peers in the table at the bottom,
you're connected to the p2p network and good to go.
If you are having trouble, try forwarding port 44158 to the miner. On AWS, double check your
security group settings.
Checking Block Height
As long as a genesis block is loaded, this query will work and return height 1 at least:
docker exec miner miner info height
If you are syncing, you should see something like this:
~$ miner info height
6889 280756
The first number is the election epoch and the second number is the block height of your miner. If you just launched an AMI instance, your Miner is been disconnected, or you simply have a slow connection, you may be a few blocks behind. To check, you can either check the mobile app, check the browser-based block explorer, or run a simple curl command to check in a Terminal:
~$ curl https://api.helium.io/v1/blocks/height
{"data":{"height":280756}}
Backing Up Your Swarm Keys
Periodically, we may release updates or you may want to migrate your miner from one machine to
another. The most important thing is that you maintain a copy of your miner's private key, i.e.
swarm_key. Fun tidbit: this is also how your three-word name is generated.
Assuming you've mounted the Docker image as detailed above, it is located at:
~/miner_data/miner/swarm_key
Another fun tidbit: for production hotspots sold by Helium, the swarm key is stored inside of a secure element and is thus unable to be migrated (or compromised/accidentally lost unless physically damaged).
Snapshots
Snapshots are summaries of the current state of the blockchain. Since you don't need every single block ever to sync with the blockchain, it's really just a summary of the last few blocks and the account balance of every account on the blockchain.
You generally want to be careful who you accept snapshots from as they might give you an incorrect state. However, we also store hashes of valid snapshots in the blockchain.
Chain-based
You can query a running Miner to see what snapshots it (1) knows about and (2) has a copy of:
$ miner snapshot list
Height 731521
Hash <<129,14,18,225,133,83,34,24,205,69,10,10,84,42,129,207,42,186,18,28,192,
157,187,76,109,233,4,108,198,197,111,176>>
Have true
Height 730801
Hash <<80,90,115,7,10,195,115,217,87,173,24,11,63,116,201,22,150,6,252,168,204,
60,65,83,106,68,94,19,7,13,72,165>>
Have true
A Miner knows about a snapshot in two ways: it either syncs a block where the snapshot hash and
height have been saved, or the configuration file of the Miner (sys.config) has the blessed
height and hash written into it (ie: blessed_snapshot_block_height and
blessed_snapshot_block_hash).
If you have a fully synced Miner, you can extract these values from snapshot list and write them
into the sys.config of another non-fully synced Miner. Restart the Miner to reload the
sys.config and it will be able to "quick sync" to that snapshot. To make things go even faster,
try manually connecting your Miner to one of your Miners that has the snapshot.
miner peer connect /ip4/192.168.0.0.1/tcp/44158
The only down-side of this approach is that snapshots happen at some rhythm defined by the blockchain (currently 720 blocks). You can use the manual approach to pass around arbitrary snapshots by hand.
Manual
You can also manually move a snapshot file around. To take a snapshot:
miner snapshot take /path/to/snapshot
Naming the snapshot something like snap-XXXXXX, where XXXXXX is the height of the blockchain is
a typical convention. Also, when using a Docker container, it's usually a good idea to save the
snapshot to /var/data so that you can easily copy the snapshot out of the Docker and into the host
system.
Loading the snapshot is as follows:
miner snapshot load /path/to/snapshot
Sometimes the load command times out. If so, try to tail the logs and see if blocks are being applied. It's working just fine! It just needs a moment to finish loading the snapshot.
Connecting a Packet Forwarder to the Miner
You're almost done. The last step after you have your Miner running is to connect to it.
With your favorite editor, open global_conf.json that lives in the packet forwarder directory:
nano ~/sx1302_hal/lora_pkt_fwd/global_conf.json
You want to change the field "server_address" from "localhost" to the IP address of the server, so in this case:
You'll need to restart the packet forwarder for the configuration change to take effect.
To verify that things are working, you can follow the logs on your miner. eg:
docker exec miner tail -f /var/data/log/console.log | grep lora
At the very least, you should see PULL_DATA messages every few seconds. If so, then you've done it!