Saving 💰 by Utilizing Free and Open Source World Maps

At the end of September, the Helium Foundation changed our map tile server to Protomaps. As a globally distributed network of physical infrastructure, maps are a critical piece of the interpretation and visualization of the network. Helium runs many services leveraging maps, including Explorer, which serves hundreds of thousands of users monthly. Through this refactor, we are on track to save several thousands of dollars a month.

Given how important maps are to us and the community, we are thrilled to be able to save this much money and want to share some insights we learned while executing this migration. Hopefully, these insights can help other developers who are thinking of taking the leap. I'd recommend a developer use this as an addendum to the existing protomaps setup documentation.

What Is Protomaps?

Protomaps is an open-source self-hosted way to serve vector tiles. The tiles are built with using OpenStreetMaps and stored in the pmtile format. This is "a single-file archive format for pyramids of tiled data." PMTiles readers can then use HTTP range-requests to get the subset of data it needs.

All you need to do in order to use protomaps:

• Upload a single planet-scale (or regional) map to your CDN.
• Use one of the multiple open-source front-end libraries to render the map itself.

Uploading Tips

You can upload your PMTiles to any S3-compatible cloud-storage platform that supports HTTP range-requests. You can find a long list of possible cloud platforms from protomaps. While those are all possible options, Cloudflare is the recommended storage platform since it has no bandwidth fees. Ultimately, that is what we did.

While Protomaps strongly recommends Cloudflare as a storage platform, it is much looser in its recommendations on how to upload the map file. It lists out the web interface (for small files), the pmtiles CLI, rclone and finally the AWS CLI. I'd recommend skipping straight to the AWS CLI. While the PMTiles CLI and rclone work great for small files, it seems that multi-part uploads currently do not work. Despite trying multiple different upload configurations (e.g. chunk size, maximum concurrency) these uploads would invariably fail with a SignatureDoesNotMatch error. Ultimately, for large files (the planet-size file is 110GB), the AWS CLI is the only option that works.

To make the AWS CLI work all you need to do is:

1. Configure an AWS profile with the Access Key ID and Secret Key that the Cloudflare R2 API provides you.
2. Upload your desired file to your R2 bucket

   aws s3 cp FILE.pmtiles s3://R2_BUCKET_NAME/ --endpoint-url https://CLOUDFLARE_ACCOUNT_ID.r2.cloudflarestorage.com --profile cloudflare

R2 Worker And CORS Setup

After uploading your pmtiles files to Cloudflare R2, you'll need to create a Worker to expose said file to the public. Protomaps provides instructions on how to do this with both the web console and through Wrangler. At the time of writing this, the instructions for the web console no longer match the UI on Cloudflare. For this reason, and for further CORS setup, I'd recommend setting up your Worker with Wrangler, per the instructions.

Your wrangler.toml already comes with an ALLOWED_ORIGINS var, which you can set to enable CORS for your front-end. As currently set up, however, ALLOWED_ORIGINS only works on exact string matches. If you are like the Helium Foundation and would also like to enable CORS for your preview deployments, then you can edit the PMTiles/serverless/cloudflare/src/index.ts file with the following code snippet.

if (typeof env.ALLOWED_ORIGINS !== 'undefined') {
  const previewRegex = /YOUR_REGEX/g
  const requestOrigin = request.headers.get('Origin') || ''

  for (const origin of env.ALLOWED_ORIGINS.split(',')) {
    if (origin === requestOrigin || origin === '*') {
      allowed_origin = origin
    }
  }

  if (!!requestOrigin.match(previewRegex)) {
    allowed_origin = requestOrigin
  }
}

Maplibre-libre-js

Since we already used Mapbox to render our front-end, we opted to use maplibre-gl-js, an open-source fork of mapbox-gl-js. For the most part, migrating from one to the other is as straightforward as installing the dependencies and adding the pmtiles protocol. There were however two minor gotcha’s.

Watch Your Urls

This one is more an indictment of myself versus the documentation itself, but be extra careful of the URLs you use when adding the sources. If you’re like me and looking at the documentation which includes the two following code samples:

{
  "sources": {
    "protomaps": {
        "type": "vector",
        "tiles": ["https://mycdn.com/tileset/{z}/{x}/{y}.mvt"],
    }
  }
}
—--
{
  "sources": {
    "protomaps": {
        "type": "vector",
        "url": "pmtiles://https://example.com/example.pmtiles",
    }
  }
}

You might think that when serving the tileset locally, that the following is correct.

"tiles": [http://127.0.0.1:8080/world.pmtiles/{z}/{x}/{y}.mvt]

It however, most definitely is not. Instead, if you are serving files locally the following two formats work.

    sources: {
      protomaps: {
        type: "vector",
        tiles: [
          "pmtiles://http://127.0.0.1:8080/world.pmtiles/{z}/{x}/{y}.mvt",
        ],
      },
    },
—--
    sources: {
      protomaps: {
        type: "vector",
        url: "pmtiles://http://127.0.0.1:8080/world.pmtiles",
      },
    },

Update Your Styling

Mapbox and Protomaps use two different sets of vector basemap layers. As such, if you are trying to migrate from Mapbox to Protomaps, your existing Mapbox styles will not render anything. I'd recommend you use or adapt one of the existing protomaps-theme-base layers as shown in the docs. Alternatively, feel free to use our modified layers (note that we have removed some layers that were not of interest to us).

Conclusion

While we ran into a hiccup or two while completing the migration, we couldn’t be happier with the end result of our migration to Protomaps. Since our migration, we have experienced equal or better performance for a fraction of the cost. We really appreciate the efforts of @bdon for this really amazing solo open source project. If you start to use protomaps, we encourage you to support his work on Github!