IPFS gateway

IPFS is a gateway to the InterPlanetary File System. It lets you fetch and display content from IPFS in your app. IPFS is a peer-to-peer network and protocol designed to make the web faster, safer, and more open. It is a decentralized alternative to the HTTP protocol. You can read more about IPFS here.

Overview

At Universal Page, we use IPFS to store and access metadata, media, and other assets. For storing content on IPFS there are several options for different use cases, such as NFT Storage for NFTs, Pinata, and Infura for general use cases. When it comes to fetching content, each of third-party service providers has its own gateway. A gateway is used privately by an app to access content from IPFS.

How it works

The gateway is a private endpoint that is used to access content from IPFS. When the app requests some content from IPFS, the request is sent to the gateway. The gateway attempts to fetch the content from a Content Delivery Network (CDN) cache. If the content is not found in the cache, the gateway communicates to a IPFS node to fetch the content from the network. Once the content is fetched, it is stored in the CDN cache and returned to the app.

The most expensive part of the process is fetching the content from the IPFS network. Third-party service providers charge based on the amount of data transferred, the number of requests made, and other variations. While it may be sufficient for most use cases, it can be expensive for apps that require a lot of data to be fetched from IPFS and displayed on the page.

Caching

Third-party service providers use a CDN cache to reduce the cost of fetching content from IPFS. It makes it possible to serve content from the cache instead of fetching it from the IPFS network. However, there are several limitations we at Universal Page have encountered when using third-party service providers:

  • If the cache is not used for a short time, it is cleared and the content is removed from the cache. This means that the content will be fetched from the IPFS network again.
  • Different service providers have different node clusters and internal policies for fetching the content. This means that the same content stored on one service provider may not be available on another service provider.

Universal Page gateway

At Universal Page, we must ensure our customers access the content from IPFS as fast as possible. We have built our own gateway that is optimized for our use cases. The gateway is only used to fetch content from IPFS. It does not store any new content. The content is fetched from IPFS and stored in the CDN cache. The cache is used to serve the content to the app. Our gateway is perfomant and reliable because we leverage edge computing and storage. Our gateway is deployed to multiple locations around the world. This allows us to serve content from the closest location to a user.

We rely on multiple IPFS providers to balance the load and ensure the content is fetched as fast as possible. When content is fetched, we store it in our own CDN cache with a long expiration time. We may still prune the cache if the content is not accessed for a long time. The content is still available on IPFS and will be fetched again if it is requested.

Usage

You must create an account to use the gateway. Once you have an account, you can create a new project to get a project id and a gateway token that you can use to access the gateway.

Example of fetching from IPFS

const projectId = '...'
const gatewayToken = '...'
const pathOrCid = '...'

const response = await fetch(
  `https://api.universal.page/${projectId}/ipfs/${pathOrCid}`,
  {
    headers: {
      'X-UniversalPage-Gateway-Token': gatewayToken,
    },
  },
)
if (!response.ok) {
  throw new Error('unexpected response ' + response.statusText)
}
const content = Buffer.from(await response.arrayBuffer())

Vercel/Next.js

If you are using Vercel and Next.js you can use edge computing to proxy requests to the private gateway. This does not require your app to proxy content's data, instead return a redirect to the gateway. Using proxy is preferred to avoid leaking a gateway token and prevent abuse of the gateway.

Example of a proxy to IPFS gateway

import type { NextRequest } from 'next/server'

// define on server only
const projectId = '...'
const gatewayToken = '...'

// deploy on edge
export const config = {
  runtime: 'edge',
}

const handler = async (req: NextRequest) => {
  // require only get requests
  if (req.method !== 'GET') {
    return new Response(JSON.stringify({ error: 'unsupported method' }), {
      status: 405,
      headers: {
        'Content-Type': 'application/json',
      },
    })
  }

  // resolve path or cid requested. Function is src/pages/api/ipfs/[...path].ts
  // e.g. /api/ipfs/QmNeJgUiSgAVAUSeuYJvABXLdAN9Q6HE2fqLLXBtncHFyf
  const url = new URL(req.url)
  const pathOrCid = url.pathname.replace('/api/ipfs/', '')

  // proxy to IPFS gateway and return redirect
  return fetch(`https://api.universal.page/${projectId}/ipfs/${pathOrCid}`, {
    headers: {
      'X-UniversalPage-Gateway-Token': gatewayToken,
    },
    redirect: 'manual',
  })
}

export default handler

Was this page helpful?