How to properly request JSON metadata stored in IPFS for your "The Graph" Subgraph

Hi, my name is JP 👋 (@J_pizzl3/@jpdev)!

I'm one of the core developers working on InterplanetaryFonts, a Web3 NFT marketplace and collaboration platform for creators and collectors of Fonts (@IPFonts/@ipfonts). Below I will describe a tricky problem we ran into when developing our subgraph on The Graph that relates to retrieving data hosted on IPFS that will hopefully help you in your Web3 journey.

Problem

Our subgraph at InterplanetaryFonts that powers our DApp on https://interplanetaryfonts.xyz/ was missing smart contract entity metadata, which is data that does not live directly in the smart contract but on IPFS as a JSON file.

This can be a frustrating and confusing problem, but with the right approach and tools, you can resolve it quickly.

Discovery

Currently, in the docs of The Graph, an API for communicating with IPFS is described here. The basic idea is that this API exposes a function called cat, which takes an IPFS CID as an argument and uses it to retrieve JSON data that describes one of our smart contract entities from IPFS.

Example from The Graph docs:

// Put this inside an event handler in the mapping
let hash = 'QmTkzDwWqPbnAh5YiV5VwcTLnGdwSNsNTn2aDxdXBFca7D'
let data = ipfs.cat(hash)

// Paths like `QmTkzDwWqPbnAh5YiV5VwcTLnGdwSNsNTn2aDxdXBFca7D/Makefile`
// that include files in directories are also supported
let path = 'QmTkzDwWqPbnAh5YiV5VwcTLnGdwSNsNTn2aDxdXBFca7D/Makefile'
let data = ipfs.cat(path)

But the docs mention this:

Note: ipfs.cat is not deterministic at the moment. If the file cannot be retrieved over the IPFS network before the request times out, it will return null. Due to this, it's always worth checking the result for null.

This behavior is because different IPFS providers have different file pinning speeds. If the provider you are using does not pin your uploaded IPFS file before your subgraph indexer runs, then the ipfs.cat function will timeout and return null.

We use web3.storage as our IPFS provider, which wasn't pinning our metadata JSON files in time for our subgraph mappings to be able to read the data using ipfs.cat. Ceci (@hyperalchemy/@hyperalchemy) pointed out this issue with web3.storage to one of our core devs at InterplanetaryFonts. It turns out that the Lens team also ran into a similar problem when they were storing and indexing their user metadata and publication content.

Searching through The Graph protocol's Discord server, we also discovered a preferred way of requesting data from IPFS during indexing called file data source templates. This method is more reliable than the ipfs.cat API function, as discussed in this Discord thread. The Graph core devs go over the file data sources in this video.

TL;DR, the file data source replaces the single inline ipfs.cat API call in your subgraph mapping logic because it could fail if your IPFS data is not ready when indexing with the creation of a file data source event. It does this by calling the create function of your file data source template class which The Graph CLI generates.

Below we will go into more detail on the solution.

Solution

To ensure the subgraph includes our IPFS-hosted metadata, we implemented the following solution:

Upload smart contract entity metadata to IPFS using an Infura IPFS node instead of web3.storage. We use Infura because they automatically pin our files, and the Lens team also recommends Infura IPFS nodes for uploading Lens profile metadata.
Create GraphQL entities in the subgraph that map to the smart contract entity IPFS hosted metadata…

type FontMetadata @entity {
    id: ID!
    name: String
    description: String
}

type FontProject @entity {
    ....
    ...
    "Font metadata data from IPFS"
    metaData: FontMetadata
}

Run The Graph CLI graph codegen command to generate all the files and classes you need for the next steps
Create an event handler function for your new entity like you would for your smart contract events, which saves your new entity.


import { json, Bytes, dataSource } from '@graphprotocol/graph-ts'
import { FontMetadata } from '../generated/schema';

// content param below is the full JSON file stored in IPFS in the form of Bytes
export function handleFontMetadata(content: Bytes): void {
  let fontMetadata = new FontMetadata(dataSource.stringParam());
  const value = json.fromBytes(content).toObject();

  if (value) {
    const name = value.get('name');
    const description = value.get('description');

    if (name) {
      fontMetadata.name = name.toString();
    }

    if (description) {
      fontMetadata.description = description.toString();
    }
  }

  fontMetadata.save();
}

Register the file data source template configuration block in your subgraph.yaml, including the handler function name, file path, and ABI config, as you would for your main smart contract data source configuration.

templates:
  - name: FontMetadata
    kind: file/ipfs
    mapping:
      apiVersion: 0.0.7
      language: wasm/assemblyscript
      file: ./src/handleFontMetadata.ts
      handler: handleFontMetadata
      entities:
        - FontMetadata
      abis:
        - name: FontProject
          file: ./abis/FontProject.json

Refactor your subgraph mappings logic to remove calls to ipfs.cat and replace them with calls to create instances of the custom file data source templates defined in subgraph.yml.

// Before 
let metadata = ipfs.cat(`${event.params.metaDataCID}/data.json`)
// ...
// ..
// .

// After
const metadataIpfsUri = `${event.params.metaDataCID}/data.json`;
newFontProjectCreated.metaData = metadataIpfsUri;

// Note we pass a single parameter to create here the URI to your IPFS-hosted JSON file, which includes CID
FontMetadataTemplate.create(metadataIpfsUri);

Update the GraphQL queries on the front end to request smart contract entity metadata through the entity metadata fields instead of directly from the main entity.

# Before
query GetFonts {
  fontProjects {
      id
      name # <-- data from IPFS JSON file
      description # <-- data from IPFS JSON file
      perCharacterMintPrice
      launchDateTime
      createdAt
      updatedAt
      fontFilesCID
  }
}

# After
query GetFonts {
  fontProjects {
      id
      metaData {
        name # <-- data from IPFS JSON file
        description # <-- data from IPFS JSON file
      }
      perCharacterMintPrice
      launchDateTime
      createdAt
      updatedAt
      fontFilesCID
  }
}

Following these steps, you can successfully fill in any missing IPFS-hosted metadata in your The Graph subgraph. Hopefully, this post helps you on your Web3 journey, and if you encounter any issues, feel free to reach out to the InterplanetaryFonts team on our Discord server 🪐🚀 Happy Building!