Getting Started With Arweave

An in-depth research on Arweave for Eden Protocol

This DevRel post arose from my research on Arweave for the Eden Protocol. Eden Protocol is a Developer DAO project centered on talent management. We're developing artificial intelligence to match the right people to the right opportunities at the right time.

BluePanda, CTO of Eden, asked me to research Arweave for permanently storing endorsement data on a blockchain. This data will be generated as a person is endorsed based on their performance on a project, so long-term storage and immutability are essential. Arweave was a natural fit because it met all the requirements for implementing this feature in Eden. The astonishing thing was how simple it was to put this into action. Thanks to Arweave's clear documentation. In this post, I will share the information I gleaned from that research.

D_D Newsletter CTA

Introduction to Arweave

Arweave is a decentralized network that allows you to store data and applications permanently. Arweave implements a data structure that it refers to as the blockweave. Each block is connected to a previous block (as in conventional blockchain) and a random block from the Arweave network history, the recall block.

The recall block is chosen based on a hash of the previous block's height and the previous block's hash. This results in a deterministic but unpredictable block selection from the block weave history. Arweave incentivizes miners through proof of access, which means that for a miner to mine a new block, that block's recall block must be present. Learn more about this on Arweave yellow paper.

The Arweave network uses the AR token as its primary unit and the Winston as its secondary unit. You pay with this token when you want to upload data to the network; the network then distributes it to the nodes that store your data.

1 AR = 1,000,000,000,000 Winston

Creating an Arweave Wallet

You need a wallet with AR tokens before uploading data to the Arweave network. Follow the following steps to create a wallet:

  • Open up arweave.app in your browser tab.
  • Once loaded, disconnect your computer from the internet.
  • Click the [ + ] button in the bottom left.
  • Click the large "Create new wallet" button on the screen that pops up.
  • Write down your passphrase on a piece of paper.
  • When the wallet generation step is complete, click the "Click to proceed" button.
  • Identify your newly created wallet on the next screen and click the "Download" button to download the key file. -Make copies of this file on multiple offline storage mediums.
  • Click the "Delete" button to remove your newly created wallet from the browser.
//content of the wallet key JSON
kty=RSA
n=""
e=""
d=""
q=""
dp=""
qi=""

You should keep the wallet key secret; you need it to perform operations on the Arweave network. The wallet has an address that you can share publicly. To get AR tokens in your wallet, visit the faucet.

Arweave Transactions

You can use a transaction to upload data to the Arweave network and transfer AR tokens between wallets. You pay for a transaction in AR tokens; the size of the data component (uploaded file) determines the fee paid for the transactions. A transaction in Arweave consists of three steps:

  • creating a transaction
  • signing the transaction using your wallet key (JSON WEB KEY)
  • submitting the transaction to the network

Making changes to previously signed transaction data invalidates the signature and causes the network to reject it.

We can create a transaction object and send it via HTTP to Nodes and the Arweave gateways through POST requests. A valid transaction may have the following format:

{
  "format": 2,
  "id": "BNttzDav3jHVnNiV7nYbQv-GY0HQ-4XXsdkE5K9ylHQ",
  "last_tx": "jUcuEDZQy2fC6T3fHnGfYsw0D0Zl4NfuaXfwBOLiQtA",
  "owner": "posmE...psEok",
  "tags": [],
  "target":"  ",
  "quantity": "0",
  "data_root": "PGh0b...RtbD4",
  "data":"  ",
  "data_size": "1234235",
  "reward": "124145681682",
  "signature": "HZRG_...jRGB-M"
}

Read more on what these fields mean and how they are generated here.

The simplest way to send a transaction to Arweave is to use the Arweave JavaScript library created by the Arweave team. We can install it on our project using NPM:

$ npm install --save arweave

Creating a Simple File Upload Express App

We'll make a simple Express application that allows us to upload a file to Arweave.

Arweave_tut_diagram.png

You can clone this repository to get started.

Installing Project Dependencies

The project consists of a client folder which contains a React application, and a server folder, which holds the Express application. We will install our dependencies into the root, client, and server folders.

The root folder dependencies are as follows:

  • concurrently: to run two or more commands at the same time
  • dotenv: for saving environment variables
  • nodemon: re-running the application after making changes to the code

At the root project, run this command:

$ npm i

The dependencies of the client:

  • axios: make HTTP requests to the server

Move into the client folder and run:

$ npm i

The server folder needs these dependencies:

  • arweave: arweave js library for creating transactions
  • formidable: package for uploading files to the server

Move into the server folder and run:

$ npm i

After installing all dependencies, we are ready to run the application.

Run the application by typing this command at the project root:

$ npm run dev`

This command starts up the server and the client of the application.

The server runs on port 4000 while the client runs on port 3000.

The server has a single endpoint it listens to: /api/uploadFile.

app.post("/api/uploadFile", uploadFile);

Managing the File Uploads to Arweave

Open the server/src/utils/uploadFileToArweave.js file. At the top of the file, we imported the following packages:

 import Arweave from "arweave";
 import * as dotenv from "dotenv";
 import fs from "fs";

We import the Arweave object from the arweave.js library and use it for initialization, where we set the necessary parameters.

const arweave = Arweave.init({
  host: "arweave.net",
  port: 443,
  protocol: "https",
});

The host/gateway we connect to is arweave.net on a port of 443; we connect via the HTTPS protocol. Remember that the Arweave JS library makes creating and submitting transactions to the Arweave network easier.

Also, at the top of the file, we created our signing key, which we named ARWEAVE_KEY. This is the JSON WEB KEY, representing our wallet's private key. We should keep our wallet key secret; that's why it is placed inside a .env file.

Create a .env file at the root of the project folder and copy and paste your wallet key inside the newly created .env file.

env-2.png

const { kty, n, e, d, p, q, dp, dq, qi } = process.env;
const ARWEAVE_KEY = { kty, n, e, d, p, q, dp, dq, qi };

Implementing getWalletAddress

async function getWalletAddress(key) {
  const walletAddress = await arweave.wallets.jwkToAddress(key);
  console.log("wallet address is ", walletAddress);
  return walletAddress;
}

We use this function to extract the wallet address from the wallet's private key. It is an async function that receives the wallet key as input.

Implementing getWalletBalance

 async function getWalletBalance(address) {
    const balance = await arweave.wallets.getBalance(address);
    let winston = balance;
    let ar = arweave.ar.winstonToAr(balance);
    console.log(winston);
    console.log(ar);
    return winston;
}

This function retrieves the balance of a wallet. It retrieves the balance in Winston (smallest Arweave unit). We can also convert the retrieved balance to AR and vice versa.

Remember: 1 AR = 1,000,000,000,000 Winston

Implementing uploadFileToArweave

  const uploadFileToArweave = async (filePath, mimetype) => {
  try {
    //check if we have a balance in wallet
    const walletAddress = await getWalletAddress(ARWEAVE_KEY);
    const balance = await getWalletBalance(walletAddress);

    if (balance < 5000) {
      //contact admin to top up the balance
      console.log("The balance is getting low");
    }
    //create a transaction:
    let data = fs.readFileSync(filePath);
    let transaction = await arweave.createTransaction(
      { data: data },
      ARWEAVE_KEY
    );
    transaction.addTag("Content-Type", mimetype);
    //sign the transaction

    await arweave.transactions.sign(transaction, ARWEAVE_KEY);

    let uploader = await arweave.transactions.getUploader(transaction);

    while (!uploader.isComplete) {
      await uploader.uploadChunk();
      console.log(
        `${uploader.pctComplete}% complete, ${uploader.uploadedChunks}/${uploader.totalChunks}`
      );
    }
    //we Successfully uploaded the file if we get here
    console.log("transaction ID", transaction.id);
    return transaction.id;
  } catch (error) {
    console.log("error uploading file", error);
  }
};

The uploadFileToArweave function takes as a parameter the path of the file we are uploading and the file mime type. We have to specify the mime type used to form the Content-Type tag when creating the transaction because if no Content-Type is supplied, the default is text/html.

Inside the function, a call is made to the utility functions to get the balance left in the wallet. We do this because the transaction will only succeed if we have enough funds.

    const walletAddress = await getWalletAddress(ARWEAVE_KEY);
    const balance = await getWalletBalance(walletAddress);

Next, we read the file using the fs module, passing the file path to the readFileSync method. This method is blocking, and the code waits until the file has been read into memory.

let data = fs.readFileSync(filePath);

After reading the file, we create a transaction, passing the file data and our wallet private key, which we named ARWEAVE_KEY, to the createTransaction method. 

let transaction = await arweave.createTransaction(
      { data: data },
      ARWEAVE_KEY
    );

We can add tags to the created transactions. Tags are metadata that gives additional meaning to uploaded data or, in the case of token transfer, can be used to embed information about the transfer. Tags are key-value pairs that are added to a transaction. Tags can be queried using GraphQL

We added an official tag called Content-type and set the value to the passed mimetype variable. This helps for the proper rendering of the file when retrieved. 

 transaction.addTag("Content-Type", mimetype);
//you can add tags as deem fit, to define the uploaded data

We have successfully created a transaction which is the first step; next, we need to sign the transaction using our private wallet key.

  await arweave.transactions.sign(transaction, ARWEAVE_KEY);

After signing the transaction, we could save the transaction object in the database if the transaction fails and we need to re-upload the file. Instead of restarting the upload from the beginning, it picks up where it left off. In this case, we will have to pass the same data we uploaded, but it will resume where it left off.

 let uploader = await arweave.transactions.getUploader(transaction);

The code above submits the transaction to the network, which, when accepted, creates an uploader object that we will use to upload the file in chunks. 

 while (!uploader.isComplete) {
      await uploader.uploadChunk();
      console.log(
        `${uploader.pctComplete}% complete, ${uploader.uploadedChunks}/${uploader.totalChunks}`
      );
 }

A loop checks if the uploader object has finished uploading the file. The uploader object uploads a chunk of data using the uploadChunk method. We use console.log to give feedback on the portion of the file uploaded and what is left to upload.

If the upload completes without error, the transaction id is returned from the uploadFileToArweave function.

File Upload to Server

Before we could upload the file to the Arweave network, we need a way to manage the file on the server side. That is where the package formidable comes in. It acts as a middleware in our Express server; it manages files uploaded to the server from the client. Open the file server/src/middleware/uploadFile.js. Inside this file, you will see a function named uploadFile.

import formidable from "formidable";
import { uploadFileToArweave } from "../utils/uploadFileToArweave.js";

const MAX_SIZE = 600 * 1024 * 1024;

const uploadFile = (req, res, next) => {
  const form = new formidable.IncomingForm();
  form.maxFileSize = MAX_SIZE;
  form.parse(req, async function (err, fields, files) {
    if (err) throw new Error("File upload failed");
    try {

      var filePath = files.fileUploader.filepath;
      const mimeType = files.fileUploader.mimetype;
      console.log("filepath", filePath);
      console.log("mimeType", mimeType);
      //upload file to arweave here 🗄
      const transactionID = await uploadFileToArweave(filePath, mimeType);
      return res.send(transactionID);
    } catch (error) {
      console.log(error);
      throw new Error(error);
    }
  });
};

export default uploadFile;

At the top of the file, we imported the formidable library and the function to upload the file to Arweave. The uploadFile is a middleware function, so it has access to the req(request) object from the client and the res (response) object from the server.

We initialized a new form object and set the value of the maximum file allowed to process. We set the maximum file size to 600MB via the MAX_SIZE variable.

const form = new formidable.IncomingForm();
form.maxFileSize = MAX_SIZE;

Next, we parse the form data. We are only interested in the files field, so we get the filepath and mimetype from the files. The value of the filepath and mimetype is passed to the async uploadFileToArweave function; we await and get back the id of the transaction to Arweave if the uploads succeed. The transaction ID is returned to the client from the function via the response object.

 return res.send(transactionID);

Uploading a File From the Client to the Server

A simple React application runs on the client that we will use to upload a file to the server. Open the client/frontend/src/App.js file. At the top of this file, we import useState and useRef hooks from react and also import axios to send HTTP request to our server.

import "./App.css";
import { useState, useRef } from "react";
import axios from "axios";

function App() {
  const [file, setFile] = useState(null);
  const [submitted, setSubmitted] = useState(false);
  const [transactionID, setTransactionID] = useState("");

  const fileRef = useRef();

  const onFileChange = (e) => {
    const file = e.target.files[0];
    const extention = ["jpg", "jpeg", "png", "gif", "mp4", "pdf"];
    //check for the type of image we are uploading
    //check extension
    const position = file.name.lastIndexOf(".");
    const fileExt = file.name.substr(position + 1, file.name.length);
    const index = extention.indexOf(fileExt);
    if (index == -1) {
      window.alert(`file extension not supported.`);
      fileRef.current.value = "";
      return;
    }
    //save the file here
    setFile(file);
  };

  const handleFormSubmit = async (e) => {
    e.preventDefault();
    try {
      const formData = new FormData();
      formData.append("fileUploader", file);
      setSubmitted(true);

      const res = await axios.request({
        method: "post",
        url: "http://localhost:4000/api/uploadFile",
        data: formData,
      });

      setSubmitted(false);
      fileRef.current.value = "";
      setFile(null);

      const transID = res.data;
      setTransactionID(transID);
      console.log("transaction ID 🏹 ", transID);
    } catch (error) {
      console.log("error message", error);
    }
  };

  return (
    <div className="App">
      <div className="div">
        <h1>Select a File to Upload To Arweave</h1>
        <form onSubmit={handleFormSubmit}>
          <input
            type="file"
            onChange={onFileChange}
            id="fileInput"
            ref={fileRef}
          />
          <br />
          <br />

          <button type="submit" disabled={submitted}>
            Upload File to Arweave
          </button>
        </form>

        {transactionID && (
          <div className="result">
            <p>Transaction ID : {transactionID}</p>
          </div>
        )}
      </div>
    </div>
  );
}

export default App;

We declare React state to track the selected file, transaction ID, and the form's submitted state. A reference to the file input is also declared using the useRef hook. The file input is placed inside a form element.

 <input
    type="file"
    onChange={onFileChange}
    id="fileInput"
    ref={fileRef}
   />

The function onFileChange is fired when we select a file using the input element. We check the file's extension to know if that file type is allowed (though this is not the best way to determine the file type).

    const position = file.name.lastIndexOf(".");
    const fileExt = file.name.substr(position + 1, file.name.length);
    const index = extention.indexOf(fileExt);
    if (index == -1) {
      window.alert(`file extension not supported.`);
      fileRef.current.value = "";
      return;
    }

If the extension is not allowed, we deselect the file using the fileRef and return it from the function. If the selected file type is allowed, it is saved to state using the setFile useState hook.

Implementing handleFormSubmit

The handleFormSubmit function is triggered when a user clicks the "Upload File to Arweave" button. We create a new instance of FormData that we send to the server and append the selected file. 

 const formData = new FormData();
 formData.append("fileUploader", file);

A post request is sent to our server on the routehttp://localhost:4000/api/uploadFile using axios passing in the formData. If we successfully uploaded the file, we get the transactionID from the server' displayed on the webpage.

Using the transactionID we can verify the status of our file upload. To keep things simple, we visit https://arweave.net/tx/{transactionID}, replacing transactionID with the transaction ID returned from the server. We can also render the file on the browser by navigating to https://arweave.net/{transactionID}.

D_D Newsletter CTA

Conclusion

We learned how simple it was to upload a file to the Arweave network in this blog post, which we did by using a React application combined with an Express app. We had to make a wallet and stock it with free 0.02 AR tokens, enough for us to try out Arweave. Our imagination only limits the possibilities for Arweave.

You should try Arweave the next time you need decentralized storage for your application.

Happy coding until the next time!

Last update: Jun 23, 2023