How to store data using web3.storage

TODO: should this be "how to store data with w3up" instead?

In this how-to guide, you'll learn how to store data programmatically for your development projects using the web3.storage client libraries in JavaScript. This includes making your data available on the decentralized IPFS network with persistent long-term storage provided by Filecoin.

If you just want to quickly store a few files using web3.storage rather than include upload functionality in an app or service you're building, you may want to skip this guide for now and simply use the Files page on the web3.storage site.

For developers, web3.storage provides a simple interface for storing data using syntax inspired by familiar web APIs such as fetch and File. This guide focuses on JavaScript client library, which is the simplest way to use web3.storage programmatically.

TODO: link to v1 docs for non-js users

Uploading data to web3.storage using a client library requires a UCAN identity key that's registered to a web3.storage account. If you already have an account and a UCAN identity, read on. If not, have a look at the quickstart guide to get up and running in just a few minutes.

Installing the client

In your JavaScript project, add the w3up-client package to your dependencies:

npm install w3p-client

Creating a client instance

TODO: Client construction example, showing how to init client with your UCAN key. Maybe include JS registration flow also? Might be enough to have it in the quickstart guide and link to it.

Preparing files for upload

TODO: figure out if this is still needed... the current w3up-client only uploads CARs, so we'd need to encode to CAR. Might change before publication tho.

Uploading to web3.storage

Once your files are ready, uploading is a simple method call on the client object.

TODO: upload code snippet

Showing progress to the user

TODO: Do we have a method for this in v2 API?

Directory wrapping

By default, files uploaded to web3.storage will be wrapped in an IPFS directory listing. This preserves the original filename and makes links more human-friendly than CID strings, which look like random gibberish.

The CID you get back from the client when uploading is the CID of the directory, not the file itself! To link to the file itself using an IPFS URI, just add the filename to the CID, separated by a / like this: ipfs://<cid>/<filename>.

To make a gateway link, use https://<cid>.ipfs.<gateway-host>/<filename> or https://<gateway-host>/ipfs/<cid>/<filename>, where <gateway-host> is the address of an HTTP gateway like dweb.link.

Once uploaded, you can retrieve the directory or list the contents without downloading it.

To avoid having your files wrapped in a directory listing, set the wrapWithDirectory: option to false when uploading using the JavaScript client.

Storing IPFS Content Archives

TODO: revise to use upload instead of put

So far we've focused on using the put method, which accepts regular files and packs them into an IPFS Content Archive (CAR) file before uploading to web3.storage. If you're already using IPFS in your application, or if you want more control over the IPLD graph used to structure your data, you can construct your own CAR files and upload them directly.

See Working with CAR files for more information about Content Archives, including how to create and manipulate them with code or command-line tools.

Once you have a Content Archive, you can use the putCar client method to upload it to web3.storage.

The putCar method accepts a CarReader, which is a type defined by the @ipld/car package.

You can create a CarReader from a Uint8Array using the fromBytes static method:

import { CarReader } from '@ipld/car';

// assume loadCarData returns the contents of a CAR file as a Uint8Array
const carBytes = await loadCarData();
const reader = await CarReader.fromBytes(carBytes);

const client = makeStorageClient();
const cid = await client.putCar(reader);
console.log('Uploaded CAR file to web3.storage! CID:', cid);

See the putCar reference documentation for more information about putCar, including optional parameters.

The Working with CAR files guide has more information about the @ipld/car package, including how to implement the loadCarData function and other ways to construct a CarReader.

Next steps

The client returns an IPFS content identifier (CID) that can be used to fetch your files over IPFS. Once uploaded, your data is immediately available for retrieval via IPFS and will be stored with Filecoin storage providers within 48 hours. To learn how to fetch your data using the web3.storage client, or directly from IPFS using a gateway or the IPFS command line, see the how-to guide on retrieval.

TODO: make sure all links resolve to v2 content