Private Data on the Open Web

This post is an introduction to the Lit Protocol x Ceramic Network Integration that allows developers to store encrypted information in a decentralized datastore (Ceramic Network) where access is limited to those that the developer or user allows via Lit Protocol's decentralized access control.

Today, in order to have an encrypted application, like Signal, you must know the intended recipient in order to send an End to End Encrypted (“E2EE”) message.

This presents an issue when attempting to have private communication with a group that’s dynamically changing, such as members of DAO or holders of a given NFT. Lit Protocol, a decentralized access control network solves this problem by granting decryption keys to users based on their blockchain credentials and asset ownership.

Furthermore, creating applications that are running on decentralized storage requires update-able and ever-changing application data. The solution to this dynamic data problem is solved by Ceramic, a permissionless data streaming network.

With the integration between Lit Protocol and Ceramic Network, private, dynamic applications with a decentralized backend are now possible.

The Processes:

As mentioned, Ceramic allows for updatable data to be stored on top of IPFS and Filecoin or Arweave. By default, this data is public. However, say we wanted to build something that not everyone could access, at least being able to select who we wanted to see it. This is where our integration of Lit Protocol and Ceramic comes into play.

Using the integration you can save data on the Ceramic Network that can only be accessed by those that meet the access control conditions, like any addresses in a list controlled via smart contract or NFT holders, that the creator sets upon initialization. For example, the team at orbis.club is using the Lit Protocol x Ceramic network integration for decentralized, encrypted messaging and gated group chats.

Additionally, this integration sets the stage for individuals to have self-sovereign data ‘hubs’. When people can own their data securely in decentralized data stores, then grant and revoke consent to decrypt that data via on-blockchain credentials.

How It Works:

Two basic processes are handled in this integration: writing encrypted data, and reading encrypted data.Each uses both Lit Protocol and Ceramic.

First, data is encrypted client side (using Lit Protocol) and then written to a stream on Ceramic Network. Second, the stream is fetched with its unique StreamID from Ceramic, checked against access control parameters using Lit, and decrypted. We’ll go through each of these processes and learn a little bit about both Lit Protocol and Ceramic in the process.

First, Initializing Clients:

Both Lit Protocol and Ceramic require you to connect with nodes from their respective networks before you interact with them. Fortunately, the Lit Protocol client connection is started via one line of code (startLitClient method) and the Ceramic node connects via the DID (Decentralized Identity) authentication process that is handled by the wallet provider (when you connect your metamask or other wallet, this happens behind the scenes, connecting the user to Ceramic). Once this has happened both of the below methods are usable.

Encrypt and Write:

Encrypting using Lit Protocol requires access control conditions to be set before encryption. At this point, the integration uses the wallet of the initiating user, such that only one with access to that wallet can decrypt any data. In future, this will expand to having further options.

Once that is complete, whatever data the user wants to encrypt is encrypted, it is packaged into an encrypted zip file with the data and accompanied by a symmetric key. Both of these (which are encoded into base 64 for easier storage in Ceramic), along with the access control conditions (stored as plaintext) are required to decrypt the data; so all three of these are written to Ceramic. The Ceramic node returns a StreamID which identifies the data on the network. If a client queried this StreamID they would retrieve the access control conditions along with the encrypted and encoded zip and symmetric key, so the data is obfuscated and safe.

Read and Decrypt:

The first step in retrieving any data that’s been encrypted and written as above is to get the StreamID. Using just the StreamID, this integration will pull the access control conditions ("ACC") and the encrypted Zip and Symmetric Key. The integration, using the ACC and Lit Protocol functionality, then decodes and decrypts the Zip and returns the data within it as a string.

Now What?

Start building apps for the new internet! You can find the code in our git repo here. There are instructions on how to install and use it as well as an example web app here .

Here are some ideas for applications that can be built:

• Setup a private proposals and project management for DAOs
• Create a self-sovereign data hub and social graph
• Build a decentralized and private data marketplace for consumers to monetize personal data

And of course, it's packaged up and it’s now available as an NPM here.

There are so many opportunities for developers to build on top of both networks using this utility, and expect it to become a useful standard for keeping one’s data under one’s own control as web3 continues to build out.

If you've got questions, ideas, or suggestions, get in touch in discord or here.