Decentralized storage concepts, practices and prospects

Source: PermaDAO

Abstract

The mainstream decentralized storage platforms currently on the market include Arweave, Filecoin and Storj, which provide services that do not rely on a single The central control point data storage method contrasts with traditional cloud storage services. Arweave focuses on long-term or permanent storage and adopts a one-time payment model; Filecoin and Storj build blockchain-based storage markets to provide flexible storage solutions. Arweave utilizes an “effective data mining” mechanism to incentivize miners, while Filecoin ensures data storage through proof of replication and proof of time and space. Arweave's storage operationEasy, while Filecoin and Storj are respectively targeted at the professional market and traditional cloud storage users. The future development prospects of decentralized storage are broad, especially in the fields of data decentralization, censorship resistance and AI applications.

Decentralized storage is a data storage method that does not rely on a single central point of control. This approach contrasts with traditional centralized storage (such as traditional cloud storage services such as Amazon S3 or Google Cloud), which are usually managed by a single enterprise or organization.

Mainstream decentralized storage

The mainstream decentralized storage currently on the market include Arweave, Filecoin, and Storj. They each have unique features and design concepts:

• Arweave focuses on long-term or permanent data storage.

• Filecoin provides a decentralized market similar to traditional cloud storage and supports flexible storage needs.

• Storj is a decentralized cloud storage service that focuses on providing security and privacy protection.

These three platforms all use blockchain technology, but their application scenarios, technical implementations and payment models are different, and each is suitable for different types of storage needs. ：

1. Arweave

• Goal:Provide A long-term, permanent data storage solution. Arweave's goal is to store data until "forever" and is primarily used for long-term data preservation.

• Technology:Using a unique blockchain technology called "Blockweave". Unlike traditional blockchains, Block Fabric contains references to earlier random blocks in each new block, designed to encourage long-term retention of data.

• Payment model: Users pay a one-time fee for data storage, and the data can theoretically be accessed permanently after it is stored.

2. Filecoin

• Goal:Aims to create a decentralized storage marketplace, similar to traditional cloud storage services.

• Technology:Filecoin is the incentive layer of IPFS (Internet File System). It uses "proof of storage" and "proof of space and time" to ensure that data is stored correctly.

• Payment Model:Users pay the storage provider based on the amount of data stored and for how long. This is a more traditional rental model where users can increase or decrease storage as needed and pay accordingly.

3. Storj

• Goal:Provide users with a decentralized cloud storage solution that focuses on security and privacy protection.

• Technology: Storj uses encryption and sharding technology to protect data security and privacy. Data is encrypted and split into multiple small chunks on the client before uploading, and then distributed and stored on nodes around the world.

• Payment model: Storj’s payment model is similar to traditional cloud storage, with billing based on storage space and bandwidth used.

In contrast, Arweave is unique in that it emphasizes permanent storage and pays more attention to data censorship resistance and durability. Filecoin and Storj both use the storage market and focus on using blockchain technology to reconstruct the storage market.

Business Architecture Analysis

Arweave’s theoretical basis for permanent data storage is similar to “Moore’s Law”. According to statistics on data storage costs from 1980 to the present, storage costs are declining at a rate of 20% every year. According to this statistical law, the cost of data storage will converge to a constant after infinite years. Arweave Perpetual Storage is based on this and calculates the cost of data storage for 200 years. Users will pay this one-time fee when storing data.

At the same time, Arweave designed a very Elegant and concise data mining mechanism. We can name it “effective data mining”.

The so-called "valid data" refers to the data that has been stored in the Arweave network in the past, and users have paid 200 years of storage fees for these valid data. Another role group in the network - miners, they use valid data to mine and provide valid data reading services. The difference from other storage blockchains is that Arweave does not force miners to store data. Instead, it establishes incentive rules to encourage each miner to maximize the storage of "valid data." In the Arweave network, the more "valid data" a miner stores, the greater the "computing power" of mining.

Assuming there are 100 TB of valid data in the Arweave network, it is not necessary for miners to store all 100 TB of data. In other words, the miner can mine by storing only 100 MB of data, but the miner's computing power is very small. If a miner chooses to store all 100 TB of data, the amount of computing power he has will be at its maximum.

In the "effective data mining" mechanism, the Arweave network incentivizes miners to store the most data possible, but does not force them to store all data. So in this incentive model, is there the possibility of data loss? The following is a simulation calculation about data loss:

The 0.5 in the first and second lines refers to a single node storing 50% of the data. Assume that the block network has 200,000 blocks and 200 nodes in the network. Each node randomly stores 100,000 blocks (50% of the block data). It can be calculated based on probability that a single block cannot The probability of access is 6.223^10-61. The data reliability provided by the cloud service is 99.9999999%, which is 10 to the 7th power. The Arweave calculus above reaches an astonishing 61st power.

Filcoin and Storj both use blockchain technology to build a data storage market. The main improvement of Storj is data privacy. This article mainly explains the principles of Filecoin.

Similar to a traditional order book, users who use Filcoin need to first go to the trading market to place bids and place orders, and indicate the data storage time and backup quantity. The miners will receive profitable orders. In order to ensure the fairness of the entire trading market, Filcoin has established a complex economic model and set up various rules such as forfeiture and small installment payments. Its core technologies are replication proof and space-time proof.

• Proof of replication: The miner proves to the user that the data has been stored on a dedicated physical device. Every time a miner performs a proof to store a user's data, the network will pay the miner a fee.

• Proof of time and space: If you only have a copy of the proof, it does not protect you The data is always stored, and miners can only store this part of the data when submitting the proof. To this end, Filecoin has added space-time proof, with the purpose of allowing miners to continuously store this data.

To summarize the above, the basis and implementation plan for Arweave’s longevity are:

• The cost of permanence is decreasing year by year

• Through "effective data mining" "Mining" incentivizes miners to achieve data permanence

Filcoin and Storj are decentralized storage markets created using block technology, and their models are both Similar to the order book of a traditional trading market, the order book provides the demand and the miners accept the order to ensure data storage. The core technical points of Filcoin are: proof of replication and proof of space and time.

Storage Practice

There are two ways to store data in Arweave. The first method directly sends data to the Arweave node and pays AR. The second way is to use the ANS-104 (Bundled) data binding protocol to batch package data into Arweave.

Store data directly into Arweave
< /h3>

Users only need to prepare an AR and hold it in their wallet to complete the action. Use the following code to store a file named file.pdf into Arweave:

More document references:

https://github.com/ArweaveTeam/arweave-js

Use ANS-104 to store data in Arweave (recommended)

Arweave’s block production rate is low, usually about 2 minutes, and one block It can only process 1,000 transactions, which greatly limits the number of transactions Arweave can store. Although the storage capacity of an Arweave transaction is unlimited, users can store 100 MB or even 10 GB of data directly into Arweave through one transaction. . In order to solve the problem of expanding the number of transactions, ANS-104 came into being.

ANS-104 is a multi-transaction binding technology that can bind tens of thousands of different data entities into a common Arweave transaction at one time. It can be compared to the Ethereum to Layer2 Rollup solution. The difference is that ANS-104 does not lose the security of the data, and the bound data is also 100% complete data stored on Arweave.

The code demonstration of using ANS-104 to store data is as follows:

This code uses the arseeding light node as the data binding service. The arseeding light node is a completely open source Arweave data node that supports all Arweave native node interfaces and expands the ANS-104 interface. At the same time, arseeding integrates the cross-chain payment protocol everPay, so in addition to using AR to pay storage fees, users and developers can also use various assets such as ETH, BNB, USDT, and USDC for data persistence.

More document references:

https://web3infra.dev/docs/Arseeding/guide/quickStart

Storage charges

Currently storage 1 GB of data on Arweave is $7.5.

The latest storage fee reference:https://ar-fees.arweave.dev/

Retrieve and download Arweave's Data

Arweave has a standardized GraphQL service interface, and any individual or organization can implement Arweave index according to the standard. The following are two typical and useful index gateways:

• ArweaveNet gateway has the most comprehensive index.

Link: https://arweave.net/graphql

• KNN3 gateway , real-time retrieval of arseeding node data, fast.

Link: https://knn3-gateway.knn3‍.xyz/ars‍eeding/graphql

To download Arweave data, you only need to know the ARID of the data or ItemID is enough, code example:

Filcoin storage method

strong>

Unfortunately, Filcoin does not provide storage tools for ordinary users and developers. For ordinary developers, Filcoin is unavailable state. From sporadic technical documents, you can find some solutions for Filecoin storage through third-party service providers. However, if you look carefully at the service provider's documents, most service providers only provide IPFS storage, and these service providers do not necessarily store data in Filcoin. Perhaps due to the limited level of the author, I really can't find a better way to store data in Filcoin, and there is no corresponding interface to directly obtain data from Filecoin.

Storj’s storage method

Storj’s storage method is the same as Web2, developers You need to register on the official website and obtain API-KEY. Storj's storage is compatible with the AWS S3 interface, so I won't go into details here. Storj’s storage fees are low, with 1 GB of storage costing just $0.004 for 1 month. However, the storage cost converted to 200 years will be slightly higher than Arweave, at $9.6.

It can be seen from the actual storage operation that Arweave's transaction processing model is consistent with blockchains such as Bitcoin/Ethereum. Filecoin does not provide usable SDK and interfaces. Unfortunately, the so-called storage leader is not available to developers, which is sad. Storj's storage method is exactly the same as Web2.

It is worth noting that Arweave is a native blockchain storage. Once the data is sent to Arweave, it cannot be deleted or tampered. Filcoin and Storj are leasing models. The project party can stop the storage leasing service at any time. In this mode, the data does not have blockchain characteristics, and the data characteristics are consistent with those stored in centralized cloud services.

In order to more clearly distinguish the difference between data storage such as Arweave and Filcoin, we can name the data on Arweave as "Consensus Data", whether it is data on BTC or Ethereum , all belong to consensus data, and these data are non-tamperable and traceable. The data stored in the Filecoin storage rental market cannot be called consensus data.

Development Prospects

Decentralized storage has two completely different business lines. Among them, the business line represented by Arweavetakes consensus data as its core and emphasizes data decentralization, censorship resistance, traceability and other characteristics. The business line represented by Filecoin is centered on the decentralized market and emphasizes allocating storage resources and proving storage success. Analogous to the development of DeFi, the early IDEX used blockchain technology to create an order book market. The order book is a very traditional business model, aiming to solve the problem of ticket exchange using the hang-up and take-order model. The explosion of DeFi is the liquidity mining technology brought by Uniswap AMM trading model. AMM allows orders to be fully automated and operates, realizing the combination of liquidity, and finally ushered in the explosion of DeFi Summer. In the current decentralized storage track, Filecoin also represents the blockchain technology that creates the order book market, while Arweave uses a unified model similar to AMM to manage data supply and demand. Arweave's unified model is more convenient for data pricing and processing. Using Arweave can more conveniently complete the transformation of ordinary data into consensus data. Data based on this consensus may usher in an explosion of "data composition".

At the same time, we have to mention the SCP theory (storage-based consensus paradigm). Its core idea is that as long as the data storage has consensus, then the applications composed of these data can also form a consensus. SCP emphasizes off-chain computing. Data can be stored on various chains such as BTC and Ethereum, and a unique state is formed by aggregating data on the blockchain. Since these states will produce the same results when run on any computing unit, why do we still need to operate on them on the chain? Wasting so much computing resources?

The currently popular BRC20 and Bitcoin Inscription both use the consensus of off-chain computing. The storage consensus emphasized by the BRC20 protocol and Arweave SCP is consistent. They both provide immutable and traceable transaction data through the blockchain as the data layer, and the calculation of the state is completely performed off-chain. With Arweave’s storage capabilities, SCP Theory can obtain a more robust consensus data set. The Arweave SCP theory has developed a complete set of engineering application solutions - Permaweb, which is equivalent to the ultimate version of the Bitcoin indexer. Permaweb can not only process assets, but also text, images and even videos. Imagine a near future where super-powerful indexers can stream media, creating a completely decentralized Douyin.

At present, the Permaweb solution supports a wide range of application types. Whether it is network disk, content co-creation, or games, it can be easily developed using this architecture. Data between Permaweb applications can be combined with each other. For example, a writer uploads the text and copyright of his creation to Arweave through content co-creation. In another game, the developer can directly quote the writer's content and allow players to pay the author for copyright.

The biggest difficulty currently encountered by DePIN is the performance of the blockchain. DePIN devices will enter thousands of households, but no blockchain can carry such huge user interactions. Most DePINs still use a centralized approach to process data, which will make DePINs lose their decentralized nature. Consensus data can bring more powerful empowerment to DePIN. Once DePIN data is made permanent, these data will also gain combinatorial characteristics. For example, a green energy certificate can offset energy consumption during blockchain PoW calculations, can become a logo in content creation, and can also become a badge in games. Data and value will flow everywhere.

Consensus data is also applicable to the field of AI artificial intelligence. Human knowledge and history should last forever, and consensus data can ensure that AI cannot pollute or tamper with human knowledge and history. Similarly, consensus data can be used as the best data raw material for AI, allowing AI to learn and process a variety of effective information.