Technical Details AO Super Parallel Computer

Author: PermaDAO

Introduction

AO is actually an extension of the Storage-based Consensus Paradigm (SCP), which is similar to sovereign Rollup. The core idea is to decouple computing and DA/storage, publishing/storing data on the chain + computing/verifying data off the chain.

Due to Arweave's extremely high data carrying capacity, DApp platforms based on AO and SCP architectures can greatly save data publishing and storage costs, making it easier to support some scenarios that have a strong pursuit of throughput.

In most people's existing cognition, Arweave focuses on the concept of permanent storage and has long been used as a storage layer by various projects. Arweave's most well-known narrative was: to become the Library of Alexandria that never disappears, with the ultimate goal of preserving the fire of human civilization.

Therefore, after the release of Arweave's ao computer, it was very unexpected that Arweave turned into a parallel supercomputer. It should be pointed out that Arweave can still be used as a storage layer, and the ao architecture is a superposition of storage functions rather than a replacement relationship.

ao is consistent with SCP, and ao's computing processes can be executed in parallel, so it can have efficient computing capabilities. The processes of ao can be interconnected, and their data formats are consistent. All data are eventually stored in the Arweave mainnet in the Bundle format of ANS-104.

All logs of each process are fully stored, and the final holographic state of the process is retained on Arweave.

AO's computing power combined with Arweave's permanent storage capacity is actually an uncensored, ubiquitous, global, high-concurrency computer, on which DApps of any type, language, and public chain can access Arweave and enjoy decentralized computing services that are cheaper than Ethereum but more efficient.

AOS is slightly different. If AO is understood as a computer architecture, then AOS is an operating system instance.

In essence, ao is an architecture system. Users need to interact with an operating system like AOS to use the corresponding functions. For the convenience of discussion, ao is used to refer to it below, but users need to pay attention to the difference between the two.

Key points of this article:

• Arweave launched the ao architecture, hoping to transform itself into an all-round player in storage and computing, and change the current situation in the market where it only has the concept of storage;

• Arweave's ao architecture is a functional superposition on storage. The ao architecture has the potential to communicate with any public chain and DApp;

• ao-related technologies include distributed architecture SSI (single system image), Actor Model (erlang language is not a model) and SU/MU/CU unit composition, among which concurrency and asynchrony are the key to understanding ao The key to the architecture;

AO/SCP has great potential and is expected to stimulate the development of the Arweave ecosystem. It is necessary to observe the continued appeal of the "off-chain computing + on-chain storage" model to the project.

Technical interpretation

Let us first introduce some prerequisite knowledge as a supplement to AO. After the recent Cancun upgrade and the activation of EIP-4844, the data storage problem of Ethereum has become increasingly important.

For example, blobs specifically used to store Layer2 DA data will not be permanently retained in the Ethereum network. Nodes can delete blob data that exceeds the time window on their own. At that time, the eliminated data will need to find another storage location.

Although there are Ethereum-based storage platforms such as EthStorage that solve the problem of blob data expiration and elimination, this is not a native solution for Ethereum and requires some additional design mechanisms.

In addition, although EIP-4844 can greatly reduce the cost of data publishing, it is still very expensive compared to Arweave.

Unlike Ethereum, Arweave, which was launched with the slogan of "Library of Alexandria", although its computing function seems simple, it natively supports data permanence and has extremely low cost (storing 1GB of data costs about tens of dollars, which is the same as the cost of 1 Ethereum transaction).

In terms of redundant data storage, Arweave associates the probability of block generation with the completeness of the node's local data set. If the storage node deletes some data, its probability of successful block generation will decrease, and the nodes that retain the most data have higher block generation "computing power", and they will get more rewards.

In this way, Arweave's incentive system can ensure that any period of historical data can be stored redundantly with a high probability.

It can be said that Arweave is suitable as a decentralized data storage and publishing layer with extremely low storage costs, while ao and SCP are modular blockchains and DApp architectures based on AR.

Although the design pattern of SCP is theoretically quite different from modular solutions such as Ethereum Rollup that focus on security, it is very feasible in terms of ease of implementation and difficulty in connecting with the Web2 platform.

Because he did not intend to limit himself to a narrow implementation path like Rollup from the beginning, he wanted to integrate the Web2 platform with Web3 facilities in a broader and more open framework.

Image source: Geek Web3《Interpreting SCP: Detrusted Infrastructure Paradigm that Breaks Out of Rollup》

The above figure is a schematic diagram of everPay using the SCP solution. The DA layer uses Arweave, which is the big circle in the figure.

The brown circle is the coordinator, which is the execution layer, similar to the sequencer of Ethereum Layer2. After the user submits the transaction to the coordinator, the coordinator will perform the calculation and submit the DA data in batches to AR.

As for the Detector, it is somewhat similar to the challenger/validator of Ethereum Layer2. They will pull the DA data submitted by the coordinator from Arweave and calculate or verify the transaction results.

The client of the Detector is open source and can be run by anyone. The Watchmen are actually the multi-signature nodes that manage the cross-chain system and will verify and execute cross-chain requests. In addition, the Watchmen are also responsible for signing governance proposals.

It is worth emphasizing that the SCP architecture does not have as strict security requirements as Ethereum Layer2, but it actually provides project parties that adopt this architecture with higher degrees of freedom, more customization options, and reduces the cost of adoption. It can be said to be a unique and creative approach.

Image source: Geek Web3 《Interpreting SCP: Breaking out of the Rollup paradigm of trustless infrastructure》

Simplifying, the framework of ao can be divided into three parts: distributed architecture, parallel computing functions and communication scheduling components. Unifying the three is a complete supercomputer function.

• Distributed architecture, ao uses a single system image (SSI) to organize the decentralized system of the ao network.

• Parallel computing function, AO uses the Actor Model theory of parallel computing theory to handle high-concurrency environments and effectively combines blockchain-related technologies. The word AO also comes from Actor Orient (oriented to Actor, imitating the word object-oriented in OOP).

• Communication scheduling component, AO designed three key parts of MU/CU/SU, Messenger Units are responsible for information transmission, Scheduler Units are responsible for process scheduling, and Compute Units are responsible for parallel computing processes.

Let's explain each of the above parts separately. First of all, SSI single system image is actually a distributed architecture.

For example, the server systems behind major Web2 applications are basically distributed systems composed of many server nodes. These servers use special message transmission and communication protocols to ensure that each other's status and views on new data are consistent.

However, at the client/front-end level, users cannot perceive that the corresponding server behind the front-end is distributed. To users, even the largest computer cluster is like a single computer.

This is actually the "abstraction" often mentioned in computer engineering, which is to unify and aggregate complex underlying components into a module. The outside world does not need to know the internal structure of this module. It only needs to pass the input information to the module to get the output result.

The SSI single system image mentioned above uses Arweave's "cheap decentralized storage" feature. It can be said that the narrative foundation of ao/SCP is mainly based on Arweave's storage price advantage compared to other public chains, and its anti-censorship and data transparency advantages compared to traditional Web2 platforms.

In the narrative of ao and SCP, AR is used as a huge data bulletin board and log recorder. The data sent by the DApp front end will be transmitted to the Arweave network and stored in the distributed blockchain network by a large number of Arweave nodes.

Compared to mainstream public chain networks with a higher degree of de-trust such as Ethereum, the storage cost on Arweave is extremely low, which can better support application scenarios that have a high demand for data throughput.

Unlike traditional Web2 platforms and consortium chains, the openness of the Arweave network is more conducive to anti-censorship and data transparency, and DApps that rely on AR are more trustless than Web2 applications.

For example, traditional Alipay can also be Web3-ized. As long as Alipay designs the interface to be compatible with the ao protocol, Alipay's interactive data will be automatically uploaded to the Arweave network, becoming a Web3 version of a trustless "Alipay".

If it is an Ethereum or EVM-based DApp, it can also be connected to the ao interface, convert the information format to ANS-104 format and upload it to Arweave.

Unlike traditional XX cloud and closed alliance chain, anyone or DApp project can request and read data from multiple nodes through P2P as long as they run nodes of third-party public chains such as Ethereum or Arweave. As long as 1 out of N nodes is willing to provide data, you can get what you need. This ultimately depends on the openness of the network.

From these two perspectives, Arweave-based DApp architecture solutions such as ao and SCP are more like a transition between Web2 and Web3.

Traditional Web3 platforms such as Ethereum and Bitcoin have achieved a high degree of anti-censorship and trustlessness at the cost of cost and efficiency, but are difficult to be adopted on a large scale.

Web2 platforms sacrifice data transparency and anti-censorship, achieve high efficiency and low cost, but cannot be trusted. Ao is more like an intermediate form between the two.

The difference between SSI and distributed architectures such as client-server architecture, three-tier architecture, N-tier architecture and peer-to-peer architecture lies in transparency. SSI can significantly improve system abstraction and user experience.

However, it should be noted that SSI relies on optimistic synchronization control, which requires the system to have a high synchronization control capability to ensure data consistency and reliability. Once the synchronization control fails, data loss may occur, thus affecting the availability of the ao architecture.

Another benefit of SSI is deployment speed. SSI can run multiple instances on a single server and does not need to rely too much on cloud services or containerization tools like microservice architecture or containerization technology, effectively reducing the complexity and deployment cost of the system.

In ao's practice, data synchronization and backup of distributed architectures rely on the Arweave network. Because of Arweave's permanent storage effect, theoretically, the data status at any point in time is retained, and there is no loss or damage to the data.

However, it should be noted that SSI will also cause new additional overhead, focusing on network communication and effective data synchronization between distributed nodes. For example, when the SSI architecture fails, in extreme cases, as long as there is a normal node, the entire network can operate normally, but in fact this will cause serious node security crises and system robustness.

Actor Orient

After briefly introducing the SSI architecture, it is necessary to study in depth the implementation of ao's parallel computing mechanism. Different from the simple "stacking" of centralized servers, ao uses the Actor model to achieve decentralized high concurrency effects, and users are basically unaware that this is a distributed system.

The efficient concurrent computing capability of the ao architecture comes from the Actor model. Carl Hewitt defined the theoretical framework of the Actor model in 1973 and used Actor as the primitive for concurrent computing. Interestingly, it was designed for computing for artificial intelligence at the time.

However, in practice, people may be more familiar with models such as OOP. In fact, according to Oracle's research, OOP is an improved version of Actor, but the two have gradually developed further and further.

The Actor model defines a series of general rules for how system components should act and interact. Each Actor is an independent entity that can make local decisions and communicate with other Actors. However, it should be noted that the Actor model emphasizes asynchronous, parallel and distributed characteristics.

Especially asynchronous and parallel, which means that the state of each component is not synchronized, and conflicts may occur, so it is necessary to rely on the message passing mechanism. This is also the reason why MU and SU are emphasized in ao. Execution is not difficult, but the difficulty is to arrange and schedule to give full play to the powerful computing power of parallelism.

Each Actor is an independent execution unit that can handle the assigned tasks on its own. If it is used, such as ensuring the atomicity and consistency of messages, it is a very powerful and flexible concurrency model.

The consideration here is the special cross-multi-node communication requirements in the blockchain. For example, the common microservice architecture often uses the node-to-node communication mode, but the RPC-based implementation method will lead to the complexity and delay of multiple data transmissions, and the ao architecture uses the unified message passing mechanism MU to uniformly allocate message formats for the final Arweave storage.

Compared with the synchronous execution of tasks by each node in CSP (Concurrent Semantics), the most typical feature of Actor is asynchronous execution. For this reason, ao does not use the common shared memory mechanism to ensure the independence of each node and communicate across nodes in a more flexible way.

Asynchrony and parallelism constitute the source of the high efficiency of the Actor model in the ao architecture, and in order to ensure this efficiency, MU/SU/CU were proposed and put into use.

In short, the combination of the Actor model and the Arweave blockchain builds an asynchronous high-concurrency computing model under efficient information transmission.

Three essentials

Under the ao architecture, both the SSI and the Actor model have put forward higher requirements for information transmission, and SU, MU and CU came into being.

First of all, to understand the ao process, it refers to the request for the corresponding computing resources when initializing the task, such as virtual machines and memory resources. The flow of any task is essentially achieved through the transmission of the process.

Whether it is SSI or Actor Model, the messages flowing between them must comply with the data standards and formats of ANS-104, so that any type of DApp can understand each other.

After producing data that meets the requirements, MU will send messages to online SUs. This process will continue until all messages are processed. Then SU ​​needs to receive the data and upload it to Arweave to link Arweave's verification capabilities.

In addition, the processing process of MU can also set up a payment mechanism to customize the message processing mechanism, such as only sending messages but not performing subsequent actions.

After SU receives the message, CU will start to access. CU is a unit that contains multiple units responsible for calculation. It should be noted that CU is also a decentralized computing power market. Similar to Akash, each CU cluster will compete with each other, and the winner of the competition has the right to calculate.

CU will respond and submit a calculation result according to the request. The results of these responses are stored on Arweave and can be verified by the original data on Arweave.

It can be found that in this mode, ao provides an efficient and competitive computing network. Users do not need to establish consensus for calculations. They only need to ensure that the transmitted messages comply with the corresponding process. This fundamentally gets rid of the high cost paid by Ethereum and other computing effectiveness.

The combination of SCP and ao

First, let's discuss verifiability. Keep in mind a consensus that the function of ao is to present the verifiable data status. The final verifiability problem is guaranteed by the consensus data on Arweave.

ao is essentially also an scp application. At this time, both the query status and the return status actions are chained to Arweave. The ao/SCP program will be loaded into these two actions, and the Mint and Slash results will be calculated through these two actions.

Specifically, based on the SCP paradigm, the rules of Mint and Slash need to be written into the index, so the node that calls the index data will naturally calculate the results of Slash and Mint (see ao spec for the data model).

After discussing the technical architecture of ao, the following will discuss its applications. Although the native cross-chain protocol aox, the decentralized stablecoin protocol astro, and the EVM-compatible project AOVM on Arweave have appeared after the ao conference, they are still in the experimental state.

It is worth mentioning that the application based on ao is in the ascendant, and many ao versions of Twitter and games are already under development and testing.

At the same time, some mature SCP projects in the Arweave ecosystem, such as everPay and Permaswap under everVision, will be adapted and transformed to ao. In theory and practice, SCP theory and ao are also homologous.

SCP is rooted in Arweave's storage capacity. You can imagine a Turing paper tape machine that never stops. SCP is responsible for uploading data to the blockchain, and it undertakes the tape recording function of the Turing machine, while the state machine can be provided by ao, and every state change can be stored on Arweave.

The problem here is state explosion, which is also a hidden disease that has plagued Ethereum for many years. Arweave does not store state, so there is no state explosion problem naturally, and all data produced by the ao process can be permanently saved.

It should be noted that the data in the ao node does not need to be calculated to reach a consensus state. In theory, as long as the relevant data is stored on the Arweave network, each state change must be recorded. Therefore, to be more precise, the data on ao can be written to the network without calculation, and calculation is only part of the data change.

Secondly, facing the enduring blockchain trilemma, that is, no blockchain can solve the problems of security, decentralization and scalability at the same time, the combination of SCP and ao can basically put an end to this dilemma.

• Security: The data provided by Arweave is the largest consensus, and the consensus data stored in Arweave provides verifiability for applications;

• Decentralization: ao brings decentralization of computing power. Any individual, organization, and NGO can join and leave the ao computing network. Thanks to data consensus, it is also very simple to restore the state after leaving;

• Scalability: Unlike Ethereum's vertical layering, ao and SCP are generally horizontally divided into blocks, which can infinitely expand computing and storage capabilities.

Based on the ao/SCP architecture, Arweave is no longer a simple storage public chain. The combination of AR+AO is a decentralized supercomputer with both storage and computing functions. Any DApp can be deployed on it and call each other.

The current ao is also a unique modular architecture, that is, horizontally expanded modularity, which not only allows smart contracts such as Warp on the Arweave network, but also smart contracts of mechanisms such as EVM to be inserted into the ao network, as long as the format of the data is kept consistent.

Or to put it this way, scp is a full-chain Layer2 built based on Arweave, which can be connected to any public chain and DApp, and ao is a super version of SCP, which can make any public chain, smart contract and DApp part of ao.

Another possibility of DeFi

In the existing EVM system, smart contracts are the core of everything. On-chain behavior is essentially the scheduling and use of smart contracts. Taking the most common DEX transaction as an example, the smart contract will perform corresponding operations in accordance with regulations when specific conditions are triggered, such as adding liquidity, finding the contract address of the token, etc.

However, it must be noted that the contract at this time is only a single-threaded process and cannot be called concurrently. All transactions will be sorted by Ethereum and determined whether they are successful or attacked by MEV.

If the ao architecture is used to transform Uniswap, a parallel, non-stop on-chain trading robot can be created. The ao version of Uniswap processes can be set to different trigger mechanisms, but they will not interfere with each other and can run using all computing resources.

It can be understood that this is a Web2 quantitative level on-chain exchange. The largest DEX project in the Arweave ecosystem, Permaswap, has adopted this method to adapt to the ao native environment.

Each process of ao has the ability to issue tokens. Taking Ethereum as an example, the issuance process of each ERC-20 token is the Token Process. It only needs to be set according to the set price range to generate orders for trading users and then complete the token exchange process.

In fact, the transfer and transaction of Ethereum tokens are extremely difficult. In essence, it is the addition and subtraction of balances in different accounts, which eventually leads to the need for repeated calculations for each transaction according to the latest requirements of Ethereum, resulting in a large amount of redundant data being deposited on the chain.

AO has changed the calculation method for token transfer. The transfer of tokens between different accounts is essentially the synchronization of different states, which depends on the message passing process of MU. It only needs each process to be sent to the Arweave network for storage, that is, the consensus occurs before the calculation, then the assets can be transferred by performing combined calculations in the end without the participation of the entire network.

That is, each token on AO is concurrent, and even countless sub-ledgers can be established for a token, each of which provides independent and parallel computing capabilities.

In the existing AO design, it is allowed to automatically wake up a specific contract. You only need to pay the node to use the process, and then the process will be calculated and executed according to the set frequency. Thanks to the cheapness of Arweave and the high speed of AO, the execution frequency can be set very densely.

For users, ao executes a computer-like program rather than an abstract smart contract. For example, after connecting to EverID, everPay connects to the ao network, so EverID operates the ao interface, and the type of program behind the ao interface is irrelevant.

Users can operate DApp applications on multiple public chains at the same time, which is highly similar to the usage logic of existing Internet terminals. The terminal will call different servers in the network to provide users with a simple and single application interface.

In essence, this is a change to the existing DeFi, unifying the user's operating logic within a set of interactive systems while preserving the degree of decentralization behind it.

Global Supercomputers

In addition to traditional encryption needs such as DeFi, ao actually opens up the ability to feed back to traditional Web2, one of the focuses of which is the trusted computing of ML (Machine learning). As mentioned above, Carl Hewitt's original intention for designing the Actor Model was AI computing, which just gave ao and AI a natural ability to combine.

In the previous combination of AI and Crypto, machine learning models could not be smoothly put on the chain, especially models with a large number of parameters such as LLM. However, ao is different from the past. Since users can choose and customize the resources allocated by ao, and can use computing services without access, and computing resources are essentially infinitely expandable and mutually cooperative, it seems that ML on the chain is not impossible.

Compared with decentralized computing markets such as Akash, ao's advantage does not lie in the number of GPU clusters, but in its ultra-large-scale parallel computing capabilities. Akash requires a strong trust mechanism to form a computing power market.

And ao does not need to sacrifice the no-access feature, and it should be noted that all of this is still based on smart contracts, that is, it runs on the chain and is stored in Arweave for state proof, and benefits from super compatibility.

Users can use the on-chain environment in their own way, such as running LLM models, and their data can be stored on the Arweave network, while solving the computing power and data decentralization requirements of large AI models.

ao is different from existing decentralized computing power platforms and cloud computing vendors. It is the first decentralized high-concurrency network, which can be understood as cloud vendors with smart contract functions. At a time when Ethereum is facing a computing and storage crisis, it can be said that Arweave has completed the wonderful loop of "decentralized computing power is decentralized data."

To put it simply, supercomputers that used to be expensive and far away from the public are now available to everyone, and no one can control its start, run, and end. As long as a process starts, it will only end under the conditions set by the smart contract, otherwise it will run forever.

Conclusion: Arweave's future

After the arrival of ao, with the superposition of the capabilities of the SCP paradigm, Arweave has the opportunity to become a permanent storage and infinite computing network, but it should be noted that the current ao running node is still in the test network state and relies on the penalty slash mechanism written in the code to operate.

Theoretical optimality does not mean feasibility in reality. ao hopes to become an infinitely extended and real-time scalable computing network, and users have full control.

However, the various ecosystems on Arweave are not active, especially the mainstream DeFi applications are still relatively scarce, not only less than Ethereum, but even behind the launch of FVM by Filecoin.

Overall, ao+SCP+Arweave does open up another possibility for blockchain, but this possibility still needs time to verify.