Neferpitou: Three Sigma’s Agent-Based Simulation Framework

•
Carolina Goldstein & Andreia Pereira
• 10 min read

Introduction

Diving into the world of agent-based modeling since early 2022, we at Three Sigma have been working to address a crucial gap in the blockchain industry. While theoretical foundations in new projects are often sound, understanding real-world participant behavior and its consequences on a project's overall success is generally overlooked. We built Neferpitou, our simulation framework, precisely to help fill this void, providing valuable insights comparable to the role of security audits in mitigating smart contract risks. Like security audits, economic audits are essential to detecting vulnerabilities and fine-tuning systems. In this blog post, we will take you through our approach and show how it can be used to stress test DeFi protocols and other blockchain-based systems.

In a nutshell, Neferpitou is a framework built to empower our economic modeling team to readily simulate any arbitrary scenario in a realistic environment with support for several blockchains and custom orchestration layers, through stochastic agent-based models.

Why agent-based?

Agent-based modeling is a computational modeling technique that focuses on simulating the actions and interactions of individual agents within a complex system. In this type of modeling, each agent is represented by a set of rules, usually called policies, that govern their behavior and interactions with other agents and with the environment. The overall behavior of the system emerges from the interactions of these agents over time. The models can be used to explore how changes in individual behavior or environmental factors can affect the behavior of the system, and to predict the impact of different scenarios and project parameterizations. The image below illustrates how complex interactions emerge from modeling individual agents in an evacuation scenario. By studying the behavior of an individual in an evacuation situation where panic might occur, the simulations illustrate the emergent behavior of crowds, to ultimately design better evacuation strategies.

Example of the initial state of a system showing agents spread uniformly in a closed space with two opposing exits, before an evacuation is triggered. [Source: https://www.youtube.com/watch?v=6--5d5WYeZU&t=133s]

State of the system after an evacuation is triggered and the simulation has been running for some time, showing a positioning of the agents around the exits and varying in panicked reactions, as indicated by the white color of the individuals. The policy of each agent urges them to leave the room as quickly as possible. As a result, crowds appear as the emergent collective behavior. By utilizing this method of simulating agents, a simulator could effectively evaluate multiple exit layouts for a building, allowing for thorough testing and analysis. [Source: https://www.youtube.com/watch?v=6--5d5WYeZU&t=133s]

In the context of blockchain, agent-based modeling can be used to simulate various scenarios and stress-test systems by exposing how agents behave and interact in such environments. For example, you could simulate different types of market conditions, such as a sudden drop in asset prices, or a spike in trading volume, and observe how the agents and, consequently, the system reacts to these stressors.

In contrast to other modeling techniques, agent-based modeling allows us to capture the complexity and interdependencies of blockchain systems, which are often highly decentralized and involve numerous interacting agents. This complexity makes it challenging to predict how the system will behave in different scenarios using traditional analytical models, which often rely on simplifying assumptions and do not account for the heterogeneity of individual behaviors.

Why stochastic?

Stochastic simulations account for the inherent randomness and uncertainty that are present in real-world systems. Through Monte Carlo runs, these simulations use random variables and probability distributions to model the behavior of agents, which allows us to capture the nature of real-world systems more accurately. The introduction of stochasticity allows for the exploration of a wider range of possible outcomes and scenarios, generating more robust predictions.

An example of how randomness can be incorporated into the system is through the use of multiple price trajectories modelled using Geometric Brownian Motion (GBM). After setting an initial price, drift, and volatility, multiple trajectories are randomly generated through this stochastic process, allowing us to depict real-world unpredictability.

In contrast, deterministic solutions rely on fixed mathematical equations and do not account for randomness and uncertainty, which can result in overly simplistic models that fail to capture the real-world complexity.

Architecture

Neferpitou's stochastic simulations are a sophisticated tool for exploring the behavior of blockchain agents. To ensure statistically relevant results, Neferpitou’s Admin module undertakes multiple Monte Carlo runs, in which the same configuration and scenarios are deployed to ensure consistency. Besides the Admin, each simulation is comprised of four essential components: the Agent model, the Snapshooter, the Mediator, and the contract call Execution Environment.

The Snapshooter serves as Neferpitou's block explorer, recording state changes at every new block and providing agents with access to the blockchain. With a window to the current blockchain state, agents can consult their behavior policies and determine which transactions, if any, they should submit - a process that mirrors real-life decision-making. Finally, the Mediator works with the Execution Environment to assemble and execute transactions.

Neferpitou was designed to be modular and flexible. With it, you can easily simulate the same scenarios in different Execution Environments - and see how varying some chain specificities such as gas prices, opcode costs, or even block gas targets, may affect your own project. Put this together with our customizable Mediator, Neferpitou can simulate different block assembly criteria and network latency.

Diagram 1: A general view of Neferpitou’s Architecture

Execution Environment

Back in Three Sigma’s early days, we wanted to move fast and break things: Neferpitou was implemented in Python for its ease of use and a relatively strong selection of community-built libraries bridging the Ethereum ecosystem over to Python developers. However, after some initial experimenting, it soon became clear that Python’s latency would be fatal for Neferpitou.

Fast forward a few months, the team decided to revive Neferpitou and start over. We knew we wanted a lightning-fast simulator that could scale and provide our partners with reliable results. Towards this goal, Neferpitou is now fully written in Rust due to the language’s safety, reliability and stellar speed - for us, this was a no-brainer, as Rust is steadily becoming the go to language for the development of blockchain tooling and infrastructure (e.g., Foundry and rETH).

Neferpitou is built to support simulations in multiple blockchain systems with distinct virtual machines. For simulations in EVM blockchains, Neferpitou uses rEVM - an open-source, Rust-based implementation of the Ethereum Virtual Machine - as its backbone execution environment for transactions being submitted during a simulation. rEVM provides a high-performance platform for simulating complex protocols, allowing our simulation framework to execute Solidity smart contracts directly, which in turn ensures our partners have a reliable testing environment that accurately reflects the Ethereum blockchain. By running the smart contracts directly as they would be ran in the mainnet, Neferpitou provides the best accuracy and can simulate any on-chain scenario.

Mediator - Neferpitou’s orchestration layer

Neferpitou uses a proprietary orchestration layer that connects the agents and the Execution Environment. This powerful layer manages the transaction mempool, assembles blocks, and ultimately executes them, resorting to the configured Execution Environment.

By building our own layer of orchestration, Neferpitou provides limitless flexibility on what we can simulate without the high latency that traditional frameworks impose. Whether it's mimicking network latency during peak demand or simulating priority gas auctions, our framework ensures you can analyse your protocol's parameterization with pinpoint accuracy. Moreover, the Mediator’s block producer can control every detail regarding block assembly - from block times and block gas targets to different transaction selection criteria. This way, Neferpitou truly empowers teams with the tools they need for optimal incentive alignment and risk management.

Modeling the real world

When designing Neferpitou, a key priority was to create a tool that could simulate a diverse range of scenarios, including collective behaviors and tendencies. To achieve this, Neferpitou supports two primary methods: modeling participant behavior through agent policies and explicitly defining specific, realistic chain scenarios within a simulation.

Modeling participant behavior

Agents are the driving force of a simulation, as they are designed to emulate the behavior of real participants. Armed with behavior policies and aware of the current chain state, our agents can stochastically determine which actions to submit to the chain at a given time.

Ultimately, the interaction between agents and the protocols on-chain is what provides protocols with actionable insights into its parameterization or underlying risks. To simulate those interactions, various policies are defined to encode how a given agent might act according to the current state of the chain (balances, protocol conditions, etc).

📎

Imagine we are simulating the Aave liquidity protocol. We may define different liquidation policies, ranging in eagerness to liquidate relative to estimated profit, for example. So, we may have multiple Liquidator agents that have different liquidation policies.

Policies can be probabilistic - we may define an agent that, given a certain set of conditions, will perform a given action with a probability given by a specific statistical distribution, derived from historical data.

📎

Aave Liquidator Policy: We may define a liquidation policy where, whenever a loan is up for liquidation and the expected profit is aligned with an agent’s goals, they will have a 60% probability to submit a transaction to liquidate that position.

Furthermore, we can parameterize each of the agent’s policies to follow a pre-observed, real-world distribution - here is where Three Sigma’s Economic Modeling and Data Analysis expertise comes into play.

To better showcase the typical profile of an actor in DeFi, Neferpitou’s agents can have multiple policies which complement each other.

📎

Aave Liquidity Provider Policy: We may define a Liquidity Provider policy which states that if an agent has liquidated a position in the last N blocks, their liquidation bonus should be re-invested in the Aave protocol. The same policy may state, in following iterations, that the agent will claim their LP gains with increasing probability as time goes on.

Our policy design gives us maximum flexibility when modeling for a specific protocol and simulation scenario, by assuring we can model rational, competitive agents that can interact with a protocol in multiple ways and with varied goals.

Running realistic scenarios

At Three Sigma, we understand that the DeFi ecosystem can be complex and challenging to navigate. That's why Neferpitou offers a comprehensive range of scenario testing options to give teams the confidence they need to move forward. By forking the mainnet to set up simulations, Neferpitou can accurately test how a protocol would respond under real historic events - providing unparalleled insight into the inner workings of DeFi protocols.

Our simulation capabilities don't stop there. Neferpitou also employs sophisticated modeling techniques like the Geometric Brownian Motion to simulate specific asset price trajectories and GARCH models to evaluate different volatility scenarios - this way we can effectively test specific edge-case scenarios that a team may worry about. By modeling our agents' policies through the lens of game theory, Neferpitou can effectively anticipate community responses to events in the DeFi space and analyse how a protocol would perform under various conditions.

If you are interested in knowing more, get in touch with us at [email protected]!

You guessed right - this is Neferpitou on the right considering all the possible scenarios while expectant projects on the left wait for recommendations. [Image taken from Hunter × Hunter series]

Sources & References