Multi-pool and events based database refactor (#1467)

This PR is the second in a collection of refactors to allow agent0 to
support multiple pools.

This PR focuses on updates to the database and related systems. In
particular, this makes the following major changes to the database.

## Database schema updates

- Instead of mapping a single pool to a single database, we now insert
all pools into a single database, indexed by the `hyperdrive_address`
field.
- A new `HyperdriveAddrToName` maps the hyperdrive address to a logical
name. The name defaults to
`<vault_share_token_symbol>_<position_duration_in_days>_day`.
- The `WalletDelta` and `HyperdriveTransactions` tables have been
removed in favor of the new events table added in
#1464.
- The `PositionSnapshot` table calculates the absolute positions from
the deltas stored in the `TradeEvent` tables. This table also stores any
additional information useful for analysis, such as realized and
unrealized values of positions, and the pnl.
- The caching `CurrentWallet`, `Ticker`, and `WalletPNL` tables have
been removed in favor of a new `PositionSnapshot` table.
- The `PoolAnalysis` table has been removed in favor of putting the
fields in the `PoolInfo` table.
- The `UsernameToUser` table has been removed, no need to map multiple
wallets to a single user.

## Analysis from remote chains

- The database container is now managed by the `Chain` object as opposed
to `LocalChain`. Likewise, database sessions is now managed by the
`Chain` object as opposed to `LocalHyperdrive`. These changes allow the
remote chain object to have access to the database for gathering events,
positions, and pnl from hyperdrive pools.

- The simulated environment uses the full set of tables in the database,
whereas the remote chain environment uses a subset of these tables. Both
workflows uses the same underlying schema, but the remote workflow
updates these tables lazily, i.e., only query and insert events and
snapshots into the db when the user calls the functions that need them.
Additionally, the remote workflow only updates the tables with any
agents initialized by agent0. In contrast, the simulated environment
updates these tables with all agents that have interacted with the
simulated pool, and the database is updated on every trade to these
pools.

## Interactive analysis interface
The interactive interface for getting information from the db has been
reworked. Note that some pool analysis functions are only available in
`LocalHyperdrive` method, due to lazy event querying on remote chains.

### Wallet objects and Positions
- `agent.get_wallet()` has replaced `agent.get_positions()` (which
replaced `agent.wallet` in
#1464). This function returns a
`HyperdriveWallet` object that contains the relevant positions (i.e.,
base, longs, shorts, lp, withdrawal shares) for a specific pool. A
future PR will allow this function to take a pool as an argument, or
require setting the agent's active pool. This function queries the
database for current positions.
- `agent.get_positions()` now returns a dataframe consisting of all open
positions across all pools held by the agent, with additional pnl,
unrealized, and realized value columns. An optional argument
`show_closed_positions`, if set to True, will also return all closed
positions and the position's realized value.
- Similarity, `hyperdrive.get_positions()` returns a dataframe
consisting of all positions across the `hyperdrive` pool. This function
is only available in `LocalHyperdrive`.
- `agent.get_trade_events()` returns a dataframe of events emitted from
the agent across all pools that the agent interacted with. An optional
argument `all_token_deltas`, if set to True, will repeat any
`RemoveLiquidity` events with a non-zero withdrawal shares, where the
additional row tracks the withdrawal share token delta.
- Similarity, `hyperdrive.get_trade_events()` returns a dataframe of
events emitted from the pool across all agents. This function is only
available in `LocalHyperdrive`.

### Historical positions
We also expose a couple of helper functions for getting positions over
time. These are exclusive to `LocalHyperdrive`, due to the underlying
table (i.e., `PositionSnapshot`) only being guaranteed to keep
historical data for the simulated environment.

- `hyperdrive.get_historical_positions()` returns a dataframe with the
history of positions over time. This function is only available in
`LocalHyperdrive`.
- `hyperdrive.get_historical_pnl()` returns a dataframe of historical
positions, aggregated across all positions to find the total pnl. This
function is only available in `LocalHyperdrive`.


## Streamlit dashboard

- The streamlit dashboard has been updated and cleaned up (with tests
for building the DataFrames for the various views) with the new db
scheams.
- The dashboard now supports looking at information from multiple pools.
- The `run_dashboard` command is now a function of the `LocalChain`.

examples/custom_policy.py

@@ -5,6 +5,8 @@
 from dataclasses import dataclass
 from typing import TYPE_CHECKING
 
+from fixedpointmath import FixedPoint
+
 from agent0 import (
     HyperdriveBasePolicy,
     add_liquidity_trade,
@@ -15,7 +17,6 @@
     redeem_withdraw_shares_trade,
     remove_liquidity_trade,
 )
-from fixedpointmath import FixedPoint
 
 if TYPE_CHECKING:
     from agent0.core.base import Trade

examples/interactive_local_hyperdrive_advanced_example.py

@@ -73,7 +73,7 @@
 
 # %%
 # Runs the dashboard in blocking mode, which waits for a keyboard press to exit the dashboard
-print(interactive_hyperdrive.run_dashboard(blocking=True))
+print(chain.run_dashboard(blocking=True))
 
 # %%
 # cleanup resources

examples/interactive_local_hyperdrive_example.py

@@ -34,31 +34,28 @@
 # %%
 # Generate funded trading agents from the interactive object
 # Names are reflected on output data frames and plots later
-hyperdrive_agent0 = hyperdrive.init_agent(base=FixedPoint(100000), eth=FixedPoint(100), name="alice")
+agent0 = hyperdrive.init_agent(base=FixedPoint(100000), eth=FixedPoint(100), name="alice")
 hyperdrive_agent1 = hyperdrive_2.init_agent(base=FixedPoint(100000), eth=FixedPoint(100), name="bob")
 # Omission of name defaults to wallet address
 hyperdrive_agent2 = hyperdrive.init_agent(base=FixedPoint(100000), eth=FixedPoint(10))
 
 # Add funds to an agent
-hyperdrive_agent0.add_funds(base=FixedPoint(100000), eth=FixedPoint(100))
+agent0.add_funds(base=FixedPoint(100000), eth=FixedPoint(100))
 
 # %%
 # Here, we execute a trade, where it's calling agent0 + gather data from data pipeline
 # under the hood to allow for error handling and data management
 # Return values here mirror the various events emitted from these contract calls
-open_long_event_1 = hyperdrive_agent0.open_long(base=FixedPoint(11111))
+open_long_event_1 = agent0.open_long(base=FixedPoint(11111))
 open_long_event_1  # pyright: ignore
 # %%
 
-# Allow for creating checkpoints on the fly
-# TODO Need to figure out how to mint checkpoints on the fly
-# checkpoint_event = hyperdrive_agent0.create_checkpoint()
-
 # Another long with a different maturity time
-open_long_event_2 = hyperdrive_agent0.open_long(FixedPoint(22222))
+open_long_event_2 = agent0.open_long(FixedPoint(22222))
 
 # View current wallet
-print(hyperdrive_agent0.get_positions())
+print(agent0.get_wallet())
+
 
 # NOTE these calls are chainwide calls, so all pools connected to this chain gets affected.
 # Advance time, accepts timedelta or seconds
@@ -68,14 +65,12 @@
 chain.advance_time(3600, create_checkpoints=True)
 
 # Close previous longs
-close_long_event_1 = hyperdrive_agent0.close_long(
+close_long_event_1 = agent0.close_long(
     maturity_time=open_long_event_1.maturity_time, bonds=open_long_event_1.bond_amount
 )
 
-agent0_longs = list(hyperdrive_agent0.get_positions().longs.values())
-close_long_event_2 = hyperdrive_agent0.close_long(
-    maturity_time=agent0_longs[0].maturity_time, bonds=agent0_longs[0].balance
-)
+agent0_longs = list(agent0.get_wallet().longs.values())
+close_long_event_2 = agent0.close_long(maturity_time=agent0_longs[0].maturity_time, bonds=agent0_longs[0].balance)
 
 # Shorts
 open_short_event = hyperdrive_agent1.open_short(bonds=FixedPoint(33333))
@@ -85,7 +80,7 @@
 
 # LP
 add_lp_event = hyperdrive_agent2.add_liquidity(base=FixedPoint(44444))
-remove_lp_event = hyperdrive_agent2.remove_liquidity(shares=hyperdrive_agent2.get_positions().lp_tokens)
+remove_lp_event = hyperdrive_agent2.remove_liquidity(shares=hyperdrive_agent2.get_wallet().lp_tokens)
 
 # The above trades doesn't result in withdraw shares, but the function below allows you
 # to withdrawal shares from the pool.
@@ -96,30 +91,25 @@
 pool_config = hyperdrive.get_pool_config()
 # The underlying data is in Decimal format, which is lossless. We don't care about precision
 # here, and pandas need a numerical float for plotting, so we coerce decimals to floats here
-pool_state = hyperdrive.get_pool_state(coerce_float=True)
-# TODO checkpoint info doesn't play nice with advancing time.
-# This is because we don't create checkpoints when we advance time.
+pool_info = hyperdrive.get_pool_info(coerce_float=True)
+
 checkpoint_info = hyperdrive.get_checkpoint_info()
 
-current_wallet = hyperdrive.get_current_wallet()
-ticker = hyperdrive.get_ticker()
-wallet_positions = hyperdrive.get_wallet_positions()
-total_wallet_pnl_over_time = hyperdrive.get_total_wallet_pnl_over_time(coerce_float=True)
+# Change this to get wallet
+wallet = agent0.get_wallet()
 
-# %%
-print(pool_state)
-# %%
-ticker  # pyright: ignore
-# %%
-wallet_positions  # pyright: ignore
-# %%
-total_wallet_pnl_over_time  # pyright: ignore
-# %%
+agent_positions = agent0.get_positions()
+agent_trade_events = agent0.get_trade_events()
 
+pool_positions = hyperdrive.get_positions()
+trade_events = hyperdrive.get_trade_events()
+pool_positions_over_time = hyperdrive.get_historical_positions()
+total_wallet_pnl_over_time = hyperdrive.get_historical_pnl(coerce_float=True)
 
+# %%
 # Plot pretty plots
 # TODO these should be in a notebook for plotting
-pool_state.plot(x="block_number", y="longs_outstanding", kind="line")
+pool_info.plot(x="block_number", y="longs_outstanding", kind="line")
 # Change wallet_address to be columns for plotting
 total_wallet_pnl_over_time.pivot(index="block_number", columns="username", values="pnl").plot()
 

examples/interactive_remote_hyperdrive_example.py

@@ -9,22 +9,30 @@
 
 from fixedpointmath import FixedPoint
 
-from agent0 import Chain, Hyperdrive, PolicyZoo
+from agent0 import Chain, Hyperdrive, LocalChain, LocalHyperdrive, PolicyZoo
 from agent0.core.base.make_key import make_private_key
 
 # %%
 # Set the rpc_uri to the chain, e.g., to sepolia testnet
-rpc_uri = "http://uri.to.sepolia.testnet"
-chain = Chain(rpc_uri)
+# rpc_uri = "http://uri.to.sepolia.testnet"
 
 # Set the address of the hyperdrive pool
 # hyperdrive_address = "0x0000000000000000000000000000000000000000"
 
 # Alternatively, look up the list of registered hyperdrive pools
 # This is the registry address deployed on sepolia.
-registry_address = "0xba5156E697d39a03EDA824C19f375383F6b759EA"
-
-hyperdrive_address = Hyperdrive.get_hyperdrive_addresses_from_registry(chain, registry_address)["sdai_14_day"]
+# registry_address = "0xba5156E697d39a03EDA824C19f375383F6b759EA"
+#
+# hyperdrive_address = Hyperdrive.get_hyperdrive_addresses_from_registry(chain, registry_address)["sdai_14_day"]
+
+# For this example, we launch a chain and local hyperdrive, and set the rpc_uri and hyperdrive address from these.
+local_chain = LocalChain()
+local_hyperdrive = LocalHyperdrive(local_chain)
+rpc_uri = local_chain.rpc_uri
+hyperdrive_address = local_hyperdrive.hyperdrive_address
+
+# Need to set different db port here to avoid port collisions with local chain
+chain = Chain(rpc_uri, config=Chain.Config(db_port=1234))
 hyperdrive_config = Hyperdrive.Config()
 hyperdrive_pool = Hyperdrive(chain, hyperdrive_address, hyperdrive_config)
 
@@ -41,7 +49,7 @@
 # agent object using the same private key, but the underlying wallet
 # object would then be out of date if both agents are making trades.
 # TODO add registry of public key to the chain object, preventing this from happening
-hyperdrive_agent0 = hyperdrive_pool.init_agent(
+agent0 = hyperdrive_pool.init_agent(
     private_key=private_key,
     policy=PolicyZoo.random,
     # The configuration for the underlying policy
@@ -51,26 +59,37 @@
 # %%
 # We expose this function for testing purposes, but the underlying function calls `mint` and `anvil_set_balance`,
 # which are likely to fail on any non-test network.
-hyperdrive_agent0.add_funds(base=FixedPoint(100000), eth=FixedPoint(100))
+agent0.add_funds(base=FixedPoint(100000), eth=FixedPoint(100))
 
 # Set max approval
-hyperdrive_agent0.set_max_approval()
+agent0.set_max_approval()
 
 # %%
 
 # Make trades
 # Return values here mirror the various events emitted from these contract calls
 # These functions are blocking, but relatively easy to expose async versions of the
 # trades below
-open_long_event = hyperdrive_agent0.open_long(base=FixedPoint(11111))
-close_long_event = hyperdrive_agent0.close_long(
-    maturity_time=open_long_event.maturity_time, bonds=open_long_event.bond_amount
-)
+open_long_event = agent0.open_long(base=FixedPoint(11111))
+close_long_event = agent0.close_long(maturity_time=open_long_event.maturity_time, bonds=open_long_event.bond_amount)
 
 
 # %%
 random_trade_events = []
 for i in range(10):
     # NOTE Since a policy can execute multiple trades per action, the output events is a list
-    trade_events: list = hyperdrive_agent0.execute_policy_action()
+    trade_events: list = agent0.execute_policy_action()
     random_trade_events.extend(trade_events)
+
+
+# %%
+
+# Analysis
+
+agent_positions = agent0.get_positions()
+agent_trade_events = agent0.get_trade_events()
+
+# %%
+# cleanup
+local_chain.cleanup()
+chain.cleanup()

examples/streamlit_example.py

@@ -0,0 +1,54 @@
+"""Example script to deploy a hyperdrive dashboard."""
+
+# pylint: disable=pointless-statement
+import time
+
+from fixedpointmath import FixedPoint
+
+from agent0 import LocalChain, LocalHyperdrive, PolicyZoo
+
+# The dashboard will run for approximately this many seconds before crashing
+DEMO_NUM_ITERATIONS = 1000
+
+# Initialization
+local_chain_config = LocalChain.Config()
+chain = LocalChain(local_chain_config)
+
+initial_pool_config = LocalHyperdrive.Config()
+hyperdrive0 = LocalHyperdrive(chain, initial_pool_config)
+
+agent0 = hyperdrive0.init_agent(
+    base=FixedPoint(100000),
+    eth=FixedPoint(100),
+    name="random_bot",
+    # The underlying policy to attach to this agent
+    policy=PolicyZoo.random,
+    # The configuration for the underlying policy
+    policy_config=PolicyZoo.random.Config(rng_seed=123),
+)
+
+initial_pool_config = LocalHyperdrive.Config()
+hyperdrive1 = LocalHyperdrive(chain, initial_pool_config)
+
+agent1 = hyperdrive1.init_agent(
+    base=FixedPoint(100000),
+    eth=FixedPoint(100),
+    name="random_bot",
+    # The underlying policy to attach to this agent
+    policy=PolicyZoo.random,
+    # The configuration for the underlying policy
+    policy_config=PolicyZoo.random.Config(rng_seed=345),
+)
+
+# Run the dashboard in a subprocess.
+# This command should automatically open a web browser that connects.
+chain.run_dashboard(blocking=False)
+
+# Make trades slowly
+for _ in range(DEMO_NUM_ITERATIONS):
+    agent0.execute_policy_action()
+    agent1.execute_policy_action()
+    time.sleep(1)
+
+# Clean up resources
+chain.cleanup()

pyproject.toml

@@ -126,7 +126,7 @@ exclude = [".venv", ".vscode", "docs"]
 
 [tool.isort]
 line_length = 120
-multi_line_output=3
+profile = "black"
 
 [tool.ruff]
 line-length = 120

scripts/run_acquire_data.py

@@ -21,5 +21,5 @@
         # TODO ideally would gather this from the deployer
         start_block=48,
         rpc_uri=rpc_uri,
-        hyperdrive_address=hyperdrive_addr,
+        hyperdrive_addresses=[hyperdrive_addr],
     )