Zellic Official Statement — 16 May 2022
UPDATE 18:44 UTC 18th May, 2022: Clarified the market mechanics
UPDATE 26th May, 2022: Drift Protocol has published an official post-mortem analysis. It is thorough and well-reasoned. We recommend all Drift users read this post-mortem.
On 11 May 2022, trading on Drift Protocol was halted amidst unprecedented market volatility, especially in the LUNA and Terra markets. Drift Protocol is a decentralized perpetual futures platform which allows users to take out long and short positions against certain tokens, including LUNA. The exchange was halted when the protocol suffered larger withdrawals than expected to protect the insurance fund and protocol funding payments. The funding payments were paid from a unified vault which also serviced users’ requests to withdraw their gains. As users raced to withdraw gains amidst turbulent markets, the funding payments were jeopardized, endangering the protocol’s incentivization structures.
We attempted to contact Drift’s team on May 11th, May 15th, and May 16th. Understandably, Drift’s team was preoccupied remediating the incident and settling users’ positions. We applaud the Drift team for their continued commitment to their fiduciary responsibilities and users’ safety. With that said, we were able to discuss the incident with them on May 16th. Following our discussion, we have prepared the following postmortem analysis of the incident.
In this analysis, we seek to answer the following questions:
- Why was the market halted?
- How did a solvency issue arise?
- What should be done to avoid similar situations in the future?
- How can Zellic improve its auditing methodology and procedures?
In a perpetuals future market, users may take out positions in a certain token. A position may either be short or long. On Drift, a short position is equivalent to a bet that the price of the token will decrease in the future. Conversely, a long position is equivalent to a bet that the token’s price will increase. A position may also be leveraged. When a position is leveraged, the gains and losses of the position are amplified by a multiplier. Because leverage creates opportunities for large gains and losses, positions are often required to be collateralized, meaning a user deposits money to back up their position. If the user “loses the bet”, and the position goes underwater, it will be liquidated, and the collateral will be forfeited.
For example, suppose you take out a 5x leveraged LUNA long position. To open the position, you first deposit $1,000 of USDC as collateral. Let’s say LUNA is worth $100 at the start. Thus, your $1,000 of USDC is equivalent to 10 LUNA at your entry price. Additionally, because your position is 5x leveraged, your overall position size is 50 LUNA. Note that you don’t actually own 50 LUNA, you own a position equivalent to 50 LUNA. If LUNA’s price fluctuates, the value of your position would be equivalent to the value of holding 50 LUNA.
Now, let’s say LUNA’s price increases from $100 to $150. Thus, each LUNA has gained $50 in value. Because your position size was 50 LUNA, its value would go up by $2,500, so your position is now worth $3,500. However, if LUNA’s price decreased from $100 to $50, you would lose -$2,500. Now your position would be underwater, with a value of -$1,500. As you can see, leverage is a powerful tool but can also be very dangerous.
In this last example, if your position goes underwater, you owe the exchange $1,500. Unless the exchange has a way to reasonably collect this debt, the exchange will be forced to eat this loss out of their own funds. This is especially true on decentralized exchanges, where users are anonymous and may choose to simply abandon a bad position.
On May 11th, the price of LUNA crashed precipitously. Due to an extreme amount of open short interest in LUNA on Drift, the market conditions uncovered a latent issue in Drift’s market mechanics.
Recall that on Drift, a position is equivalent to a bet that a token will increase or decrease in price. When you close a position (i.e., cash out that bet), where does the money come from? Typically, it comes from other users. When other users lose money on their positions, their losses would make up your winnings. In this case, other users lost their bet, while you won yours. Because it’s an AMM, the price only moves when users make a bet. Thus, in a leverage-free scenario, the winners and losers of all bets always cancel out, meaning winners can always be paid by losers. This is referred to as a zero-sum game. And because all of the positions are fully collateralized, the exchange can always liquidate losers to pay winners.
However, Drift allows users to create leveraged positions. With leverage, the AMM itself is still zero-sum, but the positions are now undercollateralized. When it comes time to pay out winners, the exchange may not be able to cover all of the winnings through liquidations alone. This is because the losers’ collateral may be insufficient compared to the large winnings.
To illustrate, suppose LUNA is trading at $100. Alice opens a large short position. Then, Bob opens a highly-leveraged 20x short position with $100 of collateral. Although Bob’s position is only collateralized by $100 (~1 LUNA), his comparatively large order size of ~20 LUNA moves the AMM price down. Since Alice is short, she can now close her position for a profit. Bob’s position would then be insolvent, and it would be liquidated in order to pay Alice’s gains. But Bob’s position was only collateralized by $100, which may not be enough to pay Alice’s gains. The exchange would have to pay for Alice’s remaining gains out of pocket. This is the fundamental issue.
Under normal market conditions, Bob’s position would have been liquidated far before it became so deeply insolvent. However, due to an unprecedented black swan event which saw the destruction of tens of billions of dollars of value and a 99% drop in LUNA’s price, the market conditions became highly erratic, uncovering this fundamental issue in the market’s structure.
During our audit, as per Drift’s documentation, it was intended for Drift’s insurance fund to cover these leveraged losses. In other words, under typical market conditions, this would be the intended behavior. Without a redesign of the fundamental market structure, there is no straightforward way to offer leveraged perpetuals products while avoiding the risks of undercollateralized leverage. An insurance fund seems necessary for a leveraged perpetuals protocol to function; meanwhile, leverage threatens to make the insurance fund insolvent.
This issue will require an updated design of Drift’s perpetuals market. As clarified below, the issue is not unique to Drift. It affects undercollateralized leverage protocols in general. We collaborated with Drift’s team to brainstorm potential designs for a new market that curtails the risk assumed by offering leverage. As a key takeaway, we learned that generally speaking, one-sided market conditions, such as the ones experienced during the LUNA crash, are incompatible with undercollateralized leverage.
We also emphasize that this solvency problem is not unique to Drift’s platform. In our discussion with the Drift team, we concluded that this issue is likely present in other undercollateralized leveraged perpetuals protocols as well. Drift is actively researching new structures and incentivization mechanisms to address the root, economic cause of the issue. We look forward to new economic designs which enable users to make leveraged trades, while guaranteeing the protocol as a whole remains solvent. This would make perpetuals safer for all users.
As an auditing firm, it was our responsibility to warn Drift of this potential problem. Although insurance fund payouts may be an intended feature, we should have raised our concerns about the exchange bearing the burden leveraged losses, when the market is unbalanced. We should have also further clarified whether this design was not just intended, but desirable. In the future, we will not hesitate to raise any potential concerns we may have of a protocol’s design we suspect to be intentional. We accept it was our fault to have assumed that this behavior was accepted based on the documentation and smart contract code alone—neither Drift nor Zellic anticipated the extraordinary market conditions on May 11th.
One proposed patch is to prevent withdrawals of realized positive PNL. We believe that does not address the root problem. It blocks users from withdrawing their gains from the exchange. In our opinion, this solution is moreso a backstop or circuit-breaker than a fix for the fundamental problem.
In the future, Drift will test their contracts under intense market conditions with an expanded suite of simulations and stress tests. We are disappointed that we were unable to uncover this issue during our fuzz testing. An investigation found that the fuzzing harness was insufficient because it only issued instructions (i.e., transactions) from a single user account. We are developing an updated harness which will exercise the contracts from multiple accounts. Simultaneously, we are also developing a generalized Solana fuzzer.
Because funding payments are critical to the functioning of a perpetual futures market, Drift halted the market when a problem threatening the funding payments was discovered. The issue arose as conditions created an extreme imbalance of withdrawals and deposits. As users raced to withdraw their gains, an issue allowed users to withdraw funds that should have been otherwise dedicated to funding payments.
According to Drift’s statement, the the insurance fund is budgeted among market payouts and other various fees such as funding payments. In essence, many sources of income for the exchange are funneled into a single account, the insurance vault. Within this unified vault account, some funds are earmarked for various purposes, including funding payments. The problem was that users were able to withdraw from the entirety of this vault, rather than just the portion not allocated to fees like funding payments. This was critical as the vault serves funding payments across all of Drift’s markets, not just LUNA. Without funding payments, there would be no stabilizing force balancing the markets.
During our audit, as per Drift’s documentation  , each token’s market has a fee pool. Due to a misunderstanding on our part, we did not realize that the intention was to earmark the fee pool funds within the unified insurance vault for fees like funding payments. The smart contract code for withdrawing collateral   did not take into account the fee pool, and we assumed that this behavior was intentional. We concluded that user PNL withdrawals were to be prioritized, as the documentation did not describe a specific allocation towards funding payments. Nevertheless, we should have clarified this with the Drift team, as funding payments are an important incentivization mechanism in a perpetual futures market. We accept responsibility for this mistake, and in the future, we will not hesitate to clarify questions of intention during an assessment.
In contrast to the previous issue, this issue can be fixed relatively easily by accounting for the earmarked funds during a withdrawal, and only allowing withdrawals within the unallocated portions of the insurance vault. Unfortunately, this issue would not have been possible to uncover through automated tests, as it ultimately arose from a misunderstanding of the intent behind the code, rather than its implementation.
We thank the Drift team for their unwavering commitment to their users. It is important to emphasize that Drift’s team has taken extensive and extraordinary measures to ensure the best possible outcome for their users, even in these extreme circumstances. We also thank the Drift team for their continued responsiveness, timeliness, and transparency.
Notwithstanding this incident, we intend to continue our mission of securing the Solana ecosystem and DeFi at large. This incident was a learning opportunity, and it uncovered several weaknesses in our assessment processes that we will rectify and ameliorate. Lastly, we are grateful to the Drift and Solana communities for their faith in us.