Hello Persistence Community,
On November 6th (Block #8647535), the Persistence Core-1 chain halted due to an appHash mismatch caused by differing Golang versions used by the validators.
The Persistence Core-1 and validators responded promptly to tackle the issue and resolve them. During the chain revival, some validators got tombstoned due to double-signing.
Stakin, along with other affected validators, is requesting to execute untombstoning (with unslashing) to safeguard the good actors in the ecosystem.
The sequence of events:
- On November 6th, at the Block Height of 8647535, the Persistence Core-1 Chain halted;
- Validators were alerted of the chain halt through their monitoring and alerting system, as well as on community chats;
- Validators started joining forces to understand the cause of the chain halt and preparing to restart the chain with other validators and the Core-1 team;
- Instructions were shared a few hours after the chain halt, which included the following:
- Compiling 4.0.0 binaries with go 1.19.3
- Restarting the chain using one of the provided snapshots
- The Core-1 chain team and validators shared the first set of instructions. After feedback, the instructions were further refined, including a backup of the validator state file. The network halt is a matter of sensitivity and urgency, so some validators acted on the initial instructions.
- To ensure a restart of the chain as soon as possible, the community of validators proceeded with the upgrade;
- Unfortunately, as the chain restarted, some validators double-signed as a result of following the restart instructions, including binary and snapshots, and not correctly backing up priv_validator_state.json keys.
Affected validator nodes:
| Validator Moniker | Operator Address | Staked amount |
|---|---|---|
| Hashquark | persistencevaloper1gydvxcnm95zwdz7h7whpmusy5d5c3ck0p9muc9 | 1,814,872 XPRT |
| Fox 99 | persistencevaloper1y2svn2zvc0puv3rx6w39aa4zlgj7qe0fz8sh6x | 1,499,287 XPRT |
| Smart Stake | persistencevaloper1qtggtsmexluvzulehxs7ypsfl82yk5aznrr2zd | 921,276 XPRT |
| Stakin | persistencevaloper1xykmyvzk88qrlqh3wuw4jckewleyygupsumyj5 | 495,871 XPRT |
| KuCoin | persistencevaloper18qgr8va65a50sdmp2yuy4y8w9p4pa2rf76mvmm | 146,481 XPRT |
| TOTAL | 4,877,787 XPRT |
Why support the affected validators and their delegators?
- The tombstone penalty was initially designed as a severe sanction for malicious behaviors, which we believe was not intended in this case.
- The validators did not intend to double-sign wilfully but rather contributed to restarting the network in the interest of the delegators and community.
- Some affected validators have impeccable governance participation records (100%) - Stakin, Smart Stake, Hashquark. Also, they have been part of the ecosystem since the launch.
- Passive delegators who have trusted to stake XPRT to these validators may take time to realize that these validators are now jailed and not producing rewards. It will negatively impact these delegators through missed staking rewards when they realize this.
- Staking is meant to be a passive activity. Based on observations from individual validator shutdown events, more than 50% of delegates take six months - one year to realize that the validator is no longer active.
- Not compensating the validators and delegators for this unfortunate event will create an unfair precedent and disincentivizes validators to join emergency upgrades in the future. E.g., Why would a validator take any potential risk to restart a network when they could participate after the restart and not risk slashing or tombstoning?
Detailed post-mortem by the Core-1 chain team: incident-reports/06-nov-2022_V4_upgrade_halt.md at main · persistenceOne/incident-reports · GitHub
What do we propose?
Untombstoning with Unslashing:
- Delegators’ funds that got slashed (5%) get reinstated
- Delegators don’t have to redelegate or unbond their remaining stake
- Affected validators can get back into the active set
- This action requires a chain upgrade
Precedents for similar incidents and resolutions:
- Secret network: Revert CoS Tombstone by Cashmaney · Pull Request #1178 · scrtlabs/SecretNetwork · GitHub
- Chihuahua: Comparing v3.0.0...v3.1.0 · ChihuahuaChain/chihuahua · GitHub
- Governance Proposal: Mintscan
Stakin is working closely with the affected validators to ensure that the proposed upgrade is well-tested and that the implementation follows the highest standards. We are open to suggestions/feedback and are willing to address any concerns/questions.