Proposal for untombstoning with unslashing of Nov. 6th validator double-signing

Hello Persistence Community,

On November 6th (Block #8647535), the Persistence Core-1 chain halted due to an appHash mismatch caused by differing Golang versions used by the validators.

The Persistence Core-1 and validators responded promptly to tackle the issue and resolve them. During the chain revival, some validators got tombstoned due to double-signing.

Stakin, along with other affected validators, is requesting to execute untombstoning (with unslashing) to safeguard the good actors in the ecosystem.

The sequence of events:

  • On November 6th, at the Block Height of 8647535, the Persistence Core-1 Chain halted;
  • Validators were alerted of the chain halt through their monitoring and alerting system, as well as on community chats;
  • Validators started joining forces to understand the cause of the chain halt and preparing to restart the chain with other validators and the Core-1 team;
  • Instructions were shared a few hours after the chain halt, which included the following:
    • Compiling 4.0.0 binaries with go 1.19.3
    • Restarting the chain using one of the provided snapshots
    • The Core-1 chain team and validators shared the first set of instructions. After feedback, the instructions were further refined, including a backup of the validator state file. The network halt is a matter of sensitivity and urgency, so some validators acted on the initial instructions.
  • To ensure a restart of the chain as soon as possible, the community of validators proceeded with the upgrade;
  • Unfortunately, as the chain restarted, some validators double-signed as a result of following the restart instructions, including binary and snapshots, and not correctly backing up priv_validator_state.json keys.

Affected validator nodes:

Validator Moniker Operator Address Staked amount
Hashquark persistencevaloper1gydvxcnm95zwdz7h7whpmusy5d5c3ck0p9muc9 1,814,872 XPRT
Fox 99 persistencevaloper1y2svn2zvc0puv3rx6w39aa4zlgj7qe0fz8sh6x 1,499,287 XPRT
Smart Stake persistencevaloper1qtggtsmexluvzulehxs7ypsfl82yk5aznrr2zd 921,276 XPRT
Stakin persistencevaloper1xykmyvzk88qrlqh3wuw4jckewleyygupsumyj5 495,871 XPRT
KuCoin persistencevaloper18qgr8va65a50sdmp2yuy4y8w9p4pa2rf76mvmm 146,481 XPRT
TOTAL 4,877,787 XPRT

Why support the affected validators and their delegators?

  • The tombstone penalty was initially designed as a severe sanction for malicious behaviors, which we believe was not intended in this case.
  • The validators did not intend to double-sign wilfully but rather contributed to restarting the network in the interest of the delegators and community.
  • Some affected validators have impeccable governance participation records (100%) - Stakin, Smart Stake, Hashquark. Also, they have been part of the ecosystem since the launch.
  • Passive delegators who have trusted to stake XPRT to these validators may take time to realize that these validators are now jailed and not producing rewards. It will negatively impact these delegators through missed staking rewards when they realize this.
    • Staking is meant to be a passive activity. Based on observations from individual validator shutdown events, more than 50% of delegates take six months - one year to realize that the validator is no longer active.
  • Not compensating the validators and delegators for this unfortunate event will create an unfair precedent and disincentivizes validators to join emergency upgrades in the future. E.g., Why would a validator take any potential risk to restart a network when they could participate after the restart and not risk slashing or tombstoning?

Detailed post-mortem by the Core-1 chain team: incident-reports/06-nov-2022_V4_upgrade_halt.md at main · persistenceOne/incident-reports · GitHub

What do we propose?

Untombstoning with Unslashing:

  • Delegators’ funds that got slashed (5%) get reinstated
  • Delegators don’t have to redelegate or unbond their remaining stake
  • Affected validators can get back into the active set
  • This action requires a chain upgrade

Precedents for similar incidents and resolutions:

Stakin is working closely with the affected validators to ensure that the proposed upgrade is well-tested and that the implementation follows the highest standards. We are open to suggestions/feedback and are willing to address any concerns/questions.

2 Likes

Smart Stake was impacted by this incident as well and fully supports untombstoning and unslashing all affected validators and their delegates. Collectively there are more than 1000 unique delegates (5%+ of all Persistence delegates) that got penalized in the incident. Given that the incident happened while validators were trying to help recover the network and there was no malicious intent involved, making 1000+ delegates whole is the right thing. Looking forward to a quick resolution.

1 Like

I just spent 2 days trying to figure out why and how my smart stake was inactive. No where on the XPRT website is there a link to get help. (That is to talk to someone)… I ended up on Telegram and talked to someone who wanted me to log into some site that wanted all my info including my key phrase.
I know I’m old (79yrs) an not the sharpest tool in the shed. So I told him there is no way I’m giving up key phrase. So I withdrew my XPRT out of Smart Stake and now waiting to redelegate it.
The problem from where I sit is:
Why isn’t there a link that can be controlled buy you to be safe and get accurate information?
Why wasn’t I notified?
Why when I see a red X buy Smart Stake I have no fucking idea what it means?
When you say ( Persistence Core-1 chain halted due to an appHash mismatch caused by differing Golang versions used by the validators) I have no idea of what you are talking about.
There are going to be alot more people like me that aren’t techie BUT HAVE MONEY and if you can’t communicate with them (or make things easier) for people to understand I fear you will be leaving a lot
on the table,

Hi there, if you go to the end of website there is option of follow us and all the links to get in touch with us can be found like telegram, Discord, reddit etc.
For this we ask users to follow our announcements channel or Twitter to get all updates of latestest happening on Persistence.

Our telegram community channel can be found below.

Smart stake validator is jailed currently.

So it doesnt give you rewards for staking.
Best to redelegate it to other validator using high gas fees.

On above website you can find information on inactive and active validators.