Collaboration with CryptoCrew for guaranteed Relayer SLAs

Hey everyone, we’re happy to provide the following progress update.

Our extended relayer systems have been successfully deployed on the following chains:

Chain name Chain ID
Persistence core-1
Cosmoshub osmoshub-4
Osmosis osmosis-1
dYdX chain dydx-mainnet-1
Kava kava_2222-10
Neutron neutron-1

image

We have set up dashboards & alerting systems and have shared them with the Persistence Labs team. We are relaying and monitoring the state of all open channels between the chains above.

After fine-tuning our configurations we’re now satisfied with the stability of all involved systems (relayer instances, RPC nodes, metrics exporters, metrics-aggregation-db, grafana-frontend & grafana-oncall). Some of the insights / learnings we can share until now:

  • IBC traffic & relayer coverage: given the reduced number of total packets at this moment, relayer coverage on most of the main channels suffices and packets are passing regularly.
  • Operational complexity due to the nature of how Interchain Accounts are used on Persistence: Due to peculiarities with how ICA channels are created on Persistence (they are added programmatically), relayers should run ICA-wildcards in their channel-filter configurations and frequently restart relayer instances to fetch the newly added channels.
  • 1 packet is pending on the Osmosis transfer channel due to a bug in packet-forward-middleware. The fix for this issue has not yet been included in the Persistence LSM fork of ibc-apps. This has been identified by our operators, a fix has been suggested to the Persistence developers. It will be possible to time out the packet and clear the channel after the next Persistence mainnet upgrade.
  • We are working in close collaboration with the Persistence Labs and Dexter teams to keep up improved service reliability on all channels.

We will use yesterday (2024/01/09) as our mandate’s effective date. A more detailed report will be published after the service term has concluded in three months.

Kind Regards,
CryptoCrew

[Service Report]

Dear Persistence Community, as the service term has completed on 2024/04/09, our IBC relaying service has hereby completed.

We’re happy to share to following Service Report.


Report Date: 2024/04/15
Service: ibc-persistence
Service Repository: cc-reports/ibc-persistence at main · cryptocrew-validators/cc-reports · GitHub

As per request by the Persistence Labs team, the scope of counterparty chains has been extended to include strategically relevant channels for pStake ICA host chains. No additional service fees have been charged for the additional counterparties.

Relayer Accounts

Total counterparty chains: 10

address chain
persistence1gx3gzk6fdxswnuqkhn6zfsuwhfefxfu9f72kz9 core-1
osmo1gx3gzk6fdxswnuqkhn6zfsuwhfefxfu90fl46n osmosis-1
cosmos1gx3gzk6fdxswnuqkhn6zfsuwhfefxfu98jv9vp cosmoshub-4
dydx1gx3gzk6fdxswnuqkhn6zfsuwhfefxfu9wtzpvk dydx-mainnet-1
neutron1gx3gzk6fdxswnuqkhn6zfsuwhfefxfu9rd98kx neutron-1
noble1gx3gzk6fdxswnuqkhn6zfsuwhfefxfu903ed50 noble-1
kava1gx3gzk6fdxswnuqkhn6zfsuwhfefxfu9m8cc6x kava_2222-10
kujira1gx3gzk6fdxswnuqkhn6zfsuwhfefxfu9k6wapt kaiyo-1
stars1gx3gzk6fdxswnuqkhn6zfsuwhfefxfu9nwmc8s stargaze-1
chihuahua1jjchjlpvq9wepzay3gthqvsszrty00m82pynkr chihuahua-1
agoric1jjchjlpvq9wepzay3gthqvsszrty00m8mftz8h agoric-3

Statistics

To actively add newly created ICA channels we perodically fetch the IBC channel state on the controller chain.

Total number of channel pairs 43
Total ICA channels 30
Total transfer channels 13
Total send packet events 64520
Total send packet events handled 21623
Total acknowledgement events 52554
Total acknowledgement events handled 16223
Average handled rate 32.3 %

Relayer Performance Dashboard (TBC)

The development of our IBC relayer performance dashboard has been finalized and the dashboard has been shared for closed testing. We’ll be publishing the final version as soon as last tests have been completed (ETA: 2024/05)

Latest previews:



Conclusion

Overall service coverage has been good with very few issues reported by users. There have been some challenges with automatically adding new ICA channels which are being addressed by engineers of the CC and Persistence Labs teams.

Continuation of Services until 2024/06

We have been asked by the Persistence Labs team to proactively continue our services for an extended period of 2 months (completion by 2024/06/09). In this timeline it is planned to set up a Infrastructure DAO to handle upcoming service contracts.

The extension of our Relayer Services will be subject to another governance proposal soon to be posted on chain.

1 Like