Hey everyone, we’re happy to provide the following progress update.
Our extended relayer systems have been successfully deployed on the following chains:
| Chain name |
Chain ID |
| Persistence |
core-1 |
| Cosmoshub |
osmoshub-4 |
| Osmosis |
osmosis-1 |
| dYdX chain |
dydx-mainnet-1 |
| Kava |
kava_2222-10 |
| Neutron |
neutron-1 |
We have set up dashboards & alerting systems and have shared them with the Persistence Labs team. We are relaying and monitoring the state of all open channels between the chains above.
After fine-tuning our configurations we’re now satisfied with the stability of all involved systems (relayer instances, RPC nodes, metrics exporters, metrics-aggregation-db, grafana-frontend & grafana-oncall). Some of the insights / learnings we can share until now:
- IBC traffic & relayer coverage: given the reduced number of total packets at this moment, relayer coverage on most of the main channels suffices and packets are passing regularly.
- Operational complexity due to the nature of how Interchain Accounts are used on Persistence: Due to peculiarities with how ICA channels are created on Persistence (they are added programmatically), relayers should run ICA-wildcards in their channel-filter configurations and frequently restart relayer instances to fetch the newly added channels.
- 1 packet is pending on the Osmosis transfer channel due to a bug in
packet-forward-middleware. The fix for this issue has not yet been included in the Persistence LSM fork of ibc-apps. This has been identified by our operators, a fix has been suggested to the Persistence developers. It will be possible to time out the packet and clear the channel after the next Persistence mainnet upgrade.
- We are working in close collaboration with the Persistence Labs and Dexter teams to keep up improved service reliability on all channels.
We will use yesterday (2024/01/09) as our mandate’s effective date. A more detailed report will be published after the service term has concluded in three months.
Kind Regards,
CryptoCrew
2 Likes
[Service Report]
Dear Persistence Community, as the service term has completed on 2024/04/09, our IBC relaying service has hereby completed.
We’re happy to share to following Service Report.
Report Date: 2024/04/15
Service: ibc-persistence
Service Repository: cc-reports/ibc-persistence at main · cryptocrew-validators/cc-reports · GitHub
As per request by the Persistence Labs team, the scope of counterparty chains has been extended to include strategically relevant channels for pStake ICA host chains. No additional service fees have been charged for the additional counterparties.
Relayer Accounts
Total counterparty chains: 10
| address |
chain |
| persistence1gx3gzk6fdxswnuqkhn6zfsuwhfefxfu9f72kz9 |
core-1 |
| osmo1gx3gzk6fdxswnuqkhn6zfsuwhfefxfu90fl46n |
osmosis-1 |
| cosmos1gx3gzk6fdxswnuqkhn6zfsuwhfefxfu98jv9vp |
cosmoshub-4 |
| dydx1gx3gzk6fdxswnuqkhn6zfsuwhfefxfu9wtzpvk |
dydx-mainnet-1 |
| neutron1gx3gzk6fdxswnuqkhn6zfsuwhfefxfu9rd98kx |
neutron-1 |
| noble1gx3gzk6fdxswnuqkhn6zfsuwhfefxfu903ed50 |
noble-1 |
| kava1gx3gzk6fdxswnuqkhn6zfsuwhfefxfu9m8cc6x |
kava_2222-10 |
| kujira1gx3gzk6fdxswnuqkhn6zfsuwhfefxfu9k6wapt |
kaiyo-1 |
| stars1gx3gzk6fdxswnuqkhn6zfsuwhfefxfu9nwmc8s |
stargaze-1 |
| chihuahua1jjchjlpvq9wepzay3gthqvsszrty00m82pynkr |
chihuahua-1 |
| agoric1jjchjlpvq9wepzay3gthqvsszrty00m8mftz8h |
agoric-3 |
Statistics
To actively add newly created ICA channels we perodically fetch the IBC channel state on the controller chain.
|
|
| Total number of channel pairs |
43 |
| Total ICA channels |
30 |
| Total transfer channels |
13 |
| Total send packet events |
64520 |
| Total send packet events handled |
21623 |
| Total acknowledgement events |
52554 |
| Total acknowledgement events handled |
16223 |
| Average handled rate |
32.3 % |
Relayer Performance Dashboard (TBC)
The development of our IBC relayer performance dashboard has been finalized and the dashboard has been shared for closed testing. We’ll be publishing the final version as soon as last tests have been completed (ETA: 2024/05)
Latest previews:
Conclusion
Overall service coverage has been good with very few issues reported by users. There have been some challenges with automatically adding new ICA channels which are being addressed by engineers of the CC and Persistence Labs teams.
Continuation of Services until 2024/06
We have been asked by the Persistence Labs team to proactively continue our services for an extended period of 2 months (completion by 2024/06/09). In this timeline it is planned to set up a Infrastructure DAO to handle upcoming service contracts.
The extension of our Relayer Services will be subject to another governance proposal soon to be posted on chain.
3 Likes
Thank you for providing a comprehensive overview of the analytics. The infrastructure has definitely become more stable and mature under this service contract.
We are interested in extending this contract for an additional two months (subject to governance). During this period, we plan to establish an infrastructure DAO to manage all similar contracts in the future so that all the service providers can be included. I will share the setup structure & details in the coming weeks.
Note: Following our discussion with CC, they have agreed to extend the contract without any increase in the funding amount. The amount will be prorated based on the terms of the previous contract for the next two months.
1 Like
Playing a bit of devils advocate here; are IBC relayers also included in the Foundation Delegation Program?
Does this funding mean that CC is excluded from earning a part of the Delegation Program for the Relayer Service that they deliver? Since it is already funded via the Community Pool. It would not be fair if it was also funded via the Delegation Program.
(note: I do not run relayers, so it does not involve my own validator. But I can surely imagine from the point of view from other validators that it should be either CP-funding or Delegation-funding, not both.)
1 Like
CC included in FDP: ✅ CryptoCrew Validators #IBCgang stats | Persistence Monitoring by Paranormal Bros.
2 080 356,53 tokens.
But to maintain the infrastructure and management, they need more finance, so they make this offer.
P.S.: I still agree that this proposal is not looks very nice in relation to other IBC operators. But we agreed to support the original proposal as a test of such interaction. On the proposal to extend the period, we voted Abstain, because our point of view is that the test has passed, it’s time to start the DAO and finance all IBC operators.
2 Likes
That’s a fair point @LeonoorsCryptoman.
For other validators, relayers will be taken under account. However for CC we will not provide the delegation related to IBC relayers till the time the current contract is active.
Note: CC should still be eligible for the other contributions they provide to the network which are mentioned in this doc.
2 Likes
The current delegation granted to CC was established during the FDP round 2 last year. Since then, no additional rounds have taken place.
To be fair, we ended up expanding the scope of the initial contract, and CC did not charge extra. They also agreed to include this expanded scope in the extension contract.
I recognize that this could be seen as unfair to some validators who manage numerous relayers for the network. Thus, forming a DAO seems like a reasonable solution.
I’ll start a new forum discussion on the infrastructure DAO by early next week, allowing all validators to collaborate on creating a sensible DAO structure.
1 Like
Engaging CryptoCrew’s professional Inter-Blockchain Communication (IBC) relay services will provide significant technical advantages and contribute to the growth of the Persistence Labs ecosystem.
- Enhanced Reliability and Performance: Shifting from volunteer relayers to CryptoCrew’s professional service ensures more reliable and stable operation of ICA channels, critical for Persistence Labs’ pSTAKE liquid staking derivative products. This translates to lower downtime risks and improved overall transaction efficiency.
- Improved Scalability: As the Persistence Labs ecosystem expands and integrates with new networks, IBC transaction volumes are soaring. CryptoCrew possesses the expertise to handle these increasing volumes and guarantee seamless relay operation.
- Guaranteed Service Level Agreements (SLAs): A contract with CryptoCrew will establish clear SLAs, guaranteeing a specific level of relay uptime and performance. This boosts reliability for developers and users building applications on Persistence.
- Extended Support: CryptoCrew offers 24/7 monitoring and incident response, allowing Persistence Labs to proactively identify and address any relay issues, minimizing user impact.
Overall, implementing CryptoCrew’s services will establish a more robust, scalable, and supported IBC relay infrastructure, fostering Persistence Labs ecosystem growth in the following ways:
- Attracting New Projects: Increased reliability and performance will entice more developers to build decentralized applications (dApps) on Persistence, expanding the ecosystem.
- Enhanced User Experience: More stable and efficient IBC relaying translates to smoother operation of existing Persistence Labs dApps and products, improving the overall user experience.
- Increased Trust: Leveraging a professional IBC relay service strengthens trust in the Persistence Labs ecosystem among users, developers, and investors.
Investing in a reliable IBC relay infrastructure with CryptoCrew is a crucial step towards ensuring the long-term growth and success of the Persistence Labs ecosystem. Hence I am fully in support of this!
1 Like
Totally agreed on this. Agreements for the validator should not intervene with potential delegations for other services delivered.
1 Like
