Collaboration with CryptoCrew for guaranteed Relayer SLAs

Governance
1

Hello again Persistence Community,

Persistence Labs is excited to present a proposal for integrating CryptoCrew’s professional IBC Relayer Services into our ecosystem. This proposal outlines the critical role of IBC and ICA relayers in enhancing the interoperability of blockchains within the Cosmos network, how CryptoCrew’s services can significantly benefit the Persistence ecosystem and their proposal/quotation for providing these services.

First, why IBC and ICA Relayers are Crucial:

Enhanced Interoperability: IBC (Inter-Blockchain Communication) relayers play a pivotal role in enabling cross-chain communication and transactions, which are essential for a network like Cosmos that consists of interconnected blockchains.

Improved Efficiency and Reliability: ICA (Interchain Accounts) channels are vital for our pSTAKE liquid staking derivative products. Having a robust IBC relayer like CryptoCrew ensures the high performance and reliability of these channels.

Support in ecosystem expansion: As an expanding ecosystem that is tightly integrated with various other chains, we continue to see higher and higher IBC volumes passing over our relayers (> 10M$ worth of volume in the last 7 days). Up till now, the ecosystem has solely relied on validators who voluntarily provide relaying services and are incentivised by foundation delegations. While this is appreciated and has been effective, it is hard to demand 100% reliability for a service that is voluntarily provided. Therefore, to optimise the relayer service and have guaranteed Service Level Agreements (SLAs) for the ecosystem, Persistence Labs suggests contracting CrytoCrew as a professional service provider for relayer services.

CryptoCrew’s Proposal at a Glance:

Expertise and Track Record: CryptoCrew has a proven history of securing and facilitating transactions across numerous interchain networks, making them a reliable partner for our ecosystem.

Comprehensive Services: Their proposal includes IBC path creation, persistent relaying service, 24/7 monitoring, and incident response, ensuring uninterrupted and efficient operation.

CryptoCrews’s support for Persistence: CryptoCrew has been running a validator node for Persistence since the inception of the network, demonstrating exceptional reliability, responsiveness, and support in various manners.

Quote:

Starting with a 6 months engagement, the proposal contains two parts:

  1. IBC relaying for 6 chains (incl. Transfers & ICA). Channels may vary, but starting with CosmosHub, Osmosis, dYdX, Noble, Kava, and Neutron = 4000$ per month
  2. Gas cost compensation: 300$ per counterparty chain = $1800 per month

Persistence Community Pool Funding Request:

It is suggested to use the 30-day TWAP (close) for the token price (at the time of the proposal going on the chain, following this forum discussion). For reference, this is what it would look like today:

  • Amount in USD: 34,800

  • XPRT 30d TWAP: 0,2652 USD

  • Amount in XPRT: 131,221.72 XPRT

  • 10% safety margin: 13,122.17 XPRT

  • Total request: 144,343.89 XPRT.

  • any unused safety margin amount will be refunded back to the Persistence Community Pool

The full proposal can be found in this proposal document

We invite the community to share their opinions and eventually vote on this proposal. Your feedback is invaluable in making informed decisions that shape the future of Persistence.

Looking forward to your thoughts and participation.

7 Likes
2

Hello Persistence community!

The reliable operation of performant Relayers is crucial to impeccable IBC user experience. At CryptoCrew, we have been focussing on building and maintaining production-grade IBC infrastructure for over 2 years.

We are honored to have been favored as the major IBC Relayer service provider by Persistence Labs, with the objective of enhancing the reliability and performance of ICA (InterChain Accounts) and important IBC-transfer channels, as they are a major part of the core functionality of the pStake and Dexter product suites.

Pending approval by the Persistence governance, CryptoCrew is committed to excellence in execution and support, ensuring reliable solutions that strengthen the Persistence ecosystem.

We power the interchain.

8 Likes
3

I am in complete support of this proposal.
While I believe community pool funds should be spent conservatively, this community pool spend proposal for funding a critical piece of infrastructure is the best use of the CP. I believe this proposal is a huge net positive for the ecosystem. With the ecosystem growing and IBC transactions and volumes expected to grow exponentially, having guarantees for relaying services is extremely important for user onboarding and retention.

Having worked with the Cryptocrew team for a while now, I believe this collaboration will be a very fruitful one.

2 Likes
4

100% in support. A step towards decentralization of app chains and their liveliness. Looking forward to more such deals this cosmos season :slight_smile:

3 Likes
5

Sounds good, with a tight SLA. :+1:
pStake should benefit from this deal.

1 Like
6

Nice proposal. But why CryptoCrew only?
According to our InterScope IBC monitoring Architect Nodes! is #1 in relaying for 2023.
If we talking about ICA transactions, then AUDIT.one and Crosnest are top, and AUDIT.one is most stable ICA relayer.
Maybe we should spread this support to few IBC operators? Decentralization, right? :wink:

3 Likes
7

Very good question and the answer is straightforward: CryptoCrew saw there was a need in the ecosystem and they came up with a proposal. While all other relayer operators are massively appreciated, and should be given a chance, everything starts by taking initiative for the better.

There will be other chains to relay and other opportunities, and I would like to see other validators and stakeholders to reach out with such initiatives, or even directly engage with the community on the forum. That’s why the forum is here. Any other proposal should also not be mutually exclusive, as long as there’s an additional benefit for the community and the value proposition is there, I believe it should be discussed.

2 Likes
8

Why only cryptoctrew? Smarter to do what the Cosmos hub is doing and work with the relayer DAO as a whole @dneorej

9

Great idea. DAO. This is the way :grinning:
CC made a good proposal, but looks like it’s just a base or first step for IBC support project. Is there anybody who is good in DAO creating and managing and can take responsibility for this? We can help this DAO with monitoring ans statistic.

1 Like
10

Persistence Team,

We agree that IBC is critical and require professional support from reliable operators for optimal user experience. Also in agreement that general incentives to run IBC at large scale is typically a significant expense that validators bear in order to support cosmos network. We also recognize CryptoCrew as an exceptional relayer operator with strong track record. I have nothing but huge respect and praise for CryptoCrew :saluting_face:

Now in order to incentivize relayer with SLA and monitoring, I recommend few points:

  • Work with top 2-3 relayer operators for Persistence based on total volume for 2023 (including Architect Nodes).
  • If relayer is willing to support IBC with SLA/monitoring requirements, then give them opportunity to run relayer for channels they can support.
  • If a relayer doesn’t want to support some channels, keep going down the list until you can find someone who can.
  • Offer same terms to all relayers, so it’s fair and competitive. I agree with the terms laid out for Infrastructure cost and tx fees as laid out by CryptoCrew.
  • Don’t put all your eggs in one basket (it’s not about CryptoCrew, they are awesome :slight_smile: , but a good business practice).

In the end, Architect Nodes has been supporting Persistence for a long time as reflected by yearly relayer numbers shared by paranormal here:

We can provide same level of service like redundant nodes in different geographical regions, monitoring/SLA and support as mentioned in original doc and would love to work with Persistence team. Our relayer infrastructure cover over 20 cosmos chains and we are always expanding our coverage. It is about time we recognize effort and time relayers put in order to support IBC, and provide them with required support to sustain their operations.

I look forward to feedback and working with Persistence team.
cc: @dneorej @ccvalidators

Architect

5 Likes
11

CryptoCrew’s proposal is thorough and CLEARLY would be eligible for an SLA, given their track record and infrastructure robustness.

That said, we need a fair and decentralized system where relayers compete and improve their infrastructure to be eligible for SLAs. Architect Node’s solution would be a good point to start - maybe a DAO?

From my point of view, relayer SLAs should be given similar to how Foundation Delegations are:

  1. Establish requirements
  2. Enter into an SLA with the “winners”
  3. Check those requirements continue being met by the SLA receivers.
  4. Continue accepting new relayers that meet the requirements

It’s easier said than done, but a fair and decentralized system would go a long way to establish Persistence’s IBC infrastructure as one of the most robust in the ecosystem.

1 Like
12

Thank you for sharing your insights @paranormal, @CosmonautStakes, and @ArchitectNodes.

The need for such a service as proposed by @ccvalidators is evident, as shown by the overall consensus in feedback. A robust IBC infrastructure is crucial to providing a seamless LSTfi experience on Persistence.

Having gone through the relayer discussion on the Cosmos Hub forum, getting more acquainted with the current status of ICS-29 (fee middleware), looking at other possible relayer incentives options, and reading through everyone’s feedback, a DAO does seem to be a better long-term solution to tackle this problem long term. Perhaps a ‘Persistence Infra DAO (PID)’ can be formed to run, expand, and bolster the core-1 infrastructure (not limited to relayer operations) actively. There should be a separate, more in-depth discussion around such a decision about the ramifications of PID, like responsibilities, members, budget, code of conduct, etc.

Considering all feedback, I suggest we ask @ccvalidators to post a revised 3-month proposal to kickstart an SLA agreement with CryptoCrew as a starting point (a short period to review its effects and benefits and involve more validators later). Simultaneously, all interested validators should come together, discuss a long-term solution for relaying operations (or beyond that) on Persistence, and proceed with the next steps with governance as required.

This way we have a short term fix (at least for Q1), and then hopefully a long term solution ready by Q2. Happy to hear further thoughts and then get this proposal on chain so that it can be in place from 1st of Jan onwards, if approved of course.

2 Likes
13

You sound like the CC proposal is already a done deal even though feedback from outside the team requests otherwise. A short period to not support other relayers. Hard to trust the persistence team if you push this.

14

I personally look at it as a quick fix to set up robust relaying to improve the Persistence IBC Ecosystem (crucial for USDC and USDT inflow, stkDYDX when it launches, ATOM and OSMO liquid staking, and stkASSET integrations such as the one on Neutron) while the community comes together to formulate a long-term focused, more validator-inclusive plan.

The feedback and interest from others also point toward a long-term resolution. It may take time to come up with all the nitty-gritty of the matter, and the community (including anyone who shares ideas on this post) should think of HOW we can do this and make it really scalable. Sharing high-level ideas is one thing, making a detailed fully functional proposal come to live is a different level.

@ccvalidators recognized the need for such an improvement (started from a validators discussion channel on the Persistence Discord) and came up with this proposal. This is not a done deal, as the ultimate power lies with XPRT governance if and when CryptoCrew decides to put this on the chain.

2 Likes
15

TLDR:

  • CC supports the notion of creating an Operations DAO to handle service contracts
  • to enhance service quality in the interim, CC proposes to limit the term of the service agreement to 3 months until the Operations DAO is established and functional
  • CC seeks feedback and plans to propose on-chain by end of this week

–

Hello Persistence community,

first of all thank you for your active participation and feedback, it is great to see such a high level of interest in the topic. We’d also like to acknowledge the hard work of every Relayer team. Relaying services in Cosmos have been heavily dependent on personal investment of time and money to date. To achieve better service quality, it is important to secure funding for professional relaying teams.

We generally favor the notion to form an Infrastructure Operations DAO to handle service contracts with various providers. This DAO would be able to maintain long-term IBC Relaying agreements, but also other relevant infrastructure services like RPC nodes, snapshot services, or similar. We will also be happy to contribute to this Operations DAO in the future.

To support the notion while still being able to secure high service quality in the interim, we propose to limit the term outlined in the original proposal to 3 months. This should give the Persistence community & governance enough time to form a robust DAO capable of handling and overseeing these agreements.

A few things we would like to expand on:

  • Why CryptoCrew?

Although there are quite a few teams offering IBC Relaying services in the Cosmos, our expertise, infrastructure to hand, and tooling position us uniquely for this project. Relaying goes beyond running nodes. Continuous monitoring of the state of the Blockchains and the Relayer process is crucial to provide impeccable service coverage. We have refined our internal processes and tooling for over 2 years, it is only because of this IP that we are able to guarantee for the proposed SLAs. In addition our team is very experienced with the peculiarities of the IBC protocol and has successfully helped to resolve a number of issues that were impacting Relayer operations on chains like Osmosis and Neutron.

  • Who else should get the opportunity?

We think that other service providers are responsible to come forward with SLAs that suit their operation, this is also the reason why we haven’t included a partnering provider “as a package deal”. We can only guarantee for our own service quality. Each agreement must be independently reviewed and overseen. We acknowledge that this goes beyond the capabilities of generalized Cosmos governance, hence we support the establishment of an Operations DAO to ensure an optimal service in the long term.

  • Aren’t more Relayers better for decentralization?

Absolutely, experience has shown that a good number to aim for are 2-3 professional Relayer teams per important channel (depending on traffic). These teams can share responsibilities over many channels.

  • But CC aren’t ranked #1 on the Paranormal Bros dashboard?

It is in fact very hard to quantify relayer efficiency, as a higher number of effected packets relayed doesn’t nessecarily equate to higher service quality. The best metric we can think of is something similar to what Skip have recently come up with: The ability to successfully handle a packet within a given timeframe. Unfortunately we are yet lacking the (onchain) infrastructure to handle service agreements in a fully permissionless matter. Having that said, CC have not rolled out full support on the Persistence channels yet, through this proposal we will be able to significantly increase our coverage.

  • Where does this lead to in the future / what’s the general plan & what are the next steps?

The IBC-go team are in the progress of finalizing the last modifications needed for channel upgradability to be rolled out in production, which is the last puzzle piece needed for ICS-29 / or possible other on-chain relayer incentivization mechanisms to work with existing channels. A current ETA is as early as Q1/2024. Of course this will need to be implemented on IBC connected chains, but also it will need some updates to the relayer software.

Until a true permissionless relayer market has been established we believe that it is advisable to establish service agreements with trusted providers.

–

If the community sentiment agrees with the updates to our proposal we are happy to go ahead and will aim for an on-chain proposal by end of this week.

Best regards,
CryptoCrew

1 Like
16

As @dneorej mentioned, this is not a done deal. Validators and delegators will eventually be voting on the proposal.

Jeroen and I are both aligned that this is something critical. Cosmos is seeing a surge in user activity, and the time to be prepared to handle high traffic was yesterday. Decentralised processes are effective but generally slow, requiring careful thought processes and planning (as is evident from this discussion).

I think a DAO would be great, but it also requires a completely independent discussion. As for the merits of this proposal, I think especially with @ccvalidators proposing to limit this proposal to a 3-month term, the proposal warrants a shot at on-chain governance.

17

My opinion is that CC is a very professional relaying crew, and this 3-month “testing phase” will set the standards for future DAO requirements. Let’s support the proposal and see how everything goes in these 3 months, and in parallel we will start developing the DAO. It would be nice if someone took responsibility for managing the creation of this DAO. Unfortunately, we are not great experts in this.

18

Not quite like this.
First, this proposal is about grant. So we are talking about money. More relaying, more networks, more txs - more spending (hardware and fees). You are asking to compensate this. But other operators are in expenses too. And compensation they get is foundation delegation.
Second, if we are talking about quality, I don’t think that #1/2/3 operator is skipping or delaying packets more that you. But I agree that we have to develop some standards to create metrics (we will definitely apply them in our dashboard as soon as the standards are developed).
For example, we see waves with zero data of txs being sent by some operators. This means that their relayer did not work for several days/weeks. This is an indicator of a decrease in quality.

19

We’d like to thank everybody for the provided feedback. The recommended changes have been added to the mandate. The proposal is now live for on-chain voting.

20

Hello Persistence Community!

Following the passing of Proposal 63 CryptoCrew would like to express gratitude for the approval of the proposed mandate.

We have liquidated the requested funds and returned the full 10% safety-margin amount to the community pool in TXID 33A609067926C03B531A37AA6DF9D796BD44E070F522CE564686A1A8E5096770 minus 0.1XPRT payed for transaction fees.

Next steps:

  • Deploy extended IBC Relayer Infrastructure on Persistence mainnet
  • Deploy monitoring & alert systems
  • Deploy private IBC metrics dashboard for Persistence Labs
  • Develop & deploy public IBC metrics dashboard (ETA: 01/24)
  • Publish service report at end of mandate term (ETA: 03/24)

Going forward we will keep the community updated about the progress in this forum post.

Best Regards,
CryptoCrew

2 Likes