Sounds good, with a tight SLA. 
pStake should benefit from this deal.
1 Like
Nice proposal. But why CryptoCrew only?
According to our InterScope IBC monitoring Architect Nodes! is #1 in relaying for 2023.
If we talking about ICA transactions, then AUDIT.one and Crosnest are top, and AUDIT.one is most stable ICA relayer.
Maybe we should spread this support to few IBC operators? Decentralization, right? 
3 Likes
Very good question and the answer is straightforward: CryptoCrew saw there was a need in the ecosystem and they came up with a proposal. While all other relayer operators are massively appreciated, and should be given a chance, everything starts by taking initiative for the better.
There will be other chains to relay and other opportunities, and I would like to see other validators and stakeholders to reach out with such initiatives, or even directly engage with the community on the forum. That’s why the forum is here. Any other proposal should also not be mutually exclusive, as long as there’s an additional benefit for the community and the value proposition is there, I believe it should be discussed.
2 Likes
Why only cryptoctrew? Smarter to do what the Cosmos hub is doing and work with the relayer DAO as a whole @dneorej
Great idea. DAO. This is the way 
CC made a good proposal, but looks like it’s just a base or first step for IBC support project. Is there anybody who is good in DAO creating and managing and can take responsibility for this? We can help this DAO with monitoring ans statistic.
1 Like
Persistence Team,
We agree that IBC is critical and require professional support from reliable operators for optimal user experience. Also in agreement that general incentives to run IBC at large scale is typically a significant expense that validators bear in order to support cosmos network. We also recognize CryptoCrew as an exceptional relayer operator with strong track record. I have nothing but huge respect and praise for CryptoCrew 
Now in order to incentivize relayer with SLA and monitoring, I recommend few points:
- Work with top 2-3 relayer operators for Persistence based on total volume for 2023 (including Architect Nodes).
- If relayer is willing to support IBC with SLA/monitoring requirements, then give them opportunity to run relayer for channels they can support.
- If a relayer doesn’t want to support some channels, keep going down the list until you can find someone who can.
- Offer same terms to all relayers, so it’s fair and competitive. I agree with the terms laid out for Infrastructure cost and tx fees as laid out by CryptoCrew.
- Don’t put all your eggs in one basket (it’s not about CryptoCrew, they are awesome
, but a good business practice).
In the end, Architect Nodes has been supporting Persistence for a long time as reflected by yearly relayer numbers shared by paranormal here:
We can provide same level of service like redundant nodes in different geographical regions, monitoring/SLA and support as mentioned in original doc and would love to work with Persistence team. Our relayer infrastructure cover over 20 cosmos chains and we are always expanding our coverage. It is about time we recognize effort and time relayers put in order to support IBC, and provide them with required support to sustain their operations.
I look forward to feedback and working with Persistence team.
cc: @dneorej @ccvalidators
Architect
5 Likes
CryptoCrew’s proposal is thorough and CLEARLY would be eligible for an SLA, given their track record and infrastructure robustness.
That said, we need a fair and decentralized system where relayers compete and improve their infrastructure to be eligible for SLAs. Architect Node’s solution would be a good point to start - maybe a DAO?
From my point of view, relayer SLAs should be given similar to how Foundation Delegations are:
- Establish requirements
- Enter into an SLA with the “winners”
- Check those requirements continue being met by the SLA receivers.
- Continue accepting new relayers that meet the requirements
It’s easier said than done, but a fair and decentralized system would go a long way to establish Persistence’s IBC infrastructure as one of the most robust in the ecosystem.
1 Like
Thank you for sharing your insights @paranormal, @CosmonautStakes, and @ArchitectNodes.
The need for such a service as proposed by @ccvalidators is evident, as shown by the overall consensus in feedback. A robust IBC infrastructure is crucial to providing a seamless LSTfi experience on Persistence.
Having gone through the relayer discussion on the Cosmos Hub forum, getting more acquainted with the current status of ICS-29 (fee middleware), looking at other possible relayer incentives options, and reading through everyone’s feedback, a DAO does seem to be a better long-term solution to tackle this problem long term. Perhaps a ‘Persistence Infra DAO (PID)’ can be formed to run, expand, and bolster the core-1 infrastructure (not limited to relayer operations) actively. There should be a separate, more in-depth discussion around such a decision about the ramifications of PID, like responsibilities, members, budget, code of conduct, etc.
Considering all feedback, I suggest we ask @ccvalidators to post a revised 3-month proposal to kickstart an SLA agreement with CryptoCrew as a starting point (a short period to review its effects and benefits and involve more validators later). Simultaneously, all interested validators should come together, discuss a long-term solution for relaying operations (or beyond that) on Persistence, and proceed with the next steps with governance as required.
This way we have a short term fix (at least for Q1), and then hopefully a long term solution ready by Q2. Happy to hear further thoughts and then get this proposal on chain so that it can be in place from 1st of Jan onwards, if approved of course.
2 Likes
You sound like the CC proposal is already a done deal even though feedback from outside the team requests otherwise. A short period to not support other relayers. Hard to trust the persistence team if you push this.
I personally look at it as a quick fix to set up robust relaying to improve the Persistence IBC Ecosystem (crucial for USDC and USDT inflow, stkDYDX when it launches, ATOM and OSMO liquid staking, and stkASSET integrations such as the one on Neutron) while the community comes together to formulate a long-term focused, more validator-inclusive plan.
The feedback and interest from others also point toward a long-term resolution. It may take time to come up with all the nitty-gritty of the matter, and the community (including anyone who shares ideas on this post) should think of HOW we can do this and make it really scalable. Sharing high-level ideas is one thing, making a detailed fully functional proposal come to live is a different level.
@ccvalidators recognized the need for such an improvement (started from a validators discussion channel on the Persistence Discord) and came up with this proposal. This is not a done deal, as the ultimate power lies with XPRT governance if and when CryptoCrew decides to put this on the chain.
2 Likes
TLDR:
- CC supports the notion of creating an Operations DAO to handle service contracts
- to enhance service quality in the interim, CC proposes to limit the term of the service agreement to 3 months until the Operations DAO is established and functional
- CC seeks feedback and plans to propose on-chain by end of this week
–
Hello Persistence community,
first of all thank you for your active participation and feedback, it is great to see such a high level of interest in the topic. We’d also like to acknowledge the hard work of every Relayer team. Relaying services in Cosmos have been heavily dependent on personal investment of time and money to date. To achieve better service quality, it is important to secure funding for professional relaying teams.
We generally favor the notion to form an Infrastructure Operations DAO to handle service contracts with various providers. This DAO would be able to maintain long-term IBC Relaying agreements, but also other relevant infrastructure services like RPC nodes, snapshot services, or similar. We will also be happy to contribute to this Operations DAO in the future.
To support the notion while still being able to secure high service quality in the interim, we propose to limit the term outlined in the original proposal to 3 months. This should give the Persistence community & governance enough time to form a robust DAO capable of handling and overseeing these agreements.
A few things we would like to expand on:
- Why CryptoCrew?
Although there are quite a few teams offering IBC Relaying services in the Cosmos, our expertise, infrastructure to hand, and tooling position us uniquely for this project. Relaying goes beyond running nodes. Continuous monitoring of the state of the Blockchains and the Relayer process is crucial to provide impeccable service coverage. We have refined our internal processes and tooling for over 2 years, it is only because of this IP that we are able to guarantee for the proposed SLAs. In addition our team is very experienced with the peculiarities of the IBC protocol and has successfully helped to resolve a number of issues that were impacting Relayer operations on chains like Osmosis and Neutron.
- Who else should get the opportunity?
We think that other service providers are responsible to come forward with SLAs that suit their operation, this is also the reason why we haven’t included a partnering provider “as a package deal”. We can only guarantee for our own service quality. Each agreement must be independently reviewed and overseen. We acknowledge that this goes beyond the capabilities of generalized Cosmos governance, hence we support the establishment of an Operations DAO to ensure an optimal service in the long term.
- Aren’t more Relayers better for decentralization?
Absolutely, experience has shown that a good number to aim for are 2-3 professional Relayer teams per important channel (depending on traffic). These teams can share responsibilities over many channels.
- But CC aren’t ranked #1 on the Paranormal Bros dashboard?
It is in fact very hard to quantify relayer efficiency, as a higher number of effected packets relayed doesn’t nessecarily equate to higher service quality. The best metric we can think of is something similar to what Skip have recently come up with: The ability to successfully handle a packet within a given timeframe. Unfortunately we are yet lacking the (onchain) infrastructure to handle service agreements in a fully permissionless matter. Having that said, CC have not rolled out full support on the Persistence channels yet, through this proposal we will be able to significantly increase our coverage.
- Where does this lead to in the future / what’s the general plan & what are the next steps?
The IBC-go team are in the progress of finalizing the last modifications needed for channel upgradability to be rolled out in production, which is the last puzzle piece needed for ICS-29 / or possible other on-chain relayer incentivization mechanisms to work with existing channels. A current ETA is as early as Q1/2024. Of course this will need to be implemented on IBC connected chains, but also it will need some updates to the relayer software.
Until a true permissionless relayer market has been established we believe that it is advisable to establish service agreements with trusted providers.
–
If the community sentiment agrees with the updates to our proposal we are happy to go ahead and will aim for an on-chain proposal by end of this week.
Best regards,
CryptoCrew
1 Like
As @dneorej mentioned, this is not a done deal. Validators and delegators will eventually be voting on the proposal.
Jeroen and I are both aligned that this is something critical. Cosmos is seeing a surge in user activity, and the time to be prepared to handle high traffic was yesterday. Decentralised processes are effective but generally slow, requiring careful thought processes and planning (as is evident from this discussion).
I think a DAO would be great, but it also requires a completely independent discussion. As for the merits of this proposal, I think especially with @ccvalidators proposing to limit this proposal to a 3-month term, the proposal warrants a shot at on-chain governance.
My opinion is that CC is a very professional relaying crew, and this 3-month “testing phase” will set the standards for future DAO requirements. Let’s support the proposal and see how everything goes in these 3 months, and in parallel we will start developing the DAO. It would be nice if someone took responsibility for managing the creation of this DAO. Unfortunately, we are not great experts in this.
Not quite like this.
First, this proposal is about grant. So we are talking about money. More relaying, more networks, more txs - more spending (hardware and fees). You are asking to compensate this. But other operators are in expenses too. And compensation they get is foundation delegation.
Second, if we are talking about quality, I don’t think that #1/2/3 operator is skipping or delaying packets more that you. But I agree that we have to develop some standards to create metrics (we will definitely apply them in our dashboard as soon as the standards are developed).
For example, we see waves with zero data of txs being sent by some operators. This means that their relayer did not work for several days/weeks. This is an indicator of a decrease in quality.
We’d like to thank everybody for the provided feedback. The recommended changes have been added to the mandate. The proposal is now live for on-chain voting.
Hello Persistence Community!
Following the passing of Proposal 63 CryptoCrew would like to express gratitude for the approval of the proposed mandate.
We have liquidated the requested funds and returned the full 10% safety-margin amount to the community pool in TXID 33A609067926C03B531A37AA6DF9D796BD44E070F522CE564686A1A8E5096770 minus 0.1XPRT payed for transaction fees.
Next steps:
- Deploy extended IBC Relayer Infrastructure on Persistence mainnet
- Deploy monitoring & alert systems
- Deploy private IBC metrics dashboard for Persistence Labs
- Develop & deploy public IBC metrics dashboard (ETA: 01/24)
- Publish service report at end of mandate term (ETA: 03/24)
Going forward we will keep the community updated about the progress in this forum post.
Best Regards,
CryptoCrew
2 Likes
Hey everyone, we’re happy to provide the following progress update.
Our extended relayer systems have been successfully deployed on the following chains:
| Chain name | Chain ID |
|---|---|
| Persistence | core-1 |
| Cosmoshub | osmoshub-4 |
| Osmosis | osmosis-1 |
| dYdX chain | dydx-mainnet-1 |
| Kava | kava_2222-10 |
| Neutron | neutron-1 |
We have set up dashboards & alerting systems and have shared them with the Persistence Labs team. We are relaying and monitoring the state of all open channels between the chains above.
After fine-tuning our configurations we’re now satisfied with the stability of all involved systems (relayer instances, RPC nodes, metrics exporters, metrics-aggregation-db, grafana-frontend & grafana-oncall). Some of the insights / learnings we can share until now:
- IBC traffic & relayer coverage: given the reduced number of total packets at this moment, relayer coverage on most of the main channels suffices and packets are passing regularly.
- Operational complexity due to the nature of how Interchain Accounts are used on Persistence: Due to peculiarities with how ICA channels are created on Persistence (they are added programmatically), relayers should run ICA-wildcards in their channel-filter configurations and frequently restart relayer instances to fetch the newly added channels.
- 1 packet is pending on the Osmosis transfer channel due to a bug in
packet-forward-middleware. The fix for this issue has not yet been included in the Persistence LSM fork of ibc-apps. This has been identified by our operators, a fix has been suggested to the Persistence developers. It will be possible to time out the packet and clear the channel after the next Persistence mainnet upgrade. - We are working in close collaboration with the Persistence Labs and Dexter teams to keep up improved service reliability on all channels.
We will use yesterday (2024/01/09) as our mandate’s effective date. A more detailed report will be published after the service term has concluded in three months.
Kind Regards,
CryptoCrew
[Service Report]
Dear Persistence Community, as the service term has completed on 2024/04/09, our IBC relaying service has hereby completed.
We’re happy to share to following Service Report.
Report Date: 2024/04/15
Service: ibc-persistence
Service Repository: cc-reports/ibc-persistence at main · cryptocrew-validators/cc-reports · GitHub
As per request by the Persistence Labs team, the scope of counterparty chains has been extended to include strategically relevant channels for pStake ICA host chains. No additional service fees have been charged for the additional counterparties.
Relayer Accounts
Total counterparty chains: 10
| address | chain |
|---|---|
| persistence1gx3gzk6fdxswnuqkhn6zfsuwhfefxfu9f72kz9 | core-1 |
| osmo1gx3gzk6fdxswnuqkhn6zfsuwhfefxfu90fl46n | osmosis-1 |
| cosmos1gx3gzk6fdxswnuqkhn6zfsuwhfefxfu98jv9vp | cosmoshub-4 |
| dydx1gx3gzk6fdxswnuqkhn6zfsuwhfefxfu9wtzpvk | dydx-mainnet-1 |
| neutron1gx3gzk6fdxswnuqkhn6zfsuwhfefxfu9rd98kx | neutron-1 |
| noble1gx3gzk6fdxswnuqkhn6zfsuwhfefxfu903ed50 | noble-1 |
| kava1gx3gzk6fdxswnuqkhn6zfsuwhfefxfu9m8cc6x | kava_2222-10 |
| kujira1gx3gzk6fdxswnuqkhn6zfsuwhfefxfu9k6wapt | kaiyo-1 |
| stars1gx3gzk6fdxswnuqkhn6zfsuwhfefxfu9nwmc8s | stargaze-1 |
| chihuahua1jjchjlpvq9wepzay3gthqvsszrty00m82pynkr | chihuahua-1 |
| agoric1jjchjlpvq9wepzay3gthqvsszrty00m8mftz8h | agoric-3 |
Statistics
To actively add newly created ICA channels we perodically fetch the IBC channel state on the controller chain.
| Total number of channel pairs | 43 |
| Total ICA channels | 30 |
| Total transfer channels | 13 |
| Total send packet events | 64520 |
| Total send packet events handled | 21623 |
| Total acknowledgement events | 52554 |
| Total acknowledgement events handled | 16223 |
| Average handled rate | 32.3 % |
Relayer Performance Dashboard (TBC)
The development of our IBC relayer performance dashboard has been finalized and the dashboard has been shared for closed testing. We’ll be publishing the final version as soon as last tests have been completed (ETA: 2024/05)
Latest previews:
Conclusion
Overall service coverage has been good with very few issues reported by users. There have been some challenges with automatically adding new ICA channels which are being addressed by engineers of the CC and Persistence Labs teams.
Continuation of Services until 2024/06
We have been asked by the Persistence Labs team to proactively continue our services for an extended period of 2 months (completion by 2024/06/09). In this timeline it is planned to set up a Infrastructure DAO to handle upcoming service contracts.
The extension of our Relayer Services will be subject to another governance proposal soon to be posted on chain.
2 Likes
Thank you for providing a comprehensive overview of the analytics. The infrastructure has definitely become more stable and mature under this service contract.
We are interested in extending this contract for an additional two months (subject to governance). During this period, we plan to establish an infrastructure DAO to manage all similar contracts in the future so that all the service providers can be included. I will share the setup structure & details in the coming weeks.
Note: Following our discussion with CC, they have agreed to extend the contract without any increase in the funding amount. The amount will be prorated based on the terms of the previous contract for the next two months.
Playing a bit of devils advocate here; are IBC relayers also included in the Foundation Delegation Program?
Does this funding mean that CC is excluded from earning a part of the Delegation Program for the Relayer Service that they deliver? Since it is already funded via the Community Pool. It would not be fair if it was also funded via the Delegation Program.
(note: I do not run relayers, so it does not involve my own validator. But I can surely imagine from the point of view from other validators that it should be either CP-funding or Delegation-funding, not both.)