Node operators issued an open letter to question, and Hyperliquid centralized API caused controversy again

Original title: A Letter To The Hyperliquid Core Team
Original author: Kam Benbrik, Researcher @Chorus One
Original compilation: Ashley, BlockBeats

Editor's note: Node operator Chorus One described in detail the multiple problems of the Hyperliquid test network in an open letter on the And put forward a number of improvement suggestions aimed at improving the transparency and decentralization of the chain. In response to these contents, Hyperliquid founder Jeff responded and emphasized that the criteria for validator selection have been stated in the announcement; Hyperliquid's official account also issued a separate post on the X platform to clarify the issues mentioned in the letter and stated that the node code will be safe. Open source.

The following is the original content (the original content has been edited for ease of reading and understanding):

The following letter is written by Chorus One to the @HyperliquidX engineering team, hoping that the team can take the time to review this feedback on Hyperliquid chain management.

TL;DR

· Due to closed-source code, lack of documentation, and reliance on centralized APIs, validators face significant challenges, resulting in frequent node jailing and unstable performance.

· The testnet incentive mechanism has led to black market trading of HYPE tokens, favoring transactions with large holders rather than fair selection of validators.

· The low returns of mainnet validators cannot cover the high self-staking requirements, and decentralization is limited because 81% of the equity is controlled by foundation nodes.

· To compete with the major layer 1s, Hyperliquid must increase transparency, decentralize staking, implement a fair validator selection mechanism, and interact more with external validators.

I started working with Hyperliquid in December 2023 and found this app to be excellent. It's easy to use, the user experience is great, and Hyperliquid offers some unique features not found elsewhere, such as vaults and the well-known HLP. HLP currently manages over $350 million in funds and allows anyone to participate in Hyperliquid in a passive manner.

After seeing the excellent performance of the platform and knowing that Hyperliquid operates independently as its layer 1, I hope Chorus One can participate as an operator on the Hyperliquid chain. I'm an employee from ChorusOne, one of the largest node operators in the industry. Chorus One has been active in the Proof of Stake industry since 2018. We have worked with many great teams, participated in the development of many blockchain designs and consensus algorithms, and played a key role in the earliest proof-of-stake chains such as Tezos and Cosmos Hub. Currently, Chorus One manages over 50 blockchains with over $3 billion in total staked assets and has worked with all major proof-of-stake blockchains since its early stages.

Chorus One joined the Hyperliquid testnet on October 17 after being whitelisted. I wanted to share our overall experience with the testnet with the Hyperliquid engineering team, because even after nearly 3 months of running the testnet, we still haven't had a chance to interact with the team. During this period, we witnessed one of the most successful token launches of 2024 – the launch of the HYPE token. At the same time, we also experienced an interesting and challenging testnet environment. I hope to mention some points of observation in the hope that they will be considered in the coming days, weeks or months.

Testnet experience

The experience on the testnet has been very challenging so far. Node operators have little information on how to run their nodes and have limited resources to consult.

Frequent node shutdowns for unknown reasons

At first we were shut down multiple times and couldn't understand why. Since the code is closed source, we cannot properly assess the reason for the shutdown. The only way is to communicate with other validators on Discord and jointly speculate on the possible reasons. After communicating with multiple validators, we learned that other validators have been shut down repeatedly, and they are not entirely clear why.

Node location problem

We later discovered that the shutdown issue might be due to our nodes not being deployed in Tokyo. Migrating nodes to Tokyo might help. Unfortunately, this was never made clear to the team and we only found out after running into the issue multiple times.

After migrating the node to Tokyo, the situation improved. This may be because many testnet nodes with large stakes are also deployed in Tokyo, so our nodes can catch up slightly and reduce missing blocks. However, even after the migration, we still faced the problem of being shut down, and the specific reasons are still unknown. This lack of understanding is largely due to the code being closed source.

Rely on automatic unshutdown script

We realize that maintaining good uptime on the Hyperliquid testnet depends on how quickly scripts can automatically undo node shutdowns. The only way to improve uptime is to rely on scripts that automatically unshut down quickly. Validators cannot fully understand or solve potential problems and can only automatically undo node shutdowns without understanding the situation.

Centralized Hyperliquid API as a single point of failure

There have been a few times where our attempts to un-shut down have failed because Hyperliquid's API was down. Since the validator must send a request to the Hyperliquid server to un-shut down, the validator cannot un-shut down on its own when the API is down.

The team may already be aware of this, but this design needs to be rethought as it makes the API a significant single point of failure in the network. If the goal is to build a Byzantine Fault Tolerant system, then no node should have special permissions, such as relying on a centralized API.

Validator selection on mainnet

Hyperliquid recently selected approximately 16 validators in the process of decentralizing its validator set. Previously, the four validators managed by the core team received a lot of criticism. Hyperliquid recently took a major step by expanding its validator set from 4 to 16.

Regarding the selection of validators, 4 validators were announced via the following Discord post:

These validators are Validao, Bharvest, Hypurrstake and Prrposefulnode. These validators are selected based on their ability to maintain greater than 90% uptime over the past 7 days or 30 days.

This is a significant achievement on multiple fronts, primarily because validator performance is also affected by external factors, such as Hyperliquid API outages, shutdown issues, and continued crashing of binaries, which have a non-negligible impact on performance. influence.

In addition to the 4 validators selected based on testnet performance, the Hyperliquid Foundation's 5 validators are also running on the mainnet. In addition, 7 other validators were also selected to participate in the mainnet, but the reasons for their selection have not been made public.

Subsequently, a black market for HYPE testnet tokens began to emerge.

The Hyperliquid testnet will initially have a set of 50 validators. Initially, specific entities were whitelisted to join the testnet, but as of December 12, the validator set is fully open.

The conditions are simple: you need 10,000 HYPE testnet tokens to register to become a validator. But to become an active validator, you must also be in the top 50, otherwise the validator will remain inactive.

This decision caused the HYPE testnet token price to surge. Initially the price rose to over 3,000 simulated USDC and a few days later it even exceeded 28,000 simulated USDC. As of this writing, the price is around 700 simulated USDC per token.

Unfortunately, the faucet only distributes 100 simulated USDC every 4 hours. To become a top 50 validator on the testnet, more than 528,747 HYPE testnet tokens are currently required. Assuming the price is 700 per token simulating USDC and relying only on faucets, the calculation is as follows:

Number of days = (528,747 × 700) ÷ (100 × 6) = 616,871.5 days

This means that relying solely on faucets, it would take approximately 616,871.5 days, or 1,690 years, to acquire enough HYPE testnet tokens to become an active validator on Hyperliquid.

However, those who received the HYPE airdrop on the mainnet also received the same number of tokens on the testnet. This provides validators with the opportunity to collaborate with these community members by allowing them to stake testnet HYPE tokens, allowing validators to secure entry into the active set.

At the same time, this situation also provides another way of thinking for those holding testnet HYPE tokens. Given the fierce competition to join the testnet validator set, many validators are eager to obtain as many HYPE testnet tokens as possible. As a result, a black market emerged, and whales holding large amounts of testnet HYPE tokens began selling their testnet tokens to validators in exchange for USDC on the mainnet.

I've never seen such chaos before. While the Hyperliquid team clearly disapproves of these practices, they are fully capable of solving this problem. One potential solution is to implement a proper testnet validator selection process.

In most other PoS networks, the core team usually shares a form that any validator can fill out to express their willingness to run the chain. The team then reviews these applications and makes an initial selection based on various criteria, such as the validator’s node running experience, past contributions, community involvement, or other factors.

This initially selected group of validators can then participate in the testnet, working closely with the engineering team to provide feedback and ensure everything runs smoothly. We have tried multiple times to provide feedback but so far have been unsuccessful.

Mainnet and decentralization

As mentioned before, the current validator set of Hyperliquid mainnet consists of 16 validators, which can be viewed at: https://app.hyperliquid.xyz/staking

· 5 validators from the Hyperliquid Foundation.

· 4 validators selected based on their performance on the testnet, maintaining over 90% uptime over the past 7 days.

· 7 validators at the discretion of the Hyperliquid team.

Among the 404,495,250 pledged HYPE tokens, approximately 329,578,724 HYPE tokens are pledged on foundation nodes, accounting for approximately 81.4% of the total pledged amount. We know very little about HyperBFT, but assuming it operates as a Byzantine Fault Tolerant system, a core assumption of most BFT systems is that no more than 33% of the voting power behaves maliciously. If a single entity controls 1/3 of the stake, they can stop the chain from functioning. If they control 2/3 of the stake, they have complete control of the network.

The Hyperliquid Foundation initially staked 60 million HYPE tokens on each foundation node. However, many HYPE holders also choose to stake on foundation nodes, which is not ideal for decentralization. The team should interact more with the community and encourage a more decentralized staking distribution.

There are three potential solutions:

· Educate the community on the importance of staking with external validators to increase the security and decentralization of the chain.

· Implement a 100% commission rate for foundation nodes to encourage users to pledge with external validators and promote decentralization.

·Redistribute foundation stakes to external validators, which is what most chains do.

Decentralizing staking to external validators will also help them become more economically sustainable. Hyperliquid is a blockchain focused on high throughput, and its infrastructure costs can be high, especially when nodes are deployed in Tokyo. Currently, validators at the bottom of the set earn between $3,000 and $5,000 per year, which is not enough to cover costs. This is particularly challenging because they must self-stake an initial 10,000 HYPE tokens (approximately $250,000 at current prices) to be validated on mainnet.

Currently, users interact with Hyperliquid by bridging USDC from Arbitrum to the Hyperliquid chain. By reviewing the bridge's contract, it appears that the bridge is still managed by 4 validators. These validators do not appear to be tied to the chain's consensus or the 16 validators on the mainnet.

Hyperliquid has a great product, but the team still needs to improve on several aspects of its infrastructure to truly compete with the major layer 1 players. Some improvements are very simple, such as:

Hear from validators experienced in managing multiple networks. While the way the team currently works independently is very effective in building its sustainable product, validators are a layer 1 backbone. It’s equally important to listen to their input to ensure everything runs smoothly.

Open source code. This will help validators better understand the issues they face when running nodes on Hyperliquid L1, while also helping users trust the product. Open source code will also enable validators to learn more about the architecture and consensus algorithm. There is currently very limited information about HyperBFT, and open source can provide much-needed transparency and understanding. Chorus One has a web brochure on the importance of open source. Operators should be able to build all software they operate from source code: https://handbook.chorus.one/node-software/open-source.html

Create a proper validator selection process to stop black market trading of HYPE testnet tokens. Selecting validators based on uptime is a fair approach, but getting good uptime should also be fair. This should not depend on whether there is a relationship to acquire testnet tokens, purchase testnet tokens, or external factors (such as reliance on Hyperliquid API uptime).

Overall, Hyperliquid doesn't need to change much to compete with the major layer 1 players. The main focus should be to interact more with external parties and incorporate their feedback. I look forward to seeing changes in the coming weeks and months, and our team is here to provide help and feedback.

Hyperliquid founder Jeff and the official account responded

In response to this letter, Hyperliquid founder Jeff responded on the X platform.

He emphasized that it is not difficult to successfully run a validator. The key lies in the setting and specialization of the validator itself. Additionally, he pointed out that the validator selection criteria, stated in the announcement, are based on high uptime performance in the early stages of the testnet. This shows that Jeff is more inclined to believe that the current problem stems more from the configuration of the verifier itself rather than a flaw in the system design.

In addition, Hyperliquid officials also issued further clarification and stated that the node code will be open source under safe conditions.

· All validators are qualified based on testnet performance and cannot obtain seats through purchase; related false statements belittle the efforts of those validators who invest time and energy in understanding the system; as the blockchain matures, the set of validators will gradually expand.

· As previously announced, the Foundation Delegation Program will be launched to support high-performing validators and further decentralize the network.

· Anyone can run an API server pointing to any node; the sample client code sends requests to a specific API server, but this is not a basic requirement of the network.

· It is unacceptable for users to attempt to create a black market for testnet HYPE; this has been stated many times; we will continue to work hard to improve the testnet onboarding process.

· The node code is currently closed source; open source is important and the project will be open source after development reaches a stable state; Hyperliquid’s development speed is orders of magnitude faster than most projects, and its scope is orders of magnitude larger than most projects; the code will Open source while safe.

· Currently there is only one binary file. Even on a very mature network like Solana, the vast majority of validators run a single client.