A letter from a validator to the Hyperliquid team: You have made a good product, but it could be better

Written by: Kam

Compiled by: Luffy, Foresight News

This letter is written to the Hyperliquid development team, hoping that the team will take the time to review this feedback on Hyperliquid blockchain management.

Key points:

• Validators face significant challenges due to non-open source code, lack of documentation, and reliance on centralized application programming interfaces (APIs), resulting in frequent jail terms and unstable performance.
• Testnet incentives created a black market for HYPE tokens that favored deals with large players rather than fair selection of validators.
• Mainnet validator rewards are too low to meet the high self-guaranteed requirements, and decentralization is limited because 81% of the pledged shares are controlled by foundation nodes.
• To compete with mainstream L1 blockchains, Hyperliquid must increase transparency, reduce staking centralization, implement a fair validator selection process, and increase interaction with external validators.

I started working with Hyperliquid in December 2023, which is an amazing encryption application. It's easy to use, has a great user experience, and offers some unique features like vaults and the famous HLP. Currently, HLP manages more than $350 million in assets, and anyone can participate in Hyperliquid passively.

After seeing how great this platform is and knowing that Hyperliquid operates as its own L1 blockchain, I hope that Chorus One (staking solution) will participate in the Hyperliquid chain as an operator. I'm an employee of Chorus One, one of the largest node operators in the space. Chorus One has been active in the proof-of-stake industry since 2018.

Chorus One joined the Hyperliquid testnet after obtaining whitelist qualifications on October 17 last year. I wanted to share with the Hyperliquid engineering team our overall experience on the testnet, because even though we've been on the testnet for almost 3 months, we still haven't had a chance to talk to them.

During this time, we witnessed one of the most successful token offerings of 2024: the launch of the HYPE token. At the same time, we also experienced a testnet environment that was both interesting and challenging. I would like to mention some observations that I hope will gain traction in the coming days, weeks, and months.

Testnet experience

The testnet experience has been extremely challenging so far. Operators have little idea of ​​how to run nodes and have limited access to resources. Beyond that, we were basically exploring in the dark, and we discovered multiple issues, including the following:

Frequently jailed for unknown reasons

At first, we were sent to jail multiple times without understanding why. Since the code is not open source, it is impossible to accurately assess the cause. The only thing we can do is communicate with other validators on Discord and guess the reasons together. After talking to several validators, we found that everyone was jailed frequently and not entirely sure why.

Node position

We later discovered that the jail issue was probably because we were not running nodes in Tokyo. Moving the node to Tokyo might help. Unfortunately, this was never made clear by the team and we only found out after running into numerous issues.

After moving the node to Tokyo, the situation improved. This is likely because many testnet nodes holding large amounts of staked tokens are also located in Tokyo, so our nodes can miss fewer blocks and keep up. However, even after moving the node, we continue to encounter jail problems, but we still don't know the exact cause. The main reason for this trouble is that the code is not open source.

Rely on automatic jailbreak script

We realize that maintaining good uptime on the Hyperliquid testnet depends on how quickly the script can automatically de-jail nodes. The only way to improve uptime is to rely on scripts that automate jailbreaks quickly. Validators cannot fully understand or solve the underlying problem and can only automatically de-jail nodes without deep understanding.

Centralized Hyperliquid API as a single point of failure

On a few occasions, our jailbreak attempts failed because of Hyperliquid API glitches. If the API fails, validators cannot de-jail themselves because they must send a request to the Hyperliquid server to de-jail.

The team may be aware of this, but this design needs to be rethought as it makes the API a significant single point of failure in the network. If the goal is to build a Byzantine fault-tolerant system, there should not be any nodes with special permissions, such as nodes that rely on a centralized API.

Mainnet validator selection

Hyperliquid recently selected around 16 validators in the process of decentralizing its validator set. Previously, Hyperliquid was managed by a core team of 4 validators, which attracted a lot of criticism. Recently, Hyperliquid took a major step by expanding the number of validators from 4 to 16.

Regarding the selection of validators, 4 validators were announced via the following post on Discord:

The four validators are Validao, Bharvest, Hypurrstake and Prrposefulnode. They are selected based on uptime, having successfully maintained over 90% uptime over the past 7 or 30 days.

This is a remarkable achievement for many reasons. The main reason is that validator performance is also affected by external factors, such as Hyperliquid API glitches, jail issues, and persistent binary crashes, which all have a non-negligible impact on performance.

In addition to these 4 validators selected based on testnet performance, 5 validators from the Hyperliquid Foundation are also running on the mainnet. In addition, 7 other validators were selected to the mainnet, but the reasons for their selection were not publicly disclosed.

Then, a black market for HYPE testnet tokens began to emerge.

The Hyperliquid testnet will initially have 50 validators. Initially, specific entities were whitelisted to join the testnet, but on December 12, it became fully open to validators.

The conditions are simple: 10,000 HYPE testnet tokens are required to register as a validator. However, to become an active validator, one must also be among the top 50, otherwise the validator will be inactive.

This caused the HYPE testnet token price to surge. At first, the price rose to more than 3,000 testnet USDC, and even rose to more than 28,000 testnet USDC a few days later. At the time of writing, the current token price is around 700 testnet USDC.

Unfortunately, the faucet only issues 100 testnet USDC every 4 hours. To become one of the top 50 active validators on the testnet, more than 528,747 HYPE testnet tokens are currently required. Assuming that the token price is 700 testnet USDC per coin and only relies on faucets to release USDC, the calculation is as follows:

Number of days required = (528747×700)÷(100×6) = 616871.5 days

This means that, relying solely on faucets to obtain the required HYPE testnet tokens, it would take approximately 616871.5 days, or 1690 years, to become an active validator on Hyperliquid.

However, those who received the HYPE airdrop on the mainnet also received an equivalent amount of tokens on the testnet. This creates opportunities for validators to collaborate with these community members, allowing them to stake testnet HYPE tokens to the validator, allowing the validator to claim a place in the active set.

At the same time, this situation also gives people who hold testnet HYPE tokens other ideas. Given the fierce competition to join the testnet validator set, many validators are eager to obtain as many HYPE testnet tokens as possible. As a result, a black market emerged, and large investors who held a large number of testnet HYPE tokens began to sell tokens to validators in exchange for real USDC on the mainnet.

I have never seen such chaos. While the Hyperliquid team clearly disapproves of these practices, they are fully capable of solving this problem. One possible solution is to implement a proper validator selection process on the testnet.

In most other proof-of-stake networks, the core team typically shares a form that any validator can fill out to express their willingness to run the chain. The team then reviews these applications and conducts an initial screening process based on various criteria, such as the validator’s experience running nodes, past contributions, community participation, or other factors.

This initially selected group of validators can contribute to the testnet, working closely with the engineering team to provide feedback and ensure everything runs smoothly. We have tried multiple times to provide feedback but so far have been unsuccessful.

Mainnet and decentralization

As mentioned previously, the current validator set on the Hyperliquid mainnet consists of 16 validators and can be viewed here: https://app.hyperliquid.xyz/staking.

Five of them are from the Hyperliquid Foundation. 4 were selected based on testnet performance, maintaining over 90% uptime. 7 chosen by the Hyperliquid team themselves.

Among the 404495250 pledged HYPE tokens, approximately 329578724 HYPE tokens are pledged at the foundation node, accounting for approximately 81.4% of the total pledge share. We don’t know much about HyperBFT, but assuming it operates as a Byzantine Fault Tolerant system, a core assumption of most BFT systems is that no more than 33% of the voting power behaves maliciously. If a single entity controlled one-third of the staked shares, they could bring the blockchain to a halt. If you control two-thirds of the pledged share, you can completely control the network.

The Hyperliquid Foundation initially pledged 60 million HYPE tokens for each foundation node. However, many HYPE holders also choose to stake on foundation nodes, which is very detrimental to decentralization. The team should engage more with the community and encourage a more decentralized staking distribution.

There are three potential solutions:

1. Educate the community on the importance of staking with external validators to increase the security and decentralization of the chain.
2. Foundation nodes implement a 100% commission rate to incentivize users to stake with external validators and promote decentralization.
3. Reallocating the foundation's staked share to external validators is what most chains do.

Distributing stake shares to external validators also helps them become more economically sustainable. Hyperliquid is a blockchain focused on high throughput, and the infrastructure costs of running a node (especially running a node in Tokyo) can be high. Currently, validators at the bottom earn between $3,000 and $5,000 per year, which is not enough to cover their costs. Particularly challenging is that they must stake 10,000 HYPE tokens (approximately $250,000 at current prices) on the mainnet to verify.

Currently, users interact with Hyperliquid by cross-chaining USDC from Arbitrum to the Hyperliquid chain. After looking at the contract of the cross-chain bridge, it seems that the cross-chain bridge is still managed by 4 validators. These validators appear to have no connection to the chain’s consensus mechanism or the 16 validators on the mainnet.

Hyperliquid has a great product, but the team still needs to improve several aspects of its infrastructure to truly compete with mainstream L1 blockchains.

Some improvements are as follows:

• Listen to validators. While the way the team currently works on their own and doesn't interact with many outsiders has been effective in building their Perp product, validators are the backbone of the L1 blockchain. Listening to their input is equally important to ensuring everything runs smoothly.
• Open source code. This will help validators better understand the issues faced when running nodes on the Hyperliquid chain, and will also help users trust the product. Open source code also allows validators to learn more about the architecture and consensus algorithm. Currently, there is very limited information about HyperBFT, and open source can provide much-needed transparency and understanding.
• Create a proper validator selection process to eliminate the black market of HYPE testnet tokens. Selecting validators based on uptime is a fair approach, but getting good uptime should also be fair. It should not depend on external factors such as having access to testnet tokens, purchasing testnet tokens, or relying on the uptime of the Hyperliquid API.

Overall, Hyperliquid does not need drastic changes to compete with mainstream L1 blockchains. The main focus should be to interact more with external parties and take their input into account. I look forward to seeing how the situation develops in the coming weeks and months, and our team is ready to help and give feedback. All the best to the Hyperliquid team!