By Shai ❤️ Deshe 💜 Wybors𐤊i 💙 (大胡子)
For those of you wondering why testnet 11 is taking so long, it is because the incredible Rusty-Kaspa team keeps coming up with exciting new ways to improve the performance of the client. However, these require deep and subtle changes to the code that require time and care, delaying the next public Testnet 11 run. In the last few weeks, two major optimizations were introduced:
1. Tracking child blocks: the node maintains for each block a list of its children blocks (that is, blocks pointing at it). When a new block arrives, it should be added to the list of children of all blocks it is pointing at. Originally, the list would be recomputed and rewritten from scratch. This requires a quadratic amount of database writes: if a block has N children, then the first child was written N times, the second one (N-1) times, and so on, summing up to N(N-1)/2 writes. This bottleneck was identified through intense profiling work carried out by @OriNewman, who, together with @MichaelSuttonIL, designed and implemented a clever way to track block children in a dependable way, meaning each child is only written once per block. For illustration, if each block has 10 children, this optimization decreases the number of DB writes per block from 55 to 10, providing an improvement of more than 80%. Relevant PR: github.com/kaspanet/rusty
2. Reachability queries: reachability queries are the mechanism that allows us to quickly determine whether one block is in the past of another block. Finding a way to do so that is efficient both in terms of storage and in terms of processing is one of the hardest roadblocks for implementing GHOSTDAG, and why many before us assumed an efficient implementation is impossible (h/t @reedvoid).
The mechanism that achieves that is rather complex and has a lot of moving parts. Without diving too much into the details, some of the data that has to be stored is also computed per block and recomputed as children are added. Adopting a similar approach as in the previous optimization allowed for a dramatic decrease in the disc pressure created by the reachability mechanism. Relevant PR: github.com/kaspanet/rusty Early benchmarks indicate that these two optimizations more than doubled the header processing rate of the client (Running on a machine with a 24-core i9 processor, the header processing rate increased from 226.03/sec to 528.81/sec) Besides these optimizations, the crew took this chance to implement dynamic cache policies. This essentially means that the client is aware of the amount of RAM available to the host machine and considers this when determining what data to cache, providing more stability (esp. on machines with lower memory). Relevant PR: github.com/michaelsutton/
The crew is now working on merging these changes and starting a cycle of internal testing.