Some Enchancements #1
Description
Hello, I have some suggestions for improvement. You've also inspired me to do my own research and learning about data compression, and to explore potentially better solutions. Additionally, your method of storing differential data is known as delta encoding.
The first suggestion is to use Huffman coding for the pre-pad or flag bits. These pre-pad bits should consist of a few unique patterns that correspond to specific encoding configurations. The most frequent results should use the shortest pre-pad patterns.
Pre-pad bit: 0 for 7 bits of data (total of 8 bits)
Pre-pad bit: 10 for 14 bits of data (total of 16 bits)
Pre-pad bit: 110 for 21 bits of data (total of 24 bits)
Pre-pad bit: 1110 for 28 bits of data (total of 32 bits)
Pre-pad bit: 11110 for 35 bits of data (total of 40 bits)
Pre-pad bit: 11111 for 35 bits of data (total of 40 bits, meant for absolute value**)
Although it may seem unintuitive, after delta encoding, data sizes that require the full 32 bits (e.g., uint32) are unlikely to occur frequently. Therefore, it's more efficient to use shorter pre-pad bits for more likely, smaller data sizes. With a unified pre-pad pattern structure, you can represent variable-length data efficiently while still allowing for large value changes when necessary.
The second suggestion would be using a very simple and widely used method for variable-length encoding—is LEB128. It is easy to implement and supports compact storage of data with varying lengths.
The third suggestion would be looking for hardware acceleration functions for MCU. Like the delta encoding and find out how many bits you need to store the delta encoded data like __builtin_clz().
I am also learning about data compression and developing my own version of it.