Performance impact because of decoded message struct

[srikartati]

Describe the bug
The message struct contains set pointer. And set pointer contains pointers to records. A single record contains pointers to multiple information elements.

Using pointers in these different levels of methods puts pressure on GC. We need to revisit the decision of using interface to support both template and data entities. This leads to the use of a pointer. Instead, move to the direct structs.
In addition for the decoded messages, allocate contiguous memory and fill the required data in it.

To Reproduce
Memory consumption with antrea and ipfix performance tests. Fragmentation because of GC stress.

Expected behavior
We should cut down the difference between sys and heap usage

Additional context
Try with a PoC and improve upon it.

[srikartati]

@zyiou @wsquan171 For closer look.

[srikartati]

Hi @dreamtalen @heanlan
Hope you are doing well. Just wondering about the resolution of this performance issue where high memory is consumed.

One thing I came across is option templates in IPFIX. It is generally used for sending metadata for different info in flow records.
For example, our flow record has complete metadata along with five tuple. However, metadata is sent multiple times unnecessarily. Instead Flow Exporter could send it as a separate record with IP of endpoint and corresponding pod name using an option template.
Using option templates for metadata seems to be industry standard.

https://docs.paloaltonetworks.com/prisma/prisma-sd-wan/prisma-sd-wan-admin/prisma-sd-wan-sites-and-devices/use-external-services-for-monitoring/prisma-sd-wan-ipfix/ipfix-template-record-options

https://kb.vmware.com/s/article/79184

The benefit of doing this is reducing the size of the record that we store in in-memory map at flow aggregator. Smaller record will require smaller allocation of memory leading lower fragmentation. We may need to have multiple maps.

Just wanted to share this for a while. Cheers!