Benchmarking Seastar-Plus

[JoelMathewC]

This discussion will focus on benchmarking seastar to show all its benefits.

[JoelMathewC]

Static Temporal Datasets

WindmillOutputLarge

1. Num Timestamps: 17472
2. Num Nodes: 319
3. Num Edges: 101761

WikiMaths

1. Num Timestamps: 731
2. Num Nodes: 1068
3. Num Edges: 27079

[JoelMathewC]

Dynamic Temporal Datasets

Varying seq len

We check how varying the sequence length for a fixed slide_size, feat_size and hidden_num affect the storage. This is to investigate when accumulating gradients how much more memory is PyG-T using in comparison to Seastar. Doubt there will be any use of the time measurements here.

Varying feat size on graph

This allows us to take a look at how the feature size affects storage and space in a graph. We will notice that seastar performs better as number of feature sizes increase.

Varying feat size on graph with slide size=0.1

1. We will also notice that PCSR and GPMA provide noticeable memory improvements for this smaller slide sizes (The improvement increases as slide_sizes decrease).
2. We will be able to see how the GPMA and PCSR tend towards the performance of naive as number of feature sizes increase

Why two different datasets

Wikitalk temporal provides more edges and should require more computational time and we should be able to see the power of GPMA more evidently there.

sx-mathoverflow

Base: 250,000
Change percent: 1-10%

When varying seq len: (slide_size=1.0%, feat_size=8, hidden=16)
We choose this set because they use up the least memory in terms of features sizes and provide the sufficient number of sequences (num_timestamps = 100).

wiki-talk-temporal

Base: 1000000
Cutoff: 2000000
Change percent: 1-10%

[JoelMathewC]

This can be closed out now, since extended benchmarking was performed in the paper that detailed the research contributions of this project.