Reproducibility issue

[seunghan96]

Thank you for sharing the code!

However, I encountered an issue with reproducing the performance reported in the paper (reproducibility issue).

I followed the settings described in

• Table 3 for the learning rate
• Table 4 for rho
• -and set the input length to 512,
  but the results I obtained are significantly worse than those reported in the paper.

For instance, in the case of ETTh1, the MSE values for horizons 96, 192, 336, and 720 are reported in the paper as 0.381, 0.409, 0.423, and 0.427, respectively. However, my results are as follows:

• model=transformer: 0.373, 0.439, 0.466, 0.531
• model=spectrans: 0.381, 0.424, 0.450, 0.560

Could you provide guidance on what might be causing this discrepancy?

It would be greatly appreciated if you could provide scripts for preparing the benchmark datasets to ensure reproducibility of the reported performance.

Thank you in advance!

[romilbert]

Hi seunghan, thank you for your interest in SAMformer.

I would like to clarify a few points that might help address the issue you’re encountering. First, the results you mentioned correspond to SAMformer, which requires setting the --use_sam flag to True. Without this argument, the model behaves like Transformer (or Spectrans), which explains the performance discrepancy.

Additionally, the repository already includes a bash script (run_script.sh) that automates the entire process — from preparing the benchmark datasets to training, testing, and displaying results. You only need to run the script as described in the README, and it will reproduce the results reported in the paper.

If you’re still encountering issues despite using the provided script, I’ll be happy to assist further.

[seunghan96]

Thank you for letting me know!

I have already followed the "run_script.sh" as instructed in README.md, and also set use_sam to True, but I still got the same results as mentioned above.

The only hyperparameter that I changed from the "run_script.sh" is the hyperparameter "rho"

• rhos=(0.7) # Default rho values for SAM optimization (modifiable as needed)

Do I have to follow the "rhos=0.7" value for all datasets in the run_script.sh, instead of "table 4"?

I would greatly appreciate your help with this!

[romilbert]

To reproduce the SAMformer results, ensure the correct ρ value is used for each dataset and prediction horizon, as specified in Table 4. Please also make sure the trainable argument is set to True; otherwise, the results will correspond to the random_transformer baseline presented in the first part of the paper.

[seunghan96]

Thank you!

Following your advice, I used different rho values based on "dataset-prediction horizon" as specified in Table 4.

I couldn't find a specific "trainable" argument in run.py, but the training process seems to work fine.
However, the results are still different from those reported.

Would you be able to provide further assistance?