the random seed for parameter selection and step data

[stylebased]

Hi,
I am really impressed by your paper. When I tried to reproduce your results with your codes and dataset, the results looks different from your results in paper eg. Figure4. Can I assume this difference is caused by randomness in hyperparameter-selection? If so, what is the random seed you used that I can exactly reproduce your results?
Best

[ojwalch]

Hi! Tragically, I didn't keep track of the random seed when I was doing this work—my bad! How different are the plots looking? Feel free to reach out at ojwalch @ umich.edu if you'd like to follow-up there.

Best,
Olivia

[stylebased]

Thank you for your quick reply.This is the figure 3 in your paper. This is the plot I got with same title

.
Does this difference make sense? Did you use Monte Carlo hyper-parameter searching method?

[ojwalch]

Yep, that difference makes sense: For those figures, I'm randomly choosing the training and testing set, evaluating performance, and then repeating the process many times. The random choices of training/testing set will cause variation in this figure, as will the number of times you repeat the simulation (I believe I did 200 times for the paper). Hope this helps!

[stylebased]

Your response is really helpful. I notice there are two hyper-parameter selection method in your codes. One is Leave-One-Out and another is Monte Carlo. Which one did you use in your paper figures such as figure 2, figure 3. Thanks

[ojwalch]

Both Figures 2 and 3 are Monte Carlo cross validation 👍

[stylebased]

thanks 获取 Outlook for iOS<https://aka.ms/o0ukef>

…

________________________________ 发件人: Olivia Walch ***@***.***> 发送时间: Thursday, July 7, 2022 9:13:54 PM 收件人: ojwalch/sleep_classifiers ***@***.***> 抄送: Hu, Xiangyu ***@***.***>; Author ***@***.***> 主题: Re: [ojwalch/sleep_classifiers] the random seed for parameter selection and step data (Issue #23) Both Figures 2 and 3 are Monte Carlo cross validation 👍 — Reply to this email directly, view it on GitHub<https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fojwalch%2Fsleep_classifiers%2Fissues%2F23%23issuecomment-1178427332&data=05%7C01%7Cxiangyu%40email.sc.edu%7C3e19c4c3aef3467ac09a08da607f20c5%7C4b2a4b19d135420e8bb2b1cd238998cc%7C0%7C0%7C637928396380098413%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ux%2FGEuyQs7Ud52iL1PBvueOyF4K4ZOdqFjxed687mw0%3D&reserved=0>, or unsubscribe<https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAK7FNGB6GQ6OXMNXSNXQP73VS56FFANCNFSM526LM2YA&data=05%7C01%7Cxiangyu%40email.sc.edu%7C3e19c4c3aef3467ac09a08da607f20c5%7C4b2a4b19d135420e8bb2b1cd238998cc%7C0%7C0%7C637928396380098413%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Cw2A8%2F3OSzNc0f3XBsf0wbSSmuOIKxw2tRrnAOO1kXc%3D&reserved=0>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[stylebased]

I have another question. In your 'clock proxy' and time-based feature part, you used the steps to estimate the light. Can you provide me more detail things about how did you use steps data to predict the circadian clock? For example, what is the threshold value you set to say there is light? In your paper, you said 'if steps were above a threshold, the light was assumed to be one of three levels'