Progress bar

[jgabry]

CmdStanPy has a nice progress bar that we can implement in R too.

We are currently using processx::run(..., echo=TRUE) so that the R user sees the output directly from CmdStan, but processx provides a lot more control than that and it looks like it has what we’d need behind the scenes for a progress bar even for chains in parallel.

[jgabry]

After discussing with @mitzimorris I found out that maybe there are some problems with the progress bar implementation inn CmdStanPy so maybe we shouldn't try to copy that until it's sorted out there.

Also, an alternative to a progress bar would be to run background processes and provide a way for the user to check the status. A progress bar isn't necessary for models that run fast, and maybe for slow models it's better to let them just run in the background and allow the user to check the status when they want? This is possible with the processx package we're already using.

Anyway, I haven't thought too hard about this yet so I'm definitely open to other ideas!

[rok-cesnovar]

So a progress bar would be fairly simple to implement after #61 is merged, provided that a user doesnt specify refresh=0. @jgabry do we still want to do this?

[jgabry]

I could go either way on this. The current output from parallelization is fine, although a progress bar could be nicer. What do you think?

[rok-cesnovar]

Do you have an example for an R package that uses progress bars? I am not completely sure how it would look. Worth a try tho.

[jgabry]

Yeah it's worth a try. Things to check out whenever you want to dive into this (definitely not super urgent though):

• I haven't tried this, but maybe it has what we need: progress package (https://github.com/r-lib/progress, https://cran.r-project.org/package=progress). Has some of same very capable authors as the processx package that we're using.
• I've used the revdepcheck package (https://github.com/r-lib/revdepcheck, for testing reverse dependencies) and that has a really nice progress bar. Looks like it uses the progress package.
• You can also look at what other packages use progress by looking at it's CRAN page: https://cran.r-project.org/package=progress (scroll down to reverse dependencies section). There are a bunch so you may be able to look at those packages for implementation details.

There are other packages for making progress bars but I haven't used them. I've heard it can be difficult to do progress bars for parallelized code in R, but I haven't gone down that rabbit hole so I don't know any details about what the challenges are unfortunately.

[mike-lawrence]

What do folks think of the approach taken by my package ezStan where I simply watch the contents of the CSV files to establish progress (inc. reporting post-warmup divergences and permitting termination on detection of these)?

[rok-cesnovar]

@mike-lawrence that sounds like a great idea.

So you essentially count the number of lines in the CSV? Do you read the entire CSV in a separate thread constantly for checking divergences?

[mike-lawrence]

So you essentially count the number of lines in the CSV? Do you read the entire CSV in a separate thread constantly for checking divergences?

Precisely. My package is a bit of a monstrosity of poorly-commented/hack-y code, so if this kind of progress indicator is of interest for cmdstanr I'd probably start from scratch. But it would be neat to finally contribute to the project beyond my forum activity, so I'm keen to do it.

[rok-cesnovar]

Cool! I am definitely in favor.

Another option that would only allow showing progress (not divergences and other diagnostic realtime stuff) is to rely on stdout. If the progress bar is enabled then set refresh = 1. That means that we get information for each iteration.

If a user wants realtime reports for divergence/treedepth then we need to read the CSV.

[mike-lawrence]

Good idea, not monitoring the csv if the user hasn't asked for any csv-content-dependent behaviour will make things more resource-friendly during sampling. I'll have to teach myself how to monitor the stdout instead, but I suspect that's within my capacity.

[rok-cesnovar]

For sampling we have a process function with a "state machine" with which we don't do much, mostly just limit that the initial Cmdstan metadata dump is not printed and to mark the transition from warmup to sampling and such: https://github.com/stan-dev/cmdstanr/blob/master/R/run.R#L650
This is called every 200ms I believe if any stdout came in.

For all the other methods the function is here https://github.com/stan-dev/cmdstanr/blob/master/R/run.R#L791
Here we only make sure the Cmdstan metadata dump is not printed.