Bootstrapping until certain values are reached

[mattnuttall00]

Hi all,

I am trying to bootstrap (with replacement) from a dataframe, repeatedly until the column sums for each of the data columns reaches a predefined value.

My dataframe is small and pretty simple. It looks like this:

with these dimensions:

The data columns I am interested in summing after each resampling loop are columns 3:8, i.e. all those starting with "eff_"

What I have tried

So far, I have tried (with some success) using a repeat loop. My first example below is where I have told it to stop after only the first data column reaches the pre-defined value of 720. The code is as follows

first <- sample_n(df_dense, 5, replace = TRUE)   ## create the first bootstrap sample
eff_10_lim <- 720   ## set the limit after which the loop should stop

repeat {
  subseq <- sample_n(df_dense, 1, replace = TRUE)  ## do the next bootstrap
  first <- rbind(first,subseq)  ## join the first sample with each subsequent sample
  N10 <- colSums(first[ ,3])   ## sum the eff_10 column

if(N10 > eff_10_lim ) {      
    break   ## tell it to stop once the column sum is greater than the defined value
  }
  
}

This appears to work. The loop stops, and first is a dataframe where the eff_10 column has a colSum of just over 720.
Just a note - I am taking a sample of 1 row in each subsequent bootstrap above on purpose - I think if I increase the number of samples then after each iteration the colSum will have jumped up a lot, meaning that when the loop stops the colSum will be quite a lot larger than the defined stop value.

Next I tried to add more conditions, using the following code:

set.seed(42)
first <- sample_n(df_dense, 5, replace = TRUE)   ## Create first bootstrap sample
eff_10_lim <- 720
eff_11_lim <- 756   ## these are the limits for the other data columns  
eff_13_lim <- 560
eff_14_lim <- 660
eff_16_lim <- 660
eff_18_lim <- 576

repeat {
  subseq <- sample_n(df_dense, 1, replace = TRUE)
  first <- rbind(first,subseq)
  N10 <- colSums(first[ ,3])
  N11 <- colSums(first[ ,4])
  N13 <- colSums(first[ ,5])  ## now I am summing all of the data columns
  N14 <- colSums(first[ ,6])
  N16 <- colSums(first[ ,7])
  N18 <- colSums(first[ ,8])
  
  if(N10 > eff_10_lim & N11 > eff_11_lim & N13 > eff_13_lim & 
     N14 > eff_14_lim & N16 < eff_16_lim & N18 > eff_18_lim) {
    break
  }
  ## above I have tried to tell it to stop when N10 is greater than its limit, AND N11 is greater that its limit, etc.etc
}

I don't think the above had worked though, as R seems to just run and run and run, so I think I may have accidentally created some infinite loop!

My questions are:

1. Is the code in the first example doing what I think/want it to do?
2. How would I add more conditions, like in the second example?
3. Is there a simpler/more efficient way of doing this? I feel like I'm over-complicating it...

Reproduce the problem

Here is some code to reproduce a simplified version of the data for you:

df_dense <- data.frame(Habitat = rep("D", times=18),
Transect=c(1:8,10:12,15,18:19,28,32:33,36),
eff_10 = rep(40, times=18),
eff_11 = rep(32, times=18),
eff_13 = rep(32, times=18),
eff_14 = rep(48, times=18),
eff_16 = rep(44, times=18),
eff_18 = rep(32, times=18))

Many thanks in advance!!!

[mattnuttall00]

UPDATE:

I think I have found the solution, using the code below:

set.seed(42)
first <- sample_n(df_dense, 5, replace = TRUE)
eff_10_lim <- 720 
eff_11_lim <- 756
eff_13_lim <- 560 
eff_14_lim <- 660 
eff_16_lim <- 660 
eff_18_lim <- 576 

repeat {
  subseq <- sample_n(df_dense, 1, replace = TRUE)
  first <- rbind(first,subseq)
  N10 <- colSums(first[ ,3])
  N11 <- colSums(first[ ,4])
  N13 <- colSums(first[ ,5])
  N14 <- colSums(first[ ,6])
  N16 <- colSums(first[ ,7])
  N18 <- colSums(first[ ,8])
  
  if(N10 > eff_10_lim) {
    if( N11 > eff_11_lim) {
      if(N13 > eff_13_lim) {
        if(N14 > eff_14_lim) {
          if(N16 < eff_16_lim){
            if(N18 > eff_18_lim){
              
            }
          }
        }
      }
    }
   
  }
   break
}

The result is a DF that appears to have met the targets, i.e. all of the colSums are above the pre-defined thresholds.

Would you mind just clarifying that it's doing what I think it's doing, and that I haven't just randomly got the correct result!

Thanks

[bradduthie]

Hi @mattnuttall00 -- Just to make sure I understand the problem, are you wanting to resample all five columns of data until each column of data fulfils the criteria of summing to some predetermined value? In other words, if it takes 10 resamples for column 1, 20 resamples for column 2, and 30 for column 3, then every column should be resampled 30 times?

I think that the way you've done it above is just fine. Here's a slightly different way that should also work (and avoids all of the nested if statements).

library(dplyr); # Looks like you're using some dplyr functions
eff_10_lim <- 720;
eff_11_lim <- 756;
eff_13_lim <- 560;
eff_14_lim <- 660; 
eff_16_lim <- 660; 
eff_18_lim <- 576; 
lims <- c(eff_10_lim, eff_11_lim, eff_13_lim, eff_14_lim, 
          eff_16_lim,  eff_18_lim); # Put all the limits in a vector
# Changed to 1 below? What if all exceed the sum immediately?
first <- sample_n(tbl = df_dense, size = 1);
exc   <- sum(first[1, 3:8] > lims) == length(lims); # Are all over?
while(exc == FALSE){
    subseq <- sample_n(tbl = df_dense, size = 1);
    first  <- rbind(first, subseq);
    colsum <- apply(X = first[, 3:8], MARGIN = 2, FUN = sum);
    exc    <- sum(colsum > lims) == length(lims);
}
which_col <- which(colsum > lims); # Should be all

I do sometimes wish that R allowed a do{}while() syntax.

[mattnuttall00]

Thanks @bradduthie for the response.

To clarify - I want to resample rows from the original dataframe, up until all of the colSums are greater than the pre-defined limit. But I have run your code and I think it does that anyway - as sample_n() is resampling rows from the dataframe. Am I correct here?

Thanks for the alternative code - it is much neater!

Just out of curiosity, any particular reason why you used a while loop instead of a repeat loop? I don't really know much about either...so perhaps it's obvious to those who understand them better!

Thanks again

[bradduthie]

Hi @mattnuttall00 -- Yes, I think that sample_n() samples the entire row. This was a new one for me though, as I would typically use the base sample() function and pull the row with an index (e.g., subseq <- df_dense[sampled_row,];). Seems sample_n() works fine though.

There's no particular reason for preferring a while loop. Either that or repeat works totally fine, so it really just comes down to personal preference.