Delayed order

[mike-mik]

Description

The delay parameter in ruleSignal seems to have no effect on execution time.

Expected behavior

The sample data contains a long signal at 2021-01-25 15:30:00.

The following strategy I would expect to issue an order at the next bar (2021-01-26 09:30:00) but to delay execution by one day.
Although the order is being delayed in the order book index the trade is still executed at time of order issue (2021-01-26 09:30:00) without any delay.

Output of example code (one day delay):

[1] "delay=86400"
           ^^^^^
[1] "2021-01-26 09:30:00 SYM 100 @ 61.24"
             ^^^^^^^^^^^?
$str2
$str2$SYM
                    Order.Qty Order.Price Order.Type Order.Side Order.Threshold
2021-01-26 15:30:00 "100"     "60.58"     "market"   "long"     NA
        ^^^^^^^^^^^ok
                    Order.Status Order.StatusTime      Prefer Order.Set
2021-01-26 15:30:00 "closed"     "2021-01-26 09:30:00" "High" NA
                    Txn.Fees Rule        Time.In.Force
2021-01-26 15:30:00 "0"      "EnterLONG" ""

Minimal, reproducible example

library(quantstrat)

suppressWarnings(rm("account.str2","portfolio.str2",pos=.blotter))
suppressWarnings(rm("order_book.str2",pos=.strategy))

oldtz<-Sys.getenv("TZ")
if(oldtz=="") Sys.setenv(TZ="UTC")

sy <- 'SYM'
fn <- paste0(sy,".csv")
tmp <- as.xts (read.csv2.zoo (fn, header=T, stringsAsFactors=F,
        dec = ".", FUN=as.POSIXct))
assign(sy, tmp)

tradeSize <- 5000
initEq <- 100000
delay_seconds <- 1 * 24*60*60

currency("USD")
stock(sy, currency="USD",multiplier=1)

initPortf('str2', symbols=sy)
initAcct('str2', portfolios='str2', initEq=initEq)
initOrders(portfolio='str2')

strategy.st <- 'str2'
strategy (strategy.st, store=TRUE)


add.rule(strategy.st, name='ruleSignal', arguments = list(
        sigcol = "Long",
        sigval = TRUE,
        ordertype = 'market',
        prefer = 'High',
        orderside = 'long',
        delay = delay_seconds,
        orderqty = 100),
  type='enter',
  label='EnterLONG')


print(paste0("delay=", delay_seconds))
out<-try(applyStrategy(strategy='str2' , portfolios='str2'))
print(getOrderBook('str2'))

updatePortf(Portfolio='str2')
updateAcct('str2')
updateEndEq('str2')

Sys.setenv(TZ=oldtz)

---

My sample data SYM.csv:

Index;Open;High;Low;Close;Volume;Long
2021-01-25 09:30:00;60.67;60.77;59.705;59.98;4841167;0
2021-01-25 10:00:00;59.98;60.31;59.955;59.992;2331090;0
2021-01-25 10:30:00;59.99;60.155;59.64;59.69;2011158;0
2021-01-25 11:00:00;59.6771;59.76;59.26;59.38;2717999;0
2021-01-25 11:30:00;59.38;59.79;59.17;59.67;1758438;0
2021-01-25 12:00:00;59.675;59.85;59.4601;59.4601;1536049;0
2021-01-25 12:30:00;59.475;59.93;59.47;59.9;1212432;0
2021-01-25 13:00:00;59.9;60.37;59.875;60.32;1461132;0
2021-01-25 13:30:00;60.32;60.42;60.185;60.23;970523;0
2021-01-25 14:00:00;60.24;60.29;59.94;60;971210;0
2021-01-25 14:30:00;60;60.24;59.88;60.06;1038065;0
2021-01-25 15:00:00;60.055;60.4;60.04;60.345;931919;0
2021-01-25 15:30:00;60.35;60.58;60.3;60.56;2135034;1
2021-01-26 09:30:00;61.2;61.24;60.087;60.335;2512210;0
2021-01-26 10:00:00;60.335;60.9;60.29;60.78;1768021;0
2021-01-26 10:30:00;60.78;60.87;60.408;60.445;1173861;0
2021-01-26 11:00:00;60.45;60.56;60.15;60.52;1085131;0
2021-01-26 11:30:00;60.51;60.53;60.19;60.33;919405;0
2021-01-26 12:00:00;60.33;60.68;60.31;60.65;1130399;0
2021-01-26 12:30:00;60.645;60.95;60.63;60.915;756772;0
2021-01-26 13:00:00;60.92;61.04;60.81;60.875;848270;0
2021-01-26 13:30:00;60.87;60.95;60.73;60.805;822825;0
2021-01-26 14:00:00;60.8;61.15;60.78;60.95;989872;0
2021-01-26 14:30:00;60.95;60.98;60.81;60.9;1001693;0
2021-01-26 15:00:00;60.9;60.93;60.73;60.86;854300;0
2021-01-26 15:30:00;60.85;60.93;60.67;60.91;3214855;0
2021-01-27 09:30:00;59.9;59.99;58.53;58.98;3972687;0
2021-01-27 10:00:00;58.97;59.43;58.74;59.37;2386966;0
2021-01-27 10:30:00;59.375;59.85;59.13;59.81;2147582;0
2021-01-27 11:00:00;59.82;59.86;59.44;59.69;1273143;0
2021-01-27 11:30:00;59.6998;59.76;59.4;59.45;1183463;0
2021-01-27 12:00:00;59.45;60;59.35;59.41;1359946;0
2021-01-27 12:30:00;59.41;59.62;59.31;59.565;570512;0
2021-01-27 13:00:00;59.565;59.868;59.44;59.76;761216;0
2021-01-27 13:30:00;59.76;59.765;59.45;59.58;623576;0
2021-01-27 14:00:00;59.585;59.7;59.37;59.42;897427;0
2021-01-27 14:30:00;59.425;59.625;59.21;59.3283;2123241;0
2021-01-27 15:00:00;59.33;59.83;59.2;59.67;1558001;0
2021-01-27 15:30:00;59.66;59.7;59.17;59.4;3507501;0

Session Info

R version 4.2.1 (2022-06-23)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Gentoo Linux

Matrix products: default
BLAS:   /usr/lib64/libblas.so.3.10.0
LAPACK: /usr/lib64/R/lib/libRlapack.so

locale:
[1] C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] quantstrat_0.16.9          foreach_1.5.2             
[3] blotter_0.16.0             PerformanceAnalytics_2.0.4
[5] FinancialInstrument_1.3.1  quantmod_0.4.20           
[7] TTR_0.24.3                 xts_0.12.1                
[9] zoo_1.8-11                

loaded via a namespace (and not attached):
[1] quadprog_1.5-8   lattice_0.20-45  codetools_0.2-18 MASS_7.3-58.1   
[5] grid_4.2.1       curl_4.3.2       boot_1.3-28      iterators_1.0.14
[9] compiler_4.2.1  

[jaymon0703]

Thanks @mike-mik for posting here. I have taken a quick look and debugged this strategy through the various stages of the quantstrat strategy execution path. Im not sure, but i suspect delay is not meant to be used as you have used it here. Using a market order will look to execute at the next observation. Perhaps delay is meant to be used within the duration of a single bar (or meant to be used with BBO data), and perhaps where ordertype is a limit for example. Perhaps @braverock can add some color.

[mike-mik]

Using a market order will look to execute at the next observation.

Without any additions to the order that's true.

But my original problem is: How do you tell quantstrat to close a position at the end of the day when you test on intra-day data? This order can be either a market-on-close (MOC) or a good-after-time (GAT). Since I think quantstrat doesn't support MOC I tried to simulate it with GAT using delay pointing to the last bar of the day.

MOC would be far more elegant here. Does quantstrat support MOC?

[jaymon0703]

No, I don't believe there is a MOC ordertype. You could request that as a feature. Would be interested to hear feedback on the idea. Thoughts on a MOC ordertype @braverock?

[braverock]

No, there is no MOC ordertype, since very few investors actually have access to the closing auction or the equivalent TAS for derivatives. Broker MOC order types are typically "we'll fill you at whatever price we can get away with". Those investors who do have access to the closing auction would likely test with tick data anyway, which follows a completely different code path.

Now, onto the issue at hand here. I see two potential problems in the code, but need to dig a bit deeper.

The price for a market order on higher than daily frequency OHLC data is determined by:

 txnprice = as.numeric(getPrice(mktdataTimestamp, prefer=prefer)[,1]) 

I'm not sure mktdataTimestamp is appropriate here vs the next market data timestamp after the order timestamp.

I'm also not sure why this order is in the ordersubset object for actionable orders, since its timestamp is in the future. This might indicate a problem in calculating ordersubset.

order.statusTime is fine, since that's the time the order status changes to closed when it is matched, so while it looks strange, it just tells us that the order executed before it should have been available to be matched.

Right now I'm guessing that the most likely root cause is that timespan isn't being handled correctly when the ordersubset object is retrieved.

[mike-mik]

No, there is no MOC ordertype, since very few investors actually have access to the closing auction or the equivalent TAS for derivatives. Broker MOC order types are typically "we'll fill you at whatever price we can get away with".

Is the closing auction the reason for the EOD close often differing from the intra-day close? Because EOD close does contain the closing auction and the intra-day close does not?

So if my strategy uses the previous day's close as a trigger for today's trade and today's close for executing this trade/calculating the P/L, do you think I should use EOD close as the trigger and intra-day close for the P/L?

[jaymon0703]

What do you mean by intra-day close...the last price printed before the start of the closing auction? P&L should be marked to the actual closing price (ignoring nuances related to after-hours trading).

So if my strategy uses the previous day's close as a trigger for today's trade and today's close for executing this trade/calculating the P/L, do you think I should use EOD close as the trigger and intra-day close for the P/L?

Your portfolio value will be marked to the close price. How you execute your strategy is up to you. If you would like to trade at the close, then it appears you will need to use daily bar data for your signal, as there is no MOC ordertype and using a delay seems to not work as expected right now.

There could be 2 pieces of work out of this:

1. Understand whether or not using delay is buggy, potentially fix. However, this will not solve your use case unless we can make the delay dynamic based on when your signal fires and the number of seconds until the close timestamp.
2. Implement MOC order type

Not sure when we get to 1 and/or 2. If the impact is big enough the effort will be more worth while.

[mike-mik]

What do you mean by intra-day close...the last price printed before the start of the closing auction?

I tested two providers of intra-day data:

• Firstratedata offers 1min bars where the close of the 15:59 bar often deviates from the daily close provided by NYSE or NASDAQ, NOCP - and their open at 09:00 often deviates from NYSE/NASDAQ, too.

• Polygon offers two sources: minute bars and daily bars. The quality of their minute bars seems to be better, but the close of the 15:59 bar also deviates from NYSE/NASDAQ close many times. BTW: Their daily bar's closes seem to be identical with NYSE/NASDAQ.

So I mean the close of the 1min bar starting at 15:59 (or 12:59 for shortened sessions).

This is the first time I deal with intra-day data and I was surprised about the differences in close prices. So my question is: Are Firstrate's and Polygon's intra-day data of mediocre quality or are these differences in close prices quite normal, e.g. because the closing auction's results are calculated shortly after 16:00 and therefore can hardly be included in the 15:59 bar?

P&L should be marked to the actual closing price (ignoring nuances related to after-hours trading).

If

1. like Brian says - most traders don't take part in the closing auction and therefore
2. their orders would be executed earlier at the close of the 15:59 intra-day bar,

this should be used for the P&L.

I'm only interested in testing the main session.

So if my strategy uses the previous day's close as a trigger for today's trade and today's close for executing this trade/calculating the P/L, do you think I should use EOD close as the trigger and intra-day close for the P/L?

[...] If you would like to trade at the close, then it appears you will need to use daily bar data for your signal, as there is no MOC ordertype and using a delay seems to not work as expected right now.

I wrote my strategy after Brian had suggested to test it with intra-day data instead of daily data.

This is the strategy (with 30min bars):

1. If a long signal occurs on day 0 (signal day) then enter long on day 1 the latest until 15:29 if the price exeeds signal day's high.
2. On execution of the entry order issue 2 exit orders combined by OCO:
   a. Stop loss at x USD or x% of entry price
   b. MOC of entry day (last bar at 15:30)
   So the strategy holds the stock for maximum one day only.

The minimal reproducible example of the strategy is available here.

There could be 2 pieces of work out of this:

1. Understand whether or not using `delay` is buggy, potentially fix. However, this will not solve your use case unless we can make the delay dynamic based on when your signal fires and the number of seconds until the close timestamp.

2. Implement MOC order type

Not sure when we get to 1 and/or 2. If the impact is big enough the effort will be more worth while.

Would 2. be a special case of 1. (with fixed time)?

[jaymon0703]

I would query any perceived discrepancies with the vendor.

It would appear that in the nextIndex() function inside applyRules() when the open order type is market we default to looking at the next index

quantstrat/R/rules.R

Line 533 in f9e4ea1

curIndex <- curIndex+1 # why isn't this put into dindex? -JMU

. This means we flag the 13th index in mktdata for the signal (i am referring to the strategy you link above in your last post), then move to index 14 for adding the trade and then (presumably) because we created a chain order to close the position we create another order which is market and subsequently add the transaction at the following timestamp at index 15. I suspect we will need to make a change here if delay is meant to work with market orders. However, when testing ordertype=stoplimit we still get a transaction at the next bar (10am) instead of at the end of the day (15:30) which makes me think we may need to update the dindexOrderProc() function in applyRules(), specifically

quantstrat/R/rules.R

Line 461 in f9e4ea1

out$move_order <- .firstCross(mktPrice, newOrderPrice, relationship, start=curIndex+1L)

we should perhaps add the delay in seconds (expressed as a function of the market data periodicity, so 1 bar would be 1800 seconds) in addition to adding 1 to the current index.

[jaymon0703]

If i add 2 instead of 1 to the curIndex in newIndex() then the trade indeed prints on the next timestamp. I suspect adding the delay here will resolve what appears to be a delay bug.

> out<-try(applyStrategy(strategy='str1' , portfolios='str1'))
[1] "2021-01-26 09:30:00 SYM 100 @ 61.2"
[1] "2021-01-26 10:30:00 SYM -100 @ 60.445"
> 
> print(getOrderBook('str1'))
$str1
$str1$SYM
                         Order.Qty Order.Price Order.Type  Order.Side Order.Threshold Order.Status Order.StatusTime      Prefer  Order.Set Txn.Fees Rule       
2021-01-25 15:30:00.0000 "100"     "60.58"     "stoplimit" "long"     NA              "closed"     "2021-01-26 09:30:00" "High"  NA        "0"      "EnterLONG"
2021-01-26 15:30:00.0000 "all"     "61.2"      "market"    "long"     NA              "closed"     "2021-01-26 10:30:00" "Close" "ocolong" "0"      "CloseLONG"
                         Time.In.Force               
2021-01-25 15:30:00.0000 "2021-01-26 15:29:00.000000"
2021-01-26 15:30:00.0000 ""