Backtester-EquityCurve-time-shifted-from-trading-positions

I can’t reproduce it. If you could email a complete strategy that exhibits the issue to support@wealth-lab.com I can investigate.

I tried reproducing it in BacktestComplete without success. It can only be reproduced in a ScoreCard object, which is weird when a plot of the equity curve (in BacktestComplete) shows the problem. Why would that be?

I'm going to email you the entire ScoreCard code. There's a statement in there that deliberately throws an error. That's the statement to watch out for. All the rest of the code is unimportant; just ignore that stuff. Thanks for looking at it.

I don't think the strategy or dataset used is that relevant, but I would pick a dataset with 100 or more stocks in it.

I think the reason PlotTimeSeries(Backtester.EquityCurve,...) sometimes fails to plot the equity curve is because it's too out of time sync with the other plots on the Chart. But equity curve time sync problems are very rare and hard to realize in BacktestComplete so I would give up trying to reproduce this sync problem there.

In contrast, the ScoreCard object is very sensitive to time sync problems (one out of 30 symbols encounter it) and for some reason the problem is more common in this context. So this is the location to focus on. If I can be of any further help, let me know. And thanks for working on it.

--
I forgot to mention, but the problem is deterministic. If I re-run the backtest on a given dataset, the exact same stocks fail. But if I move those failing stocks to their own dataset, they don't fail. So it's not the stocks themselves that are causing the failure; it's a combination of several things. And that combination falls apart if the stocks are moved to another dataset.

I'm not sure how this is surprising. The EquityCurve isn't made to plotted in a chart. It's created from synchronizing bars from all of the candidates. If the candidates are not in sync (especially if some include more bars than others in the backtest period) the EquityCurve cannot be in sync with all of the candidates - by defininition.

If you want to plot it in a chart context, you need to synchronize it to that context, which you didn't do.

QUOTE:
if some [stocks] include more bars than others in the backtest period) the EquityCurve cannot be in sync with all of the candidates

An interesting point. You're saying my PlotTimeSeries code (in the original post) needs to call

CODE:
BarHistory dailySynched = BarHistorySynchronizer.Synchronize(daily, bars);

first before it can be plotted.

My ScoreCard routine executes

CODE:
int equityCurveStartIdx = Backtester.EquityCurve.IndexOf(position.EntryDate);

And that returns a -1 (not found) on occasion (one out of 30 stocks) on a portfolio backtest. Are you saying this is expected?

If so, how would you process the -1 (not found) cases? Would you fix it at the IndexOf() method level (that seems weird) or does Gitch (or should I) have another way?

It seems to me if a stock buys on a specific date, that specific date should exist in the portfolio equity curve. Are you questioning that?

It seems odd to me that EntryDates in the Positions collection (all stocks) would not exist in the Backtester.EquityCurve. Is Glitch trying to fix a tricky problem?

I would expect the entry date to be in the equity curve. If you can isolate a specific instance that we can reproduce that will be helpful. Give us all the settings, strategy, DataSet, etc where this is occurring so we can reproduce it.

QUOTE:
If you can isolate a specific instance that we can reproduce that will be helpful. Give us all the settings, strategy, DataSet, etc ...

So you are saying you can't reproduce it with just any randomly chosen strategy and dataset? Well, that's really weird.

I don't think there's anything special about the strategy or dataset I'm using, but let me try a few others to check. If this problem is strategy and dataset specific, I would want to know why myself.

>>So you are saying you can't reproduce it with just any randomly chosen strategy and dataset?<<

Be careful assuming what someone is saying. I didn't say that, I just asked you to provide a case so we can reproduce it.

But I'll say it now, I added some code to assert that equityCurveStartIdx is always >= 0 in the Basic ScoreCard and ran it on several strategies with 100+ symbls, even a MetaStrategy, and I never ran into a case where I got a -1.

QUOTE:
I added some code to assert that equityCurveStartIdx is always >= 0 in the Basic ScoreCard and ran it on several strategies with 100+ symbls, even a MetaStrategy, and I never ran into a case where I got a -1.

Interesting.

I ran my strategy on the DOW 30 and got two cases, but when I switched Data Ranges and back, that problem disappeared. Strange. But in running on my original dataset. Clearly the choice of dataset matters. Here's the dataset I used:

ACIW ACLS ACRE ACT ACTG ADEA ADI ADSK AEIS AEM AEO AEP AER AESI AFG AFL AFRM AGCO AGL AGO AGRO AHCO AHH AI AIG AIV ALB ALC ALE ALGM ALIT ALK ALKS ALLE ALLO ALRM ALSN ALTG AMAL AMAT AMBA AMBP AMC AMD AMG AMGN AMK AMKR AMP AMPL AMSF AMT AMTB AMWD AMX ANIK ANIP APA APG APH APLE ARCC ARCH ARCO ARCT ARDX ARES ARMK ARRY ARVN ARW ASGN ASH ASIX ASLE ASO ASR ASUR ATEC ATHM ATI ATKR ATNI ATOM ATR ATRC AVB AVNT AVTR AVY AWI AWK AXAHY AXGN AXL AXON AXP AXS AXTA AZN AZPN AZTA

But here's the thing. If I change the Data Range from Most Recent N Bars (650) to Most Recent N Weeks (120) and back, the problem disappears. So does that mean my strategy instance is corrupted? Could it be the strategy code is okay, but the strategy XML is messed up...somehow?

So you're asking about the strategy XML and not the C# code itself. I didn't realize that. Let me investigate.

>>Could it be the strategy code is okay, but the strategy XML is messed up...somehow?<<

I feel that's very unlikely.

If you come up with a reproducible case let us know the settings, DataSet, etc.

QUOTE:
An interesting point. You're saying my PlotTimeSeries code (in the original post) needs to call... [some code]

Yah, I didn't say that either.

I said you needed to synchronize it to the chart context, and you did not. You elected to plot in BacktestEnd(), which doesn't have a reference to the chart's BarHistory (it's only called once for the whole backtest).

QUOTE:
You elected to plot in BacktestEnd(), which doesn't have a reference to the chart's BarHistory

So how can I "correctly" plot the Backtester.EquityCurve when there isn't a specific BarHistory for the entire backtest? Or is your point that it's impossible? But what if you're testing a single stock. Shouldn't you be able to plot the Backtester.EquityCurve?

By the way, I have been able to reproduce the -1 not-found date problem with the S&P500. So maybe there is something about my strategy code that breaks WL. How can I be so blessed? :-)

I'm going to try an earlier, simpler, WL6 version of the strategy to see if that still breaks it. It's the Voss Predicter by John Ehlers with some of my enhancements. (Maybe I over enhanced it.)

You could plot it in the Cleanup method using the BarHistory passed in the parameter.

Well, I have interesting news. I tried reproducing the problem in earlier strategies without success, but I was using a larger Data Range (1000 to 1300 daily bars) when I did. Then I went to the newest strategy and couldn't reproduce it either. So what was different?

It's the Data Range. A Data Range of 650 daily bars reproduces the problem with the dataset in Post #9, but Data Ranges of 1000 to 1300 bars (or more) does not. Don't ask me why.

But now that I know how to avoid the problem, I think I'm good. If I can figure out more useful information, I'll post it here.

I suspect your scorecard has a synchronization bug that doesn't properly align bar numbers (a.k.a. index). If you do nothing, a bar index give you the index in a BarHistory that is not synchronized to another context, like an equity curve. There are 2 solutions - 1) use DateTimes instead of a bar number/index, or if appropriate in the context, 2) use GetCurrentIndex to get the bar number of a BarHistory in a larger context, like Pre/PostExecute.

QUOTE:
I suspect your scorecard has a synchronization bug that doesn't properly align bar numbers (a.k.a. index). If you do nothing, a bar index give you the index in a BarHistory that is not synchronized to another context, like an equity curve.

You are correct that the position.EntryBar is not sync'd with the Backtester.equityCurve. If you don't correct for that, nothing works.

The code below relies on the Backtester.EquityCurve.IndexOf(position.EntryDate) to return an offset. The problem is it returns a "-1" (not found) because there's no such position.EntryDate in the Backtester.EquityCurve. The catch{...} code below prints these relevant values out in Post #14 when the throw statement (below) is reached.

CODE:
         {
             //below, calculate time sync offset (in bars) between "position" and "EquityCurve"
             int equityCurveStartIdx = Backtester.EquityCurve.IndexOf(position.EntryDate);
             if (equityCurveStartIdx < 0)
                 throw new ArgumentOutOfRangeException();
             equityCrvBarOffset = position.EntryBar - equityCurveStartIdx;
             previousStock = position.Bars; //record corresponding stock ID for equityCrvBarOffset
             //WLHost.Instance.AddLogItem("ScreenCard", equityCrvBarOffset.ToString() + "=EquityCurve bar offset for " + position.Bars.Symbol, WLColor.Green);
         }
         xEquityTradePoints[EquityTradePointIdx] = position.EntryBar;
         yEquityTradePoints[EquityTradePointIdx++] = bt.EquityCurve[position.EntryBar - equityCrvBarOffset];
     }
     catch (ArgumentOutOfRangeException ex) //Index out of range. Must be non-zero and < size of collection
     {
         string errMsg = string.Format("{0}=EntryBar {1}=barOffset {2}=EqCrvSize  ", position.EntryBar, equityCrvBarOffset, bt.EquityCurve.Count);
         string errMsg2 = position.EntryDate.ToShortDateString() + " " + Backtester.EquityCurve.IndexOf(position.EntryDate).ToString();
         WLHost.Instance.AddLogItem("ScreenCard", "Error assigning yEquityTradePoints for symbol " + position.Symbol + ": " + errMsg + errMsg2, WLColor.Red, ex);
         xEquityTradePoints[EquityTradePointIdx] = Double.NaN;
         yEquityTradePoints[EquityTradePointIdx++] = Double.NaN;
     }

Next, you're wondering in the IndexOf(...) method has a bug, which is causing it to fail at finding the position.EntryDate in the Backtester.EquityCurve. I've printed the dates for both out and can confirm there's no matching entry date in the equity curve.

So the question remains, "Why are there missing Dates in the Backtester.EquityCurve when the Date Range is less than two years (Daily scale)?" I don't know.

I sent the complete ScoreCard code to support@wealth-lab.com. If you can spot a problem that Glitch and I can't see, please say something. I still can't understand why the Date Range is even involved in this? Simply increasing the Date Range fixes the problem without changing the strategy, dataset, or ScoreCard.

superticker, you'll need to provide us with a concrete case where this is happening, we can't reproduce it locally. Try to boil it down to the simplest solution possible and send us the settings, DataSet, etc so we can reproduce what you are seeing.

QUOTE:
Try to boil it down to the simplest solution possible

I totally understand that. I was trying to do that yesterday by looking for a WL6 version of the Voss Predictor strategy that doesn't employ my Local.Components library. But none exist on the WL8 side. I'll have to strip out all the Local.Components calls from the early version I do have to see if I can still reproduce the problem with a less-than-two-year Date Range.

And in this process, I may discover more about this problem, which would be revealing.

Somehow the Backtester's equity curve gets out of time sync with the Positions collection when the Date Range is small. But how the strategy would influence that I don't know. John Ehlers' Voss Predictor strategy isn't doing anything weird.

Well, this hasn't been your "average" debugging problem. Let's start with some background.

I have a robust statistical analysis routine that set dynamic threshold points for trade entries. That routine must first gather enough statistical data before StartIndex can be assigned. The problem is every stock in the dataset is different, so the StartIndex for each stock is also different. So can't you emulate this operation with ...?

CODE:
StartIndex = random.Next(40, 160);  //Date Range: 450 Daily bars

Well, not exactly because trading starts shortly after the StartIndex point and the random number generator doesn't know anything about that. Moreover, this random number generator generates a new sequence with each strategy execution. (We need a different random number generator that generates the same sequence each time.)

I've modified the SeventeenLiner sample strategy to demonstrate the equity curve sync problem when you set the Date Range to 450 Daily bars. And I don't think the dataset matters, but I used the dataset listed in Post #9 for my testing.

But you may have to execute the sample strategy a half a dozen times before you get a failure because the random number generator isn't creating the same random sequence with each execution.

What we really need is a way to set the StartIndex relative to the first trade so we get reproducible behavior. Give it a try; otherwise, I can get my strategies working if I just use a large enough Date Range--and that works okay.

CODE:
using WealthLab.Backtest;
using System;
using WealthLab.Core;
using WealthLab.Indicators;

namespace WealthScript4 
{
    public class SeventeenLiner : UserStrategyBase
    {
      Random random = new();
      
        //create indicators and other objects here, this is executed prior to the main trading loop
        public override void Initialize(BarHistory bars)
        {
         StartIndex = random.Next(40, 160);  //Date Range: 450 Daily bars
         //StartIndex = dxSmoothedThres.FirstValidIndex;
         WriteToDebugLog("@" + bars.Symbol + " " + StartIndex.ToString("#=StartIndex"));
      }

        //execute the strategy rules here, this is executed once for each bar in the backtest history
        public override void Execute(BarHistory bars, int idx)
        {
         if (Lowest.Series(ROC.Series(bars.Close, 1), 2)[idx] > 0)
            foreach(var pos in OpenPositions)
               ClosePosition(pos,OrderType.Market);
         else if (
            (52 * Highest.Series(ROC.Series(bars.Close, 1), 3)[idx] <
            (15 + 12 * StdDev.Series(ROC.Series(bars.Close, 1), 31)[idx]))
            & (3 * LRSlope.Series(ROC.Series(bars.Close, 1), 3)[idx] < 19)
            & (Highest.Series(RSI.Series(bars.Close, 31), 31)[idx - 1] > 60)
            & (46 * StdError.Series(bars.Close, 31)[idx] > bars.Close[idx])
            & (53 * SMA.Series(bars.Volume, 31)[idx] > 24 * bars.Volume[idx])
            & (bars.Open[idx] > bars.Low[idx])
            & (3 * ATRP.Series(bars, 31)[idx] > 7))
         {
            priority = bars.Close.TurnsUp(idx) ? 1 : 0;
            priority = bars.Close.TurnsDown(idx) ? -1 : 0;
            priority = priority - ROC.Series(bars.Close, 4)[idx];
            int n = OpenPositions.Count + 1;
            if (n > 16) n = 1;
            if (n < priority) n = Math.Abs((int)priority);
            for (int i = 1; i < n + 1; i++)
            {
               var p = priority - i;
               var t = PlaceTrade(bars, TransactionType.Buy, OrderType.Market, default, (int)p);
               if(t != null)
                  t.Weight = p;
            }
         }
        }

      //declare private variables bebars.Low
      double priority = 0;
   }
}

I thought about later generating a Dictionary lookup of random numbers (symbols vs StartIndexes) for my testing dataset. If that would be helpful so you get a failure every time, let me know. It would be deterministic.

Well, this solution doesn't work as well as the random number generator, but at least it's deterministic. And that's what matters for debugging. For some reason, the random number generator amplifies the bad behavior, but I don't know why.

I "think" what happens is the StartIndex gets assigned on either the bar of the first trade or the bar just before the first trade. After that, everything gets messed up from that point forward.

Keep in mind, it's the DateTimes between the Positions and EquityCurve that are out-of-sync in the ScoreCard extension that's the problem. The bar numbers will always be offset between these two and that's expected.

UPDATE: Oops, I should have used DateTimes rather than bar numbers so results are reproducible over days. Oh well.

CODE:
    public class SeventeenLiner : UserStrategyBase
    {
      //Random random = new();
      Dictionary<string,int> dic = new Dictionary<string,int>();

      public SeventeenLiner()
      {
         dic.Add("ALE", 138);
         dic.Add("ATR", 100);
         dic.Add("AFG", 109);
         dic.Add("ANIP", 110);
         dic.Add("ARW", 125);
         dic.Add("AMC", 61);
         dic.Add("ALGM", 150);
         dic.Add("ACIW", 67);
         dic.Add("ATRC", 154);
         dic.Add("AFL", 43);
         dic.Add("ASGN", 42);
         dic.Add("APA", 96);
         dic.Add("ALIT", 145);
         dic.Add("AMD", 153);
         dic.Add("ACLS", 91);
         dic.Add("AVB", 59);
         dic.Add("AFRM", 65);
         dic.Add("AVNT", 140);
         dic.Add("AGCO", 136);
         dic.Add("ASH", 86);
         dic.Add("APG", 83);
         dic.Add("AMG", 87);
         dic.Add("ALK", 111);
         dic.Add("ACRE", 67);
         dic.Add("AMGN", 152);
         dic.Add("AVTR", 80);
         dic.Add("AGL", 65);
         dic.Add("APH", 70);
         dic.Add("ASIX", 136);
         dic.Add("ALKS", 150);
         dic.Add("ACT", 43);
         dic.Add("AMK", 68);
         dic.Add("AVY", 86);
         dic.Add("APLE", 98);
         dic.Add("AGO", 140);
         dic.Add("ASLE", 94);
         dic.Add("ALLE", 140);
         dic.Add("ACTG", 59);
         dic.Add("AMKR", 108);
         dic.Add("AWI", 54);
         dic.Add("ARCC", 154);
         dic.Add("AGRO", 69);
         dic.Add("ASO", 79);
         dic.Add("ALLO", 76);
         dic.Add("ADEA", 122);
         dic.Add("AMP", 77);
         dic.Add("AWK", 124);
         dic.Add("ARCH", 110);
         dic.Add("ARCO", 50);
         dic.Add("AHCO", 82);
         dic.Add("ALRM", 72);
         dic.Add("ASR", 114);
         dic.Add("ADI", 142);
         dic.Add("AMPL", 68);
         dic.Add("AXAHY", 112);
         dic.Add("ARCT", 137);
         dic.Add("AHH", 88);
         dic.Add("ALSN", 69);
         dic.Add("ASUR", 130);
         dic.Add("AMSF", 131);
         dic.Add("ADSK", 42);
         dic.Add("AXGN", 152);
         dic.Add("ARDX", 75);
         dic.Add("AI", 108);
         dic.Add("ALTG", 159);
         dic.Add("ATEC", 56);
         dic.Add("AMT", 138);
         dic.Add("AEIS", 87);
         dic.Add("AEM", 75);
         dic.Add("AXL", 109);
         dic.Add("ARES", 73);
         dic.Add("AIG", 53);
         dic.Add("AMAL", 146);
         dic.Add("AMAT", 74);
         dic.Add("ATHM", 54);
         dic.Add("AMTB", 102);
         dic.Add("AEO", 155);
         dic.Add("AXON", 55);
         dic.Add("ARMK", 64);
         dic.Add("AIV", 94);
         dic.Add("AMBA", 117);
         dic.Add("ATI", 109);
         dic.Add("AMWD", 66);
         dic.Add("AEP", 114);
         dic.Add("AXP", 46);
         dic.Add("ARRY", 47);
         dic.Add("ALB", 142);
         dic.Add("AMBP", 126);
         dic.Add("AMX", 74);
         dic.Add("ATKR", 53);
         dic.Add("AER", 130);
         dic.Add("AXS", 67);
         dic.Add("ARVN", 140);
         dic.Add("ALC", 129);
         dic.Add("AXTA", 105);
         dic.Add("ATNI", 149);
         dic.Add("ANIK", 94);
         dic.Add("AESI", 41);
         dic.Add("AZN", 104);
         dic.Add("ATOM", 128);
         dic.Add("AZPN", 126);
         dic.Add("AZTA", 77);
      }
      
        //create indicators and other objects here, this is executed prior to the main trading loop
        public override void Initialize(BarHistory bars)
        {
         StartIndex = dic[bars.Symbol];  //Date Range: 450 Daily bars
         //StartIndex = random.Next(40, 160);  //Date Range: 450 Daily bars
         //StartIndex = dxSmoothedThres.FirstValidIndex;
         WriteToDebugLog("@" + bars.Symbol + " " + StartIndex.ToString("#=StartIndex"));
      }