- ago
I can't seem to figure out why my backtest results are different for each run (with the exact same parameters). In the metrics report I see that there are 0 NSF positions and my position size is set to 1 share on a 200,000k account and trading 12 symbols. I went through the FAQ's and can see that NSF can lead to random/different positions/results however that doesn't seem to be the case here. Position count seems to stay the same as well.

Video below:
https://drive.google.com/file/d/1ybnnITFyewUqfMCnfktbgi8WJMgXbDJ1/view?usp=sharing
0
953
Solved
15 Replies

Reply

Bookmark

Sort
- ago
#1
Since it's a coded strategy we have little idea what may be going on there. 🤷‍♂️
0
- ago
#2
It's the ORB from the sample strategies, I just copied it over to my own file. Even running the one under sample strategies returns me different results each time with no NSF positions.
0
Glitch8
 ( 11.81% )
- ago
#3
Not sure, my results are the same running the ORB Strategy on 1 minute data, provided my position size is small enough to take all the signals. Perhaps you turned off "Retain NSF Positions" in Advanced Strategy Settings?
0
MIH8
- ago
#4
Let me be a little sarcastic, please: If you configure zero in the position sizing you will always get the same result.

Now seriously. What is the idea, the rationale to reduce the position sizing?
It just doesn't make sense. Besides, it doesn't always lead to the goal.

@adrianbegi

Here is my suggestion.
In Post #34 you will find a short summary that references different posts in the thread.
At the end you can find a alternative implementation of Post #32.
You can decide yourself if it is useful for you. In Post #36 the comparision is done.

https://www.wealth-lab.com/Discussion/How-to-configure-deterministic-backtests-8203

0
Glitch8
 ( 11.81% )
- ago
#5
The reason is to ensure you have enough simulated equity to take all of the trades.
0
MIH8
- ago
#6
Ok, so the effect of this action is to be able to execute all potential trades.
Your statement implicitly includes that the basis for the result and the significance are changed.

Further, one then does this indirectly via position sizing to ensure equity.
So you could also increase the equity directly, if I interpret that correctly now.

Adjusting the position size changes equity. Ultimately, the capital is the decisive factor whether a trade can be executed. This in turn changes in the course of the backtest anyway. The initial event that leads to different course may well occur again when the capital develops close to the initial state.

In summary, this procedure leads to the fact that with some luck an event is not repeated at a special time.
However, there is no guarantee for it, since the capital develops also during the backtest.

But what really should be considered is that getting an identical result is not an aim in itself.
One wants to have an identical result only under certain constraints.

A restriction is for example to leave the backtest parameters, including the position sizing, unchanged.

The success of strategies is also dependent on the capital.
Strategies with 100K capital can be successful if commissions are captured in the profit. But the same strategy with 10K can fail.

Thanks for your feedback.
1
- ago
#7
Sorry I am completely lost. As you can see in the video there is no NSF positions and all other factors are staying the same but the results are different every time. I am using a simple 1 share entry on a 200k account so I know there is enough capital.

Can anyone paste the ORB sample strategy code so I can make sure I haven’t made any accidental adjustments?

Could the dataset effect this? I run it on AMD,MSFT,TSLA,AAPL,NVDIA,SPY,TQQQ,SQQQ,SHOP,SQ but also has the same issue when I run it on 1 symbol.
0
- ago
#8
QUOTE:
Can anyone paste the ORB sample strategy code so I can make sure I haven’t made any accidental adjustments?


As a side note. If you've modified the "Opening range breakout" canned strategy, follow these steps to reset:

1. Choose "Open WL User Data Folder" from the File menu.
2. Delete the canned strategy "Opening range breakout".
3. Close WL.
4. Edit Settings.txt, deleting the line that starts with StrategiesCopied= or (less invasive) just this piece: "Opening range breakout;"

On next start your strategy(ies) should be copied over.
1
- ago
#9
Ok so I did make a change, correct me if I am wrong but the sample Opening Range Breakout will never take a short position correct?

line 101
CODE:
               if (PlaceTrade(bars, TransactionType.Buy, OrderType.Stop, HighRange) == null)                   PlaceTrade(bars, TransactionType.Short, OrderType.Stop, LowRange);


When I remove the if and PlaceTrade to both long and short side is when I get different results everytime. Any idea why this happens or why the sample strategy is set to only go long?
0
Glitch8
 ( 11.81% )
- ago
#10
Would you mind posting your complete code, or emailing it to support@wealth-lab.com?

Also, the ORB sample Strategy is faulty. PlaceTrade never returns a null, this logic it seems is based on how WL6 and earlier used to work, not the new WL7+ paradigm.
1
- ago
#11
Hey Glitch, I don't mind posting it here. The only difference is that I added the choppiness indicator as it showed as a promising filter in the analysis series and I removed the if statement from placetrade on the long position as I mentioned above so that it takes shorts as well.

CODE:
using WealthLab.Backtest; using System; using WealthLab.Core; using WealthLab.Indicators; using System.Drawing; using System.Collections.Generic; namespace WealthScript4 {    public class ORB : UserStrategyBase    {       public ORB()       {          AddParameter("Range", ParameterType.Int32, 30, 30, 60, 1);          AddParameter("Profit Target %", ParameterType.Double, 3, 1, 10, 1);          AddParameter("Stop Loss %", ParameterType.Double, 2, 1, 10, 1);       }       //create indicators and other objects here, this is executed prior to the main trading loop       public override void Initialize(BarHistory bars)       {          if(!bars.IsIntraday)             return;          //PlotStopsAndLimits();          range = Parameters[0].AsInt;          target = Parameters[1].AsDouble;          stop = Parameters[2].AsDouble;          int n = bars.Scale.Interval;          num = (range / n) - 1; // the N-minute bar          firstIntradayBar = -1;          lastChartBar = bars.Count - 1;          nextToLastBarTime = Convert.ToInt32(bars.Market.GetCloseTimeMarketTime().ToString("HHmm")).AddIntegerTime(-bars.Scale.Interval);          choppiness = Choppiness.Series(bars, 14);          //PlotIndicator(choppiness,WLColor.Blue);       }       //execute the strategy rules here, this is executed once for each bar in the backtest history       public override void Execute(BarHistory bars, int idx)       {          if (bars.IntradayBarNumber(idx) == 0)          {             firstIntradayBar = idx;             canEnterToday = true; //   one position per day          }          if (bars.IntradayBarNumber(idx) <= num) // highlight the opening range             SetBarColor(bars, idx, WLColor.Silver);          if (bars.IntradayBarNumber(idx) == num) // get the highest high and the lowest low after first hour          {             HighRange = Highest.Series(bars.High, num + 1)[idx];             LowRange = Lowest.Series(bars.Low, num + 1)[idx];             if (firstIntradayBar > -1)             {                DrawLine(idx, HighRange, firstIntradayBar, HighRange, WLColor.Blue, 1, LineStyle.Dashed);                DrawLine(idx, LowRange, firstIntradayBar, LowRange, WLColor.Red, 1, LineStyle.Dashed);             }          }          if (HasOpenPosition(bars, PositionType.Long) || HasOpenPosition(bars, PositionType.Short))          {             canEnterToday = false;             Position p = LastPosition;             var dt = bars.DateTimes[idx].Date;             //   Exit last bar of day             if ((idx == lastChartBar) && (dt.GetTime() >= nextToLastBarTime))             {                SetBarColor(bars, idx, WLColor.Orange);                ClosePosition(p, OrderType.Market, 0, "Exit at open of last bar");             }             else if ((idx != lastChartBar) && bars.IsLastBarOfDay(idx + 1))             {                SetBarColor(bars, idx, WLColor.Orange);                ClosePosition(p, OrderType.Market, 0, "Exit at open of last bar");             }             else             {                //   Stop Loss (optimizable)                //   Profit Target (optimizable)                if ( p.PositionType == PositionType.Long)                {                   ProfitTgt = p.EntryPrice * (1 + target / 100.0d);                   StopPrice = p.EntryPrice * (1 - stop / 100.0d);                   //StopPrice = bars.Low[idx] - .01;                }                else                {                   ProfitTgt = p.EntryPrice * (1 - target / 100.0d);                   StopPrice = p.EntryPrice * (1 + stop / 100.0d);                   //StopPrice = bars.High[idx] + .01;                }                ClosePosition(p, OrderType.Stop, StopPrice, "Stop Loss");                ClosePosition(p, OrderType.Limit, ProfitTgt, "Profit Target");             }          }          else          {             if (bars.IntradayBarNumber(idx) >= num && !bars.IsLastBarOfDay(idx) && canEnterToday && choppiness[idx] < 65)             {                PlaceTrade(bars, TransactionType.Buy, OrderType.Stop, HighRange + .01,1,"long entry");                PlaceTrade(bars, TransactionType.Short, OrderType.Stop, LowRange - .01,1,"short entry");             }          }       } //declare private variables below int lastChartBar, firstIntradayBar, range, num, nextToLastBarTime;       bool canEnterToday = false;       double HighRange = 0, LowRange = 0, target = 0, stop = 0, ProfitTgt = 0, StopPrice = 0;       Choppiness choppiness;    } }
0
Glitch8
 ( 11.81% )
- ago
#12
It looks like the reason is that it's possible that your Strategy can get a long and a short position at the same time, if both orders are filled on the same bar. Yet, the exit logic is assuming that this will never be the case. I modified the code to fix this, when running with Pre/Post Market data filtered out I get the same results on each run now using your DataSet and a 1 minute scale.

CODE:
using WealthLab.Backtest; using System; using WealthLab.Core; using WealthLab.Indicators; using System.Drawing; using System.Collections.Generic; namespace WealthScript4 {    public class ORB : UserStrategyBase    {       public ORB()       {          AddParameter("Range", ParameterType.Int32, 30, 30, 60, 1);          AddParameter("Profit Target %", ParameterType.Double, 3, 1, 10, 1);          AddParameter("Stop Loss %", ParameterType.Double, 2, 1, 10, 1);       }       //create indicators and other objects here, this is executed prior to the main trading loop       public override void Initialize(BarHistory bars)       {          if(!bars.IsIntraday)             return;          //PlotStopsAndLimits();          range = Parameters[0].AsInt;          target = Parameters[1].AsDouble;          stop = Parameters[2].AsDouble;          int n = bars.Scale.Interval;          num = (range / n) - 1; // the N-minute bar          firstIntradayBar = -1;          lastChartBar = bars.Count - 1;          nextToLastBarTime = Convert.ToInt32(bars.Market.GetCloseTimeMarketTime().ToString("HHmm")).AddIntegerTime(-bars.Scale.Interval);          choppiness = Choppiness.Series(bars, 14);          //PlotIndicator(choppiness,WLColor.Blue);       }       //execute the strategy rules here, this is executed once for each bar in the backtest history       public override void Execute(BarHistory bars, int idx)       {          if (bars.IntradayBarNumber(idx) == 0)          {             firstIntradayBar = idx;             canEnterToday = true; // one position per day          }          if (bars.IntradayBarNumber(idx) <= num) // highlight the opening range             SetBarColor(bars, idx, WLColor.Silver);          if (bars.IntradayBarNumber(idx) == num) // get the highest high and the lowest low after first hour          {             HighRange = Highest.Series(bars.High, num + 1)[idx];             LowRange = Lowest.Series(bars.Low, num + 1)[idx];             if (firstIntradayBar > -1)             {                DrawLine(idx, HighRange, firstIntradayBar, HighRange, WLColor.Blue, 1, LineStyle.Dashed);                DrawLine(idx, LowRange, firstIntradayBar, LowRange, WLColor.Red, 1, LineStyle.Dashed);             }          }          if (HasOpenPosition(bars, PositionType.Long) || HasOpenPosition(bars, PositionType.Short))          {             canEnterToday = false;             Position p = LastPosition;             var dt = bars.DateTimes[idx].Date;             // Exit last bar of day             if ((idx == lastChartBar) && (dt.GetTime() >= nextToLastBarTime))             {                SetBarColor(bars, idx, WLColor.Orange);                ClosePosition(p, OrderType.Market, 0, "Exit at open of last bar");             }             else if ((idx != lastChartBar) && bars.IsLastBarOfDay(idx + 1))             {                SetBarColor(bars, idx, WLColor.Orange);                ClosePosition(p, OrderType.Market, 0, "Exit at open of last bar");             }             else             {                // Stop Loss (optimizable)                // Profit Target (optimizable)                foreach (Position pos in OpenPositions)                {                   if (pos.PositionType == PositionType.Long)                   {                      ProfitTgt = p.EntryPrice * (1 + target / 100.0d);                      StopPrice = p.EntryPrice * (1 - stop / 100.0d);                      ClosePosition(pos, OrderType.Stop, StopPrice, "Stop Loss");                      ClosePosition(pos, OrderType.Limit, ProfitTgt, "Profit Target");                   }                   else                   {                      ProfitTgt = p.EntryPrice * (1 - target / 100.0d);                      StopPrice = p.EntryPrice * (1 + stop / 100.0d);                      ClosePosition(pos, OrderType.Stop, StopPrice, "Stop Loss");                      ClosePosition(pos, OrderType.Limit, ProfitTgt, "Profit Target");                   }                }             }          }          else          {             if (bars.IntradayBarNumber(idx) >= num && !bars.IsLastBarOfDay(idx) && canEnterToday && choppiness[idx] < 65)             {                PlaceTrade(bars, TransactionType.Buy, OrderType.Stop, HighRange + .01, 1,"long entry");                PlaceTrade(bars, TransactionType.Short, OrderType.Stop, LowRange - .01, 1,"short entry");             }          }       }       //declare private variables below       int lastChartBar, firstIntradayBar, range, num, nextToLastBarTime;       bool canEnterToday = false;       double HighRange = 0, LowRange = 0, target = 0, stop = 0, ProfitTgt = 0, StopPrice = 0;       Choppiness choppiness;    } }
1
Best Answer
- ago
#13
Beat me too it, I just realized that is possible (Even when using Granular Limit/Stop).

I appreciate the help. Going to be running this live next week again as it did really well last week.
0
- ago
#14
Couple of questions

1. What is the purpose of this if else? If the purpose is to exit on the last bar of the day then wouldn't it always hit the else if first?
CODE:
            if ((idx == lastChartBar) && (dt.GetTime() >= nextToLastBarTime))             {                SetBarColor(bars, idx, WLColor.Orange);                ClosePosition(p, OrderType.Market, 0, "Exit at open of last bar");             }             else if ((idx != lastChartBar) && bars.IsLastBarOfDay(idx + 1))             {                SetBarColor(bars, idx, WLColor.Orange);                ClosePosition(p, OrderType.Market, 0, "Exit at open of last bar");             }

2. I noticed we are still using Position p = LastPosition; when looping through all the positions to create the ProfitTgt and StopPrice, should we not be using pos in the foreach?
0
Glitch8
 ( 11.81% )
- ago
#15
Yes we should! My mistake.
0

Reply

Bookmark

Sort