Hi, today just to double check reran a backtest, got radically different results, clicked again, different again and so forth every time. Here is video of about 5 clicks in a row, screen cap. https://youtu.be/y6uHy0jRa4s
Is this a known problem?
Is this a known problem?
Rename
Hi Darp, no it’s totally expected. If you read our FAQ under the Support menu you’ll learn why it’s happening.
Glitch I see
In some cases it's possible that Strategy's StartIndex starts before the longest period indicator. See the Best Answer post here for explanation and workarounds"
I was not optimizing, was just running the same parameters in backtest. I went back into Surface parameters and all but one parameter was hardwired, so turned off the one that still had a parameter that could be tested. And clicked "non-parallel" Then saved workspace. Rebooted, tried again and exact same problem, every run is very different result. Also the last link in FAQ does not seem to apply as I am not optimizing.
I did some experiments. If I move to just 1 symbol, not Nasdaq100, then results stay the same. Also I took a RSI strat WL comes with and it stayed the same on a small portfolio and a single stock. Two of mine fail to stay stable on that same small portfolio. Hope that info helps.
I then opened another strategy that uses RSI not MFI and again every backtest run is different. I had not before done many of these reruns, just did it today as a double check and noticed this. What do you think?
QUOTE:
"Why my backtest and optimization results don't match? Why exhaustive optimization may return unexpected results?
Inconsistency may be caused by the Exhaustive optimization running in parallel to speed things up. Use the Exhaustive (Non-Parallel) Optimizer and your results should match. Also, be careful when optimizing strategies with static variables using a parallel optimizer and either adopt in your code a design pattern outlined here or use a non-parallel optimizer.
In some cases it's possible that Strategy's StartIndex starts before the longest period indicator. See the Best Answer post here for explanation and workarounds"
I was not optimizing, was just running the same parameters in backtest. I went back into Surface parameters and all but one parameter was hardwired, so turned off the one that still had a parameter that could be tested. And clicked "non-parallel" Then saved workspace. Rebooted, tried again and exact same problem, every run is very different result. Also the last link in FAQ does not seem to apply as I am not optimizing.
I did some experiments. If I move to just 1 symbol, not Nasdaq100, then results stay the same. Also I took a RSI strat WL comes with and it stayed the same on a small portfolio and a single stock. Two of mine fail to stay stable on that same small portfolio. Hope that info helps.
I then opened another strategy that uses RSI not MFI and again every backtest run is different. I had not before done many of these reruns, just did it today as a double check and noticed this. What do you think?
Sounds to me like you're overallocating in your position sizing.
After a backtest, go to the Backtest Result and then the Metrics Report tab. If the Maximum Exposure > 100% then you're likely going to get different results on each run.
After a backtest, go to the Backtest Result and then the Metrics Report tab. If the Maximum Exposure > 100% then you're likely going to get different results on each run.
Hi Paul,
I was working at finding better results with the various Position sizing options when noticed this. You can see the parameters in the short video linked. I had gone to 1.5 margin. I just took that to 1.0 but still have the issue.
I think would have noticed this before, as would backtrack aftre getting worse results to older parameters on the strategy. I am going to do some experiments based on your concept. So in your experince you get very steady results on backtest?
Edit, I copied the Startegy Settings from one strat that does get same results to the two that I see the problem in. Again on one stock the two work fine, same results in backtest, but on list of stocks, again big diffrences ever run. Hard to figure.
I was working at finding better results with the various Position sizing options when noticed this. You can see the parameters in the short video linked. I had gone to 1.5 margin. I just took that to 1.0 but still have the issue.
I think would have noticed this before, as would backtrack aftre getting worse results to older parameters on the strategy. I am going to do some experiments based on your concept. So in your experince you get very steady results on backtest?
Edit, I copied the Startegy Settings from one strat that does get same results to the two that I see the problem in. Again on one stock the two work fine, same results in backtest, but on list of stocks, again big diffrences ever run. Hard to figure.
Darp it’s very simple. It has nothing to do with optimization. Look at the Metrics Report and find NSF Position Count. If it’s above zero it means there were more signals in the simulation than available simulated capital. In this case you’ll always get different results each run because WL will randomly pick signals to fill from amongst the candidates.
Reduce the position size and/or increase the margin factor until NSF Positions go to zero for stable results.
Reduce the position size and/or increase the margin factor until NSF Positions go to zero for stable results.
ah, will try.
Glitch, yes like 300. I added a bunch of money and dropped it down to 5% max size and it went away. Even at 7-8% it would vary a little like + or - 3% on results. But it was varying +-15% before that at 15% capital per position.
Thanks!
I have two suggestions.
A. Which is to have it select which trades to use in alpha order, and user can even say start with A or Z backwards. Then it would get identical backtest results each time. Correct me if wrong on that. Anything to get it to pick the same stocks instead of random would be helpful.
B. Maybe this exists already? There could be a way after say 50% of capital is gone to reduce size progressively so it never runs out of money. The profit per bar % would still be right for all stocks, just that AAPL might be 3 times more $s than ADBE. Advantage is when looking at trades on chart every signal would show, that would be most helpful to evaluate how a system works on a stock.
2nd thing is Meritocratus. I was thinking of doing exact same concept before finding it. Very cool. I just tested it and it would seem it would pick the same stocks each time, but it does not, guess because too many are over the bar of winning percent. But it does get better results.
Cheers
Glitch, yes like 300. I added a bunch of money and dropped it down to 5% max size and it went away. Even at 7-8% it would vary a little like + or - 3% on results. But it was varying +-15% before that at 15% capital per position.
Thanks!
I have two suggestions.
A. Which is to have it select which trades to use in alpha order, and user can even say start with A or Z backwards. Then it would get identical backtest results each time. Correct me if wrong on that. Anything to get it to pick the same stocks instead of random would be helpful.
B. Maybe this exists already? There could be a way after say 50% of capital is gone to reduce size progressively so it never runs out of money. The profit per bar % would still be right for all stocks, just that AAPL might be 3 times more $s than ADBE. Advantage is when looking at trades on chart every signal would show, that would be most helpful to evaluate how a system works on a stock.
2nd thing is Meritocratus. I was thinking of doing exact same concept before finding it. Very cool. I just tested it and it would seem it would pick the same stocks each time, but it does not, guess because too many are over the bar of winning percent. But it does get better results.
Cheers
You can assign a transaction Weight to control the priority. If unassigned it assumes a random number. That’s how we handle that.
QUOTE:
Glitch I see
WAY above what you see there re: optimization is a paragraph that we actually expected you to see:
Every time I run a Strategy I get a different result. Why?
Thanks Eugene
Your Response
Post
Edit Post
Login is required