Difference-between-optimization-results-and-a-backtest-run-with-these-values

Hi,

My first assumption is this FAQ > "Every time I run a Strategy I get a different result. Why?"
https://www.wealth-lab.com/Support/Faq

Please read it and review/adjust your strategy.

Hi Eugene,

thank you for your message but it is not this. I have set transaction weight. So, I always get the same result. The optimization shows much higher APRs than I got before optimization. But when I choose "Run a backtest with these parameter values" after right clicking a line in the optimization results, I always get a result significantly lower - however, always the same.

When I choose the +38.43% APR I get +14.88%. But when I choose the +35.80% result, I always get +13.71%.

But I think, I just solved the problem. I ran the optimization again. And this time the "run a backtest with these parameter values" worked. I guess, I must have changed something (unconsciously) in the strategy after have done the optimization...

But thanks anyway for helping me on a weekend!

Werner

I encountered a similar problem.

Sometimes when I made a right-click to "Run a Backtest with these Parameter Values" I got got different results from the optimization, but in some cases the reason could be tracked down to the transfer of wrong parameters to the Backtest. This could be solved by just going back to the Tabular and restart the Backtest with a right-click. Not a big thing.

But in some cases the deviation remains. The optimization can be redone several times, always generating the same results, just the same with the backtest. The transfer from the optimization to the backtest seams to work, but the results differ in APR, Profit, PositionCount.

I can provide the code of a sample strategy and all parameters to show the effect.

Have you upgraded to the latest version? There was a known bug fixed right in build 15:
https://www.wealth-lab.com/Discussion/-quot-Run-a-backtest-with-these-parameters-quot-uses-the-wrong-parameters-6573

Just doublechecked it with build 15: Same result. The "Standard Opt Params" in the "Strategy Parameters" are the correct ones, but the deviations in all results in the metrics report between the optimization and the backtest remain the same.

Can you describe your workflow? What I did was optimize a strategy, then sorted by Profit to get the best result. I then opened the same strategy again, set the parameters via the sliders, and ran the backtest. The results matched to the penny.

Also, how many NSF Positions are there in your Metrics report?

I did just the same, I optimized a strategy, sorted by Profit to get the best result, opened the strategy with the same parameters and ran the backtest. In the backtests I get the same results if I make a backtest by rightclick on the optimization line or by entering the parameters via the sliders.

As I saw you tested the optimization with one symbol I did the same. In that case of all the 3 symbols I tested the results in the optimzation and in the backtest matched perfectly. But when I changed to a Portfolio Backtest with DOW 30, the deviations were back again.

In the Portfolio Backtest I got NSF Position Count 0, in the optimization with the same parameters 185, with Single Symbol in all cases 0 (as was to be expected).

When I did the same test on the Dow 30, ensuring a position size such that I would get no NSF, my results are still identical:



Maybe you have Retain NSF turned off in Backtest Settings but you really are getting NSF? What's your position size and what DataSet?

The DataSet is the Wealthdata DOW30, the position size is 25% of equity, Margin Factor 1.1. I can provide a code with all data to check if you get the same deviation.

If the position size is 25% of equity then likely your strategy has positions that cannot be taken due to insufficient equity. Try reducing the position size to 1% as a test and see if that makes a difference.

With the reduced position size the results match perfectly.

As the reduction of the position size is not solving my problem I made some additional tests with "Retain NSF positions". I investigate strategies usually without retaining NSF positions and in some strategies there is a significant difference between retaining or not retaining the NSF positions.

According to my tests the optimization seems to neglect the "Retain NSF Positions" checkbox. The optimization result is the same if the box is checked or not, but when the checkbox in the advances settings is checked, optimization and backtest match, if the checkbox is unchecked, there are deviations between optimization and backtest.

I suppose this should be fixed.

Thank you. And thank you for working for us even on the weekend.

I am running throught the same difficulties as described by quantarum -- under build 18. whether NSF is checked or not in advanced settings, I continue to have major discrepancies between optimization values and backtest runs (right-click) for an optimized value. I was using Wealthdata DOW30 as dataset. I suspect the issue lies with WD data as Wealthdata has some kind of historical DOW30 component.
If you use Yahoo - US DJ 30 Industrials the issue goes away

QUOTE:
I suspect the issue lies with WD data as Wealthdata has some kind of historical DOW30 component.

Exactly. I mentioned this recently on the forum. The dynamic index's composition changes with time, so this is not an issue.

you need to ensure that you have 0 nsf trades, period. The retain nsf option doesn’t make a difference. but it’s best to leave it on so you can see your nsf count on the metrics report. only when you have enough capital to take all candidate trades will the results match.

Thank you Eugene - as usual
Dynamic vs fixed index compositions will deliver different backtest results - however not clear why optimization vs backtest with dynamic index (ie same dataset) produces different outcomes

It would be because your strategy probably had some nsf positions. Unless you assign a Weight, WL7 will determine which ones to include randomly.

makes sense - thank you Glitch !