Scorecard-discrepancy-between-Strategy-and-Long-Only-columns

There must be something else at play that isn't mentioned.

I ran multiple backtests of a long-only system, having switched Scorecards, changed the strategy parameters many times, checking "Retain NSF" and changing the position size in between runs. The Strategy vs. Long Only figures always match.

Tell us more about the strategy kind (compiled, editor, Blocks etc.) and your Strategy settings.

Yes, I see similar from time to time. Still I have nothing to add to reproduce. Here is my topic about this:

https://www.wealth-lab.com/Discussion/Strange-values-in-Backtest-Results-Metrics-Report-6477

QUOTE:
Tell us more about the strategy kind (compiled, editor, Blocks etc.) and your Strategy settings.

I've had it occur on both Block and Coded strategies.

I thought I would test a WL7 example strategy so that we can rule out any custom strategy issues. Results below from the Simple Channel Breakout 50.

It took just 6 backtest runs to produce the issue. However, I have retraced my steps in producing this and was unsuccessful the second time. I am wondering if it has anything to do with the timing of viewing the scorecard tab. I have noticed if i open the Scorecard tab mid-way through a backtest, sometimes the values are all 0's. Othertimes it is populated. Is it posible for the scorecards are being triggered / populated before backtest is complete?

Note: I was not optimizing or running any other backtests at the time of producing this. I do have another strategy open, but dormant. I also have a chart open (daily / static with a couple of indicators added - not streaming)

Thanks for the follow-up.

QUOTE:
I have noticed if i open the Scorecard tab mid-way through a backtest, sometimes the values are all 0's.

It happened to me too when I had the Metrics Report open during a backtest and started switching the Scorecards back and forth. Which doesn't make sense to do. If I just had the tab open, the values were populated.

QUOTE:
Note: I was not optimizing or running any other backtests at the time of producing this. I do have another strategy open, but dormant. I also have a chart open (daily / static with a couple of indicators added - not streaming)

I did the same (opened a dormant strategy). Also exposed a few parameters to Simple Channel Breakout 50 and started changing them between backtests. With debug build 33 the results were rock solid, no mismatch. Let me try the production build too.

UPDATE. No change, production build 32 works as expected after half a dozen parameter permutations.

Maybe @splodge and @replikant have included their own ScoreCard extensions?

Good guess @MaRe. I thought about not having the Finantic ScoreCard installed but forgot to tell.

I don't have something like that installed now. But I had Finantic ScoreCard installed before. I removed it in a couple of days or something, but maybe there are dlls that still there? Anyway I don't see extra scorecards in WL7 menus.

I have my own scorecard installed, but not the Finantic Scorecard on the notebook that produced the above screenshots.

Considering my scorecard is called after the Basic and Extended, and I am not altering scorecard metrics, just adding my own., I don't see what interaction from my code could cause the issue. But, my assupmtions have been wrong before :)

I'll install build 33, remove my card and test again.

Side note: When the issue occurs on the basic and extended cards the values generated by my card are correct in both columns, but any imported values from the basic or advanced card are not. (ignore the impactindex and qualitycurve as are always 0 in long only as they reference the equity curve data which is not available in short or long column initializations. Also, the rounding of the values in the longOnly column to 2 decimals is WL7 not my code. This seems to happen with all Long/Short column values)

Update.
- Build 33 installed
- All WL extensions up-to-date
- No Finantic extensions installed
- No Score cards or proprietary code (dll's) in program directory

It took about 30 backtests but the issue has surfaced again.

This instance was with a brand new building block strategy. (A Simple Wm%R cross) . Please ignore the different values on the design and settings screenshots. I took these just after I began trying to duplicate the issue with different values

I've been trying to lock down the cause but no joy yet.

Still working with Build 33, no thirdparty code, extensions, indicators, etc. Only WL7 Shipped code or code generators.

No other strategies, code editors, tools or extensions open.

Here is another example after making some block changes to the strategy shown above. I must note that after the occurance reported earlier with this strategy I treid about 50 or 60 times to recreate the issue with no luck.

I then added a new exit block and conditions and the issue poped up again immediately after adding the "For N Consequtive Bars" qualifier to the sell block (and made it an optimizable param). I can't say that is the cause, its just the last thing I did.

Could you export the problematic strategy (Strategies > right click on it > choose "Export strategies") and send the file to support@wealth-lab.com?

I can, but it's not limited to one strategy. I've had this occur on C-Coded strategies, building blocks, and examples shipped with WL7.

I appreciate that it is difficult to duplicate. There does not seem to be any rhyme or reason for it.

Interestingly, and perplexingly, when you get a discrepency like those I've shown and simply hit the backtest button again (no changes to values or code/blocks etc) it will return matching columns.

The new matching results seem to reflect the Long Only column values when the discrepency occured.

-------------------------------------
Thinking outloud:

Ovbiously my understanding is limited, but its almost as though data is changing in the BT object between calling the metric calculations for each column.

Is there any chance of threads spawned for the BT calculations not completing before the scorecard population is called?

This however doesn't explain why Basic and Extended metrics are affected but not those generated by my personal scorecard (which was removed for several tests as shown above) and If you are processing in alphabetical order

- Basic (Srat ,Bench,Long,Short)
- Extended(Srat ,Bench,Long,Short)
- UTM (My scorecard)

then why does the basic long column appear correct when the extended strategy column shows discrepencies?

Considering this, some sort of aborting of calculations seems more likely, but unless you are handling every possible error for each calculation I would expect some metrics to be "skipped".

I'll think on this some more and see if I can divise a test that will help us better understand what is happening.

An update:

I've created a metric in my scorecard that compares bt.Metrics with bt.LongResult Metrics. This is then reported to the Optimization results if the discrepency occurs.

The function is simple. It check Basic Scorecard Metric values and return 0 for match and 1 for discrepency. Code below. Note that I am not comparing my own metrics, and the function returns 0 value (default match=true) if error occurs to prevent any false positives.

The call to the function is at the top of my scorecard ninitialization, before any metrics are calculated.

CODE:

     
 ' calling code from initialization
 ' check for discrepency in result of Long and Strategy Columns.
                    Try
                        .Discrepency = CheckForDiscrepency(bt)
                    Catch ex As Exception
                        .Discrepency = -99 ' set to error
                    End Try

    Private Function CheckForDiscrepency(bt As Backtester)

            Try
                ' check long only values against strategy total values, they should match for all Long Only Strategy, otherwise raise flag.
                If bt.Metrics.Profit <> bt.LongResults.Metrics.Profit Then Return 1
                If bt.Metrics.ProfitPct <> bt.LongResults.Metrics.ProfitPct Then Return 1
                If bt.Metrics.ProfitPerBar <> bt.LongResults.Metrics.ProfitPerBar Then Return 1
                If bt.Metrics.APR <> bt.LongResults.Metrics.APR Then Return 1
                If bt.Metrics.Commission <> bt.LongResults.Metrics.Commission Then Return 1
                If bt.Metrics.Dividends <> bt.LongResults.Metrics.Dividends Then Return 1
                If bt.Metrics.PositionCount <> bt.LongResults.Metrics.PositionCount Then Return 1
                If bt.Metrics.AvgProfit <> bt.LongResults.Metrics.AvgProfit Then Return 1
                If bt.Metrics.ProfitFactor <> bt.LongResults.Metrics.ProfitFactor Then Return 1
                If bt.Metrics.AvgBarsHeld <> bt.LongResults.Metrics.AvgBarsHeld Then Return 1
                If bt.Metrics.NSFPositionCount <> bt.LongResults.Metrics.NSFPositionCount Then Return 1
                If bt.Metrics.MaxDrawdown <> bt.LongResults.Metrics.MaxDrawdown Then Return 1
                If bt.Metrics.MaxDrawdownPct <> bt.LongResults.Metrics.MaxDrawdownPct Then Return 1
                If bt.Metrics.RecoveryFactor <> bt.LongResults.Metrics.RecoveryFactor Then Return 1
                If bt.Metrics.GrossProfitWinners <> bt.LongResults.Metrics.GrossProfitWinners Then Return 1
                If bt.Metrics.CountWinners <> bt.LongResults.Metrics.CountWinners Then Return 1
                If bt.Metrics.WinRate <> bt.LongResults.Metrics.WinRate Then Return 1
                If bt.Metrics.AvgProfitWinners <> bt.LongResults.Metrics.AvgProfitWinners Then Return 1
                If bt.Metrics.CountLosers <> bt.LongResults.Metrics.CountLosers Then Return 1
                If bt.Metrics.LossRate <> bt.LongResults.Metrics.LossRate Then Return 1
                If bt.Metrics.AvgLossLosers <> bt.LongResults.Metrics.AvgLossLosers Then Return 1

                Return 0
            Catch ex As Exception
                Return 0
            End Try

    End Function

I've had this running on an optimization today for several hours.

The results:
- 1221 backtest run by the optimizer.
- 22 Positive Discrepency values where ALL PositionCount = 0 (highlighted lilac in image)
- 9 Positive Discrepency values with positioncount > 0 (highlighted yellow in image)
- 31 Total



Considering all ZeroCount results produced a discrepency value of 1 (true) I believe this is suggestive of a global difference between metics if no trades are present. There were no zero-trade results that produced a 0 discrepency value.

I am most intereted in the 9 positive discrepencies where PositionCount >0 this is, i believe, what we are seeing in the scorecards.

The next test will eliminate my score card (Though I have already shown the issue is present without my Scorecard installed - see earlier post). To do this I will use the Strategy Runner Class to loop through backtests with semi-randomized strategy paramters and settings. This will alow me to write to file the basic scorecard metrics without my card installed - hopefully this will provide further insight into why these discrepencies might be happening.

It apears i have found the cause of the above discrepenncies... But, unfortunately, it does answer the question of discrepencies in the scorecards.

While researching the potential cause of the ZeroTrade records producing false positive discrepency values I discovered that NaN==NaN will always return false. The expectation in my code is that this would return true.

If you are interested in the rationalle there is an interesting stack overflow article here:
https://stackoverflow.com/questions/1565164/what-is-the-rationale-for-all-comparisons-returning-false-for-ieee754-nan-values

The response from the member of the IEEE-754 committee is particularly enlightening.

Interestingly, the 9 results target for further investigate also had NaN values in several compared metrics. Therefore it was most likely the cause.

I have updated my function to first check for NaN/NaN metric combinations and if present mark them as a match, rather than discrepency. I've conducted about 1000 backtests on a single strategy since with no instance of discrepency.

I have also added routines to my scorecard to export metircs where discrepencies appear on individual backtests (outside of the optimize routines). If the metrics are NOT exported when a discrepency is visually confirmed on a scorecard, it may indicate that the issue is localized to the population of the control or some aspect of code AFTER all metrics are generated that either modifies the Metrics ListBox or the MetricsValues

New function code below:

CODE:
 
Private Function CheckForDiscrepency(bt As Backtester)

   Try

      ' METRIC VALUES WITH NO TRADES
      '-----------------------------------------------
      '| Metric           | Strategy | Long 
      '-----------------------------------------------
      '| NetProfit        | 0.00 | 0.00
      '| Profit           | 0.00 | 0.00
      '| ProfitPct        | 0    | 0
      '| ProfitPerBar     | NaN  | NaN
      '| APR              | 0    | 0
      '| Commission       | 0    | 0
      '| Dividends        | 0    | 0
      '| Postion Count    | 0    | 0
      '| AvgProfit        | NaN  | NaN
      '| ProfitFactor     | NaN  | NaN
      '| AvgBarsHeld      | NaN  | NaN
      '| NFSPostionCount  | 0    | 0
      '| MaxDrawdown      | 0    | 0
      '| MaxDrawdownPct   | 0    | 0
      '| RecoveryFactor   | NaN  | NaN
      '| GrossProfitWinner| 0    | 0
      '| CountWinners     | 0    | 0
      '| WinRate          | 0    | 0
      '| AvgProfitWinners | NaN  | NaN
      '| CountLosers      | 0    | 0
      '| LossRate         | 100  | 100
      '| AvgLossLosers    | NaN  | NaN
      ' --------------------------------------

      ' only check long and strategy metrics
      If bt.Metrics.Profit <> bt.BenchmarkBacktestResults.Metrics.Profit And bt.Metrics.Profit <> bt.ShortResults.Metrics.ProfitThen Then
         With bt.LongResults.Metrics
            ' before we progress to numeric checks, FirstDayOfWeek check NaN values of doubles. 
            ' NaN will always return false when compared with itself (NaN<>NaN) will return true. 
            ' This would be a false positive i nthe context of this checking function.

            If Double.IsNaN(bt.Metrics.ProfitPerBar) And Double.IsNaN(.ProfitPerBar) Then Return 0 ' Both are NaN = Match
            If Double.IsNaN(bt.Metrics.AvgProfit) And Double.IsNaN(.AvgProfit) Then Return 0 ' ' Both are NaN = Match
            If Double.IsNaN(bt.Metrics.ProfitFactor) And Double.IsNaN(.ProfitFactor) Then Return 0  ' Both are NaN = Match
            If Double.IsNaN(bt.Metrics.AvgBarsHeld) And Double.IsNaN(.AvgBarsHeld) Then Return 0  ' Both are NaN = Match
            If Double.IsNaN(bt.Metrics.RecoveryFactor) And Double.IsNaN(.RecoveryFactor) Then Return 0  ' Both are NaN = Match
            If Double.IsNaN(bt.Metrics.AvgProfitWinners) And Double.IsNaN(.AvgProfitWinners) Then Return 0  ' Both are NaN = Match
            If Double.IsNaN(bt.Metrics.AvgLossLosers) And Double.IsNaN(.AvgLossLosers) Then Return 0  ' Both are NaN = Match


            ' check long only values against strategy total values, they should 
            ' match for all Long Only Strategy, otherwise raise flag.

            If bt.Metrics.Profit <> .Profit Then Return 1
            If bt.Metrics.ProfitPct <> .ProfitPct Then Return 1
            If bt.Metrics.ProfitPerBar <> .ProfitPerBar Then Return 1
            If bt.Metrics.APR <> .APR Then Return 1
            If bt.Metrics.Commission <> .Commission Then Return 1
            If bt.Metrics.Dividends <> .Dividends Then Return 1
            If bt.Metrics.PositionCount <> .PositionCount Then Return 1
            If bt.Metrics.AvgProfit <> .AvgProfit Then Return 1
            If bt.Metrics.ProfitFactor <> .ProfitFactor Then Return 1
            If bt.Metrics.AvgBarsHeld <> .AvgBarsHeld Then Return 1
            If bt.Metrics.NSFPositionCount <> .NSFPositionCount Then Return 1
            If bt.Metrics.MaxDrawdown <> .MaxDrawdown Then Return 1
            If bt.Metrics.MaxDrawdownPct <> .MaxDrawdownPct Then Return 1
            If bt.Metrics.RecoveryFactor <> .RecoveryFactor Then Return 1
            If bt.Metrics.GrossProfitWinners <> .GrossProfitWinners Then Return 1
            If bt.Metrics.CountWinners <> .CountWinners Then Return 1
            If bt.Metrics.WinRate <> .WinRate Then Return 1
            If bt.Metrics.AvgProfitWinners <> .AvgProfitWinners Then Return 1
            If bt.Metrics.CountLosers <> .CountLosers Then Return 1
            If bt.Metrics.LossRate <> .LossRate Then Return 1
            If bt.Metrics.AvgLossLosers <> .AvgLossLosers Then Return 1
         End With

      End If

      Return 0
   Catch ex As Exception
      Return 0
   End Try

End Function

Yes it’s something that tripped me up years ago, and then I learned about Double.IsNan 😅

Always gives me a taste for Indian bread!

First up... thank you for all the recent improvements. My attention has been elsewhere for a couple of weeks. But when I booted up WealthLab there were some serious improvements and enhancements!. Thank you...

I believe we are still "hunting" the cause of this issue. So I thougth I would post an update.

Not long after installing Build 40 it surfaced again.

This time I was running a backtest on one of the WL Published (Locked) strategies. No optimizable values. No Edits with the exception of disabling the stoploss building block. Some settings adjusted (Datset & Period, POsition Size %age, and set to strategy to MultiPosition)

Not only did the discrepency apear on the basic scorecard, (and in the refferenced basic and extended metrics on my scorecard) but I also caught it with the above code in my card which suggests the data held in the metrics is incorrect and this not a caching or similar issue with the listbox control of the GUI.

I've have not yet caught this discrepency during an optimization.

I've installed the new WL7 build and updated all extensions. This issue persists.



I have run at least 100 thouand backtests through the Exhaustive Optimizer with a check in my scorecard for the discrepency. I have not had a single positive discrepency check result from the optimizer backtests. However, I see it regularly (almost daily) when running backtests through the GUI (like this example).

An additional note:

In all the examples I have shown of this issue the status bar $profit / profit % seem to always match the Long ONLY column.



I assume you are using bt.NetProfit for the statusbar data and bt.Metrics.Profit for the Strategy Column - which would add further evidence to the strategy metrics being incorrect.

I don't fully understand the process flow or the Backtester-->Metrics interaction. But the only other posibility I can think of is that somehow the code is exiting before the strategy metrics are calculated in full. Perhaps if the LongOnly Metrics Task completes first? I know I'm grasping at straws here. Does any of this help?

I’m working on wrapping up some other things atm so haven’t had much time to scrutinize, but will do so asap, thanks!

Thanks Glitch.... If there is anything mroe I can do to help pinpoint the issue please let me know.

You're not switching the Scorecards mid-way or opening the Metrics Report during a backtest now as in Post #3?

QUOTE:
You're not switching the Scorecards mid-way or opening the Metrics Report during a backtest now as in Post #3?

No, not at all. I do nothing in WL7 while Backtest is running. Since our earlier discusion on changing cards being a possible contributer I allow the backtest to complete before looking at any metrics/scorecards or visualizers.

Scorecard is sometimes open from previous bactest run, but selection is not changed during. Othertimes, scorecard is not open, instead building blocks or settings are displayed when initiating backtest.