Nonreproducible-errors-during-some-parameter-optimization-runs

UPDATE: I only see the non-reproducible error problem with optimization runs with those that run in parallel. I haven't seen it with single-threaded runs at all. So it could be a thread safety problem since the problem isn't reproducible (i.e. is nondeterministic). But if this is the case, then others should be seeing it too with their parallel optimization runs. Are they?

---
If this is the case, and the parallel optimizations are being corrupted "slightly", then the single-threaded runs should be of better optimization quality. Hmm.

QUOTE:
then others should be seeing it too with their parallel optimization runs.

Maybe, depending on the strategy. Does your strategy (or libraries) use static variables?

Did the errors show up in the Log Viewer?

QUOTE:
Does your strategy (or libraries) use static variables?

Not this particular strategy. (Hmm. The library code may.) However, there are some ReadOnly variables in the main strategy.

CODE:
    public class VossPredictStrategy0 : UserStrategyBase
    {
      //StringBuilder preferredValues = new StringBuilder();
      WLTrading WLTradingExtras;
      ResyncPort posTrack = new ResyncPort();
      readonly double[] peakReboundMinRangeLimits = { 0.12, 0.6 };
      readonly string[] tabooSymbols = WLUtil.ExcludeSymbolsByDataSetName(new String[] { ",e" }); //{ "screen","ETF" }

      public VossPredictStrategy0()
      { ...

QUOTE:
Did the errors show up in the Log Viewer?

I'm not even sure the strategy thread can even generate Log Viewer messages. That brings me to another question. If I create Try/Catch blocks for the greater strategy, would the Log Viewer be able to capture the WriteLog statements the Catch block creates in the first place during parallel optimization?

Understand, these are non-reproducible errors that don't show up in single-threaded optimizations, so we know the strategy itself is not a problem (although "shared-write" static variables might be an issue). It's more like a thread safety problem that needs to be located. And I have seen screenshots of others' optimizations with these errors (But those errors could be for a different reason.).

I'm gathering your parallel optimizations are not seeing these non-reproducible errors with either Shrinking Window or SMAC?

There were similar errors in a previous build, but they've been fixed. I just verified a parallel optimization with the strategies that were throwing these errors in the past and they're still error-free in the build 22 optimizations.

QUOTE:
There were similar errors in a previous build, but they've been fixed.

The errors I've seen in other screenshots (from other posts) were around Build 19.

Let me take a look at my library code for shared-write static variables. There was one case where I didn't want to pass by reference, so I wrote some of the output into static placeholder variables and then queried those static variables on a separate call. But that code has been replaced with a library call that has multiple "out" parameters so those static placeholder variables are no longer needed. The problem is some order strategies may not be using the new multiple-out-parameter call.

I found the problem. If I comment out the static method below, ExcludeSymbolsByDataSetName(...), the non-reproducible errors created by the optimization threads go away.


Here's the library code from my static WL utility class that defines the ExcludeSymbolsByDataSetName(...) method. Could someone please tell me why this method is not thread safe? Is there some kind of garbage collection problem with the strings on the heap that are preventing thread safety? If so, does that mean all string operations are off limits to static methods? Thanks for your help.

CODE:
   public static class WLUtil //common autonomous Wealth-Lab utility functions
   {   //developed by Superticker

      public static string[] ExcludeSymbolsByDataSetName(string[] tabooDataSetNameParts)
        {
         SortedSet<string> tabooSymbols = new SortedSet<string>();
         foreach (DataSet dataSet in WLHost.Instance.DataSets)
         {
            foreach (string tabooNameSnippet in tabooDataSetNameParts)
            {
               if (dataSet.Name.Contains(tabooNameSnippet))
                  tabooSymbols.UnionWith(dataSet.Symbols);
            }
         }
         string[] tabooSymbolsArray = new string[tabooSymbols.Count];
         tabooSymbols.CopyTo(tabooSymbolsArray);
         return tabooSymbolsArray;
      }

Nothing is jumping out at me. If you hover the mouse over the little red exception triangles you should see a stack trace, that might shed some light?

Nothing is jumping out at me either. And if the heap gets corrupted by the garbage collector, the whole program would crash (which isn't happening).

There isn't any stack trace, but there is an error message, which is exactly the same for each Run with a red explanation mark. It talks about an enum collection that has been modified. Which collection would that be? And an enum is an array, not a collection. What's it talking about? I don't understand the error message.

It doesn’t mean an Enum (enumerated type) it means an IEnumerable, so it’s one of the foreach that is causing the issue. You could try replacing the foreach statements with for loops.

The MyStrategy field variable for tabooSymbols should get separate storage for each thread .... Correct? So each thread gets an independent copy when the MyStrategy constructor executes. Right?

CODE:
readonly string[] tabooSymbols = WLUtil.ExcludeSymbolsByDataSetName(new String[] { ",e" }); //{ "screen","ETF" }

The reason you can use foreach with arrays is because they are reference types that implement IList and IEnumerable. If there's a chance of it being modified, you need to make a deep copy before using foreach (or use a for loop).

QUOTE:
If there's a chance of it being modified, you need to make a deep copy before using foreach (or use a for loop).

I understand your point. I learned that (the hard way) when trying to use a foreach with a TimeSeries. The compiler remarked that a TimeSeries wasn't IEnumerable. Yes, I know, the Count on a TimeSeries is constantly changing with each new bar.

But I thought the WLHost.Instance.DataSets collection was IEnumerable and its Count would be fixed during optimization. And its foreach statement does work for non-parallel optimizations. But for a parallel optimization, the Count begins to change according to the run-time error. That confuses me and is unexpected.

I did some digging and saw that the DataSets list was performing some sorts that I was able to avoid, so it should not throw these exceptions in Build 24. But the DataSets list is subject to change when certain DataSetProviders have completed their processing and add DataSets shortly after WL8 starts, but after a couple of minutes it should become stable and usable for parallel optimizations.

Thanks for that update/finding. I did try parallel optimization with Build 23 and the problem is still there. So the DataSets List is being sorted during parallel optimizations? That's interesting--and unexpected. Hmm. I'm thinking doing a FOR loop on DataSets wouldn't help if they are undergoing a sort operation? Perhaps a List.ToArray() "snapshot" operation to string[] would be safer.

I wonder if I need to "Lock execution" to get an atomic List.ToArray() copy? I guess that won't be a concern in Build 24.

Thanks for all the help with my really weird problem with parallel optimizations.

No problem at all, yes let's wait for Build 24, but if you need to get going before then, a copy before doing the foreach should be a short term solution.

I tried converting the List.DataSets to an array, then passing that array into the foreach loop. See the screenshot with the code. But this approach fails.

Build 23 provides a more detailed error report, and it's naming the local variable tabooSymbols as the problem. Is that even possible? Am I reading this error report wrong? Optimization threads won't know about each others' local variables. I'm confused by this error report.

I think the problem is the UnionWith with the dataset.Symbols property. The Symbols is being enumerated and that might be getting regenerated as the property is accessed.

Let me understand this right. You're saying DataSet.Symbols is a moving target during parallel optimization, but not during serial optimization?

In my mind, DataSet.Symbols shouldn't be changing during any optimization or strategy execution. Is that not the case?

So what's the proposed fix when these DataSet.Symbols are constantly changing during parallel optimization?

That would be fair to assume, but Symbols is actually not safe to use in a foreach in a parallel optimization. Symbols performs a sort before returning to ensure the symbols are in the property order.

Let me refactor this for Build 24 and make Symbols safe to use this way.

QUOTE:
Symbols performs a sort before returning to ensure the symbols are in the property order.

Why would a symbol list need to be re-sorted if the list (during parallel optimization) didn't change in the first place? That seems to be a waste of cpu cycles.

Let me suggest you change the symbol list to a SortedSet datatype so sorting is automatically done whenever symbols are added or removed. Alternatively, you could add a dirty-page Boolean to each symbol list indicating a change has been made so a re-sorting would eventually be necessary.

And thanks for addressing this problem (somehow) in Build 24.

Thanks for the suggestion but the matter is already closed and a change has been made for Build 24 addressing it.