Optimization-File-Collection-was-modified-enumeration-operation-may-not-execute

Please send the file to support@wealth-lab.com, thank you.

Unfortunately, to reproduce it we would need access to the strategy code.

Strategy Code Export Sent via email.

You will need to set the %equity as an optimizable variable to duplicate the optimization run.

Optimization Type: Shrinking Window 100 outer, 50 inner (5000 total)
Dataset: Dow 30 (WLD)
Year Range: 1995-2015

Below are my backtest preferences. Do you require anything else?

I searched through all of my saved optimizations and didn't find such an error, but I don't think we will be able to reproduce your setup because of all the custom performance metrics.

Do you use any foreach loops when computing any of your metrics?

I thought the structure of the file would give you a clue as to whether the error occurs in place of one of my metrics, or a default WL metric.
I dont have access to the output code that determines the order of values when saved.

I have a number of for loops that iterate through arrays I have created, and threefor / foreach loops that traverse BT object property lists.

BT Oject Properties Taversing
---------------------------
bt.Positions
bt.DrawdownPctCurve
bt.EquityCurve

The error text is just a generally captured exception message stored from the optimization run. It doesn't give more of a clue about where it happened, it could have been during metrics calculation or some place else.

Does the position of the error message in the file provide any clues? We know what was calculated before... up to the point of exception. RIght? And what the next value should be acording to other optimization runs that did not produce the exception

No, the exception message is persisted at the end of the tokenized record, regardless of where it was caught.

ok, well, that does make it a little more challenging

I've just gone through my code again.

- All loops are contained within broader try/catch. In fact, all code is contained within try/catch as a result of debuging the conflict with the finantic card and the changed bt.metric names from the outlier calculation updates.
- I'm not throwing exceptions I'm handling them in-code. by setting metrics to personalized error codes (integers) appropriate for each metric.
- My error handling simply sets the metric value to an integer and calls a debug errorlogger. No loops or complex handling there that could could cause unhandled exceptions.
- My debug log is turned off with a simple boolean value that wraps the entire Sub (multithreaded optimizations create file access errors so its disabled). Even if enabled, it's a simple append string to file function (no iteration or enumeration).

There might be something I'm missing, but I've checked multiple times.

I'll update my code with additional try/catch specific to each loop. But considering they are already wrapped I'm not convinced it's being caused by an iteration in my scorecard.

The strategy used is a simple conversion from block strategy to C#, then one indicator source value altered to create an indicator of indicator. No foreach or loops introduced at all.

Thought: I've assumed the exception messages are the cause of the failure to load the file. If so, could the load routine simply skip records where an exception message exists and alert the user at load_complete of # of Skipped records.

This would give user access to successful optimizer run data while alerting them to an issue that needs to be resolved.

I was able to determine why the saved optimization file couldn't be opened, it's because the Strategy wasn't saved first, it was an "Untitled Strategy." The reload logic uses Strategy name to re-load the Strategy, so I'm afraid it won't work on an unsaved Strategy. Save a Strategy before optimizing is the moral here I guess.

Interesting. And good advice, clearly.

Perhaps I forgot to hit save when I coppied the code over to the workstation.

Could we have a check/prompt prior to optimization start if untitled?

I'm running an optimization again now. I have tested cancelling it, and tested both the resume and load features.

- The resume threw an error to the log (below) and seemed to "stand still".
- The Load and Resume seemed to work as expected.



Back to the original file contents... I am wondering if the enumeration exception could also be the cause of the scorecard discrepencies I have noted in a seperate thread. If so, I might be able to confirm it in my scorecard. Wish me luck :)

Breadcrumbs, breadcrumbs... if they were skittles we could at least sort them by color!

QUOTE:
Could we have a check/prompt prior to optimization start if untitled?

Is it really necessary? Optimizing an Untitled strategy can be a way to run a quick proof of concept. It's more likely that one would be optimizing and reoptimizing their saved strategy.

I was thinking a reminder rather than a requirement. something like:

"AutoRecover is disabled for this strategy. To enable AutoRecover please save the strategy, then begin your otimization. Would you like to optimize with Autorecover disabled?" Yes/No.

If yes, continue. (but turn off the autosave feature to improve performance)
If no, don;t start optimization.

Necessary, I guess not. But potentially very helpful. (and prevents future support questions.)

The idea (and greatest value for me at least) of an autosave/recover/resume feature is to act as a safety in the event of system, software or other failure (power outage etc). What's the point of saving optimization result for recovery if the strategy was lost in the system/environment outage?

I can only speak for myself... I have a lot going on: constant interuptions and juggling of multiple hats and responsibilities. Hitting the save button is something that is (and was in this case) easily missed. Fortunately I had the code backed up on my development machine, so no loss there.

It's completely up to you ofcourse.

Update on the new optimization... Collection Modified Exception is still apearing in the AutoSaveOpt file.



This is with My Scorecard installed and all code wrapped in try/catch.

I have confirmed that all my metrics have populated across the optimzation set. None are missing, suggesting my catch/handles are working for all calculations.

I'm running another test now without my scorecard to see if that produces the issue. Should have results within the hour.

Results are in.... and I believe I have stumbled on a solid clue as to the cause...

I removed my scorecard (which means shutdown and restart of WL7.) And have run the same strategy with same settings and same optimizer.

But, in my haste to attend to another pressing task, I skipped a step when I set the optimization running. Which apears to have given a key peice of information we lacked before.

When I returned I looked at the first 500KB of data exported and to my suprise there was no instance of the Collection Modified Exception. I was begining to wonder if it was my scorecard code afterall.

I double checked the strategy and settings an noticed I didn't have "Interim Updates" checked on the optimizer results tab (which I have had turned on in previous tests) <-- this is the step I missed. So, I switched "Interim Updates" ON and decided to review my code again line by line.

Once again I couldn't find anything in my code that could cause the error. I'm not modifying any lists, collection or array during a foreach loop. But when you've been staring at the same code for hours these things can easily be missed.

By the time I had finished my code review the optimization had produced another 300kb of data. To my suprise (and relief), when I checked if again for the collection modified exception I found 6 instances all in the second half of the file.....

QUOTE:
Could it be that the collection being modified is the Optimization Results that are being enumerated during the export? (or the contents of the result list box?) If multiple threads are runing and one returns result while the autosave or listbox update are taking place, could that modify the collection and cause the exception?

Below is a screen grab of the result file with my score card NOT active or installed.

I think I've done as much as I can in determining the cause. I don;t know for certain that it is the results list, but the evidence suggests the my score card is not the source and it is happening when the interim updates are turned on.

I've narrowed it down as best I can. I'll hand it over to you to locate the cause.

It could very well be, in my Build Highlights video I recommend leaving interim updates off and turning it on only long enough to catch one update cycle, it is a performance bottleneck and makes things sluggish overall. Will spend some time looking into this but for as best practive leave interim updtes off, and just turn it one long enough to capture some data if needed.

Understood. And agreed. Unless you've built a solid custom control, populating listboxes with thousands of records is never a fast process :)

I haven't had a chance to look at the highlights video for that build yet. It's on my list :)

I'm actually changing the UI from a checkbox to a button. The button will pull one interim update. This is better than leaving it "on" for unsuspecting users who aren't aware of the consequences. Will keep looking further to see why the interim updates might be causing the exceptions too.

That could work... the realtime results are a nice feature. I originally had it turned on while i was checking that all my metrics were populating properly. But its' easy enough to click a button when you want an update.

Now I know why I had that error popping up too from that check box!

on another note: I really like the stop and start function of optimizing.

It has allowed me to pause optimizations and see the results without going all the way through for WFO to tell if it will work or not without having to wait for the full optimization time, resulting in saving me hours of analysis time! Thanks WL7 team!

@Glitch, I like the error symbol addition in build 34 for this Colection Modified Exception. Have you been able to identify the cause?

No not yet!

Not sure if this helps, but I've noticed a couple of things with recent test runs.

1) It appears that when a collection modified event occurs the list item is duplicated. Once with the alert and a second with the data. Was this by design? Image below. This suggests the routine adding the result to the list is failing then retying with succes. I have only noticed duplication, not triplicate or greater. In the image below you'll see a total result run number of 1493. The acutal TotalRuns was 1250. Leaving 243 occurrences of the exception.

2) The Collection Modified Exception is occurring in Build 33 when interim updates is unchecked (and checked). Confirmed in Build 34 with full optimization run with no "clicking the button". Perhaps this suggests it is not in the listbox update routine, but maybe in another foreach of the master list.

3) When running backtests the first few hundred results seem to be ok, but as the number of results increase so does the frequency of exception occurrence. Logically this makes sense if you are looping through a list of Backtester or Optimizer Results and the optimizer threads are completing and updating (adding or removing list items - not editing an item) a list accessed elsewhere . While the list is short a full iteration set happens faster... as the list grows the time to loop through it increases along with the probability of an update occurring in another method simultaneously.

4) Speed of optimization/backtest (due to difference in active threads/cores etc between machines as well as size of dataset and complexity of the strategy - in my experience) appears to be a factor in frequency of exceptions, I believe this is to be a sister issue to the size of the master list and time taken to iterate through it.

Thoughts: VB.net has a syncLock feature for the blocking of updates to an object while it is being enumerated via foreach. Is there such a feature in C++? If so, does it solve the issue if you are unable to find the cause of the collection modification

https://docs.microsoft.com/en-us/dotnet/visual-basic/language-reference/statements/synclock-statement

QUOTE:
VB.net has a syncLock feature for the blocking of updates to an object while it is being enumerated via foreach. Is there such a feature in C++?

Sure, it's called lock: https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/keywords/lock-statement

And for the record we don’t code WL7 in C++, it’s a C# code base.

Ooops! Of course you do... let's think of that as a typo :)

Do the above observations give you any further insight as to the cause?

I've been reading through the Optimizer API documentation and came across an interesting passage under Internal State that might help us here. From memory, this issue surfaced after the autosave feature was added..

QUOTE:
Internal State: If WL7's StrategyOptimizer sees a result with the same Parameter values already in its result list, it will return that instance instead of executing another backtest run with the same set of parameters.

This implies a loop through the results list for a paramter match before the Optimization run is fired.

If an OptimizationRun completes and adds results to the ResultsList while this check is occuring it would cause the exception. Not sure if that is the cause, but it would make sense.

Thought: If this check is happening on every ExecuteOptimizationRun call then performance would slowly degrade as the number of calls increase. Especially when dealing with 10's of thousands of results items/records. Perhaps a property could be set programatically prior to the call to "skip" this check (StrategyOptimizer.SkipDuplicateResultsCheck = True) or an optional byVal (Optional skipDuplicateResultsCheck As Boolean = False). This could be implemented when resumePrevious is False without causing a breaking change for existing optimizers as the default would be false (matching the current state).

If you do choose to implement this (or already have something similar inplace) please let me know. I'm building a series of optimizers currently that are beter suited to my objectives and I'd like to implement anything that provides a performance boost.

Update on this issue... I noticed today that the error seems to occur on the second instance for each parameter set (higher run number)

I'm assuming run number is being assigned after the run is complete, as I see this issue in my optimizer also (based on the exhaustive example) and I am most definatley not triggering two runs per parameter set.

Initially I thought perhaps the first add was failing and the second was successful, when infact the failure is the higher run number.

QUOTE:
Update on this issue... I noticed today that the error seems to occur on the second instance for each parameter set (higher run number)

Just for the record, what optimizer is this? Or to reprhase, does the issue affect all optimizers?

I've been using the prepacked Exhaustive optimizer for mos tests, but will test hte shrinking window also.

I can confirm that an optimizer extension built from your exhaustive example code also produces the issue.