Since I updated WLP to version 6.9.21, WLP crashes a couple of time a day every trading day.
I updated to version 6.9.22 but the issue persists.
The crash is due to unhandled exception coming from the thread executing the strategy on a form when invoking it.

Any idea how to fix it or workaround it?

CODE:
Please log in to see this code.

CODE:
Please log in to see this code.

Presently there is not enough detail to decide on what it may be caused by - let alone suggest a workaround. But since it affects just you, the problem is most likely rooted with your workflow (of a power user) and/or your strategy code (of above average complexity).

Please provide the following information:

1. Strategy or sample code used (an "as-small-as-possible" script that demonstrates the anomaly). If you're using a un-modified public Strategy, the name is sufficient.
2. Wealth-Lab tool in use (Strategy Monitor, Strategy Window, Streaming Strategy, charts, Ranking etc.)
3. Single symbol or Multi-symbol mode, Position Sizing, and Data Loading settings
4. Symbol(s) and Data provider

To clarify explanations, you might want to attach send a screenshot (in PNG format) of the WLP main workspace at the time of crash.

If you post the Windows "Application" log for the .NET run-time WL was using, that would provide more detailed information including a time stamp of the crash. Also, I would include the log4net.txt log around the time of the crash to see if WL wrote anything to the error log just prior to the crash. Be sure you have the error logger set to <level value="Error"/> so it captures all hard (i.e. nonrecoverable) errors.

Are you using any "custom windows" of your own design as part of your strategy execution?

I agree Windows.Forms is causing the error. So please tell us which window is doing it? Start closing windows until the problem goes away. Why is this window executing as part of a background thread? Is it performing reoccurring updates? If you switch those reoccurring updates off, does that fix the problem?

It mystifies me why you're the only one seeing this problem. Perhaps there's a window instance that's corrupted. Have you modified the WL default windows somehow? Try closing all the windows, then open only the ones you need and saving that new windows configuration as your new default Workspace.

QUOTE:
I agree Windows.Forms is causing the error.

That's incorrect. It appears that the error is thrown by the Strategy Monitor rather than by an open Strategy window:

CODE:
Please log in to see this code.

QUOTE:
Have you modified the WL default windows somehow?

Nice observation. In the past he may have attempted read access: Accessing text in the main status bar. (As pure offtopic, write access is possible too: Write to status bar on chart window)

QUOTE:
Nice observation. In the past he may have attempted read access: Accessing text in the main status bar. (As pure offtopic, write access is possible too: Write to status bar on chart window)

I used to have a code accessing to WLP main form in the past but not any more.

QUOTE:
That's incorrect. It appears that the error is thrown by the Strategy Monitor rather than by an open Strategy window.

Yes, apparently the exception is coming from Strategy Monitor.

I have two virtual machines running on two computers, the total four virtual machines in which WLP is running. One machine is Windows 10 + VMware and the other machine is macOS + Parallels. They all exhibit the same exception. Approximately, three times a day every day.

I have a small code as follows which runs in Strategy Monitor. There are 50 instances running in 1 minute scale and they are sharing the last price to Strategy Chart Window.

CODE:
Please log in to see this code.

My gut feeling is that there is a bug in Strategy Monitor when updating the strategy running status on UI winodw. Perhaps it's a timing window or a race condition, and the issue might be exposed by running many instances simultaneously.

In addition to those virtual machines, I just got the same exception on a physical machine which I actually trade and place real orders.
So this issue is not virtual machine specific but it happens more often on virtual machines for some reasons (probably due to timing).

QUOTE:
There are 50 instances

I interpret that as "this strategy was added 50 times (i.e., 50 lines) in the S. Monitor". 1 symbol each? Either way, that's spawning 50 threads. That's a lot of thread switching just to get a quote, isn't it?

What do you think about assigning maybe 5 threads to 10 symbols each?

QUOTE:
I interpret that as "this strategy was added 50 times (i.e., 50 lines) in the S. Monitor". 1 symbol each?

Yes, that's correct.

QUOTE:
That's a lot of thread switching just to get a quote, isn't it?

It depends. My machines are 3.5GHz-4GHz(5GHz turbo boosted) 8 core CPUs with 32GB-64GB memory. I think it's a piece of cake to run those tiny threads. In fact, CPUs spike for just a second or two on CPU performance chart on Task Manager.

QUOTE:
What do you think about assigning maybe 5 threads to 10 symbols each?

Unfortunately, it doesn't work for me. I tried that logic at the beginning. Even if just one symbol doesn't get a bar (no volume), the strategy won't run and the other 9 symbols won't get an update.

QUOTE:
Even if just one symbol doesn't get a bar (no volume), the strategy won't run and ...

So add a watchdog timer (say a 2-second timed wait) that would skip that inactive symbol (for now) and cycle on to the next one. The problem is there will be some jitter between one bar and the next since the stocks you skip over for the first 1-minute bar will see immediate execution for the next bar (possibly just short of a minute late). Is this sub-minute jitter a problem? If so, could you use 3-minute bars instead for stocks with low volume to reduce the jitter?

Understand the ticks coming out of the exchange are not going to be minute aligned, so you're going to have some arrival jitter anyway.

Hmm, I'm wondering if WL would have to support asynchronous I/O (or asynchronous intra-bar arrivals) to make this watchdog timer work? If you look at the interval (time) stamp on the last bar on the chart {i.e. Bars.IntradayBarNumber(bar) == nextInterval} for the "current" symbol, will it tell you if the most recent bar has arrived yet or not? If so, then you're in business with the watchdog timer approach.

QUOTE:
My gut feeling is that there is a bug in Strategy Monitor when updating the strategy running status on UI winodw. Perhaps it's a timing window or a race condition, and the issue might be exposed by running many instances simultaneously.

Maybe. At least we can be sure that SetGlobal itself is immune as being thread-safe.

I hit a slight different exception. Did it come from Strategy Monitor?

CODE:
Please log in to see this code.

No, this one's from Paper Trading feature. Are you paper trading with Fidelity streaming data and/or is it related to the code in post #5 now that you get this exception?

Yes, I'm paper trading with Fidelity streaming data on virtual machines. These are the ones I install the latest WLP to test the stability and confirm the changes in the strategy code before deploying to the trading platform which runs on physical machines.

I was able to duplicate a crash running 11 of those threads on a Win 7 machine a day or two ago.

I had a Quotes window running with 100 streams, but I never have any problem with that. I also had another 5-minute strategy Auto-Paper Trading on the Nasdaq 100 symbols, which unfortunately could have been a contributor to this. (We're also investigating a frozen U.I. while Auto-Trading a paper account.)

I'll come back to this, because I've got 2 machines doing other tests. For now, I'll post my crash image for posterity and easy reference.

Perhaps it was just a coincidence that the market plunged today but I ran into this exception for number of times on almost all machines regardless whether WLP was running in virtual or physical.

Fortunately, Cone was able to duplicate the problem and hopefully it is being investigated and it is fixed soon.

It was so scary for waiting WLP to restart each time WLP crashed while the market was diving.
You know how it feels.

QUOTE:
Cone was able to duplicate the problem and ...

... he wishes he could make it happen again. I'm up to 40 instances of the SetGlobal strategy and haven't seen the problem since. I think my crash wasn't related to the bug you're seeing because the "crash" signature wasn't the same as either of yours, the latter of which indicates an error in the Paper trader - I'm also investigating another report of WLPro "freezing" when auto-paper-trading.

There are WLPro updates coming soon for the PaperProvider and for login "keep-alive". It could be that one of the reasons that I didn't see this again last week is because I've been evaluating 6.9.23 with those fixes.

Aside: I'm watching those 40 Strategy Monitor items taking up to 30 seconds to complete - that's terrible! I'm certain this would work much faster - probably under 10 seconds if all of these were included in the same thread. I'm going to look at that now and I'll let you know.

I can't get in front of loading 40 or 50 1-minute symbols this time in the day (save to disk takes too long for these huge files), but I did the test with the Nasdaq 100 using 5-minute bars. All 100 symbols are updating within 24 seconds (and I'm in the Canary Islands). It's still underwhelming, but way better than each symbol as its own line item. Either way, let's cross our fingers that the new Streaming Provider (coming later this years) will improve this situation.

I'm using 1-minute bars but I set the range to 1 week and that's probably why they all run fast (5 to 7 seconds)?

Today, one of my trading machine running WLP 6.9.20 started crashing due to the exception thrown by Strategy Monitor at noon and it kept crashing through the day. I lost $1500 because I had to close 17 positions manually.

I'm figuring out how to mitigate this problem.

One idea would be to use Strategy Chart instead of Strategy Monitor but I'm not comfortable having 50 to 60 charts at once.

Any suggestion?

I modified Default.ws to have 50 chart windows running the small code (SetGlobal) mentioned above.
They are all 1 Minute scale and 1 Week data range.

When I open this default workspace, some of the chart windows won't run and they stuck with the spinning circle.
Apparently it is running into thread deadlock or something.

QUOTE:
What do you think about assigning maybe 5 threads to 10 symbols each?

Unfortunately, it doesn't work for me. I tried that logic at the beginning. Even if just one symbol doesn't get a bar (no volume), the strategy won't run and the other 9 symbols won't get an update.

Why is this important to run 50 threads passing the current price on 1 minute scale?

I have a strategy which requires two external symbols.
I usually have 25 symbols running the strategy therefore I need 50 threads to share the external symbol via SetGlobal.
Because GetExternalSymbol delays the bar data by 1 bar or 2 bars (Ticket #18663), I have been using StrategyMonitor to run the SetGlobal strategy.

Re: my comment in post #17, I was referring to updating the initial historical data update takes too long in the middle of the day to get in front of the 1-min end-of-bar update for multiple symbols. I'm only loading 100 bars.

For the last 2 or 3 weeks I've been using a test build for 6.9.23 and cannot make this happen. Make sure to avoid using Paper Trading - there were fixes for that and at least by my account it would appear to have made a difference.

As for 50 strategy windows - requires a lot of resources but shouldn't be a problem. At a minimum, make sure to disable all the visualizers and then save that Workspace to use later.

I'm running into the issue using both Paper Trading and Fidelity Account. Especially today I ran into this problem number of times using Fidelity Account.

Disabling all the visualizers so that there are only [Chart], [Strategy Summary], [Editor] and [Alerts] tabs doesn't fix the 50 strategy windows problem.

I may have to switch back to simply getting the 2nd and 3rd symbol bar data from GetExternalSymbol but they are 1 bar or 2 bars behind.
It's been 10 years since I reported the problem but I believe the issue hasn't been fixed.
Any workaround or mitigation developped for the behind bar data issue since then?

If I'm not mistaken, the problem is simply that the historical bar isn't ready fast enough for the GetExternalSymbol call. The only thing that occurs to me is to put a delay at the start of the Execute method. I'd imagine 5 to 10 seconds would be more than sufficient, but you'd have to test to see what works.

If you're having trouble with 50 charts in a Workspace - break it up into several workspaces. I'm running 5 workspaces with 8 streaming charts each without any problem on an old Win 7 machine.

fwiw,
GetExternalSymbol and 1-minute streaming data

QUOTE:
I'd imagine 5 to 10 seconds would be more than sufficient, but you'd have to test to see what works.

Does it require "Update Data on Demand" to be enabled?

The reason asking is because I don't enable "Update Data on Demand" but often (20% approximately) GetExternalSymbol returns the last closed bar without any delay even though there is no chart window and no strategy monitor accessing the symbol. Does it make sense?

That's a good question.

I tested it yesterday but don't remember if I had on-demand enabled - I'll repeat that today. The historical request for the primary symbol from a streaming chart uses the current date/time (or one in the future) for the end date. I'd assume it's the same for GetExternalSymbol(), but we'll find out for sure today.

I did some tests with GetExternalSymbol using Paper Account this morning and so far I found:

On Execute, call GetExternalSymbol each second for 15 seconds until ExternalSymbol.Date[bar].Minute == Date[bar].Minute.

Once GetExternalSymbol returns the bar data, the bar data never changes in the bar interval.
So polling GetExternalSymbol within the bar interval seems meaningless.

I have "Update Data on Demand" disabled but sometime GetExternalSymbol did return the last bar within the first few seconds in the bar interval. This is very interesting because I have no chart window and no strategy monitor accessing the external symbol. Also this is symbol independent as some symbols got the last bar yesterday didn't get it today.

In most case, GetExternalSymbol returns with the bar data in which the last bar is duplicated from the previous bar (one bar behind).

Do they all make sense?

By the way, I have "Update Data on Demand" disabled all the time because I think it was often recommended in the forum but I don't recall the reason. Any drawbacks for enabling "Update Data on Demand" in the live trading?

I forgot about this yesterday, but will try to remember to do my own test today. It appears that you found what I suspected though - that the historical request is for the most-recent data. I'm a little surprised to hear that the polling doesn't work, but some caching could explain it. If that's really the case, waiting 8 to10 seconds before processing the request would be the only way to go.

QUOTE:
Any drawbacks for enabling "Update Data on Demand" in the live trading?

No real drawbacks unless you're doing intraday backtesting. It makes sense to turn this on during the market session for ad-hoc charting.

QUOTE:
waiting 8 to10 seconds before processing the request would be the only way to go.

Could you exlain why sometime GetExternalSymbol can return the last bar data even if neither Chart Window nor Strategy Monitor is accessing the symbol? If the delay gives a chance to Data Provider to download the last bar data, how Data Provider knows which symbol to download if it was never accessed?

QUOTE:
No real drawbacks unless you're doing intraday backtesting.

You mean intraday backtesting is a backtest during the session? No issue for backtesting in intraday bar scale with "Update Data on Demand" enabled as long as market is closed (no new bar is coming)?

QUOTE:
why sometime GetExternalSymbol can return the last bar data

Because it creates a static request for new data.

QUOTE:
If the delay gives a chance to Data Provider to download the last bar data, how Data Provider knows which symbol to download if it was never accessed?

What do you mean? GetExternalSymbol specifies the symbol that you want data for.

QUOTE:
You mean intraday backtesting is a backtest during the session?

I'm talking about backtesting, strategy development. If you want to wait for a round-trip request for data, leave it on. If you don't want to wait, disable it. It's as simple as that. Obviously if you're running a backtest on S&P 500 for 1-min bars, it's going to be a problem in the middle of the day if you're using round-trip requests for new data.

QUOTE:
In most case, GetExternalSymbol returns with the bar data in which the last bar is duplicated from the previous bar (one bar behind).

It depends on the symbol, but yes, you can't count on getting the latest bar using GetExternalSymbol() only. However, if you Sleep thread for say, 5 seconds before that request, it should work - probably every time.

For the best solution with most minimal delay, see the link in my post #25 above.

QUOTE:
Because it creates a static request for new data.

QUOTE:
What do you mean? GetExternalSymbol specifies the symbol that you want data for.

The reason asking is becasue sometime GetExternalSymbol can return the latest bar at 1st second of the streaming bar. I guess a round-trip request would take way more than a second unless Data Provider requests the data ahead even if the symbol is never accessed during the session.

Maybe I'm missing something but Data Provider happens to request the latest bar for all the symbols in the DataSets through the day from the session open regardless if the symbols are ever accessed by the strategies? I was under the impression that GetExternalSymbol creates a static request only once for the bar but not for the subsequent bars.

QUOTE:
Obviously if you're running a backtest on S&P 500 for 1-min bars, it's going to be a problem in the middle of the day if you're using round-trip requests for new data.

So the propblem backtesting in the middle of the day with it enabled is that the backtesting runs very slow due to the round-trip requests, correct?

QUOTE:
I forgot about this yesterday, but will try to remember to do my own test today.

Did you have a chance to test this?

I have been testing some numbers and "Update Data on Demand" option but none of them worked very well so far.
I tried 5 seconds, 7 seconds, 8 seconds and 10 seconds delay but they all return the last bar of the previous day (more than 1050 minutes delay).

Re: "backtesting". Right - slow if you have to keep requesting new data.

I did test it last week and thought I had answered (maybe I did it in the wrong thread).

QUOTE:
Once GetExternalSymbol returns the bar data, the bar data never changes in the bar interval.

I found the same. Caching is the problem here. Therefore, you can't rush this call. It takes some finite amount of time for the data to become available in the historic data server to respond to a static data request. Recalling from years-ago experience, the longer the interval, the more time you need to allow. e.g., For 1-Minute bars, 3 or 4 seconds should be enough, whereas you might need 10 or 15 seconds for a 10-minute bar.

For the fastest and reliable way to get external intraday data, I'd still recommend streaming the external symbol(s) in their own strategy window using the template given here (see post #3, method 3):
GetExternalSymbol and 1-minute streaming data

fwiw, I cannot duplicate S. Monitor crashes with the 6.9.23.x test version, and I'm anxious to get it released for you to try... it's just that the gears or production are moving slowly. Hopefully soon.

Further testing revealed some interesting behaviors in Fidelity Static Provider.

1) I had to use GetExternalSymbol which doesn't take dataSetName parameter. If I use GetExternalSymbol which takes dataSetName, it never update the data sets regardless of the on-demand option, therefore it returns the last bar from the previous day instead of the latest bar. This is the reason why GetExternalSymbol didn't work well yesterday as I was specifying dataSetName.

2) 5 to 7 seconds delay works 90%. 8 to 10 seconds delay works 100% (so far tested today).

3) If I call GetExternalSymbol which doesn't take dataSetName parameter at every 1-minute bar, the data sets are updated regardless of the on-demand option. I confirmed this by monitoring the WL files under "FidelityStaticProvider\1 minute". If it updates the data sets all the time, what's the purpose of the on-demand option?

So in order to make GetExternalSymbol to return the latest bar, I have to:

A) Do not specify dataSetName to GetExternalSymbol.
B) "Update Data on Demand" option doesn't matter.
C) Need 8 seconds delay at least.

Do the all make sense to you?

Unfortunately, sleeping the thread and calling GetExternalSymbol from many chart windows (I have 14 chart windows) doesn't sound like a good idea. It causes a lot of exceptions like this:

CODE:
Please log in to see this code.

This doesn't occur so often with Paper Account but but it does occor very frequently with Fidelity Account for some reasons.

I think I have to get back to the original option that is to use Strategy Monitor to share the bar data via SetGlobal().

QUOTE:
What do you think about assigning maybe 5 threads to 10 symbols each?

How many symbols can Strategy Monitor handle at 1-minute scale?
I just tried 50 symbols but it doesn't run at all. What's going on in this case?
I just figured it out. It was due to the data range. I used 1 week range and it didn't work but 1 day range works fine.