What-does-27-symbols-not-processed-mean-Strategy-Monitor

Same question here. Below is my SM after today's 7:00 PM (43 minutes ago) run:



The run worked as expected the first time through but seemed to try to run a second time leaving the page shown above. I think I have correct output from the first pass through the 4 activated strategies.

I guess no matter how many tickers is used. Also I saw it with streaming, streaming bar, polling.

And it also looks like strategies processing or data processing takes much time. E.g. I use 7 tickers, 1 simple strategy. What I expect: instant execution when time gets to the very begining of the new candle, becaus you deal with streaming data - and they are already there by the time of new candle start, and the strategies are not processing-consuming. What I see - something strange (according to Status field of the strategy (Strategy Monitor)) is going on and on for a long time (not millisecond or second or a couple (which is still too much for the context), but much longer).

@sdbens20 - Which Historical Provider did you select to use? We'll have to look at that. I'll have to admit to concentrating on Intraday while testing the S. Monitor.

@Replikant - The strategy is run as soon as possible after the data bar updates. When that happens depends on a lot of things, especially the streaming provider updating the bar using a heartbeat. If your provider is waiting for a tick to start the next bar, you could be waiting a long time, and that is not desirable.

I think your expectations are too high. Each strategy has to get a hold of a Task so it can run parallel, create an instance of the strategy, run the Initialize method which most likely needs to create several indicators, then go through the Execute loop. All of this is overhead, and expecting each strategy to run nearly instantaneously is not realistic.

Glitch it looks like a bug more than just a overhead described. Or this overhead is very big)

It’s quite a bit faster than WL6 and certainly not a bug, but I’m sorry it didn’t meet your expectations.

@Cone , but if we use Streaming, not Streaming bar, is the strategy triggered by the time, not the new tick of the new time quantum? So we collect data, received by subscription and when the time comes we just use data, collected by this moment - does it work this way?

I feel inner need to discuss it via Discord)).

QUOTE:
if we use Streaming, not Streaming bar, is the strategy triggered by the time, not the new tick of the new time quantum?

Not by "time" alone. The streaming provider needs call UpdateHeartbeat(datetime) to close bars for instruments that haven't already received a tick to open the new bar. If it's not clear I can expand with an example.

QUOTE:
So we collect data, received by subscription and when the time comes we just use data, collected by this moment - does it work this way?

Yes, if you call UpdateHeartbeat() at or just after the closing time of the interval. Otherwise, it's waiting for a tick in the next interval to close the previous bar.

Cone,
The top 4 strategies use WealthData. Please don't spend any time on the 5th strategy. It uses Q-Data and I'm not concerned about any problems with that dataset at the moment. Thanks.

Early indications point to a problem with the Schedule Run for Daily.

For now, avoid the schedule and run it on demand with right click > Run Now.

Look for a fix in Build 8.

@Repl..
Here's a video made while testing in January - this is 100 symbols in 7 seconds, but you can see processing dozens of symbols in quick spurts. The last 20+ symbols are executed in just msecs.



As I recall, this was using IQFeed with Streaming Bars. We had to work on that one a lot in order to get the "streaming bars" to close "near" the end of interval because IQFeed does not close and send the bar (for streaming bars) unless a tick occurs in the next interval. Although I argued that this is nonsensical for "streaming bars", their reason was to ensure that these bars match their historical bars.

Consequently in our implementation, for illiquid instruments an IQFeed streaming bar could be delayed up to 3 seconds.

Your one looks pretty good. Mine - not). I'll send the video to support@.

This is the result after the iteration:



Nothing in logs to track for this:

GRNT that was not processed is a ticker low liquidity - maybe it wasn't processed because there vere no trades within certain period, but my developer says that he uses heartbeats on the first second of every minute...

Please give it a try in Build 7 next week. More here (click)

Thanks!)

Bug: Strategy Monitor stopes streaming after the first run.
- Build 7 with IQFeed 1-minute scale.
- Dataset: VIX.XO VIX3M.XO
- strategy: see code below.
- Test code: see code below.

-Drop the strategy into SM
- Set parameters to the above and click 'Activate'.
- watch SM.
- After the first Bar it stops processing.
- Run the debugger and break into the test code.
- You will see that the bars are not updated (all the bars since the first one. Not only last one).

Test Code:

CODE:
private TimeSeries GetBarsFromStreaming(BarHistory bars, string pairSymbol)
        {
            string s2Bars = "";
            if (pairSymbol.StartsWith("."))
            {
                s2Bars = pairSymbol.Substring(1) + "(UpToDate)";
            }
            else
            {
                s2Bars = pairSymbol + "(UpToDate)";   // global memory key
            }

            DateTime lastDate = bars.DateTimes[bars.Count - 1];
            BarHistory resBars;                  // retrieved from global memory in loop below
            DateTime s2date;

            int watchdog = 0;
            do
            {
                System.Threading.Thread.Sleep(100);      // 100 msec delay
                resBars = (BarHistory)GetGlobal(s2Bars);      // these bars are unsynched
                if (resBars == null || resBars.Count < 1)
                {
                    return null;
                }
                s2date = resBars.DateTimes[resBars.Count - 1];

                if (++watchdog > 100)          // 5 secs max
                    break;
            } while (s2date != lastDate);

            TimeSeries close = TimeSeriesSynchronizer.Synchronize(resBars.Close, bars.Close, 0, null); 
            return close;      
        }

Strategy:
using WealthLab.Backtest;
using System;
using WealthLab.Core;
using WealthLab.Indicators;
using System.Drawing;
using System.Collections.Generic;

namespace WealthScript4 
{
    public class MyStrategy : UserStrategyBase
    {
        //create indicators and other objects here, this is executed prior to the main trading loop
        public override void Initialize(BarHistory bars)
        {
         string vixBars = bars.Symbol;
         if (vixBars.StartsWith("."))
         {
            vixBars = vixBars.Substring(1);
         }

         vixBars = vixBars + "(UpToDate)";   // global memory key
         SetGlobal(vixBars, bars);
      }

        //execute the strategy rules here, this is executed once for each bar in the backtest history
        public override void Execute(BarHistory bars, int idx)
        {

        }

        //declare private variables below

    }
}

I wouldn't bother with the S. Monitor for only 2 symbols, but ...
I noticed something weird today too with IQFeed. Please try this, because this worked for me:

1. Restart everything
2. First, open a chart and start IQFeed Streaming
3. Then, Activate the S. Monitor.

I followed the steps you described and even reset the computer. I got the same results.
I will try it again tomorrow. I don't care much about running SM itself, but I have a strategy that uses those two symbols (VIX and VIX3M) and I want them always up to date.
In WL6, I used the sample code above (I think upon your recommendation) to make sure I have the latest data for supporting symbols in a strategy. However, in WL7 I don't know if the strategy picks and updates latest bar automatically.

After I reset everything, it worked great - I was running 700 symbols in under 10 seconds.

But I think there's something we need to look at, because even I had unexpected trouble the first time I ran it today. Thanks for the report.

It's not looped, so you have to refresh to see it again. The Streaming strategy both finish their 100 symbols in 3 seconds. The 500 symbols takes about 8 seconds, but leaves 1 - I forgot to remove VAR from the S&P 500, it was delisted last week.

Cone,
I narrowed the issue to those two symbols: VIX.XO and VIX3M.XO.
The problem could be related that both have dot in their symbols.
IQFEED requires subscribing to "CBOE Indexes", which I have.

Here how I test it.
- Create a data set of these symbols: AAPL AMZN IBM VIX.XO VIX3M.XO
- Drop a sample strategy in SM and set time interval to one minute.
- Activate and wait for few minutes.
- Error "2 Symbols left to Process". See log below.
- I think if you just try to stream VIX.XO in a chart it will have the same issue.


log:
4/21/2021 11:23:50: Status = LoadingData
4/21/2021 11:23:50: Start Streaming Thread
4/21/2021 11:23:50: Populating Data
4/21/2021 11:23:50: Calling GetHistories Pass 1
4/21/2021 11:23:52: GetHistories returned with 5 symbols
4/21/2021 11:23:52: Calling GetHistories Pass 2
4/21/2021 11:23:54: GetHistories returned with 5 symbols
4/21/2021 11:23:54: VIX3M.XO has 500 bars from IQFeed
4/21/2021 11:23:54: IBM has 500 bars from IQFeed
4/21/2021 11:23:54: AAPL has 500 bars from IQFeed
4/21/2021 11:23:54: AMZN has 500 bars from IQFeed
4/21/2021 11:23:54: VIX.XO has 500 bars from IQFeed
4/21/2021 11:23:54: NextRun set to 4/21/2021 2:24 PM
4/21/2021 11:23:54: Status = Idle
4/21/2021 11:23:54: VIX3M.XO got Partial Bar from IQFeed
4/21/2021 11:23:54: AMZN got Partial Bar from IQFeed
4/21/2021 11:23:54: IBM got Partial Bar from IQFeed
4/21/2021 11:23:54: VIX.XO got Partial Bar from IQFeed
4/21/2021 11:23:54: AAPL got Partial Bar from IQFeed
4/21/2021 11:24:00: Status = Processing
4/21/2021 11:24:03: Ran Strategy on VIX.XO,AAPL,VIX3M.XO,IBM: 0 Signals, Run Time=2.74sec
4/21/2021 11:24:50: NextRun set to 4/21/2021 2:25 PM
4/21/2021 11:24:50: Status = Incomplete
4/21/2021 11:24:50: 1 Symbol not Processed: AMZN
4/21/2021 11:25:00: Status = Processing
4/21/2021 11:25:02: Ran Strategy on AMZN,AAPL,IBM: 0 Signals, Run Time=1.48sec
4/21/2021 11:25:50: NextRun set to 4/21/2021 2:26 PM
4/21/2021 11:25:50: Status = Incomplete
4/21/2021 11:25:50: 2 Symbols not Processed: VIX3M.XO,VIX.XO
4/21/2021 11:26:00: Status = Processing
4/21/2021 11:26:02: Ran Strategy on AMZN,AAPL,IBM: 0 Signals, Run Time=1.60sec
4/21/2021 11:26:50: NextRun set to 4/21/2021 2:27 PM
4/21/2021 11:26:50: Status = Incomplete
4/21/2021 11:26:50: 2 Symbols not Processed: VIX3M.XO,VIX.XO
4/21/2021 11:27:00: Status = Processing
4/21/2021 11:27:02: Ran Strategy on AMZN,AAPL,IBM: 0 Signals, Run Time=1.40sec
4/21/2021 11:27:50: NextRun set to 4/21/2021 2:28 PM
4/21/2021 11:27:50: Status = Incomplete
4/21/2021 11:27:50: 2 Symbols not Processed: VIX3M.XO,VIX.XO
4/21/2021 11:28:00: Status = Processing
4/21/2021 11:28:02: Ran Strategy on AMZN,AAPL,IBM: 0 Signals, Run Time=1.34sec
4/21/2021 11:28:50: NextRun set to 4/21/2021 2:29 PM
4/21/2021 11:28:50: Status = Incomplete
4/21/2021 11:28:50: 2 Symbols not Processed: VIX3M.XO,VIX.XO
4/21/2021 11:29:00: Status = Processing
4/21/2021 11:29:01: Ran Strategy on AMZN,AAPL,IBM: 0 Signals, Run Time=1.41sec
4/21/2021 11:29:50: NextRun set to 4/21/2021 2:30 PM
4/21/2021 11:29:50: Status = Incomplete
4/21/2021 11:29:50: 2 Symbols not Processed: VIX3M.XO,VIX.XO
4/21/2021 11:30:00: Status = Processing
4/21/2021 11:30:02: Ran Strategy on AMZN,AAPL,IBM: 0 Signals, Run Time=1.59sec
4/21/2021 11:30:50: NextRun set to 4/21/2021 2:31 PM
4/21/2021 11:30:50: Status = Incomplete
4/21/2021 11:30:50: 2 Symbols not Processed: VIX3M.XO,VIX.XO
4/21/2021 11:31:00: Status = Processing
4/21/2021 11:31:01: Ran Strategy on AMZN,AAPL,IBM: 0 Signals, Run Time=1.36sec
4/21/2021 11:31:50: NextRun set to 4/21/2021 2:32 PM
4/21/2021 11:31:50: Status = Incomplete
4/21/2021 11:31:50: 2 Symbols not Processed: VIX3M.XO,VIX.XO
4/21/2021 11:32:00: Status = Processing
4/21/2021 11:32:02: Ran Strategy on AMZN,AAPL,IBM: 0 Signals, Run Time=1.37sec

- VIX and VIX3M are updated every 15 seconds.
- 'Streaming Bars' works as expected. I will use it.
- Streaming Chart stops close to the time you start the chart and never updates unless I refresh the chart manually.

As far as I am concerned, this issue is resolved since with 'Streaming Bars' I get the quotes updated as expected.

Thank you for your help.

Thanks for the feedback. I'm thinking the problem is that there are no "quote" messages for indices. IQFeed has something else called a "summary" message that we'll probably need to use for index streams. We should be able to get this fixed for you soon.

Hi Mohammad -
Please check your email and let me know if you didn't get an invitation to download something to test.

i replaced the dll as instructed but got the same issue both in SM and Streaming Charts.



4/23/2021 07:52:50: NextRun set to 4/23/2021 10:53 AM
4/23/2021 07:52:50: Status = Incomplete
4/23/2021 07:52:50: 2 Symbols not Processed: VIX.XO,VIX3M.XO
4/23/2021 07:53:00: Status = Processing

So, I keep on experimenting with my custom Russian Alor Broker SDP. And I've figured out that only the first tick of a new bar triggers strategy execution for a ticker, although it's expected the first second of a new bar would do it as well.

Here is some code of my SDP project, could you please suggest something to make it work as expected. Maybe it's something on your side, did you try experimenting with symbols with rare trades?

CODE:

protected override bool Connect()
        {
            Task task = Task.Factory.StartNew(() =>
            {
                while(true)
                {
                    if (DateTime.Now.Second == 1)
                        UHB();
                    Thread.Sleep(250);
                }
            });
         ...
         }

CODE:

public void OnMessageReceived(object sender, MessageReceivedEventArgs e)
{
       public void UHB()
        {
            UpdateHeartbeat(DateTime.Now);
        }
        ...
}

Rep_m, Is this a new issue?
Your post was highjacked with that IQFeed streaming index issue, which was resolved.

I'd prefer if we started fresh with a well-defined issue, but if you're creating a heartbeat on a tick event, then it's no wonder that your strategy waits for that tick to occur in the next interval.

The idea of the heartbeat is to send it precisely at the time the interval ends, or every 1 minute, for example. Many feeds include heartbeats, or a clock that updates frequently, like every second that you can use for a heartbeat event trigger.

I'm using the Strategy Monitor with 30 minute Scale and that works just fine. However I also have another line in SM with Daily Scale and get message "74 Symbols not processed". I'm using IQFeed and Dummy Broker. I updated the IQFeed API to the latest, but still no symbols getting processed. See the Log file
Help

My interpretation of the results is that the Daily bars are not yet available for many (all?) of the symbols at the run time (seconds or a few minutes after 4pm).

We recommend not to run Daily+ strategies until well after the market closes. NYSE stocks often don't have an official close for several minutes after 4pm, and most Historical providers don't run daily corrections for several hours.

This is why the default value for Preferences > Data/Trading > Strategy Monitor Daily+ scale processing delay is 180 minutes. You can reduce that if you want, but I wouldn't set it for less than 60 minutes.

But I'm on the west coast, the market closes at 1300 hours. 1600 is 3 hours after the market closes.

How about I change the time to 1400 hours. That would be 1 hour after the market closes.

What time are you refering to?

This is the time to run the strategy, it is to run Daily.

There is no place to specify such a time. Instead, you specify a number of minutes after market close.

OK then, I'll enter 60 minutes.