Data-loading-for-IQFeed-in-the-Strategy-Monitor-is-slow-during-market-hours

That's a broad statement without any details. Let's start by adding at least some:

1. data provider(s)
2. symbols and market(s)
3. data range
4. bar scale
5. streaming / polling / streaming bars
6. when does that happen (all the time, since build X, periodically, etc.)

Thanks Eugene. Here are the info:

- Data provider: IQFeed
- Symbols and market: about 500 symbols (non - OTC) in US market. I can provide the symbols if you want. The symbols broken into 5 groups with about 100 for each identical strategy and they run in parallel. There a total of 2 unique strategies and since each has 5 duplications, there at total of 10 strategies active in the Strategy Monitor.
- Data range: 500 bars
- Bar scale: 3 minute
- Use Streaming data
- It happens almost all the time, there are a few times where it loads faster than the rest. This morning I have been waiting for about an hour and it is still loading.

The strategy monitor takes a long time to load data. There are 5 strategies, each use about 100 symbols. Most of the time, it takes about 15-20 minutes to complete loading data for all strategies. In some days, it takes half an hour or longer or stop loading data completely.

When a strategy is Activated, you can see new symbols being loaded under the Status tab, but then shortly, the new symbol loading rate slow down considerably or stop completely. This is frustrated to stay and wait for all the loading to be completed. I am using the latest IQFeed provider and WL7.

Edited:
Time scale - 3 minutes. Tonight the data loading has been running over an hour after 3 WL restarted and it seems stopping. This and most data loading for the Strategy Monitor take place during after hours.

#BugReport

In build 10 of the IQFeed adapter, we removed parallel request support. I don't remember why we did that, but I seem to remember that the performance wasn't any better or worse. In any case, the IQFeed client does it own load balancing/pacing and we can't get the data any faster than it hands it over.

Aside:
I've been working on the IB Data Adapter for the last week, and there might be a bug in WL7's base that's requesting too much data when not required. If that's actually the case, we'll have it nailed down by next week.

I just reviewed our notes - we removed parallel support because intraday updates were using excessive amounts of memory leading to crashes. Maybe we can make parallel requests an option for IQFeed (disabled by default).. we'll look into it.

Edit -
Parallel Updates option added for IQFeed Build 16.
In any case, parallel requests are used only for scales higher than 1 Minute.

Just testing this on a bulk update.. updates started quickly, but then slowed down. IQFeed is pacing the requests/data.

Hi Cone,

Thank you for looking into this on your busy schedule. I am using the 3 minutes data and will look forward to testing out the Parallel Updates option and hope it will speed the download. It seems the slow download rate might have something to do with data synchronization because the rate was initially okay but as a minute or two into it, it starts to slow down from there. I don't recall I have this issue with WL6.

I will follow up on how it goes with the Parallel Updates option. Thanks.

This question is for my clarification as I am somewhat in the same position as tomphm having about 500 symbols I need to call on an intraday basis (over 1 min). Can I assume that the WL7 adapter will accept 10 parallel requests and IQFeed will service them?

If that is correct then the clarification is that I will need to post those requests via one of Microsoft's parallelization frameworks, correct? But, I think those frameworks only exist in .Net 4 and higher.

Thanks

Going from this

CODE:
foreach (string sym in ds.Symbols)
            {
                BarHistory bh = GetHistory(bars, sym, myDataSet);
                if (bh != null)
                    _dsBarHistories.Add(bh);
           }

To This

CODE:
Parallel.ForEach(ds.Symbols, sym =>
            {
                BarHistory bhh = GetHistory(bars, sym, myDataSet);
                if (bhh != null)
                    _dsBarHistories.Add(bhh);
            });

Helped somewhat (not so much on intraday) but is there anything else I can do?

QUOTE:
But, I think those frameworks only exist in .Net 4 and higher.

You're talking about something (Parallel Extensions, if I remember correctly) that had appeared before WL6.9 was released. In other words, .NET 4.x which included it for the first time is a distant ancestor to .NET Core 3.1.

I don't know your specific application, but I think a better way to do this is using IQFeed Streaming Bars in the Strategy Monitor in order to avoid polling for data, which is slow by nature.

The first script would simply save each BarHistory to Global Memory like this:

CODE:
        public override void Initialize(BarHistory bars)
        {
         WriteToDebugLog(bars.SymbolKey);   //e.g. "QQQ;5 Minute"
         SetGlobal(bars.SymbolKey, bars);
        }

The other strategy script would need a small delay (about 4 seconds for the Streaming Bars to complete) and then retrieve the bar histories, sort of like this -

CODE:
using WealthLab.Backtest;
using System;
using WealthLab.Core;
using WealthLab.Indicators;
using System.Drawing;
using System.Collections.Generic;
using System.Threading; 

namespace WealthScript123
{
    public class MyStrategy : UserStrategyBase
    {        
        public override void Initialize(BarHistory bars)
        {
         Thread.Sleep(5000);
         const string interval = ";5 Minute";   // set as required
         _dsBarHistories.Clear();
         
         DataSet ds = WLHost.Instance.FindDataSet("S&P 100");

         foreach (string sym in ds.Symbols)
         {
            string key = sym + interval;
            if (HasGlobal(key))
            {
               WriteToDebugLog("found the key: " + key);
               BarHistory bh = (BarHistory)GetGlobal(key);
               if (bh != null)
                  _dsBarHistories.Add(bh);
            }
         }
        }

        public override void Execute(BarHistory bars, int idx)
        {
            
        }

      //declare private variables below
      List<BarHistory> _dsBarHistories = new List<BarHistory>();

   }
}

Thanks!!

Please clarify how I would use two scripts as per your comments:

QUOTE:
The first script would simply save each BarHistory to Global Memory like this:

QUOTE:
The other strategy script would need a small delay...

How would the strategy(s) update real time on a streaming chart? Would one call the other? Sorry I'm sure I'm missing something.

The first script just runs on the DataSet in the Strategy Monitor using Streaming Bars, storing and updating their BarHistories using SetGlobal. Your Strategy script uses the technique in the second script to access the updated BHs.

But like I said, I don't really know your application. What are you doing with the BarHistories of an entire DataSet in a Strategy run?

I have a custom dataset that i use to calculate the movement (something like correlation) between the symbols, bar by bar, and there are about 400 symbols. In WL6 i did this with files during an end of day run, which was difficult. WL7 seems to have all the pieces i need to do it real time now but I am having trouble understanding some WL7 features. Here, while i have used Set Global many times, I just don't get

QUOTE:
first script just runs on the DataSet in the Strategy Monitor using Streaming Bars

. Is Streaming Bars a separate entity that runs globally in Strategy Monitor? Sorry for my misunderstanding.

BTW, when i drop my compiled strategy (Testing) on Strategy Monitor, Use Streaming Bars is greyed out.

I think i understand what you've been saying no need for you to spend anymore time here. Really sorry for wasting your time here.

No problem.
Streaming Bars isn't available for Daily bars. It will be enabled for an intraday scale. Run that first script on your 400-symbol DataSet with Streaming Bars to put the BH's in global memory for your correlation script to access.

Note - you won't be able to start so many symbols in the middle of the day using 1 or 2-minute bars since it will take too long to update the histories. Start with 5-minute bars or higher right after the end of an interval, for example.