Streaming-vs-Historical-Data-in-IB

Of course we expect minor differences in streaming, and, we recommend that you do not use IB Streaming in the Strategy Monitor in favor of Streaming Bars.

However, this difference is so great it makes me wonder if,
1. the stream stopped, or,
2. the market or the market's closing time is not properly set up in the item's activation settings.

Sorry, I didn't look closely enough. Since you mentioned Streaming data I thought we were talking about the Strategy Monitor.

Streaming data has nothing to do with it. We're talking about yesterday's closing price, right?

For me, it's correct.


When did you update the data? It seems that a partial bar found it's way through.

I already must apologize because the situation was different. I did not stream today, which means i did not use the strategy monitor.

I took the signals for today from a strategy window (backtest). IB was the provider, so i would assume the data is from IB. There "should" no other data source be involved as the provider was choosen explicitly (but not exclusivly).

1. Which number would you expect? The "12.42" value or the historical close from the previous day 12.09? (guess 12.09)

Now in this scenario another question came up?



The order price in the quotes tool is based on a 4% dip on the close, but none of the above prices can be the source for the order price. Any idea?

We are writing at the same time :-).

Well, I think I did the update quite soon after yesterday's session? I don't normally do that. But look at your close.

I have no idea with the information given.

Your close (12.8) is also different from the IB chart (12.42)

Thanks for the screenshot. What the hell ...

Well, your IB shows you 12.8 and your historical data is 12.8 too, i see everything is fine for you. On my side i have a different historical price than you, a different price in the IB chart than you, and different closes between historical and live data. Damn.

And a "hidden" price that was used for the order price calculation. None of the prices is the result of close * 0.96.

It turns out that 12.42 is the correct price. My chart and TWS is returning 12.80 because the chart is configured to return data outside of market hours. There was an aftermarket tick at 12.80. I had no idea that would affect the Daily bars for the API.

Ok, if I had updated at a later time, there is a chance that I would have got the correct 12.42. So the first question of #3 can be ignored until I have a reason to look for another cause.

EDIT: OK, the order price "11.61" is of course the result of the 12.09 close of my (wrong) close. Thanks a lot for the help. I might delay IB updates a bit in the future.

I already had two issues with IB updates. One was within the pre-market hours (i reported it) and now direclty after the session. On the fly we saw a side effect for you too. Need to find a good time for IB updates.

Can you tell me if there is a way to clean\correct the IB data. I am not sure how often i did an update in the past after the session or during pre-markes ... ?

I would like to remove such erroneous data. Do i need to delete and reload it?
Data integrity tab won't help i guess ?

As you know, IB is slow to load, so I wouldn't recommend a refresh unless absolutely required. Probably you can truncate the last several days or weeks, but you need to do it by DataSet and Scale. Select the DataSet/Scale and use Data Truncation at the bottom.

Hello Cone,
Hello Eugene,

I find it time consuming to keep the historical data for IB clean. For this reason, I mainly use WD, QD, EODHistorical, Firstrate data for the historical data. Until the data provider for EODHistorical data is ready, I use ASCII imports for this data.

For the streaming data I am using IB, but I might consider EODH once it is available. So now my question.

When using the strategy monitor, the data range can be set in the settings.
Do the bars that are loaded come from the local files or are they freshly downloaded from the provider (before the streaming starts)?

For EOD trading, I only use WD+QD (EODH) to generate signals. But for intraday trading, I don't want to use historical IB data to generate signals based on the previous day's close (unless they are based on a fresh download). I see two options for me.

1. using local intraday data in the SM. If I have my ascii provider before IB in the data manager, will the SM load the data from the ASCII files or from the local IB source in this special context?

2. if the bars are loaded from IB (not from local files), I just use IB.

QUOTE:
I find it time consuming to keep the historical data for IB clean.

Why is that? What are you doing to clean it? (After reading further, I suspect you're updating Daily data too soon before the settled close and corrections are applied. You can avoid that by scheduling updates or EOD Strategy runs for the next morning.)

QUOTE:
Do the bars that are loaded come from the local files or are they freshly downloaded from the provider (before the streaming starts)?

They're combined. Cached data are loaded and then new bars are requested from the end of that local data (and before if required) and combined with what comes back. If it's not up-to-date, it goes through the Historical Data Provider priority, as usual.

1. Unless you're dynamically updating ASCII data, it's not an option (read: don't use it) for intraday strategies in the Strategy Monitor. Data aren't combined from different providers.... at least I don't think so!

2. Yes, and Streaming Bars or Polling is recommended so that the result (the new bars added) matches the historical charts that you will request at some later date.