ADP intraday data shown/used in WL7 does not seem to match its IQFeed source (from 2007 to 2014). There are spikes at Open seen every day in the intraday data in WL7 that indicate some data issue.
I've tried data truncation, physical file deletion, chart data reload - on 5, 10 and 30 minute scales. Relaunched WL7 after each of those operations before trying again, with no change in result.
WL7 5 min ADP - Highlighted for 11/17/2011
Corresponding IQFeed chart
WL6 shows the right data that matches IQFeed. Not sure if it's related to Splits/Dividend handling in WL7. Didn't think that was any different for IQFeed though in WL7 vs. WL6. Is there some setting to be tweaked in WL7 for this to match the source exactly?
I've tried data truncation, physical file deletion, chart data reload - on 5, 10 and 30 minute scales. Relaunched WL7 after each of those operations before trying again, with no change in result.
WL7 5 min ADP - Highlighted for 11/17/2011
Corresponding IQFeed chart
WL6 shows the right data that matches IQFeed. Not sure if it's related to Splits/Dividend handling in WL7. Didn't think that was any different for IQFeed though in WL7 vs. WL6. Is there some setting to be tweaked in WL7 for this to match the source exactly?
Rename
This is what I get for those dates, are you using the "Regular Session Only" option? My suspicion is it's maybe that.
Look at the price range for the bars in your image.... they are all in the 40s, while the intraday range for that entire day (11/17/2011) stays close to 50 (Low is 49.67) based on the daily bar, and as per 5 min on the IQFeed chart itself.
Thought about the RSO option too but the bogus open prices are too patterned.
That's not what I see in my daily chart for ADP ...
It looks like there is some kind of dividend adjustment going on, let me ask Cone to take a look here since he's the expert with this IQFeed provider.
Here's what I see on the Daily for ADP -
Yes, sounds good.
QUOTE:
let me ask Cone to take a look here since he's the expert with this IQFeed provider
Yes, sounds good.
QUOTE:It is RSO, but with a twist.
Thought about the RSO option too but the bogus open prices are too patterned.
Background (main points):
1. IQFeed Daily data are split-adjusted
2. IQFeed intraday data are never split-adjusted, so we must do that operation.
3. IQFeed only supplies the last 2 split for a company; and it does it poorly using 2-decimal place text in reverse, e.g., a 3:1 split is represented as 0.33 only.
4. To overcome #2 and #3, Wealth-Lab uses a resource file of split data that we've created.
So this ADP and several others are problematic. The split resource includes a split for the ADP Dealer Services spinoff on October 1, 2014. You can see it here in Wealth-Data:
Consequently, we split the intraday data, but that adjustment is missing in IQFeed Daily data.
I don't think there's a fool-proof way to combat this, because it's possible that IQFeed adjusts for some company actions, but there's no way to tell unless it's one of the last 2 splits. We're going to have to think about this one.
You can turn off RSO and refresh the intraday chart. The intraday will still be adjusted, but you won't have that RSO opening-price correction issue.
QUOTE:
You can turn off RSO and refresh the intraday chart. The intraday will still be adjusted, but you won't have that RSO opening-price correction issue
The spike in opening price isn't the primary issue with this for me. Yes, that does take away the spike at open, but the opening price (and the rest of the intraday bar data) still doesn't match the daily range. So the granular processing logic can't work as intended and that makes the (MSB) backtests inaccurate for the affected data range.
WL6 still shows the right intraday data that matches the source IQFeed (and the Daily bar). Is there a difference in the WL7 data provider that might be causing this?
I'm sure that you'll find that if you refresh the Daily chart in WL6, the data will match WL7. The difference must be that sometime in the past you manually applied the Yahoo! (or W-D) split indicated in the chart to the ADP data.
Here's my IQFeed-refreshed data in WL6 -
Implementing manual splits could be a interim solution for the problem.
Here's my IQFeed-refreshed data in WL6 -
Implementing manual splits could be a interim solution for the problem.
Speaking of manual splits, I had a trouble with automatic split adjustment on WL6 and since then I have been using my own manual splits in my code.
On WL7, however IQFeed extension is missing an option to disable the automatic split adjustment that is avaiable on WL6 as "Split Adjustment Intraday Data".
On WL7, however IQFeed extension is missing an option to disable the automatic split adjustment that is avaiable on WL6 as "Split Adjustment Intraday Data".
I couldn't imagine a scenario that you wouldn't want intraday data split. IQFeed doesn't give you the option to not split Daily data, and frankly I still don't understand why they won't split intraday in the same way.
If you have trouble with specific intraday splits, then tells us what they are and we'll get them corrected in our Split resource file.
For the topic starter issue, the solution I'm leaning to is using a resource file of the special splits, which include corp. actions like spinoffs/mergers and 1-time special dividends to apply to the Daily data. .. unless someone can make a good case why it's important to keep 14%-like cliffs in the data they're using to test and trade.
If you have trouble with specific intraday splits, then tells us what they are and we'll get them corrected in our Split resource file.
For the topic starter issue, the solution I'm leaning to is using a resource file of the special splits, which include corp. actions like spinoffs/mergers and 1-time special dividends to apply to the Daily data. .. unless someone can make a good case why it's important to keep 14%-like cliffs in the data they're using to test and trade.
QUOTE:
and frankly I still don't understand why they won't split intraday in the same way.
http://forums.dtn.com/index.cfm?page=topic&topicID=2815
Let me clarify - the reasons they gave:
- Breaking current client software isn't a reason either because splitting intraday could just be made an option in a new protocol
QUOTE:It's a statement that's just not true.
The reason is that to simulate trades correctly, you need to know the actual price you would have paid ON THE DAY OF THE TRADE. You'll still have to track splits so that you can work out this NON-split-adjusted price.
QUOTE:This part is true IFF (if and only if) you're basing your trade decision on a price range - it's why we provide the SplitRev indicator for that purpose!
Furthermore, the non-split-adjusted price might even be needed to determine IF you'll make a trade -- for example, if your system wants to trade equities that fall in a certain price range, or your money-management algorithm needs to know an accurate account balance or transaction cost to decide what size transaction to make.
- Breaking current client software isn't a reason either because splitting intraday could just be made an option in a new protocol
QUOTE:
The difference must be that sometime in the past you manually applied the Yahoo! (or W-D) split indicated in the chart to the ADP data.
I've never had W-D for WL6. I did try Yahoo (for a day) last year when Fidelity was going to go away, before signing up for IQFeed. I don't have any Yahoo data files downloaded in WL6 any longer and don't have it selected on the "Update Data" tab in WL6.
Here's my Yahoo split setting in WL6 -
And the IQFeed split setting -
Does the Yahoo setting apply to the IQFeed as well? Is that what you mean by "manually applied the Yahoo! split...."?
QUOTE:
I'm sure that you'll find that if you refresh the Daily chart in WL6, the data will match WL7.
I deleted the physical files on WL6 for ADP Daily and 5 Min. Reloaded the charts. Below Here are the images after the data refresh (using IQFeed).
ADP (Daily) WL6 -
ADP (5 Min) WL6 -
QUOTE:No. I mean that if you saw that there was a split, you might have applied it manually to IQFeed.
Does the Yahoo setting apply to the IQFeed as well? Is that what you mean by "manually applied the Yahoo! split...."?
Your image for Daily cannot be IQFeed-sourced. (You turned on IQFeed streaming, but that's doesn't influence the history.) In WL6, to see the IQFeed Daily data, you need to create a Daily DataSet and request the data there. Otherwise you're just rescaling your 5-minute data.
WL7 circumvents that process somewhat because if possible, it grabs data in the scale you requested.
QUOTE:
No. I mean that if you saw that there was a split, you might have applied it manually to IQFeed.
I don't even know how to do that. Also, I wouldn't have started this thread if I'd known there was a split and manually applied it already. :)
QUOTE:
Your image for Daily cannot be IQFeed-sourced. (You turned on IQFeed streaming, but that's doesn't influence the history.) In WL6, to see the IQFeed Daily data, you need to create a Daily DataSet and request the data there. Otherwise you're just rescaling your 5-minute data.
A Daily dataset is what highlighted this issue on my 10 year backtest in WL7. I have the same daily dataset in WL6 that is updated and used every day. The image is after clicking on ADP in that dataset in WL6. I even deleted the 5 minute data file for ADP before trying that chart again just now (just to be sure it wasn't picking up the 5 minute data and scaling it), and there was no change.
Anyway, you seem to know what the right solution is, so no doubt it will make it in sooner than later. Thanks for the support!
After reviewing a few of the recent Spinoffs, I found the IQFeed has the correction for BBWI and DTE, but not for IAC or, of course, ADP. Further complicating the issue is that the correction (which is basically applied with a split factor) is not transmitted in the fundamental message.
In short, IQFeed does correct for some spinoffs (BBWI, DTE), but not others (ADP, IAC), and there's no way to tell from the data feed which is which.
Solution #1
Send IQFeed a message to apply missing corrections for spinoffs. I've found them to be responsive. I'll send them the ones I know about.
Solution #2
Feature Request - allow users to apply split corrections to charts, like in v6.
In short, IQFeed does correct for some spinoffs (BBWI, DTE), but not others (ADP, IAC), and there's no way to tell from the data feed which is which.
Solution #1
Send IQFeed a message to apply missing corrections for spinoffs. I've found them to be responsive. I'll send them the ones I know about.
Solution #2
Feature Request - allow users to apply split corrections to charts, like in v6.
Your Response
Post
Edit Post
Login is required