External symbols are not updating on my system. There are only two ways I have found to force an update, and these work every time. Nothing else seems to work.
1. Use DataManager to update provider
2. Plot the external symbol on a chart
This was a problem with WL7 and its solution deferred to pivot development to WL8. The test provided by Glitch fails on my system.
You can find an extensive history on this problem at https://www.wealth-lab.com/Discussion/Last-date-value-repeating-on-external-symbol-7516
@Eugene, please look at your reply #44 in this post and let us know if it indicates that you confirmed the problem on your own system. @Glitch apparently was unable to.
1. Use DataManager to update provider
2. Plot the external symbol on a chart
This was a problem with WL7 and its solution deferred to pivot development to WL8. The test provided by Glitch fails on my system.
CODE:
using WealthLab.Backtest; using System; using WealthLab.Core; using WealthLab.Indicators; using System.Drawing; using System.Collections.Generic; namespace WealthScript1 { public class MyStrategy : UserStrategyBase { //Initialize public override void Initialize(BarHistory bars) { IndicatorBase vix = SymbolData.Series(bars, "^VIX", PriceComponents.Close); PlotIndicator(vix); } //Execute public override void Execute(BarHistory bars, int idx) { } } }
You can find an extensive history on this problem at https://www.wealth-lab.com/Discussion/Last-date-value-repeating-on-external-symbol-7516
@Eugene, please look at your reply #44 in this post and let us know if it indicates that you confirmed the problem on your own system. @Glitch apparently was unable to.
Rename
That script isn't going to work in WL8 because WL7's enum PriceComponents has been changed to PriceComponent (singular) for WL8. Maybe it's just that the script isn't running due to the syntax error? Please correct that and try again.
@innertrader, of course your code (with Cone's correction above) is working for me in WL8. The Migrator should have translated it for you; how come it's "PriceComponents"?
I copied the code right out of the original discussion, not my code editor. I didn't realize that would create confusion. I am running the new, properly migrated, version but it fails for me.
@Eugene I never understood if your reply (#44) in the post linked above indicated that you were able to confirm the issue or not. I realize it is difficult to fix a problem you can't reproduce.
If you have not been able to reproduce, would someone be able to troubleshoot on my system remotely?
@Eugene I never understood if your reply (#44) in the post linked above indicated that you were able to confirm the issue or not. I realize it is difficult to fix a problem you can't reproduce.
If you have not been able to reproduce, would someone be able to troubleshoot on my system remotely?
Hey @innertrader, still have not been able to reproduce this at my end, here is a video of my latest attempt, maybe you can point out something I'm doing differently?
https://drive.google.com/file/d/1W7uV2IDXo4TCdSHKPL1VdLgamZB0RY1o/view?usp=sharing
https://drive.google.com/file/d/1W7uV2IDXo4TCdSHKPL1VdLgamZB0RY1o/view?usp=sharing
Hi @Glitch. The best way to test is to run your program and see if the last two most recent prices for VIX are duplicated in a chart for SPY in the VIX window.
I don't know what the initial state of your symbols were at the start of the video. I see the problem when I run my strategies first thing in the morning. External symbols do not update with yesterday's price.
Reviewing Reply #25 in the above linked post, it looks like the date-time stamp is not updated for VIX when there is a failure. The data is not really duplicating. WL is filling it in because there is no data for that day.
Are all your daily symbols being updated automatically by the DataManager Auto-Update option by chance? Mine are not. Can you try first thing in the day before symbols have updated?
What did you use to record your video? Maybe I can make a video for you.
I don't know what the initial state of your symbols were at the start of the video. I see the problem when I run my strategies first thing in the morning. External symbols do not update with yesterday's price.
Reviewing Reply #25 in the above linked post, it looks like the date-time stamp is not updated for VIX when there is a failure. The data is not really duplicating. WL is filling it in because there is no data for that day.
Are all your daily symbols being updated automatically by the DataManager Auto-Update option by chance? Mine are not. Can you try first thing in the day before symbols have updated?
What did you use to record your video? Maybe I can make a video for you.
How about Zoom call and you can share screens with us?
You can see exactly what i did in the video. I didn’t update using the Data Manager. I’ll try the same thing tomorrow morning.
QUOTE:
How about Zoom call and you can share screens with us?
Yes. What are some good times for you. I'm on PDT
I'm 9 hours ahead of you, so it would have to be market hours for me. Pick a time tomorrow and I'll send you an invitation.
How about 8 AM PDT. I want to check after market open to make sure problem is occurring tomorrow morning and that I can duplicate it.
Okay, I sent you an invite by email. See you at 8am PDT.
QUOTE:
We finally fixed your External Symbol update issue, will be in Build 5 (knock wood!)
That's great news @Glitch!
I'm sorry to report that the External Symbol Not Updating issue is still occurring with WL8 (B5), although not every day. One thing that's different now is that I can make the symbol update by restarting WL8 and re-running the strategy that uses external symbols. (WL8 was left running overnight).
I am still using the original method
I am still using the original method
CODE:
IndicatorBase vix = SymbolData.Series(bars, "^VIX", PriceComponent.Close);
It's a problem on Yahoo Finance side. When you notice it next time, open their website and download the CSV history file. The last bar isn't there.
QUOTE:
When you notice it next time, open their website and download the CSV history file. The last bar isn't there.
It is for me. See attached PNG's.
And here's the @Glitch test failing. Note duplication of last two ^VIX values.
However, if I restart WL8 (I leave it open overnight), the problem goes away -- ^VIX is updated. This behavior is slightly different from it's previous incarnation, as restarting WL8 never fixed it before the last Glitch update to fix this problem.
Also, to take your idea to the next step, before restarting, I replaced ^VIX with $VIX (Q-Data) in my code, and the VIX external value updated correctly. The current problem may be specific to Yahoo data as you suspect, but there's no evidence (for me) that it is on the Yahoo side.
Hmm looks like you're in good company with the ASCII crowd now?
Last bar missing in ASCII data backtest, not charts
Last bar missing in ASCII data backtest, not charts
Haha. I just saw that. I don't think it's the same symptom because this topic
is limited to fetching external symbol data.
is limited to fetching external symbol data.
This morning external symbol ^VIX failed again. Restart didn't fix it, despite what I wrote before. "Glitch Test" also failed.
Yahoo Web historical and download has the updated data for ^VIX.
I subbed $VIX (Q-Data) for ^VIX in Glitch Test strategy code and it ran successfully.
Yahoo Web historical and download has the updated data for ^VIX.
I subbed $VIX (Q-Data) for ^VIX in Glitch Test strategy code and it ran successfully.
We'll open up the bug again and keep trying to duplicate it - this is the problem!
In the meantime, please just switch to the other method.
In the meantime, please just switch to the other method.
Give build 7 a try. I notice you're on B6.
If one symbol is ok (QData) and the other isn't (Yahoo) then it points a provider-specific issue. So maybe Eugene is not using the same web page you are when trying to download the updated data. In any event, it's good that one provider is working for you.
It's possible that we may be hitting different servers in their server farm or time of update matters. :/
The question is why the failure seems to be tied to SymbolData() and not GetHistory().
It would be useful to know if innertrader ever sees the problem when using GetHistory().
It would be useful to know if innertrader ever sees the problem when using GetHistory().
QUOTE:
Give build 7 a try. I notice you're on B6.
Just happened again this AM. I'm already on B7.
QUOTE:
If one symbol is ok (QData) and the other isn't (Yahoo) then it points a provider-specific issue. So maybe Eugene is not using the same web page you are when trying to download the updated data. In any event, it's good that one provider is working for you.
There is so much apparent randomness around failures and fixes that I wouldn't want to draw too strong a conclusion from that one data point.
QUOTE:
The question is why the failure seems to be tied to SymbolData() and not GetHistory().
I need a troubleshooting strategy. Can't change more than one thing at a time. Here are the alternative troubleshooting strategies I can implement. Let me know which you prefer.
1. Run with only Q-Data for external symbols (^VIX => $VIX, ^VXN => $VXN)
2. Change code to GetHistory()
3. Contact you next morning it happens to do a Zoom call while problem is happening.
Note: we are not sure right now that #1 or #2 solution will not fail.
Since I replaced Yahoo symbols with Q-Data Symbols about a year ago, I have only rarely had failures with External Symbols not updating. But starting about two weeks ago, it is consistently failing to update yesterday's close value.
The Q-Data external symbols are $VIX, $VXN, $RVX, and $VXEEM, and none are updating with last day's close. I am running the daily strategy 30 minutes before market open. I can fix the problem and re-run the strategy successfully by doing one of the following:
1 Use Data Manager / Historical Providers to update Q-Data provider. This adds another day to the above symbols.
2. Open each of the above symbols in a chart window, where they immediately update, correctly showing previous day's close
I have confirmed that these symbols are available at Yahoo (using "^" prefix instead of "$") at the time of failure, by search or history download.
The Q-Data external symbols are $VIX, $VXN, $RVX, and $VXEEM, and none are updating with last day's close. I am running the daily strategy 30 minutes before market open. I can fix the problem and re-run the strategy successfully by doing one of the following:
1 Use Data Manager / Historical Providers to update Q-Data provider. This adds another day to the above symbols.
2. Open each of the above symbols in a chart window, where they immediately update, correctly showing previous day's close
I have confirmed that these symbols are available at Yahoo (using "^" prefix instead of "$") at the time of failure, by search or history download.
I noticed that WL8 starts up on 3:18pm on a daily schedule. It may have something to do with this. Could you shut it down and restart next a.m. (it's a good practice anyway)?
Hi Eugene. The reason I do that is because IB requires daily restart of TWS, and restarting WL is how I get WL to reconnect to IB after TWS-scheduled daily restart. WL restart is a scheduled Windows task that runs after market close, so it doesn't affect strategy execution.
Further, I auto-trade daily trades for this strategy the day before (after extended hours), which are executed on market open. This gives me the opportunity to make sure the strategy gave signals as expected and we have expected pending trades for the next day. Since I am using the cancellation Code function, I have to make sure WL is not restarted after running strategy and before market open, or the Wait for Close leg in the Order Manager will be lost.
I'm happy to try anything to troubleshoot. Just wanted you to know the constraints I'm currently working imposed by IB. If this strategy was running to TDA, I would do it differently. Given this background, let me know what you would like me to try.
UPDATED
Eugene, ignore the previous explanation. It is accurate but not relevant. The strategy that runs at the time you indicated does not use these symbols or any external symbols. The strategy where the symbols are outdated runs 30 minutes before market open.
Further, I auto-trade daily trades for this strategy the day before (after extended hours), which are executed on market open. This gives me the opportunity to make sure the strategy gave signals as expected and we have expected pending trades for the next day. Since I am using the cancellation Code function, I have to make sure WL is not restarted after running strategy and before market open, or the Wait for Close leg in the Order Manager will be lost.
I'm happy to try anything to troubleshoot. Just wanted you to know the constraints I'm currently working imposed by IB. If this strategy was running to TDA, I would do it differently. Given this background, let me know what you would like me to try.
UPDATED
Eugene, ignore the previous explanation. It is accurate but not relevant. The strategy that runs at the time you indicated does not use these symbols or any external symbols. The strategy where the symbols are outdated runs 30 minutes before market open.
The problem fetching current prices for External Symbols is still occurring and has been since the last update in this thread, but not every day. I moved away from Yahoo symbols to Q-Data symbols but then I started getting random failures with Q-Data. (Please review previous posts to review background as they are still relevant.) I then switched to TDA data which worked for a while and then failed occasionally.
This is a daily strategy that runs after the market open. The symbols I am fetching are volatility values for SPX, NDX, RUT, and EEM (VIX, VXN, RVX, and VXEEM). I use the SymbolData.Series() method to get the data for these symbols. I use GetHistory() for other symbols that are not volatility indices. I believe I previously used GetHistory() for the volatility symbols, but that was failing as well. That being said, I don't know if it matters that these are volatility indices.
I wrote code in my strategy to detect update failures by comparing current Open/Close prices to the previous day to insure they are not duplicates. I pop up an alert and write an error message to the Debug log when a duplicate is detected. (I also added code to retry 5 times upon a failure, but it doesn't appear to make any difference.)
I recently wrote a separate test program to run in case of failure, which tests all three data providers (Yahoo, Q-Data, and TDA) I have used. (Note that the failures with each of the data providers is identical, but not all fail on a given day.) Today, I got a failure (using TDA data). Here is what the debug messages look like:
Note that all symbols fail on the same day.
I ran my test program. It confirmed that TDA was failing, but Yahoo and Q-Data updated correctly. (Note that Yahoo = ^SYMBOL, Q-Data = $SYMBOL, and TDA = $SYMBOL.X for index symbols.)
Here's what that looks like with a plot of NDX with the volatility value charted below the bar. Note the duplication of the last two volatility values (the last one was not updated).
The way I can fix the problem is to chart a volatility symbol that is failing (e.g., $VXN.X). The symbol charts correctly with the correct current value, as you can see below.
This fixes the problem with $VXN.X as indicated by the debug output, when the strategy is rerun.
This is further verified by running the test again.
I hope there is enough here to identify a suspect cause for this behavior or to suggest additional testing I could do. I am fairly certain these results would be randomly repeated if I switched the data provider again.
This is a daily strategy that runs after the market open. The symbols I am fetching are volatility values for SPX, NDX, RUT, and EEM (VIX, VXN, RVX, and VXEEM). I use the SymbolData.Series() method to get the data for these symbols. I use GetHistory() for other symbols that are not volatility indices. I believe I previously used GetHistory() for the volatility symbols, but that was failing as well. That being said, I don't know if it matters that these are volatility indices.
I wrote code in my strategy to detect update failures by comparing current Open/Close prices to the previous day to insure they are not duplicates. I pop up an alert and write an error message to the Debug log when a duplicate is detected. (I also added code to retry 5 times upon a failure, but it doesn't appear to make any difference.)
I recently wrote a separate test program to run in case of failure, which tests all three data providers (Yahoo, Q-Data, and TDA) I have used. (Note that the failures with each of the data providers is identical, but not all fail on a given day.) Today, I got a failure (using TDA data). Here is what the debug messages look like:
Note that all symbols fail on the same day.
I ran my test program. It confirmed that TDA was failing, but Yahoo and Q-Data updated correctly. (Note that Yahoo = ^SYMBOL, Q-Data = $SYMBOL, and TDA = $SYMBOL.X for index symbols.)
Here's what that looks like with a plot of NDX with the volatility value charted below the bar. Note the duplication of the last two volatility values (the last one was not updated).
The way I can fix the problem is to chart a volatility symbol that is failing (e.g., $VXN.X). The symbol charts correctly with the correct current value, as you can see below.
This fixes the problem with $VXN.X as indicated by the debug output, when the strategy is rerun.
This is further verified by running the test again.
I hope there is enough here to identify a suspect cause for this behavior or to suggest additional testing I could do. I am fairly certain these results would be randomly repeated if I switched the data provider again.
It doesn't have anything to do with being external symbols. Unlike Wealth-Data stock symbols, index symbols are "outsourced" and won't update until after approximately 1 am ET. Your post is hours before this time.
Edit -
You're using other EOD sources too (e.g.,, $VIX.X comes in on TDA; ^VIX on Y!), but the explanation is probably the same.
In any case, run your script now. The data are updated. Works?
Edit -
You're using other EOD sources too (e.g.,, $VIX.X comes in on TDA; ^VIX on Y!), but the explanation is probably the same.
In any case, run your script now. The data are updated. Works?
QUOTE:
Your post is hours before this time.
No, the post was written about 12 hours *after* 1 AM ET. The strategy uses daily data running in the morning (~ 6 hours after 1 AM), operating on yesterday's close, which is when I took the screen shots.
Edit -
QUOTE:
In any case, run your script now. The data are updated. Works?
I fixed it yesterday using the method noted above and it worked properly this morning except for VXEEM. I'm currently looking into why that single symbol failed.
This morning (10:30 ET) volatility symbols for the 3 providers I use (TDA, Q-Data, Yahoo) all failed to update with *yesterday's* closing daily price. I ran my test program and here's what it showed.
I then checked ^VIX online with Yahoo, to insure that yesterday's data was available at the site. It was, as was today's real-time pricing.
I then daily charted ^VIX in WL and it showed yesterday's close price correctly (fetching from Yahoo1 data). I then reran my test program and got the following results.
Note that ^VIX is now fixed in WL historical data. Other symbols are not. Charting the symbol seems to be resetting the historical data, something that is not happening by running the SymbolData.Series() method.
I then checked ^VIX online with Yahoo, to insure that yesterday's data was available at the site. It was, as was today's real-time pricing.
I then daily charted ^VIX in WL and it showed yesterday's close price correctly (fetching from Yahoo1 data). I then reran my test program and got the following results.
Note that ^VIX is now fixed in WL historical data. Other symbols are not. Charting the symbol seems to be resetting the historical data, something that is not happening by running the SymbolData.Series() method.
UPDATE
(I am putting this update in a new post, as I was unable to select an image when I tried to edit the previous post.)
in WL I went to Data Manager>Historical Providers and selected TDA for update. Here were the results of the update. See WL notation on the volatility symbols updated.
I then ran my test program again. Note that all the TDA symbols are now corrected for yesterday's date.
(I am putting this update in a new post, as I was unable to select an image when I tried to edit the previous post.)
in WL I went to Data Manager>Historical Providers and selected TDA for update. Here were the results of the update. See WL notation on the volatility symbols updated.
I then ran my test program again. Note that all the TDA symbols are now corrected for yesterday's date.
UPDATE 2
I couldn't fit the $VXEEM.X update info in the previous image. Here it is.
I'm surprised that it seems to show that it was not updated, unlike the other symbols. This is an unexpected result, as this symbol failed my test prior to the update, and passed following it.
The strategy that relies on these symbols is now running successfully with TDA data after the update.
I couldn't fit the $VXEEM.X update info in the previous image. Here it is.
I'm surprised that it seems to show that it was not updated, unlike the other symbols. This is an unexpected result, as this symbol failed my test prior to the update, and passed following it.
The strategy that relies on these symbols is now running successfully with TDA data after the update.
Seems like a pretty good clue... it seems as if the Data Valet is sleeping on the job when it comes to external symbols.
Here's another possible clue. All 4 of my vol symbols failed to update this morning (TDA). Running my test confirmed this, but also showed an error with ^VIX (Yahoo). It also generated a log error I didn't understand.
Using the DM to update TDA Historical corrected the problem, but as with yesterday, it showed $VXEEM.X did not require an update (not true). As, expected, it did *not* fix the ^VIX (Yahoo) failure.
Here is my Historical Provider priority in the DM:
Why is it trying to access ^VIX from IB instead of rolling over to Yahoo for the symbol?
QUOTE:
8/17/2023 07:09:35:675
InteractiveBrokers
200: No security definition has been found for ^VXEEM
8/17/2023 07:09:35:691
Interactive Brokers
IB request aborted. Could not identify a contract for ^VXEEM
Using the DM to update TDA Historical corrected the problem, but as with yesterday, it showed $VXEEM.X did not require an update (not true). As, expected, it did *not* fix the ^VIX (Yahoo) failure.
Here is my Historical Provider priority in the DM:
Why is it trying to access ^VIX from IB instead of rolling over to Yahoo for the symbol?
QUOTE:Because the Data Valet determined that Yahoo! was not up-to-date, so it went to the next provider in the hierarchy.
Why is it trying to access ^VIX from IB instead of rolling over to Yahoo for the symbol?
I'll look into this issue now.
Then, the Data Valet incorrectly determined Yahoo was not up to date? Because I'm sure it was, based on the site at least. (I don't know if the site historical data on Yahoo is the same as data WL sees or if it is a different source.)
RIght, that's why were talking about this.
Fixed for Build 43.
The bug we found occurred during market hours when the cached data hadn't already been updated.
The bug we found occurred during market hours when the cached data hadn't already been updated.
Great news! Thanks.
Incidentally, you can workaround the problem just by updating those external symbols in the Data Manager - before you have requested them as secondary symbols in a script.
Since B43 I have a new problem with External Symbols. It is not pulling in the last daily bor. Instead, it is stopping at last daily bar minus 1 day.
On Saturday afternoon (ET) I ran my test program for External Symbols, and it did not return Friday's data with any of the three data providers.
Instead of updating to 9/1/2023, it stopped at Thursday (8/31/2023).
I found that if I run in single symbol mode (instead of the Portfolio mode with 4 index symbols in the DataSet), it returns the correct date.
This appears to reset something, because after doing this, Portfolio mode also works.
All three of these tests were run within 2 minutes of each other.
ADDITION
I am using Q-Data as the provider for the index symbols in the Portfolio DataSet
On Saturday afternoon (ET) I ran my test program for External Symbols, and it did not return Friday's data with any of the three data providers.
Instead of updating to 9/1/2023, it stopped at Thursday (8/31/2023).
I found that if I run in single symbol mode (instead of the Portfolio mode with 4 index symbols in the DataSet), it returns the correct date.
This appears to reset something, because after doing this, Portfolio mode also works.
All three of these tests were run within 2 minutes of each other.
ADDITION
I am using Q-Data as the provider for the index symbols in the Portfolio DataSet
Was is a fresh run with the need to "Reload Data"?
Because that would have happened after you switched the symbol mode (or adjusted the Date Range) as you described.
In other words, did you run the script on Friday, left WealthLab open overnight, and then hit Run Strategy in the window that was left open from Friday?
Because that would have happened after you switched the symbol mode (or adjusted the Date Range) as you described.
In other words, did you run the script on Friday, left WealthLab open overnight, and then hit Run Strategy in the window that was left open from Friday?
QUOTE:
In other words, did you run the script on Friday, left WealthLab open overnight, and then hit Run Strategy in the window that was left open from Friday?
WL is closed and opened every day after market close, which would have been Friday afternoon. I do have the test program currently open in the default Workspace, so it is possible it ran when WL started up on Friday afternoon, and the strategy would still have been open on Saturday morning. I will remove this test program from the Workspace to eliminate this possibility.
I do recall that it went through a data reload when I went to single symbol.
QUOTE:
I will remove this test program from the Workspace to eliminate this possibility.
@Cone it looks like that worked. Thanks for the catch.
Your Response
Post
Edit Post
Login is required