When I run data update for today (10/25) after market close, the latest date for the data shown on the data manager is 10/22 and not 10/25. Plotting daily data indicates that 10/22 is the latest bar. Intraday data however seems to have the latest date (10/25) data.
Restart WL7 and the PC couple times do not help. This seems to occurs after installation of WL7 Build 28 and IQFeed Build 8. Reinstall IQFeed Build 7 does not have an effect.
Restart WL7 and the PC couple times do not help. This seems to occurs after installation of WL7 Build 28 and IQFeed Build 8. Reinstall IQFeed Build 7 does not have an effect.
Rename
Here is the screen shot of the data manager while data update is running.
Did you try to highlight IQFeed on Historical Providers tab, right click and choose "Delete local files" and then "Perform bulk data update" from the same context menu?
Hi Eugene,
Yes, the IQFeed box is always checked on the Historical Providers tab. I have not performed 'Delete local files and perform bulk data update'. Why do I need to do this? Is it because data from IQFeed is corrupted? Thanks.
Yes, the IQFeed box is always checked on the Historical Providers tab. I have not performed 'Delete local files and perform bulk data update'. Why do I need to do this? Is it because data from IQFeed is corrupted? Thanks.
When I clicked on "Perform Bulk Data Update", I got an instructional message - to choose desired data scale and update provider dataset first. Does this means that I need to select individual dataset from the DataSets tab and select the 'Daily' scale in this case then run 'Update DataSet'?
I selected individual dataset and run update on it with Daily scale selected and the updated data shown the correct latest date (10/25). Charting symbols from the dataset on intraday scale also produced the latest date data also. It seems I have to go through each dataset and perform update on them. It is working now. Thank you.
I had the same issue with my IQFeed data and I followed your "Delete local files" and then "Perform bulk data update" but I got the following when I tried to perform the update.
I have so many DataSets and scales, so it's painful to update each DataSet manually.
I would like to update them all at once like I can update on WL6.
On WL6, local data files and DataSet files are managed separately, so you can delete the local data files while the update can download each DataSet for the bar scale as specified in the DataSet files.
On WL7, DataSet file doesn't know the bar scale. Once the local data files are deleted, you cannot download the data until you manually update each DataSet for each bar scale.
QUOTE:
Updating Provider: IQFeed
Please choose desired bar scales and update provider DataSet(s) first
Historical Provider Update completed.
I have so many DataSets and scales, so it's painful to update each DataSet manually.
I would like to update them all at once like I can update on WL6.
On WL6, local data files and DataSet files are managed separately, so you can delete the local data files while the update can download each DataSet for the bar scale as specified in the DataSet files.
On WL7, DataSet file doesn't know the bar scale. Once the local data files are deleted, you cannot download the data until you manually update each DataSet for each bar scale.
You cannot update them all at once because WL7 is bar scale agnostic. Updating DataSets in each desired bar scale makes WL7 "recognize" the bar scales. This is by design.
In WL6 you have to create numerous DataSets just to be able to capture each bar scale for that provider and it does not support Unlinked DataSets.
In WL7, this update is only performed once. Typically, "Clear internal request tracking info" or restarting WL7 helps (I didn't suggest it because topic starter had restarted the PC already). Also you can try truncating the last few bars (highlight a DataSet > "Data truncation").
In WL6 you have to create numerous DataSets just to be able to capture each bar scale for that provider and it does not support Unlinked DataSets.
In WL7, this update is only performed once. Typically, "Clear internal request tracking info" or restarting WL7 helps (I didn't suggest it because topic starter had restarted the PC already). Also you can try truncating the last few bars (highlight a DataSet > "Data truncation").
I'm running into the same kind of issue with "Update DataSet" but for Wealth-Data now. A dozen of German symbols are seen as not up to date (last date 10/25/2021) despite:
- they are all up to 10/26 in WL6 (the German market is already closed)
- 'Clear internal request tracking info' is clicked
- WL7 has been restarted
- affects production build as well as internal build 30
- they are all up to 10/26 in WL6 (the German market is already closed)
- 'Clear internal request tracking info' is clicked
- WL7 has been restarted
- affects production build as well as internal build 30
The issue turns out to be intermittent because I wasn't able to replicate it under debugger. :/
Once I opened up a chart of such "stuck" symbol it has updated to 10/26 and a subsequent "Update DataSet" has done its job as designed.
We'll keep an eye on it.
Once I opened up a chart of such "stuck" symbol it has updated to 10/26 and a subsequent "Update DataSet" has done its job as designed.
We'll keep an eye on it.
The data update is not working today. It is not updating to the latest bar (11/4). Restart WL7 does not help. I am going to perform the steps in Post #2 from Eugene above. This problem seems to come up more recently has not occurs in the past.
I just recalled on Post #7 above from Eugene, doing a "Clear Internal Request Tracking Info" on IQFeed does not help however doing a data truncation (truncate only on the latest few days) on each effected dataset and afterward do a "Update Dataset" works and it was able to update to the latest bar (11/4). This needs to be repeated for each time scale.
Since you've already truncated it's too late to ask now. Should you encounter this issue again, please choose "Open Wealth-Lab User Data Folder" (WL7 > File menu). As this opens an Explorer window, go to IQFeed subfolder and find StartDateRegister.txt there, then copy/paste this file's contents along with your local timestamp (e.g. "It's 11/5/2021 2:13am CST") in a new reply to this topic. Thanks.
Will do.
Eugene, the data update is not working right tonight - it's not update to the most current date (1/16/2021). This is the 3rd time in about 4 weeks. I have attached the info you request below.
The content of the StartDateRegister.txt is shown below.
{...skipped...}
The content of the StartDateRegister.txt is shown below.
{...skipped...}
Apparently you don't have no Daily data by IQFeed in principle. You don't seem to have ever updated its DataSet(s) in Daily scale.
Dion, can this be a clue?
Tom, could you switch the DataSet(s) to Daily scale in Data Manager and update them?
Dion, can this be a clue?
Tom, could you switch the DataSet(s) to Daily scale in Data Manager and update them?
Start Dates aren't used/required for Daily.
Are there (were there) any messages in the Log Viewer? If there's a problem with the login, connection, request, etc. it will show up there.
Do you use IQFeed on another machine? Maybe it robbed the connection - although an obvious dialog on the affected machine would appear.
Are there (were there) any messages in the Log Viewer? If there's a problem with the login, connection, request, etc. it will show up there.
Do you use IQFeed on another machine? Maybe it robbed the connection - although an obvious dialog on the affected machine would appear.
QUOTE:
Apparently you don't have no Daily data by IQFeed in principle. You don't seem to have ever updated its DataSet(s) in Daily scale.
Normally, I would click on "Update All Checked Providers" and I assume that data of all scales would be update automaticallly.
QUOTE:
Tom, could you switch the DataSet(s) to Daily scale in Data Manager and update them?
I did switch to both daily and intraday scales when trying to update individual data set last night and this did not work.
QUOTE:
Are there (were there) any messages in the Log Viewer? If there's a problem with the login, connection, request, etc. it will show up there.
I am sorry Cone, as I copy the messages on the log viewer, WL7 become non-responsive and after about 15+ minutes, I gave up and restarted it. I did captured a screenshot of the log viewer but I don't think it contains the info you are looking for.
QUOTE:
Do you use IQFeed on another machine? Maybe it robbed the connection - although an obvious dialog on the affected machine would appear.
No, I only use one machine running WL7 and IQFeed.
QUOTE:
I am sorry Cone, as I copy the messages on the log viewer, WL7 become non-responsive and after about 15+ minutes, I gave up and restarted it.
These general messages ("WL7") are sourced from errors in your strategy and thus are not very helpful in this context. Ideally you could leave only IQFeed checked in the Log Message Sources.
That's what I thought. I will capture those from IQFeed in the Log Viewer next time.
QUOTE:Only data for those scales that you've already downloaded explicitly.
I assume that data of all scales would be update automaticallly.
The data update again not update data to the latest bar today (11/18/21). Below are the content of the StartDateRegister, date/time stamp, and content of the Log Viewer (IQFeed)
{...skipped...}
{...skipped...}
Thanks for your bug report. No need in StartDateRegister.txt anymore.
The data update is not update data to the latest date (11/29/2021) again. I noted that after the PC wakeup from sleep, the scheduled data update was running and it wasn't running correctly based on looking at dates of the update bars which are not the current date. Manual update of each data set does not correct the problem.
Below is a copy of the Debug Log.
Below is a copy of the Debug Log.
QUOTE:
Timestamp Source Message Exception
"11/28/2021 09:47:02:845" "IQFeed" "Connected"
"11/28/2021 09:51:40:026" "IQFeed" "Error parsing: P,CLOW,5.010100,1,11/26/2021,09:51:11.985000,3,0,,,,,Ov," "Input string was not in a correct format."
"11/29/2021 07:31:30:552" "IQFeed" "Error parsing: Q,CLOW,5.010100,150,11/29/2021,09:31:29.067000,3,150,,,5.0101,,Cohl," "Input string was not in a correct format."
"11/29/2021 07:36:08:731" "IQFeed" "Error parsing: Q,CLOW,5.0100,1,11/29/2021,09:36:07.228000,3,151,,,5.0101,,O," "Input string was not in a correct format."
"11/29/2021 08:34:30:491" "IQFeed" "Error parsing: Q,CLOW,5.0100,1,11/29/2021,10:34:29.029000,3,152,,,5.0101,,O," "Input string was not in a correct format."
"11/29/2021 08:52:45:004" "IQFeed" "Error parsing: Q,CLOW,5.0100,1,11/29/2021,10:52:43.348000,3,153,,,5.0101,,O," "Input string was not in a correct format."
"11/29/2021 10:05:27:769" "IQFeed" "Error parsing: Q,CLOW,5.0100,1,11/29/2021,12:05:26.392000,3,154,,,5.0101,,O," "Input string was not in a correct format."
"11/29/2021 10:05:28:070" "IQFeed" "Error parsing: Q,CLOW,5.0100,1,11/29/2021,12:05:26.711000,3,155,,,5.0101,,O," "Input string was not in a correct format."
"11/29/2021 10:07:14:812" "IQFeed" "Error parsing: Q,CLOW,5.0100,1,11/29/2021,12:07:13.437000,3,156,,,5.0101,,O," "Input string was not in a correct format."
"11/29/2021 10:07:40:069" "IQFeed" "Error parsing: Q,CLOW,5.0100,1,11/29/2021,12:07:38.686000,3,157,,,5.0101,,O," "Input string was not in a correct format."
"11/29/2021 10:12:39:171" "IQFeed" "Error parsing: Q,CLOW,5.0100,1,11/29/2021,12:12:37.817000,3,158,,,5.0101,,O," "Input string was not in a correct format."
"11/29/2021 10:12:39:171" "IQFeed" "Error parsing: Q,CLOW,5.0100,1,11/29/2021,12:12:37.819000,3,159,,,5.0101,,O," "Input string was not in a correct format."
"11/29/2021 10:41:19:855" "IQFeed" "Error parsing: Q,CLOW,5.0100,1,11/29/2021,12:41:18.515000,3,160,,,5.0101,,O," "Input string was not in a correct format."
"11/29/2021 11:21:44:354" "IQFeed" "Error parsing: Q,CLOW,5.0100,1,11/29/2021,13:21:43.056000,3,161,,,5.0101,,O," "Input string was not in a correct format."
"11/29/2021 11:21:44:354" "IQFeed" "Error parsing: Q,CLOW,5.0100,1,11/29/2021,13:21:43.057000,3,162,,,5.0101,,O," "Input string was not in a correct format."
"11/29/2021 11:21:44:354" "IQFeed" "Error parsing: Q,CLOW,5.0100,1,11/29/2021,13:21:43.059000,3,163,,,5.0101,,O," "Input string was not in a correct format."
"11/29/2021 11:21:44:375" "IQFeed" "Error parsing: Q,CLOW,5.0100,1,11/29/2021,13:21:43.061000,3,164,,,5.0101,,O," "Input string was not in a correct format."
"11/29/2021 11:46:04:911" "IQFeed" "Error parsing: Q,CLOW,5.0100,1,11/29/2021,13:46:03.611000,3,165,,,5.0101,,O," "Input string was not in a correct format."
"11/29/2021 12:29:36:289" "IQFeed" "Error parsing: Q,CLOW,5.0100,10,11/29/2021,14:29:35.043000,3,175,,,5.0101,,O," "Input string was not in a correct format."
"11/29/2021 12:49:38:236" "IQFeed" "Error parsing: Q,CLOW,5.0100,1,11/29/2021,14:49:36.977000,3,176,,,5.0101,,O," "Input string was not in a correct format."
"11/29/2021 14:02:02:790" "IQFeed" "Error parsing: P,CLOW,5.0100,1,11/29/2021,14:49:36.977000,3,176,,,5.0101,,Cohlv," "Input string was not in a correct format."
"11/29/2021 22:39:02:836" "IQFeed" "IQFeed request timed out: AAUKF, Daily"
"11/29/2021 22:39:20:720" "IQFeed" "Error parsing: P,CLOW,5.0100,1,11/29/2021,14:49:36.977000,3,176,,,5.0101,,Cohlv," "Input string was not in a correct format."
Could it be that the IQFeed client won't reconnect properly after a disconnect caused by that putting the PC to sleep? Have you tried not doing so since this is problematic?
The errors are from a Streaming connection you must have had running - the responses are missing a closing price. We could "eat" those errors, but in this case it's better that to know something is wrong.
You can Sleep you computer before updates, but Close Wealth-Lab before you do it. Make sure that IQFeed disconnects and closes too, which it will do automatically when it doesn't have any clients like WL7 connected.
You can Sleep you computer before updates, but Close Wealth-Lab before you do it. Make sure that IQFeed disconnects and closes too, which it will do automatically when it doesn't have any clients like WL7 connected.
Thanks Eugene and Cone.
I will close WL before putting the PC to sleep from now on and see if the problem will go away.
I will close WL before putting the PC to sleep from now on and see if the problem will go away.
Data update is not working today - it is not update to the latest date (12/13/2021). WL is restarted each time before running data update. This time the data update seems to able to update intraday data but not daily data. Previously, it wasn't able to update neither. The Log Viewer is empty - there is no error or warning messages.
Hi Tom,
1. Did you verify the daily data availability for Dec 13 in IQ's test app?
2. What IQFeed provider build are you running?
1. Did you verify the daily data availability for Dec 13 in IQ's test app?
2. What IQFeed provider build are you running?
I think it's more likely that somehow we're passing an old date intermittently. We've got a minor build correction coming up (to handle index ticks without a Size field), so let's add some logic to make sure the data request has tomorrow's date.
It would help to know if you're using the IQFeed configuration for "Regular Session Only". Are you?
-- Edit --
We're already adding an end date 2 days into the future based on the local machine's date/time.
It would help to know if you're using the IQFeed configuration for "Regular Session Only". Are you?
-- Edit --
We're already adding an end date 2 days into the future based on the local machine's date/time.
QUOTE:
1. Did you verify the daily data availability for Dec 13 in IQ's test app?
Hi Eugene,
No, I did not verify the availability of daily data via the IQ's test app. I am not sure how to do this and have never done this step before. I run data update usually very late in the evening, so the data should be available by then.
QUOTE:
2. What IQFeed provider build are you running?
I am running the latest build for the IQFeed provider.
QUOTE:
It would help to know if you're using the IQFeed configuration for "Regular Session Only". Are you?
I don't think so. The Regular Session Only check box for IQFeed under the Streaming Providers tab is unchecked. This box is left unchecked by default I think.
Should I check the Regular Session Only box?
The data update is act up again and not update data to the latest date (12/20/2021). WL7 is restarted before running the data update. The regular session only box is checked. Below is the message from the Log Viewer for IQFeed. The problem seems happen in interval of few to several days apart. I am hoping we can find a solution to this problem soon. Thanks.
Edited:
Previously I clicked on Update al Checked Providers and the data update was not working. Manual update of each data set also did not work. A moment later, I clicked on Update IQFeed Data and the data update is working - it updates data for 12/20/2021. I don't recall if this scenario has taken place before.
The only other data provider beside IQFeed that are selected is Crytocompare. I have this provider selected because I have data sets that are containing cryto currency symbols.
I will try this sequence of steps later if the data update is not working to see if it is repeatable.
QUOTE:
Timestamp Source Message Exception
"12/20/2021 22:21:49:039" "IQFeed" "Connected"
"12/20/2021 22:22:59:142" "IQFeed" "Error parsing: P,HRC,,,,,,0,,,,155.7200,acv," "Input string was not in a correct format."
"12/20/2021 22:23:09:336" "IQFeed" "Error parsing: P,KSU,,,,,,0,,,,,bv," "Input string was not in a correct format."
"12/20/2021 22:23:41:437" "IQFeed" "Error parsing: P,SNR,,,,,,,,,,,," "Input string was not in a correct format."
"12/20/2021 22:23:50:521" "IQFeed" "Error parsing: P,UFS,,,,,,0,,,,55.4900,acv," "Input string was not in a correct format."
Edited:
Previously I clicked on Update al Checked Providers and the data update was not working. Manual update of each data set also did not work. A moment later, I clicked on Update IQFeed Data and the data update is working - it updates data for 12/20/2021. I don't recall if this scenario has taken place before.
The only other data provider beside IQFeed that are selected is Crytocompare. I have this provider selected because I have data sets that are containing cryto currency symbols.
I will try this sequence of steps later if the data update is not working to see if it is repeatable.
My gut feeling is that somehow WL7 is determining that the U.S. market is still open. The next time it happens, open a Daily chart. If the current day's bar isn't displayed, select Stream. Does that make it appear?
What's your time zone and about what time do you try to update?
What's your time zone and about what time do you try to update?
The data update is not working today (12/21/2021) again. I clicked on Update IQFeed Data instead of Update all Checked Providers and the daily data does not updates. However, the intraday data seems to be updating correctly.
Opened a daily chart and the current day bar is not displayed. And clicked on Stream does not display the current day bar.
I am in (Arizona) Mountain Time Zone and usually update data late in the evening.
QUOTE:
My gut feeling is that somehow WL7 is determining that the U.S. market is still open. The next time it happens, open a Daily chart. If the current day's bar isn't displayed, select Stream. Does that make it appear?
Opened a daily chart and the current day bar is not displayed. And clicked on Stream does not display the current day bar.
QUOTE:
What's your time zone and about what time do you try to update?
I am in (Arizona) Mountain Time Zone and usually update data late in the evening.
Cone,
This problem does not occur in WL6. Another piece of info is that for the data update to work again, I have the data sets truncate its few most recent bars (as mentioned by Eugene)
#BugReport
This problem does not occur in WL6. Another piece of info is that for the data update to work again, I have the data sets truncate its few most recent bars (as mentioned by Eugene)
#BugReport
Are you leaving WL7 open for days at time by chance? It's possible the internal tracking of when symbols were last updated is at play here. Instead of truncating the data next time, try instead the "Clear Internal Tracking Info" option in the same popup menu. If that works then we'll know how to proceed to fix this.
Hi Glitch,
No, WL7 is restarted each night before running data update.
Will do that and will follow up with the results. Thanks.
Edited:
This problem was not there before around the end of last October (date of Post #1), and since then the frequency of failures seems to increase.
QUOTE:
Are you leaving WL7 open for days at time by chance?
No, WL7 is restarted each night before running data update.
QUOTE:
Instead of truncating the data next time, try instead the "Clear Internal Tracking Info" option in the same popup menu.
Will do that and will follow up with the results. Thanks.
Edited:
This problem was not there before around the end of last October (date of Post #1), and since then the frequency of failures seems to increase.
Glitch,
The data update is not working again tonight. Where do you find the "Clear Internal Tracking info"?
The data update is not working again tonight. Where do you find the "Clear Internal Tracking info"?
QUOTE:
Where do you find
In the built-in Help, the first link.
Sorry Eugene. I am still not sure where you were referring to. Is it located in the Data Manager window?
Sorry for not being clear. Type in "clear" in the Help and the first link to come up will explain the Data Manager feature Dion referred to.
Eugene, I found it. Thanks.
Edited:
I found it under: Help/Data Manager/Historical Providers. However, in WL7, under the Data Manager > Historical Providers, the "Clear Tracking Info" tab/button could not be found. Am I in the right location?
Edited:
I found it under: Help/Data Manager/Historical Providers. However, in WL7, under the Data Manager > Historical Providers, the "Clear Tracking Info" tab/button could not be found. Am I in the right location?
Right click on the Provider for which you want to clear tracking info.
Thanks Cone. This morning (after a restart of WL), the data update is working (?) - I could not get to try the "Clear Internal Tracking Info" method.
Edited:
Just to be cleared, WL7 is restarted at end of day (such as last night) before doing data update.
Edited:
Just to be cleared, WL7 is restarted at end of day (such as last night) before doing data update.
Data update is not working again. Performed "Clear Internal Request Tracking Info" for IQFeed does not resolve the problem - the data update still not able to update to the latest date (1/10/22). Are there any other suggestions on what I can try to identify this problem?
It's unlikely to be it, but please check that your machine's date and time is accurate for the configured time zone.
Also, please run this script on any one symbol and post the debug output. Let's see if there's something about Arizona time (also unlikely)...
Also, please run this script on any one symbol and post the debug output. Let's see if there's something about Arizona time (also unlikely)...
CODE:
using WealthLab.Backtest; using System; using WealthLab.Core; using System.Collections.Generic; namespace WealthScript123 { public class TimeCheck : UserStrategyBase { private static TimeZoneInfo _tziEST = TimeZoneInfo.FindSystemTimeZoneById("Eastern Standard Time"); public override void Initialize(BarHistory bars) { DateTime localDate = DateTime.Now; DateTime estDate = TimeZoneInfo.ConvertTime(localDate, _tziEST); TimeSpan offsetLocalToEST = localDate - estDate; TimeSpan offsetMarketToEST = WealthLab.Data.DataProviderFactory.FindDataProvider("IQFeed").GetSymbolMarket("IBM").Offset; WriteToDebugLog(_tziEST.ToString() + ", " + estDate + ", " + localDate + ", " + offsetMarketToEST); foreach (TimeZoneInfo tzi in TimeZoneInfo.GetSystemTimeZones()) { if (tzi.ToString().Contains("Arizona") || tzi.ToString().Contains("Eastern")) WriteToDebugLog(tzi); } } public override void Execute(BarHistory bars, int idx) { } } }
Stop the presses! We have an update -
Since this is not IQFeed, clearly the problem is centralized to WealthLab.Data control. Unfortunately this daily update wasn't running a debug session of WL7, so I can't directly determine the cause either, but it made me consider that just maybe WealthLab.Data is sending providers an update date that's only 2 days advanced from the last update, not "today". I'm not sure if that's the case yet, but looking back at your posts, the problem most frequently occurs on Tuedays -
Posted Tue. 10/26; last bar: Fri. 10/22
Posted Fri. 11/5; last bar: Thurs. 11/4
Posted Fri. 11/19; last bar: Thurs. 11/18
Posted Tue. 11/30; last bar: (presumably) Fri. 11/26
Posted Tue. 12/14; last bar: Fri. 12/10
Posted Tue. 12/21; last bar: (presumably) Fri. 12/17
Posted Tue. 1/11; last bar: (presumably) Fri. 1/7
Since this is not IQFeed, clearly the problem is centralized to WealthLab.Data control. Unfortunately this daily update wasn't running a debug session of WL7, so I can't directly determine the cause either, but it made me consider that just maybe WealthLab.Data is sending providers an update date that's only 2 days advanced from the last update, not "today". I'm not sure if that's the case yet, but looking back at your posts, the problem most frequently occurs on Tuedays -
Posted Tue. 10/26; last bar: Fri. 10/22
Posted Fri. 11/5; last bar: Thurs. 11/4
Posted Fri. 11/19; last bar: Thurs. 11/18
Posted Tue. 11/30; last bar: (presumably) Fri. 11/26
Posted Tue. 12/14; last bar: Fri. 12/10
Posted Tue. 12/21; last bar: (presumably) Fri. 12/17
Posted Tue. 1/11; last bar: (presumably) Fri. 1/7
The update above was launched from the Task Scheduler. When I ran a scheduled update in my debug session, everything was okay.
Just to see if that's a common theme, are you also running automatic updates from the Task Scheduler using ..\WealthLab7.exe /U?
Just to see if that's a common theme, are you also running automatic updates from the Task Scheduler using ..\WealthLab7.exe /U?
Hi Cone, thanks for your help and insights into the problem. I ran the script you provided to check on the time zone anyway and the debug log from it is shown below.
A screenshot of the time/date setting is attached below for completeness.
That's great that you noticed the pattern on the date (Tuesday) of when the problems arise - it might related to the root cause of the problem.
If I understand your question correctly, No, I do not use Task Scheduler to schedule WL7 data update via using Wealthlab7.exe. However, I do use Task Scheduler to wake up the PC in the morning. WL was put to sleep the night before. Would the task scheduler used in this way be an issue with running WL7?
However, before I run data update, WL7 is restarted, so if the task scheduler introduced any incorrect information into the operation of WL - would the restart reset and nullify it?
Edited:
Do you have a way for me to check this?
QUOTE:
---Symbol by Symbol Debug Logs---
---SPY---
(UTC-05:00) Eastern Time (US & Canada), 1/11/2022 10:12:26 AM, 1/11/2022 8:12:26 AM, 02:00:00
(UTC-07:00) Arizona
(UTC-05:00) Eastern Time (US & Canada)
A screenshot of the time/date setting is attached below for completeness.
That's great that you noticed the pattern on the date (Tuesday) of when the problems arise - it might related to the root cause of the problem.
QUOTE:
are you also running automatic updates from the Task Scheduler using ..\WealthLab7.exe /U?
If I understand your question correctly, No, I do not use Task Scheduler to schedule WL7 data update via using Wealthlab7.exe. However, I do use Task Scheduler to wake up the PC in the morning. WL was put to sleep the night before. Would the task scheduler used in this way be an issue with running WL7?
However, before I run data update, WL7 is restarted, so if the task scheduler introduced any incorrect information into the operation of WL - would the restart reset and nullify it?
Edited:
QUOTE:
it made me consider that just maybe WealthLab.Data is sending providers an update date that's only 2 days advanced from the last update, not "today"
Do you have a way for me to check this?
QUOTE:It's not likely, but there are some environment variables that are different when running a program from the Task Scheduler. It could be a clue, but I think it would only apply if the app were launched from TaskSch.
Would the task scheduler used in this way be an issue with running WL7?
I'll be ready for this next Tuesday with a debug session. We're going to solve it!
Note to myself:
Next Monday is a holiday, so the day to look will be Wednesday.
Everything looked fine today, and I could see that WL7 is passing a DateTime.MaxValue to the Data Providers for updating. Providers detect and massage that for the actual request for data. We'll try again next Tuesday.
Thank you Cone. The data update for yesterday (1/18) also worked fine on mine.
Well, it is not working properly 1 day late :). The data update is not working right for Wednesday (1/19). Some daily data were able to update tonight though. I don't think this was the case with previous problems.
It's working for me again today, but we'll keeping watching it every day until we catch it in the act.
Data update does not update daily data for today (1/24).
There are some symbols, somehow, got updated during the update process.
Could be a clue. PMHG last traded on 1/18 before 1/24.
I was trying to be diligent and update IQFeed everyday as you do, but I missed updating last Friday's session because I had unknowingly disabled the Task :( Of course, updates are up-to-date for me.
It doesn't make sense on the surface (or in the code), but it's a good clue that the updates may only fail on Tuesday for symbols whose Friday session is up-to-date.
I was trying to be diligent and update IQFeed everyday as you do, but I missed updating last Friday's session because I had unknowingly disabled the Task :( Of course, updates are up-to-date for me.
It doesn't make sense on the surface (or in the code), but it's a good clue that the updates may only fail on Tuesday for symbols whose Friday session is up-to-date.
I just tried updating again after truncating yesterday's bar from my S&P 500 Daily list, but it didn't make a difference. Update requests "always" start using the date from the last bar in the data and end 2 days in the future. I put "always" in quotes until we can prove that's not the case!
Cone,
As you might noticed, yesterday was Monday (differed from others - occured on Tuesday). The symbol is PMHG is one of the symbols got updated yesterday, however, there are other symbols that are got updated. I tried 'Ctrl + L' to get the data update log but didn't see the complete log.
Those symbols that weren't able updated last night were updated fine this morning - this is consistent with other time earlier.
As you might noticed, yesterday was Monday (differed from others - occured on Tuesday). The symbol is PMHG is one of the symbols got updated yesterday, however, there are other symbols that are got updated. I tried 'Ctrl + L' to get the data update log but didn't see the complete log.
Those symbols that weren't able updated last night were updated fine this morning - this is consistent with other time earlier.
It looks like the data update is not working right on Tuesday (1/25) also. So, this time, it did occur on a Tuesday. There seems to be a mixture of symbols got updated and those don't. Refer to the attached pic, those symbols on the bottom of the pic with dates (1/25) are those got updated and those with dates (1/24) were not.
I noticed that 4 out of 5 times the failure pattern is usually updating Monday's session - now I'm convinced it's a red herring.
What do you notice different about my update below and yours?
-->> 1 bar added
Probably this whole issue comes down to Wealth-Lab 7's Internal Request Tracking that's meant to keep from re-requesting data that you already have - to speed up backtests. I think the issue is that either you're updating before IQFeed has the Daily bars, or, you updated earlier in the session.
In any case, you can get around this by clicking this option at the end of the day -
Shutting down Wealth-Lab and letting the Task Manager perform the update should work too. We suggested this in Post #28 on November 30th and you didn't come back for 2 weeks. Let's hope this is it.
Update...
We're reviewing the Data Manager update processes and I think we found we're automatically clearing the tracking only for DataSet updates, but not provider updates. If that's the case, we'll change it!
What do you notice different about my update below and yours?
-->> 1 bar added
Probably this whole issue comes down to Wealth-Lab 7's Internal Request Tracking that's meant to keep from re-requesting data that you already have - to speed up backtests. I think the issue is that either you're updating before IQFeed has the Daily bars, or, you updated earlier in the session.
In any case, you can get around this by clicking this option at the end of the day -
Shutting down Wealth-Lab and letting the Task Manager perform the update should work too. We suggested this in Post #28 on November 30th and you didn't come back for 2 weeks. Let's hope this is it.
Update...
We're reviewing the Data Manager update processes and I think we found we're automatically clearing the tracking only for DataSet updates, but not provider updates. If that's the case, we'll change it!
QUOTE:
What do you notice different about my update below and yours?
If the data update was done on 1/24, I see that all data were either updated or were discontinued. There is no data that is not updated.
Clear Internal Request Tracking Info did not work for me. Please see Post #12 and #47.
QUOTE:
Shutting down Wealth-Lab and letting the Task Manager perform the update should work too. We suggested this in Post #28 on November 30th and you didn't come back for 2 weeks. Let's hope this is it.
Here is the sequence of the PC goes through: At end of stock market session, PC goes to sleep with WL still on, sometimes later in the evening, PC wakes up and WL got restarted, then data update is performed.
QUOTE:It's impossible that you updated on the 24th and got data for the 25th. See the bottom of your image. No bars are being added, yet it started showing bars updated to the 25th.
If the data update was done on 1/24, I see that all data were either updated or were discontinued. There is no data that is not updated.
I sure hope you're wrong about the tracking info, but your previous comment about it not working but then working after truncating data sure throws a wrench into it.
I am sorry if I mis-stated it, the data was updated last night (1/25), so there are some data that got updated (shown dates 1/25) and those that don't got updated (shown dates 1/24).
I hope the tracking info might be the clues. However, in the past events when data update not working, I would to go the IQFeed provider and clicked the "Clear Internal Request Tracking Info" and afterward, would run the data update again. In both cases (Post #12, #47), the update did not updates the data to the correct dates.
All data updates that I done are done manually and no Task Manager involved. The Task Manager only use to wake up the PC and occasionally to put the PC to sleep.
Truncating data always work for me to get the data update working.
Are you thinking the problem might lies with the internal tracking info? Would the # of days look ahead being used might be related to the problem as you might indicated earlier?
QUOTE:
I sure hope you're wrong about the tracking info, but your previous comment about it not working but then working after truncating data sure throws a wrench into it.
I hope the tracking info might be the clues. However, in the past events when data update not working, I would to go the IQFeed provider and clicked the "Clear Internal Request Tracking Info" and afterward, would run the data update again. In both cases (Post #12, #47), the update did not updates the data to the correct dates.
All data updates that I done are done manually and no Task Manager involved. The Task Manager only use to wake up the PC and occasionally to put the PC to sleep.
Truncating data always work for me to get the data update working.
Are you thinking the problem might lies with the internal tracking info? Would the # of days look ahead being used might be related to the problem as you might indicated earlier?
Cone,
Data update is not working tonight (1/26). Like last night there are some that got updated (see attachments) and most are not.
The above statement is from Post #63, I missed to reply earlier. I update data usually late in the evening (10pm AZ time or after) and stock market closes at 2PM AZ time, so by the time I do the data update (8hours passed from the time the market closed), IQFeed data should be available.
Looking at the pics below, why are some symbols got updated and others do not?
Both images below are capture last night (1/26) while the update is running.
Data update is not working tonight (1/26). Like last night there are some that got updated (see attachments) and most are not.
QUOTE:
I think the issue is that either you're updating before IQFeed has the Daily bars, or, you updated earlier in the session.
The above statement is from Post #63, I missed to reply earlier. I update data usually late in the evening (10pm AZ time or after) and stock market closes at 2PM AZ time, so by the time I do the data update (8hours passed from the time the market closed), IQFeed data should be available.
Looking at the pics below, why are some symbols got updated and others do not?
Both images below are capture last night (1/26) while the update is running.
It looks like the "< APP" symbols that didn't update last time are updating this time, and the symbols after "APP" that were updated last time didn't update this time. Truly bizarre.
I still want to believe it has something to do with clearing the Internal Requests, and that will be done automatically for all Data Manager updates with the next build (later today). Note that if you select a specific DataSet and do a DataSet update, the tracking is already cleared automatically.
It's already "tomorrow" in Arizona now (so all updates will probably work), but if there's a "next time" before the next build, try a DataSet update to see if there's a difference.
I still want to believe it has something to do with clearing the Internal Requests, and that will be done automatically for all Data Manager updates with the next build (later today). Note that if you select a specific DataSet and do a DataSet update, the tracking is already cleared automatically.
It's already "tomorrow" in Arizona now (so all updates will probably work), but if there's a "next time" before the next build, try a DataSet update to see if there's a difference.
Silly question, is it possible IQFeed just hasn’t made the data available yet for those symbols?
No, it's not [repeatedly] possible.
And, I've seen the issue less frequently for other providers, which lends weight to the internal register.
And, I've seen the issue less frequently for other providers, which lends weight to the internal register.
QUOTE:
It's already "tomorrow" in Arizona now (so all updates will probably work), but if there's a "next time" before the next build, try a DataSet update to see if there's a difference.
I don't know if this answers the question above, in the past (including last night) when data update is not working, I have tried to go to individual dataset and run update on it, however this step did not work. I also have tried to click on Update IQFeed Data and this too does not work. Typically, I would start the update by click on Update all Checked Providers.
Would it help if I capture and provide the entire data update log each day for say a week? I use 'Ctrl + L' but this only provides a brief list of events.
I can do a Ctrl + A to select all texts in the log and will post a complete log of the data update each day for the next week and starting tonight (1/27).
Here is the complete log that I just ran this afternoon (1/27) - market still open.
QUOTE:
Updating Provider: IQFeed
Updating Scale: Daily
A 5,583 1/26/2022
AA 1,318 1/26/2022
AABB 1,666 1/26/2022
AACG 3,422 1/26/2022
AAGC 2,177 1/26/2022
:
:
GTC.USD 2,001 1/27/2022 6:00 AM
NAX.USD 2,001 1/27/2022 6:00 PM
EDN.USD 2,001 1/27/2022 6:00 PM
Historical Provider Updates completed.
Just to be completed, last night there a few datasets (daily and 3 minutes) that were updated using the truncate method. So some of the symbols on this log that were updated might be resulted from that operation.
That's pretty painful to scroll through here (too bad we can't attach a document - next feature request!) .
I don't see anything out of the ordinary. Do you?
I don't see anything out of the ordinary. Do you?
I agree it is pretty long. Yes, it would better if we can do a file attachment.
Looking through it, almost all of the daily symbols are updated (as of 1/26). There are only few OTC symbols like WINSF, UNDR, etc. are still not updated. Last night I only updated about 500 (daily & 3 minutes) symbols manually using the truncate technique, which is about 10% of all the symbols. This past morning (1/27), when I re-run the update, the remaining 90% pretty much all got updated. Beside that I didn't see anything unusual.
Sorry for the long log again, here is the data update log for 1/27.
Updating Provider: IQFeed
Updating Scale: Daily
A 5,583 1/26/2022
AA 1,318 1/26/2022
AABB 1,666 1/26/2022
AACG 3,422 1/26/2022
AAGC 2,177 1/26/2022
AAGFF 1,079 1/26/2022
AAGH 1,507 1/26/2022
AAIC 6,062 1/26/2022
AAL 2,048 1/26/2022
AAME 5,962 1/26/2022
AAN 291 1/26/2022
:
:
TRUE.USD 2,004 1/28/2022 5:00 AM 3 bars added
TDS.USD 2,001 1/28/2022 5:00 AM
TDS.USD 2,001 1/28/2022 5:00 AM
Historical Provider Updates completed.
A look at the log, almost all the daily symbols are not updated. This is something new - typically in the past, it only fails for a day in the week and would work the next day, but this week, it did not work for all week (1/24 - 1/27). Those few symbols that got updated seems are all OTC symbols.
Looking through it, almost all of the daily symbols are updated (as of 1/26). There are only few OTC symbols like WINSF, UNDR, etc. are still not updated. Last night I only updated about 500 (daily & 3 minutes) symbols manually using the truncate technique, which is about 10% of all the symbols. This past morning (1/27), when I re-run the update, the remaining 90% pretty much all got updated. Beside that I didn't see anything unusual.
Sorry for the long log again, here is the data update log for 1/27.
Updating Provider: IQFeed
Updating Scale: Daily
A 5,583 1/26/2022
AA 1,318 1/26/2022
AABB 1,666 1/26/2022
AACG 3,422 1/26/2022
AAGC 2,177 1/26/2022
AAGFF 1,079 1/26/2022
AAGH 1,507 1/26/2022
AAIC 6,062 1/26/2022
AAL 2,048 1/26/2022
AAME 5,962 1/26/2022
AAN 291 1/26/2022
:
:
TRUE.USD 2,004 1/28/2022 5:00 AM 3 bars added
TDS.USD 2,001 1/28/2022 5:00 AM
TDS.USD 2,001 1/28/2022 5:00 AM
Historical Provider Updates completed.
A look at the log, almost all the daily symbols are not updated. This is something new - typically in the past, it only fails for a day in the week and would work the next day, but this week, it did not work for all week (1/24 - 1/27). Those few symbols that got updated seems are all OTC symbols.
It's almost impossible to follow this conversation with these long logs and I can't deal with them in the web page. I'll cut the logs out and save them aside.
If only OTC symbols were updated, that would point to IQFeed.
If only OTC symbols were updated, that would point to IQFeed.
Tell you what. I'm going to change the way we request Daily data from IQFeed. Instead of passing dates, we'll calculate the number of calendar days spanned by the request for the max number of datapoints, which should always return more bars than required. Let's see if it makes a difference. A new round of builds are scheduled to go out today or tomorrow.
Thanks Cone. Is there a way that perhaps I can send the log files to you via email if you want.
This morning (1/28), I run the data update and thus far, it seems it runs fine -data are updating correctly to the right date. So somehow, the time difference from last night of (1/27) and this morning of (1/28) makes a difference.
I will keep saving the data update log here and won't posted it but can send it to you if you want it.
Yes, let see how the new build you are making will affect this. Thanks.
This morning (1/28), I run the data update and thus far, it seems it runs fine -data are updating correctly to the right date. So somehow, the time difference from last night of (1/27) and this morning of (1/28) makes a difference.
I will keep saving the data update log here and won't posted it but can send it to you if you want it.
Yes, let see how the new build you are making will affect this. Thanks.
Build 17 is ready to download.
The logs really aren't helping - they're just telling us we have a problem that we already know about.
Re: morning (1/28)
If the problem is actually the Internal Request Tracking, one day makes all the difference.
Glitch just told me we didn't get the auto-clear in Build 46 for bulk updates, which is probably good so that we don't change 2 things at once that could affect the problem.
The logs really aren't helping - they're just telling us we have a problem that we already know about.
Re: morning (1/28)
If the problem is actually the Internal Request Tracking, one day makes all the difference.
Glitch just told me we didn't get the auto-clear in Build 46 for bulk updates, which is probably good so that we don't change 2 things at once that could affect the problem.
Cone, thanks for all the help. I will get the latest build and will keep you posted.
With the IQFeed latest build 17, the data updater updates today correctly all non-OTC symbols to (1/28). There are a number of OTC symbols that are not updated. This is a much better working step in the right direction. Will see how it goes in the coming days.
Data update is not working for today (1/31). Almost all daily data (OTC and non-OTC) are not updated. There are small number of daily OTC symbols got update correctly. The intraday data seems to update correctly. The whole data update took about few to several minutes to complete - possibly because it skipped most of daily symbols.
This does not occur in WL6, what are the differences in the two WLs data update that might cause this?
This does not occur in WL6, what are the differences in the two WLs data update that might cause this?
Internal Request Tracking.
We'll make sure that's automatically cleared for bulk updates in Build 47.
We'll make sure that's automatically cleared for bulk updates in Build 47.
After performed "Clear Internal Request Tracking Info" for IQFeed then clicked on individual data set does updated daily data correctly.
However, after performed "Clear Internal Request Tracking Info" for IQFeed then clicked on "Update IQFeed Data" does not updated daily data correctly.
However, after performed "Clear Internal Request Tracking Info" for IQFeed then clicked on "Update IQFeed Data" does not updated daily data correctly.
Yes, we know it's not cleared yet for "bulk updates", but it is auto-cleared now for DataSet updates. Next time you have trouble, Clear, then try clicking on a DataSet, and then Update DataSet.
Also, if you Clear Request Tracking, and just run your Strategies, you should get the updates now too.
Also, if you Clear Request Tracking, and just run your Strategies, you should get the updates now too.
Thanks Cone. With Clear Request Tracking set to auto cleared in build 47, Data Update should work without manually performing Clear Request Tracking?
That's the idea for all updates using the Data Manager.
This change didn't make Build 47 due to an oversight. Look for it in Build 48. My apologies.
Thank you Cone, look forward to checking it out.
Although it's not mentioned in the release notes, Build 48 should automatically clear request tracking for bulk updates. Let's hope that puts this problem behind us once and for all!
Cone, sorry to tell you that with WL build 48, the data update did not update data correctly tonight. I tried to do both the bulk update and individual dataset update and neither update to today (2/8/22) data.
I just also want to mention that in the last few days, I have been running data update before restarting WL, and it worked correctly every time. Maybe this is a coincidence - since it is not making sense. Anyhow, I will try it this way for the next week and see if it is working consistently.
Anyhow, this problem is frustrating for you and me and I thought you have it narrowed down to the internal tracking info request and were hoping we are done with it.
At the beginning of WL7, this problem wasn't there and starting late October last year, it started to appear, and it get worse since. Does anyone else has this issue? Data update on WL6 does not has this issue.
I just also want to mention that in the last few days, I have been running data update before restarting WL, and it worked correctly every time. Maybe this is a coincidence - since it is not making sense. Anyhow, I will try it this way for the next week and see if it is working consistently.
Anyhow, this problem is frustrating for you and me and I thought you have it narrowed down to the internal tracking info request and were hoping we are done with it.
At the beginning of WL7, this problem wasn't there and starting late October last year, it started to appear, and it get worse since. Does anyone else has this issue? Data update on WL6 does not has this issue.
Sorry to hear that. I don't know what to think now.
Let's work on giving you a special build with some logging so that we can see what's going on with the data requests. We'll contact you by email when we have something ready.
Let's work on giving you a special build with some logging so that we can see what's going on with the data requests. We'll contact you by email when we have something ready.
I am ready to throw in the towel on this for now. I can do the truncate method if needed and maybe down the road something will become more appearance.
Yes, please let me know whenever you have the special build ready. No hurry, I am sure you have many other important items to take care. Thanks.
Yes, please let me know whenever you have the special build ready. No hurry, I am sure you have many other important items to take care. Thanks.
Is there any chance that you've adjusted market hours or have the symbols assigned to a 24-hour market? That could probably only happen in a global way if you've used a bunch of wildcards in the Markets & Symbols > Symbols view.
Just to check - using IQFeed open a chart of AAPL or anything and check the market in the status bar. Does it say "US Stocks" by default?
Just to check - using IQFeed open a chart of AAPL or anything and check the market in the status bar. Does it say "US Stocks" by default?
No, the Market is NOT set to 'US Stocks'. It was set to 'My_Markets' - probably custom market that I created earlier that might have wild cards to look at the OTC symbols.
Would you like me to set it to 'US Stocks' by default? How would I do that?
Would you like me to set it to 'US Stocks' by default? How would I do that?
Thanks for the image. It's a good clue.
After inserting logging in the IQFeed history request path, I realized that none of the statements would be hit - the problem is not IQFeed, it's in the logic for WL's base provider that requests data. And if the problem can no longer be request tracking, it points to Market hours (that's why I asked).
Wealth-Lab 7's bulk update won't request daily data if the symbol's market is open - i.e., not for the day that's currently open.
Now I don't see a problem with [your] My_Markets, but it was an old bug on our part to have allowed Markets with the same name to be created. You must have done that early on before we got that fixed. I'm not sure what effect the naming conflicts can have, but it can't be good.
Please recreate your custom markets that have the same name as the core markets (CBOT, CME, NYMEX), give them unique names (that will be forced now anyway), and delete the old ones. Optionally, you'll need to reassign Symbols to your new markets.
After inserting logging in the IQFeed history request path, I realized that none of the statements would be hit - the problem is not IQFeed, it's in the logic for WL's base provider that requests data. And if the problem can no longer be request tracking, it points to Market hours (that's why I asked).
Wealth-Lab 7's bulk update won't request daily data if the symbol's market is open - i.e., not for the day that's currently open.
Now I don't see a problem with [your] My_Markets, but it was an old bug on our part to have allowed Markets with the same name to be created. You must have done that early on before we got that fixed. I'm not sure what effect the naming conflicts can have, but it can't be good.
Please recreate your custom markets that have the same name as the core markets (CBOT, CME, NYMEX), give them unique names (that will be forced now anyway), and delete the old ones. Optionally, you'll need to reassign Symbols to your new markets.
Cone, I have renamed the "My_Markets" market file to a more unique name and delete the old one as you have suggested.
After I rename the Market file "My_Markets" to a different name and delete the original file, when run the data update, I got the following errors.
All my strategies in the Strategy Monitor are gone - after restarting WL. Reloading the Workspaces does not bring back. It seems deleting the "My_Markets" market file might have something to do with this.
Re-installing the original Market file solved the "Object referencing" issue in the data update and the strategies also recovered in the Strategy Monitor.
While getting rid of the duplicates, you should simply reinstate everything based on them - like recreate strategies in the Strategy Monitor, picking the new market etc.
Thanks Eugene. After removing the old Market, how would I redirect WL to use a new market?
I got it now - the one step I missed earlier is to update the Symbol.txt file to reflect the new market.
Sorry for the confusion, but I said that My_Markets wasn't a problem, but that you should rename the duplicates - (CBOT, CME, and NYMEX).
It's my fault too for not studying the list of Markets carefully. I am not sure how the duplicates markets (CBOT, CME, NYMEX) got there but I didn't create them. The only custom market that I created is the "My_Markets".
I will remove all the duplicates.
Edited:
WL won't allow me to delete CBOT and CME, even after I signed in as admin. I was able to delete NYMEX.
I will remove all the duplicates.
Edited:
WL won't allow me to delete CBOT and CME, even after I signed in as admin. I was able to delete NYMEX.
It was probably an old bug that duplicated those. Click the button "Open Markets.txt".
Do you see them there?
If so, and when it's convenient, close Wealth-Lab, delete them from that file, and save.
Do you see them there?
If so, and when it's convenient, close Wealth-Lab, delete them from that file, and save.
Yes, I was able to see it in the 'Open Markets.txt' and deleted from there. Below is a screen shot of the updated list of Markets. Thanks.
Great. Let's see if that makes a difference.
Cone, the bulk data update did not work this morning - it did not update data to the correct date (2/11), ie., the latest data date is still as of last Thursday (2/10). It seems like it skipped over updating daily data when running bulk update. I tried to update individual data set but the update stops and freeze part way of the download process.
It also takes a while (longer than usual) to open and display a chart on any symbol.
I also performed WL repair with build 48 and that did not help.
Somehow, delete those markets (CBOT, CME) might related to this. Do you have any suggestions?
It also takes a while (longer than usual) to open and display a chart on any symbol.
I also performed WL repair with build 48 and that did not help.
Somehow, delete those markets (CBOT, CME) might related to this. Do you have any suggestions?
The freeze isn't Markets-related, it's probably lack of memory. Restart Wealth-Lab and disable Parallel updates in the IQFeed settings.
Here's something -
There's a Market ICEUS that's open from Midnight to 4pm every day. Symbols that are assigned to that market will only update their Daily bars between 4pm and midnight. Any other time, they'll be skipped.
Even though I think this is preloaded by Wealth-Lab and you probably haven't assigned anything to it, if you don't mind, please share the contents of your Symbols.txt file.
QUOTE:This could be Markets-related, but updating on Saturday in Arizona, I don't see how, yet this is the problem. Wealth-Lab thinks the Market is Open.
It seems like it skipped over updating daily data when running bulk update.
Here's something -
There's a Market ICEUS that's open from Midnight to 4pm every day. Symbols that are assigned to that market will only update their Daily bars between 4pm and midnight. Any other time, they'll be skipped.
Even though I think this is preloaded by Wealth-Lab and you probably haven't assigned anything to it, if you don't mind, please share the contents of your Symbols.txt file.
ICEUS should still update whenever the data is found to be out of date. The fact that a market is considered open doesn’t prevent an update, if it did then none of the crypto DataSets would ever update.
QUOTE:
The freeze isn't Markets-related, it's probably lack of memory. Restart Wealth-Lab and disable Parallel updates in the IQFeed settings
The Parallel updates box is NOT checked - so it is disabled the whole time.
I can tell that after the delete the CBOT and CME markets from the Symbols.txt, WL behaves not normal. The data update seems not responsive at all (it skipped entirely over the IQFeed provider and only updating the crypto symbols, charting symbols takes 20 or more seconds to load a symbol, strategies would run in infinite loop).
Markets.txt
QUOTE:
*V2,CFE,Central Standard Time,Sun+Weekdays,,2,0,Stock,USD,00:00:00,15:30:00,,USA
*V2,COMEX,Central Standard Time,Sun+Weekdays,,2,0,Stock,USD,00:00:00,16:00:00,,USA
*V2,Copy of CBOT,Central Standard Time,Sun+Weekdays,,2,0,Stock,USD,00:00:00,16:00:00,,
*V2,Copy of CME,Central Standard Time,Sun+Weekdays,,2,0,Stock,USD,00:00:00,16:00:00,,
*V2,Copy of NYMEX,Central Standard Time,Sun+Weekdays,,2,0,Stock,USD,00:00:00,16:00:00,,
*V2,ICEUS,Central Standard Time,All,,2,0,Stock,USD,00:00:00,16:00:00,,USA
*V2,US_Stocks_high_precisions,Eastern Standard Time,Weekdays,IWC,5,5,Stock,USD,09:30:00,16:00:00,,
Symbols.txt
QUOTE:
BG###,CFE,1,2443,0.005,2,0
BT###,CME,5,12830,0.5,0,0
DX###,ICEUS,1000,1815,0.005,3,0
E###,,1,1,0.1,5,5
ES###,,50,6600,0.25,2,0
NQ###,,20,7700,0.25,2,0
ZB###,CBOT,1000,2530,0.01,2,0
##,US_Stocks_high_precisions,1000,10000,0.01,4,4
###,US_Stocks_high_precisions,1000,10000,0.01,4,4
####,US_Stocks_high_precisions,1000,10000,0.01,4,4
#####,US_Stocks_high_precisions,1000,10000,0.01,4,4
######,US_Stocks_high_precisions,1000,10000,0.01,4,4
Edited:
I just noted 'Copy of CBOT', Copy of CME' and 'Copy of NYMEX' are in the Markets.txt - should I delete these?
QUOTE:Glitch, Daily bars are not updated if the market is open. DataProviderBase checks TimeSeriesBase.IsUpToDate(). If the market is open, the checkDate becomes PreviousTradingDay,.
The fact that a market is considered open doesn’t prevent an update, if it did then none of the crypto DataSets would ever update.
This Symbols.txt is a problem. All symbols will match the 2, 3, 4, 5, and 6 wildcard records, so at best there will be conflicts. Even so, since US_Stocks_high_Precision market has the proper hours and days defined, theoretically it should not be "the" problem. I'll play around with it and see where these conflicts lead.
It will update the previous day if needed. that’s what I meant. Just because a market is open, it won’t outright prevent a required update.
Maybe I said something like "won't update", but this whole discussion is about the failure to update the most-recent daily bar, so that's what I meant.
Is there a 'default' or workable Symbols.txt file that I can use in place or the current file?
Would uninstall/re-install WL7 help to reset the Symbols.txt file and maybe bring things back to default state? If so, I would like the Strategies and data files to be preserve and un-touch.
You can rename Symbols.txt to be safe, and WL7 should copy the default file back over the next time it starts up. Reinstalling is unnecessary.
Thanks Glitch.
Finally we've got this solved for Build 49!
Aside and fyi,
I'm going to roll back the change in the IQFeed Provider to request by date ranges again, and also include a change not to continually request data for old expired futures contracts.
Aside and fyi,
I'm going to roll back the change in the IQFeed Provider to request by date ranges again, and also include a change not to continually request data for old expired futures contracts.
Changing the Symbols.txt and deleting duplicates markets earlier somehow broke WL. Connection to IQFeed is broken and strategies, charting, data update are not working. Restore default Symbols.txt and Markets.txt did not help.
I finally did a Window restore and WL works now except there still the problem with data update. I will just leave the Markets settings at that.
I look forward to checking out build49. Thanks for your help.
I finally did a Window restore and WL works now except there still the problem with data update. I will just leave the Markets settings at that.
I look forward to checking out build49. Thanks for your help.
Symbols.txt and Market.txt aren't really made to edit manually, but if you do, every character has to be in the right place. After installing Build 49, I'd recommend that you start fresh - just delete those files. We removed the duplicates that we were creating.
Build 49 updates data correctly for Last Friday (2/18). Where do you delete the Symbols.txt and Markets.txt files? There are multiple copies of these files at different folders within Windows.
In the User Data folder, you can open it using the WL7 File menu.
Thanks.
Just an update, the data update on build 49 is working correctly for this week.
Your Response
Post
Edit Post
Login is required