I have an intraday strategy using 10-min bars with Tradier broker and streaming. The strategy generates trade signals based on indicators crossing, using historical close data of 10 min bars.
I was getting erroneous signals when running in the SM, (works fine with SW), particularly in the early hours of market open. Outputting the indicator values to the Log Viewer, I could see the indicator values were not correct when there was a false signal, as determined by comparing these values to a streaming chart I run in parallel (for viewing only, with 10-min streaming data also from Tradier). I noted that if I deactivated the strategy and reactivated, it would begin to provide accurate indicator values and it discarded the pre/post market data from the strategy. There is no pre/post market data in the chart.
In troubleshooting further, I exported to a file the historical close values the strategy was using to populate the indicators. I examined this file after a false signal resulting from inaccurate indicator values. I found the cause was that the strategy., which comes online via restart of WL premarket, had premarket intraday data in it. The first executable bar of the day was 9:40 (ET), but it was using post market data from 4:10 - 8:00 PM, and pre-market data starting a 12:10 am. This is despite the fact that I have the "Filter Pre/Post Market Data" check box selecting in the SM settings.
How can I filter out pre/post intraday data in this application? Could this have something to do with starting up WL pre-market with this strategy activated?
Notes
Symbol is QQQ
Tradier does not have 10-min data natively, but it does have 5-min
I was getting erroneous signals when running in the SM, (works fine with SW), particularly in the early hours of market open. Outputting the indicator values to the Log Viewer, I could see the indicator values were not correct when there was a false signal, as determined by comparing these values to a streaming chart I run in parallel (for viewing only, with 10-min streaming data also from Tradier). I noted that if I deactivated the strategy and reactivated, it would begin to provide accurate indicator values and it discarded the pre/post market data from the strategy. There is no pre/post market data in the chart.
In troubleshooting further, I exported to a file the historical close values the strategy was using to populate the indicators. I examined this file after a false signal resulting from inaccurate indicator values. I found the cause was that the strategy., which comes online via restart of WL premarket, had premarket intraday data in it. The first executable bar of the day was 9:40 (ET), but it was using post market data from 4:10 - 8:00 PM, and pre-market data starting a 12:10 am. This is despite the fact that I have the "Filter Pre/Post Market Data" check box selecting in the SM settings.
How can I filter out pre/post intraday data in this application? Could this have something to do with starting up WL pre-market with this strategy activated?
Notes
Symbol is QQQ
Tradier does not have 10-min data natively, but it does have 5-min
Rename
The Pre/Post filter works - that's easy to check right now. I don't know how it can be different than a Streaming Chart (it uses the same code) but maybe the SM is allowing new bars to be built premarket. I'll check it tomorrow.
That was a quick test. If you enable the Pre/Post filter here, it won't even schedule the first run until the market opens, and the data is filtered.
In short, I can't duplicate it unless I assign the symbol under test to the wrong market.
In short, I can't duplicate it unless I assign the symbol under test to the wrong market.
Here's the result from a strategy that exports the BarHistory when it runs. I did this with 4 different symbols and 4 different Data Loading settings. All the same - no premarket data even though Tradier streaming was running premarket.
This one is TTD -
This one is TTD -
CODE:
20240927 1540 109.27 109.325 109.20 109.22 29870.0 20240927 1550 109.21 109.44 109.21 109.31 57196.0 20240927 1600 109.29 109.57 109.25 109.47 185051.0 20240930 0940 109.08 109.08 108.15 108.81 46593.0 20240930 0950 108.83 109.3286 108.74 109.3286 47101.0
@Cone Thank you for setting up a test. Did you use Tradier 10 min data for streaming and Tradier for historical priority?
yes - and seriously would make no difference because [insert Provider Name here] Streaming "is a" StreamingProviderBase. These bars are built and pushed by one piece of "agnostic" code.
The only things I can think of that could have happened are that the "Filter" option was not set in the SM Configuration (saving it with the Strategy Settings doesn't matter), or, the wrong Market was applied - possibly by selection or by error.
The only things I can think of that could have happened are that the "Filter" option was not set in the SM Configuration (saving it with the Strategy Settings doesn't matter), or, the wrong Market was applied - possibly by selection or by error.
OK. I will run another test tomorrow and see if I can duplicate the issue. Thanks.
The first signal I got today was an erroneous Buy. My strategy outputs the historical data for the time period that the strategy is using to a file when there is a Buy or Sell signal.
At 2 PM (eastern) I got a Buy signal. The file clearly shows that there is post / pre market bars in the data.
Here is some Tradier log activity that preceded the Buy and file output. Possibly the disconnect is triggering a data update that includes pre / post market data, ignoring the filter.
Addendum
SM Log
When strategy is deactivated / reactivated, the strategy starts working correctly again. No doubt because the data is (correctly) refreshed.
Here's what the file looks like after the reactivation
At 2 PM (eastern) I got a Buy signal. The file clearly shows that there is post / pre market bars in the data.
QUOTE:
10/2/2024 4:00:00 PM 481.94
10/2/2024 4:10:00 PM 482.61
10/2/2024 4:20:00 PM 482.82
10/2/2024 4:30:00 PM 482.86
10/2/2024 4:40:00 PM 483.1
10/2/2024 4:50:00 PM 483.32
10/2/2024 5:00:00 PM 483.16
10/2/2024 5:10:00 PM 483.24
10/2/2024 5:20:00 PM 483.29
10/2/2024 5:30:00 PM 483.25
10/2/2024 5:40:00 PM 483.13
10/2/2024 5:50:00 PM 483.089
10/2/2024 6:00:00 PM 483.0006
10/2/2024 6:10:00 PM 483.32
10/2/2024 6:20:00 PM 483.02
10/2/2024 6:30:00 PM 483.11
10/2/2024 6:40:00 PM 483
10/2/2024 6:50:00 PM 482.9
10/2/2024 7:00:00 PM 482.82
10/2/2024 7:10:00 PM 482.82
10/2/2024 7:20:00 PM 482.8
10/2/2024 7:30:00 PM 482.79
10/2/2024 7:40:00 PM 482.8205
10/2/2024 7:50:00 PM 483.001
10/2/2024 8:00:00 PM 483.1
10/3/2024 12:10:00 AM 480.76
10/3/2024 12:20:00 AM 480.85
10/3/2024 12:50:00 AM 480.5
10/3/2024 1:00:00 AM 480.5
10/3/2024 1:10:00 AM 480.75
10/3/2024 1:20:00 AM 480.69
10/3/2024 1:30:00 AM 480.83
10/3/2024 1:40:00 AM 480.98
10/3/2024 1:50:00 AM 480.43
10/3/2024 2:00:00 AM 480.45
10/3/2024 2:20:00 AM 480.81
10/3/2024 2:30:00 AM 480.38
10/3/2024 2:40:00 AM 480.15
10/3/2024 2:50:00 AM 480.3
10/3/2024 3:00:00 AM 481.1
10/3/2024 3:10:00 AM 480.88
10/3/2024 3:20:00 AM 479.46
10/3/2024 3:30:00 AM 479
10/3/2024 3:40:00 AM 478.95
10/3/2024 3:50:00 AM 478.81
10/3/2024 4:00:00 AM 479.06
10/3/2024 4:10:00 AM 479.38
10/3/2024 4:20:00 AM 479.7
10/3/2024 4:30:00 AM 479.67
10/3/2024 4:40:00 AM 479.91
10/3/2024 4:50:00 AM 479.74
10/3/2024 5:00:00 AM 479.73
10/3/2024 5:10:00 AM 479.51
10/3/2024 5:20:00 AM 479.8
10/3/2024 5:30:00 AM 479.85
10/3/2024 5:40:00 AM 479.97
10/3/2024 5:50:00 AM 479.79
10/3/2024 6:00:00 AM 479.89
10/3/2024 6:10:00 AM 479.96
10/3/2024 6:20:00 AM 479.72
10/3/2024 6:30:00 AM 480.04
10/3/2024 6:40:00 AM 479.98
10/3/2024 6:50:00 AM 480.03
10/3/2024 7:00:00 AM 480.55
10/3/2024 7:10:00 AM 480.6865
10/3/2024 7:20:00 AM 480.2996
10/3/2024 7:30:00 AM 480.76
10/3/2024 7:40:00 AM 480.84
10/3/2024 7:50:00 AM 480.85
10/3/2024 8:00:00 AM 481.26
10/3/2024 8:10:00 AM 481.041
10/3/2024 8:20:00 AM 481.34
10/3/2024 8:30:00 AM 481.2
10/3/2024 8:40:00 AM 480.4525
10/3/2024 8:50:00 AM 480.65
10/3/2024 9:00:00 AM 480
10/3/2024 9:10:00 AM 480.15
10/3/2024 9:20:00 AM 479.81
10/3/2024 9:30:00 AM 479.77
10/3/2024 9:40:00 AM 481
10/3/2024 9:50:00 AM 481.67
10/3/2024 10:00:00 AM 482.46
10/3/2024 10:10:00 AM 484.42
10/3/2024 10:20:00 AM 482.52
10/3/2024 10:30:00 AM 481
10/3/2024 10:40:00 AM 481.47
10/3/2024 10:50:00 AM 481.29
10/3/2024 11:00:00 AM 480.44
10/3/2024 11:10:00 AM 481.18
10/3/2024 11:20:00 AM 481.1224
10/3/2024 11:30:00 AM 481.695
10/3/2024 11:40:00 AM 482.2061
10/3/2024 11:50:00 AM 482.57
10/3/2024 12:00:00 PM 482.7913
10/3/2024 12:10:00 PM 482.7087
10/3/2024 12:20:00 PM 482.6201
10/3/2024 12:30:00 PM 482.6
10/3/2024 12:40:00 PM 481.99
10/3/2024 12:50:00 PM 480.39
10/3/2024 1:00:00 PM 479.97
10/3/2024 1:10:00 PM 479.8896
10/3/2024 1:20:00 PM 479.88
10/3/2024 1:30:00 PM 480.5566
10/3/2024 1:40:00 PM 480.595
10/3/2024 1:50:00 PM 480.46
10/3/2024 2:00:00 PM 480.98
Here is some Tradier log activity that preceded the Buy and file output. Possibly the disconnect is triggering a data update that includes pre / post market data, ignoring the filter.
QUOTE:
"10/3/2024 06:38:23:352" "Tradier" "06:38:23.352 Initiating Tradier accounts socket...Start"
"10/3/2024 08:10:26:407" "Tradier" "Streaming Disconnected - attempting Reconnect: Lost, CloseStatus: , SessionId: 42386eb8-c692-4e3e-ab88-7496c36c3016"
"10/3/2024 08:10:27:677" "Tradier" "Tradier Streaming Reconnected after Lost on sessionID: 42386eb8-c692-4e3e-ab88-7496c36c3016"
"10/3/2024 08:10:27:977" "Tradier" "Streaming Reconnect Successful"
"10/3/2024 08:10:28:213" "Tradier" "Tradier Streaming Reconnected after Lost on sessionID: 42386eb8-c692-4e3e-ab88-7496c36c3016"
"10/3/2024 11:00:03:521" "Position Inactive" ""
"10/3/2024 11:00:03:521" "02:00 PM" "Signal is Buy: Ki is 0.0028 SMAK is -0.0587"
"10/3/2024 11:00:08:739" "Tradier" "user/profile Get empty"
"10/3/2024 11:10:04:366" "Position Active"
Addendum
SM Log
QUOTE:
10/3/2024 11:00:00: Status = Processing
10/3/2024 11:00:00: Status = Processing
10/3/2024 11:00:00: Pass 1 - Requesting updates for 1 symbols...
10/3/2024 11:00:02: Pass 1 - UpdateHistories returned 1 symbols
10/3/2024 11:00:07: Ran Strategy on QQQ,QQQ241010C00481000: 1 Signals, Run Time=4.96sec
10/3/2024 11:00:07: NextRun set to 10/3/2024 11:10
10/3/2024 11:00:07: Status = Completed
10/3/2024 11:00:07: Processed all Symbols
10/3/2024 11:10:01: Status = Processing
10/3/2024 11:10:01: Pass 1 - Requesting updates for 1 symbols...
10/3/2024 11:10:02: Pass 1 - UpdateHistories returned 1 symbols
10/3/2024 11:10:04: Ran Strategy on QQQ: 0 Signals, Run Time=1.67sec
10/3/2024 11:10:04: NextRun set to 10/3/2024 11:20
10/3/2024 11:10:04: Status = Completed
10/3/2024 11:10:04: Processed all Symbols
10/3/2024 11:20:00: Status = Processing
10/3/2024 11:20:00: Pass 1 - Requesting updates for 1 symbols...
10/3/2024 11:20:02: Pass 1 - UpdateHistories returned 1 symbols
10/3/2024 11:20:03: Ran Strategy on QQQ: 0 Signals, Run Time=1.94sec
10/3/2024 11:20:03: NextRun set to 10/3/2024 11:30
10/3/2024 11:20:03: Status = Completed
10/3/2024 11:20:03: Processed all Symbols
10/3/2024 11:26:41: Status = Deactivated
10/3/2024 11:26:42: Leaving Polling Thread
10/3/2024 11:32:06: Status = LoadingData
10/3/2024 11:32:06: Start Polling Thread
10/3/2024 11:32:06: Populating Data
10/3/2024 11:32:06: Calling GetHistories Pass 1
10/3/2024 11:32:08: GetHistories returned with 1 symbols
10/3/2024 11:32:08: Calling GetHistories Pass 2
10/3/2024 11:32:08: GetHistories returned with 1 symbols
10/3/2024 11:32:08: QQQ:400 bars to 20241003 1430|Tradier
10/3/2024 11:32:08: NextRun set to 10/3/2024 11:40
10/3/2024 11:32:08: Status = Idle
When strategy is deactivated / reactivated, the strategy starts working correctly again. No doubt because the data is (correctly) refreshed.
Here's what the file looks like after the reactivation
QUOTE:
10/2/2024 3:50:00 PM 482.98
10/2/2024 4:00:00 PM 481.94
10/3/2024 9:40:00 AM 481
10/3/2024 9:50:00 AM 481.67
10/3/2024 10:00:00 AM 482.46
10/3/2024 10:10:00 AM 484.42
10/3/2024 10:20:00 AM 482.52
10/3/2024 10:30:00 AM 481
10/3/2024 10:40:00 AM 481.47
10/3/2024 10:50:00 AM 481.29
10/3/2024 11:00:00 AM 480.44
10/3/2024 11:10:00 AM 481.18
10/3/2024 11:20:00 AM 481.1224
10/3/2024 11:30:00 AM 481.695
10/3/2024 11:40:00 AM 482.2061
10/3/2024 11:50:00 AM 482.57
10/3/2024 12:00:00 PM 482.7913
10/3/2024 12:10:00 PM 482.7087
10/3/2024 12:20:00 PM 482.6201
10/3/2024 12:30:00 PM 482.6
10/3/2024 12:40:00 PM 481.99
10/3/2024 12:50:00 PM 480.39
10/3/2024 1:00:00 PM 479.97
10/3/2024 1:10:00 PM 479.8896
10/3/2024 1:20:00 PM 479.88
10/3/2024 1:30:00 PM 480.5566
10/3/2024 1:40:00 PM 480.595
10/3/2024 1:50:00 PM 480.46
10/3/2024 2:00:00 PM 480.98
10/3/2024 2:10:00 PM 480.91
10/3/2024 2:20:00 PM 481.07
10/3/2024 2:30:00 PM 480.7099
10/3/2024 2:40:00 PM 480.82
We need to discover how to make that happen then.
Meaning the disconnect? If you think it would happen with the sandbox, then I think it's as simple as running a 10-min strategy all day that calculates an indicator or just populates a chart. I'm not sure it matters. It happens frequently, as you know. Based on the Log Viewer notifications, it looks like the streaming data is disconnecting, not the broker connection.
Aren't we talking about Pre/Post data when there should not be any?
I don't see how your streaming disconnects can affect that. If anything that would make less data.
Also, the image shows you selected Polling, so that's not even the same as what you've described as the problem.
I don't see how your streaming disconnects can affect that. If anything that would make less data.
Also, the image shows you selected Polling, so that's not even the same as what you've described as the problem.
QUOTE:
Aren't we talking about Pre/Post data when there should not be any?
Yes. Which is corrupting the values of the custom indicator (should be based on market-open data only).
QUOTE:
I don't see how your streaming disconnects can affect that. If anything that would make less data.
In #post7 I wrote:
QUOTE:
Here is some Tradier log activity that preceded the Buy and file output. Possibly the disconnect is triggering a data update that includes pre / post market data, ignoring the filter.
My theory is that the Reconnect appears to be updating the history data, but it ignores the "Filter Pre/Post Market Data" setting. Indirectly the disconnects could be the catalyst but Pre/Post market data should not be downloaded with Reconnect in any case, which is the problem.
QUOTE:
Also, the image shows you selected Polling, so that's not even the same as what you've described as the problem.
I think you may be referring to the title post where I wrote I was using Tradier streaming. That was referring to the selection in the "Broker and Streaming Providers" section of the SM settings.
The irony is the strategy is not even using streaming data. although the chart (which is separate from the strategy but connected to Tradier) is showing bars being built, so the chart must be streaming data.
One caution is I can't prove yet that the Pre/Post data problem is being caused by Disconnect / Reconnect. Since my strategy only outputs data to file with a signal, it's possible it was corrupted before market open. My theory is based on circumstantial evidence so far (timeline correlation between disconnect and data filter fail).
I am going to modify the strategy to start outputting to file on the first bar of the last day. That way I can see immediately on the first bar if the data is already corrupted overnight. If it is not, and it becomes corrupted later in the day, it may synchronize with some disconnect event in the Log Viewer.
Streaming doesn't have anything to do with it.
You only need to let me know your work flow, and I'm guessing now that you probably load from a Workspace. This is likely the missing piece (and likely cause), because I didn't do that when testing it.
You only need to let me know your work flow, and I'm guessing now that you probably load from a Workspace. This is likely the missing piece (and likely cause), because I didn't do that when testing it.
Loading from a Workspace is the issue - we should be able to get that corrected for the next build. Sorry for the bug and thanks for the observation!
Ah.... I restarted WL pre-market this morning and the file from the test I discussed in Post #11 shows that post / pre-market data was downloaded. But it sounds like you don't need to see that anymore.
No thanks. And the weird part about all of it is that Pre/Post is working for everything except the last Pre/Post period (starting from yesterday's market close).
Yes, I noticed that, and it confused me. Weird.
Is this just a Tradier thing or is this true with all data providers?
Is this just a Tradier thing or is this true with all data providers?
The Pre/Post filter is a WealthLab client feature; the provider wouldn't matter unless it went out of its way to do it's own Pre/Post filtering - like the IQ Provider with the Regular Session Only option.
QUOTE:
My theory is that the Reconnect appears to be updating the history data, but it ignores the "Filter Pre/Post Market Data" setting.
I had several Disconnects / Reconnects today and my program outputted the bar data to a file after these events. There were no pre/post market bars in the data, so my theory is incorrect and the only cause of these bars appearing is the one you already found.
Yes, you shouldn't have gone through the trouble. We're already testing the fix .. and since you'll ask >> ensures that the Pre/Post filter is applied everywhere it should be.
This appears to be fixed in WL B101 as it is working normally for me. Showing no OOM bars with intraday data at start of day.
Your Response
Post
Edit Post
Login is required