Tradier-signals-not-getting-to-Order-Manager-Resolved

We're trying to get to the bottom of the Tradier order status issue, but this sounds more insidious.
Anything from Tradier in the Log Viewer? Sounds like something disconnected.

[corrected]
Nothing in the log viewer.

Let's run a poor man's (person?) connection monitor in WealthLab. This script just pings google's DNS 6 or 7 times every 8 seconds. You can run it in the S. Monitor on 1-minute bars, any symbol, polling is fine. If there's a dropout, you'll see a "Ping Test" in Red in the Log Viewer. It also keeps a running log of the pings and the "Roundtrip Time". If you're game, let's see if it finds something on Thursday.

CODE:

/* IMPORTANT! Add References (Tools > Assembly References) to: 
 * System.Net.Ping
 * System.Net.Primitives
 * System.ComponentModel.Primitives
 */
 
using System;
using System.Diagnostics;
using System.IO;
using System.Net.NetworkInformation;
using System.Threading.Tasks;
using WealthLab.Backtest;
using WealthLab.Core;
using WealthLab.Data;
using System.Threading;


namespace WealthScript2 
{
    public class PingTest : UserStrategyBase
    {   
        public override void Initialize(BarHistory bars)
        {
         string logFilePath = Path.Combine(WLHost.Instance.DataFolder, $"{DateTime.Now:yyyy-MM-dd}_connection_log.txt");
         string pingAddress = "8.8.8.8"; // Google DNS
         int pingInterval = 8000; // msec
         DateTime dte = DateTime.Now; 

         using (StreamWriter logWriter = new StreamWriter(logFilePath, append: true))
         {
            logWriter.WriteLine("Timestamp\t\t\t\tStatus\tRoundtrip Time (ms)");

            while((DateTime.Now - dte).TotalSeconds < 56)
            {
               Ping ping = new Ping();
               PingReply reply = ping.SendPingAsync(pingAddress).Result;

               string logEntry = $"{DateTime.Now:yyyy-MM-dd HH:mm:ss.fff}\t{reply.Status}\t{reply.RoundtripTime}";
               logWriter.WriteLine(logEntry);
               WriteToDebugLog(logEntry);

               if (reply.Status != IPStatus.Success)
                  WLHost.Instance.AddLogItem("Ping Test", $"Connection issue detected!\n{logEntry}", WLColor.Red);

               Thread.Sleep(pingInterval);
            }
         }
        }
        
        public override void Execute(BarHistory bars, int idx)
        {  }   
    }
}

10:43 AM (E) Signal to close IWM option in SM not transmitted to Order Manager (2x). Manually closed position.
10:58 AM Streaming data has stopped. Restarting WL.
11:03 AM Broker manager once again receiving orders from signals
11:38 AM Noticed that streaming data has stopped again. Restarting WL.
11:59 AM Noticed that streaming data has stopped again. Restarting WL.
1:18 PM Order Manager errors on opening position for EEM option
"Unexpected character encountered while parsing value: A. Path '', line 0, position 0.
An error occurred while communicating with the backend." repeated 4 times. Restarting WL
1:22 PM Order Manager is working again.
1:29 PM Order Manager places close on EEM option but it never reaches Tradier. Manually closing position.
2:11 PM Streaming data has stopped again.
Notice Ping stopped running in SM based on WL8 Log File. See gap in timestamp. Probably stops running if streaming data is lost. I will switch to Polling:

QUOTE:

Timestamp Status Roundtrip Time (ms)
2024-06-20 11:12:02.259 Success 27
2024-06-20 11:12:10.286 Success 25
2024-06-20 11:12:18.314 Success 26
2024-06-20 11:12:26.342 Success 25
2024-06-20 11:12:34.370 Success 26
2024-06-20 11:12:42.397 Success 25
2024-06-20 11:12:50.425 Success 25
Timestamp Status Roundtrip Time (ms)
2024-06-20 11:39:08.636 Success 25
2024-06-20 11:39:16.663 Success 26
2024-06-20 11:39:24.690 Success 25
2024-06-20 11:39:32.715 Success 25
2024-06-20 11:39:40.743 Success 26
2024-06-20 11:39:48.769 Success 25
2024-06-20 11:39:56.796 Success 25
Timestamp Status Roundtrip Time (ms)
2024-06-20 12:09:53.538 Success 25
2024-06-20 12:10:01.564 Success 25
2024-06-20 12:10:09.590 Success 25
2024-06-20 12:10:17.617 Success 25
2024-06-20 12:10:25.644 Success 26
2024-06-20 12:10:33.671 Success 25
2024-06-20 12:10:41.697 Success 25

Restarting WL.

3:37 PM Trades for EEM and SPY options in Order Manager never make it to Tradier.
Looks like no disruption in Ping log file:

QUOTE:
Timestamp Status Roundtrip Time (ms)
2024-06-20 09:35:01.660 Success 26
2024-06-20 09:35:09.711 Success 25
2024-06-20 09:35:17.737 Success 25
2024-06-20 09:35:25.765 Success 25
2024-06-20 09:35:33.791 Success 25
2024-06-20 09:35:41.819 Success 26
2024-06-20 09:35:49.845 Success 25
Timestamp Status Roundtrip Time (ms)
2024-06-20 09:36:02.359 Success 25
2024-06-20 09:36:10.385 Success 25
2024-06-20 09:36:18.412 Success 25
2024-06-20 09:36:26.438 Success 25
2024-06-20 09:36:34.465 Success 25
2024-06-20 09:36:42.503 Success 36
2024-06-20 09:36:50.531 Success 25
Timestamp Status Roundtrip Time (ms)
2024-06-20 09:37:02.082 Success 25
2024-06-20 09:37:10.108 Success 25
2024-06-20 09:37:18.135 Success 25
2024-06-20 09:37:26.162 Success 26
2024-06-20 09:37:34.190 Success 26
2024-06-20 09:37:42.216 Success 25
2024-06-20 09:37:50.243 Success 25
Timestamp Status Roundtrip Time (ms)
2024-06-20 09:38:01.661 Success 25
2024-06-20 09:38:09.687 Success 25
2024-06-20 09:38:17.715 Success 25
2024-06-20 09:38:25.741 Success 25
2024-06-20 09:38:33.768 Success 25
2024-06-20 09:38:41.794 Success 25
2024-06-20 09:38:49.821 Success 25

Restarting WL

It's abnormal. Run the connection test and let's see some ping data.

It's been running in SM since the market opened. Can you see log file?

Addendum

Ony one error in Log Viewer from 1:23 PM

QUOTE:
6/20/2024 10:23:09:200
Tradier
Error Replacing Order (EEM240628C00042500 Buy 14 at Market,EEM240628C00042500 Buy 18 at Market): Unexpected character encountered while parsing value: o. Path '', line 0, position 0.
order not in valid state for modifications
JsonReaderException
Unexpected character encountered while parsing value: o. Path '', line 0, position 0.
at Newtonsoft.Json.JsonTextReader.ParseValue()
at Newtonsoft.Json.JsonReader.ReadAndMoveToContent()
at Newtonsoft.Json.JsonReader.ReadForType(JsonContract contract, Boolean hasConverter)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings)
at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value, JsonSerializerSettings settings)
at WealthLab.Tradier.TradierBroker.ReplaceTrade(Transaction oldOrder, Transaction newOrder)

Addendum 2
The above error repeated on EEM option

QUOTE:
6/20/2024 10:51:14:271
Tradier
Error Replacing Order (EEM240628C00042500 Buy 14 at Market,EEM240628C00042500 Buy 21 at Market): Unexpected character encountered while parsing value: A. Path '', line 0, position 0.
An error occurred while communicating with the backend.
JsonReaderException
Unexpected character encountered while parsing value: A. Path '', line 0, position 0.
at Newtonsoft.Json.JsonTextReader.ParseValue()
at Newtonsoft.Json.JsonReader.ReadAndMoveToContent()
at Newtonsoft.Json.JsonReader.ReadForType(JsonContract contract, Boolean hasConverter)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings)
at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value, JsonSerializerSettings settings)
at WealthLab.Tradier.TradierBroker.ReplaceTrade(Transaction oldOrder, Transaction newOrder)

I ran Ping Test in SW and got this error

QUOTE:
Initialize Exception (QQQ) Line 24 - The process cannot access the file 'C:\Users\inner\AppData\Roaming\WealthLab8\2024-06-20_connection_log.txt' because it is being used by another process.
at Microsoft.Win32.SafeHandles.SafeFileHandle.CreateFile(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options)
at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable`1 unixCreateMode)
at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable`1 unixCreateMode)
at WealthScript1.PingTest.Initialize(BarHistory bars) in :line 24
at WealthLab.Backtest.UserStrategyExecutor.CreateService(List`1 symbols)

Addendum 3

Interesting. While EEM option buy continues to fail, WL successfully traded an IWM option. Maybe related to this specific EEM option on backend?

Restarting WL now to see if it will trade EEM option. It did.

Addendum 4
Ping test ran successfully in SW after WL restart

If there's a ping failure, you'll see it in the Log Viewer. Don't open the file unless you want to drill down after the market is closed.

Tradier support said they can't duplicate any of this. We can't do it nearly as regularly as you, but I have seen it a couple times now. Even so, it just leads us back to messaging from Tradier, which is sort of a dead end for now.

There are multiple issues here:
1. Streaming (1-min) disconnects multiple times a day
2. Completed trades not updating the Accounts page, requiring manual updates by click.
3. Trading signals not getting to Order Manager
4. Order Manger placed trades not getting to Broker

This is the first time I have run this type of strategy that uses 1-min streaming data. I haven't documented it as extensively, but I was having similar problems with IB running the same strategy, such as streaming disconnects, in the paper account. However, the Accounts page not updating definitely appears unique to Tradier.

Addendum

I wish I had a way to change to another configuration, or do some additional testing, but it doesn't look like there's an easy path to do that. I would have to re-modify the Tradier strategy to run with IB but perhaps I should and run it with a paper account to see if these problems are duplicated.

Since I Beta-tested the Medved extension, I noticed that once a bug with Medved and in in the extension was fixed, I was no longer getting streaming disconnects. But I don't know if that fix applies to the Tradier implementation.

It would be nice to know whether these are Tradier failures or extension implementation. It's hard to believe Tradier, with their stock-in-trade being their API support, has this many problems completing simple tasks. It seems I'm the first one to report these types of issues to Tradier. I don't know how many other WL users are running this type of strategy with Tradier. If they're out there, I would love to hear their experience.

In the meantime, regardless of the cause, I can't see continuing to run this strategy at Tradier with real money until we have resolutions. Unfortunately, their paper account does not support options (not sure that's relevant) or streaming data (relevant).

Addendum2

What I see happening is that a symbol that fails to get to Tradier Broker has a CancelPending status in the Order Manager that never goes away. The timestamp on the order changes with each new signal, but the status doesn't, which is probably why the multiple orders for that symbol never get to the Tradier Broker.

Addendum3
I just confirmed my theory in Addendum 2. After a signal failed to trade, I went into the Order Manager and "Killed" the CancelPending status trade for the symbol. On the next signal, it traded properly. So it looks like this could be another symptom of this unresolved issue

The subject issue in this post is a symptom of another issue, the status of which you can find here..

There are so many of these threads, I don't know which is most appropriate to respond to.
I want you avoid connecting to Tradier Streaming while trading today.

Don't use Tradier for:
1. Quotes
2. Streaming Charts
3. Streaming for the S. Monitor
4. and, avoid Order Manager manual orders, but if you must ...



Let's see if streaming is interrupting the broker connection.

No need to do this. It was based on something I saw that I've already debunked.

Made good progress on this and seeing the light at the end of the tunnel.
If you're game, give this a try to test a hypothesis and see if it works out.

1. Connect to Tradier.
2. Within one minute, send in a limit order away from the market so that it doesn't execute.
3. If all is as I'm seeing it now, it should go Active, and you can Cancel it.
4. I think you'll be good to go for the day, as long as you have a good connection.

Although we have some work to do (already mostly done) for Detect/Reconnect, It looks like the initial problem is keeping the market socket alive on Tradier's end if you don't get an order in before it's sends the first heartbeat (should occur each minute).

This is preliminary, but I've repeated the test about 4 times and it works each time... and also after a Reconnect.

That's encouraging news! Of course, I'm game to test whatever you want to throw at me. Unfortunately, as the Tradier gremlins would have it, today it is working perfectly. If it breaks again tomorrow, I can try something new.

However, please be aware that I have been also placing limit trades with Tradier and they work fine (placing and executing) but they do not update (when Tradier is broken) the Account page or Order Manager with the fill.

When you wrote to Cancel the limit order as a test, do you care how it is cancelled, i.e., programmatically or using Order Manager manually, or using Tradier's Web interface? For that matter do you care if the limit order is placed programmatically or through Tradier's Web interface?

Addition

I am aware that there are multiple discussions that are related. Initially, it wasn't clear to me that these were all caused by the same underlying issue until I watched trades very carefully. Since then, I have been attempting to terminate related discussions by inserting a link to the "mother" discussion. If you would prefer that I link to a different one, let me know.

I'm still waiting on an answer on this from Tradier, but what I mentioned above was a consistent scenario last Friday and even when testing on Saturday.

Canceling the order is optional (any way you want). What appeared important was receiving a heartbeat in the first minute - that's in the background, but if it doesn't occur it's an early indication that the session is already lost.

Now (in dev) we're successfully reconnecting on a different sessionId after detecting this condition. Even so, the failed heartbeat may occur again immediately. What I found was that placing an order within the first minute always resulted in a successful stable connection thereafter...

...for at least some hours - 2 or even maybe 6 -before getting a disconnect again. There's always a chance these disconnects are somehow related to my ISP, but it seems doubtful. Imho, Tradier needs to find the primary cause on their end.

WL shut down unexpectedly (internal error). When it restarted, Tradier was failing again. Here's what I tried based on my understanding of your request:

1. I placed an order using Tradier website for an option contract with an OOM limit.
2. Tradier showed order as pending.
3. I successfully cancelled order using Tradier website.

Tradier still did not process fills correctly after trying this. Of course, the Order Manager was not involved in the test transaction. Let me know if there's anything else you want me to try.

Well, you need to place the order with WealthLab within the first minute after "Connect".

The idea is to "jump start" the accounts session that just connected.

With Wealth-Lab programmatically or manually from Order Manager?

Either. Both are placed the same way.

I placed a limit order within 1 min after WL restart, then cancelled it. No effect on fixing Tradier sync issue on subsequent trades.

okay, good to know. That technique hasn't failed for me yet, even though later the connection can still drop off - maybe 30 minutes later or maybe 6 hours later.

I've run B28 for 3 days now with a few dozen auto-trades. No errors. Thank you, thank you, thank you!

I'm curious whether this was a shortcoming on Tradier's side, or just another typical issue that arises when implementing a new broker extension. Is what you found and coded applicable to the Schwab WebSocket implementation as well?

Great, thanks for the confirmation.

I thought I explained it in Post #21. The difference with Tradier is that the [typical] 1-minute heartbeat is relative to the connect time instead of "regular" intervals, like 09:31:00, 09:32:00, etc. If you connect at 09:28:23, then Tradier's heartbeat for your connection is at 09:29:23, 09:30:23, etc.

The problem was that the socket timeout (no activity) was also 60 seconds. So milliseconds were the difference of keeping or resetting a connection. The account socket is inactive most of the time, unless you're sending orders in every minute; the heartbeat keeps it active so that it's not reset.

But even after a reset, the same issue would keep occurring. If the heartbeat were sent at any other time (as we're accustomed to with every other integration), it's very unlikely to have ever had an issue.

Interesting. Thanks for the detail. I can see why it might have appeared to be intermittent. More like random.

Was this a hidden "feature" of Tradier's implementation that was undocumented?

It was just an integration issue that wasn't easy to get a handle on. Again, if you set up a strategy to trade every minute (like we're prone to do when trying to duplicate a problem like this one) you'd never have seen it.

Good detective work! These integrations are not straightforward. Thanks for the backstory.

I'm sorry to say that this problem started happening again today. Placed and executed trades status in the Order Manager remains as "Placed". Trades are not reflected in the Accounts window unless and until "Request Updates from Broker(s)" is clicked.

WL restart does not fix problem.

If you're running into the "too many sessions" issue, then you don't have a fix for that yet.
Anyway, what does the Log Viewer show? (Filter for Tradier)

QUOTE:
If you're running into the "too many sessions" issue, then you don't have a fix for that yet.

That issue has not shown up today.

QUOTE:
Anyway, what does the Log Viewer show? (Filter for Tradier)

No errors in the Log Viewer. That is typical with this problem.

Okay, it's a different probelm then. Let's start over.
For the following we're talking only about orders that show "Placed" status.

1. Are the orders making it to Tradier?
2. What types of orders are they? Be specific.
3. Are you using symbol mapping? (Non-Tradier Data Provider mapped to a Tradier symbol.)

QUOTE:
1. Are the orders making it to Tradier?

Yes

QUOTE:
2. What types of orders are they? Be specific.

Buy / Sell Market orders for options

QUOTE:
3. Are you using symbol mapping? (Non-Tradier Data Provider mapped to a Tradier symbol.)

No

Addendum

QUOTE:
Okay, it's a different probelm then.

If you mean different than the "too many sessions" issue, then yes. I don't think they are related. But if you mean the topic issue of this post, then no. It's acting the same way. I have a feeling something changed on the Tradier side that made the post topic fix stop working. Nothing has changed on the WL side, including the API extension, since the last release, and it was working perfectly.

@Cone this morning it appears to be working again. Since you've worked with Tradier tech team directly on this problem previously, could you reach out to them to ask what they changed on their side in the last couple of days to break the fix and then unbreak it?

You're only assuming that something changed but probably didn't. Let's just hope it keeps working for you. We are not and haven't been seeing any issues. I have seen intermittent reconnects, though very infrequent - maybe twice a day (and usually overnight). I'd call that normal because internet dropouts can happen.

If you're correct, that would mean the fix made it happen less often but did not eliminate the issue completely.

Update
It appears to be working again today. I will post again if it fails.