Tradier-broker-not-auto-connecting

"Tradier Broker socket disconnected ByServer".
This is an external disconnect from the socket. I can't tell you why, only that this is what the socket reported. You have to restart WealthLab like it says to reinitialize it.

My guess is that the Tasks are creating another instance of WeatlhLab when one is already running and that's causing the disconnect.

Ah. Interesting thought. I will probably have to get on early and actually watch what happens.

I Did change a setting in the Task Scheduler to not open new session if one is not open. Didn't make any difference. I've tried restarting after market open. Same problem. I've watched Task scheduler Live and it's working properly. It just seems to reject the first connect of the day and works on the second one. It reminds me of a previous Tradier problem (now fixed) that required two fetches for it to work (download option chain?).

QUOTE:
This is a recent problem, possibly since I upgraded to B34 of the Tradier extension. It will connect once I restart WL again.

I am in discussion with Tradier support, but they have no clue what is causing this. This is a new problem I didn't have before, and it consistently fails each morning. Is it possible for me to undo the Tradier B34 update to test if the problem disappears? Or would I have to revert to B99 of WL as well?

QUOTE:
I Did change a setting in the Task Scheduler to not open new session if one is not open.

This only refers to other sessions that the Task Manager has opened. If WealthLab is already open and connected, then your Task is not going to work. I can't tell if that's your scenario or not.

No. First task is to shut down WL. 2 min later second task to start WL runs. Both working properly.

Okay, I'll see if I can duplicate that scenario.
Are you using TASKKILL or something else to shut down WealthLab?
Give me the details.

I've gone through a few cyclesand it works fine for me. The named Workspace I'm loading just has an Order Manager with Tradier selected and a streaming chart.

The last time it went through a "Lost" disconnect about 1 minute after connecting, but that lasted only about 1 second before it reported that it reconnected. Stable ever since.

yes.

I suspect it may have nothing to do with the Task Scheduler. To test this, I will manually shut down and restart WL tomorrow to see if the problem happens (i.e., take TS out of the picture).

Why do you think you are getting so many disconnects?

You're calling one transient 0.5 second Disconnect "so many"?

It was one transient 0.5 second disconnect. I'm not going to try investigate the cause of that.

... although it did happen almost exactly 60 seconds after the Connect, so there could be something there - I will check that because it could be part of the default auto-reconnect logic working that should have been disabled.

Sorry. It looked like more. I see now only one. I was just surprised because I thought I was the only one getting disconnects and I still would like to know if it's my Internet connection, something happening at the broker side, or something between WL and broker.

Regarding Post #8

QUOTE:
I suspect it may have nothing to do with the Task Scheduler. To test this, I will manually shut down and restart WL tomorrow to see if the problem happens (i.e., take TS out of the picture).

This morning, I had Task Scheduler (TS) shut down WL before 9am (Market time) and I manually started WL about 2 min later. It did not connect automatically with Tradier. No log error message When I manually connected (green connect button), It failed. Log message below:

QUOTE:
Timestamp Source Message Exception
"10/9/2024 06:07:58:769" "Tradier" "06:07:58.769 Initiating Tradier accounts socket...Start"
"10/9/2024 06:08:00:164" "Tradier" "06:08:00.164 Tradier Broker socket disconnected ByServer. Restart WealthLab to Reinitialize."

So, the error has nothing to do with TS. This is what happened when I restarted WL.

QUOTE:
Timestamp Source Message Exception
"10/9/2024 06:18:14:870" "Tradier" "06:18:14.870 Initiating Tradier accounts socket...Start"
"10/9/2024 06:18:16:268" "Tradier" "06:18:16.268 Tradier Broker socket disconnected ByServer. Restart WealthLab to Reinitialize."
"10/9/2024 06:18:30:348" "Tradier" "06:18:30.348 Initiating Tradier accounts socket...Start"
"10/9/2024 06:18:32:583" "Tradier" "06:18:32.583 Tradier Broker socket disconnected ByServer. Restart WealthLab to Reinitialize."
"10/9/2024 06:18:38:378" "Tradier" "06:18:38.378 Initiating Tradier accounts socket...Start"
"10/9/2024 06:18:39:770" "Tradier" "06:18:39.770 Tradier Broker socket disconnected ByServer. Restart WealthLab to Reinitialize."
"10/9/2024 06:18:46:148" "Tradier" "06:18:46.148 Initiating Tradier accounts socket...Start"
"10/9/2024 06:18:47:533" "Tradier" "06:18:47.533 Tradier Broker socket disconnected ByServer. Restart WealthLab to Reinitialize."
"10/9/2024 06:18:56:352" "Tradier" "06:18:56.352 Initiating Tradier accounts socket...Start"
"10/9/2024 06:18:57:779" "Tradier" "06:18:57.779 Tradier Broker socket disconnected ByServer. Restart WealthLab to Reinitialize."

What do you think of my idea in Post #3?

QUOTE:
This is a new problem I didn't have before, and it consistently fails each morning. Is it possible for me to undo the Tradier B34 update to test if the problem disappears? Or would I have to revert to B99 of WL as well?

Finally connected at 9:26 am

Let me ask again. What process are you using in the TS to shut down WealthLab, and, are you sure that it shuts down.
I'm talking about a Windows Process - not just if there's a WealthLab window showing or not.

Re: Post #3 - We don't keep old builds. You can try if you downloaded it though.

QUOTE:
Let me ask again.

I responded in Post #8 but now I see it was separated by another post (likely added while I was responding) and was probably not clear what I was answering "Yes" to.

QUOTE:
What process are you using in the TS to shut down WealthLab

QUOTE:
are you sure that it shuts down. I'm talking about a Windows Process - not just if there's a WealthLab window showing or not.

Yes, I'm sure it's shut down. Here are snaps of Task Manager after I ran the TS shut down. Wealth-Lab does not appear.

Apps


Background processes

QUOTE:
We don't keep old builds. You can try if you downloaded it though.

Would I have to install *both* WL build (99) and Tradier (33) together?

No, without changing WL you can just go back to Tradier Build 33, and put up with the "Too many sessions" problem if you want, but please don't post about that.

The thing is, the "Server" is disconnecting you. I don't see that - ever (at least not that I can recall). This indicates a problem with your login, your session token, or something like that - like multiple logins.

QUOTE:
please don't post about that.

Hahaha. I promise I won't.

I am not planning to use B33 long-term. This idea is just to see if the connect problem appears with B33, as a quick troubleshooting step. Maybe it's just a coincidence. I'll be able to test tomorrow morning, assuming the downgrade works, and let you know what I find.

Unfortunately, I could not find the old install files in my download folder in my image backups. Either WL changed the install to fetch from the cloud instead of dl'ing the file or I don't have the right name for the install file.

At any rate, the symptom is consistent. My first WL startup of the day will not connect (auto or manual) with Tradier broker and will generate log error shown above. My second and every subsequent WL startup of the day will auto-connect with Tradier without error.

If I could make it happen, we would dig into it.

If you tell me what's in your start-up workspace and how it's configured, I'd have something to try... And by "configured" I mean precisely how it's configured...

1. is streaming enabled on the chart windows?
2. are there active items in the S.Monitor?
3. if so, how is each configured for data and update mode
4. is Tradier the selected broker in the Order Manager?
5. what order of checked Hist. Providers,
6. Is Auto-Run and/or Auto-Reactivate enabled in the Workspace menu.

Or, a better idea is to simplify a Workspace that only has only:
1. the Order Manager with Tradier selected.
2. Streaming chart with Tradier

Try starting up to just that "mostly empty" workspace.
If that works, start building up your trading workspace by adding items one by one to try to find the "breaking point".

Bad, but not atypical, day with Tradier. Here's the abbreviated log:

QUOTE:
"10/10/2024 09:19:41:058" "Tradier" "Streaming Disconnected - attempting Reconnect: Lost, CloseStatus: , SessionId: e95b45c6-96ed-41db-803e-6147d747ae4e"
"10/10/2024 09:19:42:423" "Tradier" "Tradier Streaming Reconnected after Lost on sessionID: e95b45c6-96ed-41db-803e-6147d747ae4e"
"10/10/2024 09:19:42:649" "Tradier" "Streaming Reconnect Successful"
"10/10/2024 09:19:42:885" "Tradier" "Tradier Streaming Reconnected after Lost on sessionID: e95b45c6-96ed-41db-803e-6147d747ae4e"
"10/10/2024 09:34:08:789" "Tradier" "Streaming Disconnected - attempting Reconnect: Lost, CloseStatus: , SessionId: e76e6b1f-7c35-446a-835e-c1a881e93e69"
"10/10/2024 09:34:08:789" "Tradier" "09:34:08.789 Tradier Broker socket disconnected: Lost; Reconnecting..."
"10/10/2024 09:34:10:152" "Tradier" "Tradier Streaming Reconnected after Lost on sessionID: e76e6b1f-7c35-446a-835e-c1a881e93e69"
"10/10/2024 09:34:10:371" "Tradier" "Streaming Reconnect Successful"
"10/10/2024 09:34:10:612" "Tradier" "Tradier Streaming Reconnected after Lost on sessionID: e76e6b1f-7c35-446a-835e-c1a881e93e69"
"10/10/2024 09:34:12:661" "Tradier" "09:34:12.661 Tradier Broker reconnected after Lost"
"10/10/2024 09:34:14:241" "Tradier" "09:34:14.241 Tradier Broker socket disconnected ByServer. Restart WealthLab to Reinitialize."
"10/10/2024 10:30:05:985" "Tradier" "user/profile Get empty"
"10/10/2024 10:30:05:986" "Tradier" "10:30:05.986 Initiating Tradier accounts socket...Start"
"10/10/2024 10:30:07:987" "Tradier" "10:30:07.987 Tradier Broker socket disconnected ByServer. Restart WealthLab to Reinitialize."
"10/10/2024 10:40:08:214" "Tradier" "user/profile Get empty"

Had to restart WL to get it going again. The irony is that I do not have Tradier streaming selected for data. It's not selected in the streaming chart, and it's not selected in the SM settings for Streaming: (Broker and Streaming Provider settings). Besides, I am using polling data. So why is it streaming and what kind of streaming is failing? Is the broker connection streaming?

Bad? If you didn't have that log, you would have never noticed the disconnects - except for the last one, I guess. Initially, they were transient disconnects and it reconnected in less than 2 seconds.

I don't know why you keep getting disconnected "By Server". That's abnormal and seems to be only happening for you. If anyone else sees that, I'd like to hear about it.

Because I'm always programming, investigating, etc., I don't often get 12 consecutive hours on a machine without restarting WealthLab. But today I did, and the result was one solid Tradier connection for more than 12 hours straight.


At other times, I have seen transient disconnects that correspond to internet dropouts, but Tradier reconnects quickly when it becomes available again.

And since fixing the "Too many sessions" bug, I can't recall being disconnected "By Server" - this seems to be unique to your scenario/usage.

QUOTE:
this seems to be unique to your scenario/usage.

@innertrader
I've been following your networking problems. Some thoughts from a computer engineer.

1) A passive component failure (cable connections, reflections, wiring) would last fractions of seconds due to the ringing in the transmission lines. This doesn't seem like the problem you are having.

2) An active component failure (like a digital computer chip) would last a second or two. This sounds more like your problem. Moreover, as the failing chip overheats and fries, it will eventually fail altogether, so it needs to be addressed. This means the problem is either in your computer or router.

I think it's unfortunate your local ISP can't send a tech with a digital network analyzer to precisely locate the networking problem source. That being said, I would hire an engineering consultant with a network analyzer to do this instead. It's worth the couple hundred bucks to get this done.

I'm "guessing" the problem is in the networking hardware of your computer itself because those chips get much hotter than those in your router. You don't need to replace the entire motherboard. For about $70 you can buy an Ethernet adapter PCIe card that plugs in your motherboard to bypass the problem. All the best to you.

You think hardware is a reason that the "Server" is disconnecting him?
I don't think so.

QUOTE:
You think hardware is a reason that the "Server" is disconnecting him?

The server many be disconnecting him because to too many retries. Actually, there's a firewall type of device that's in charge of fending off Denial-of-Service attacks that may be kicking in with all the reconnecting retries from the same IP address. (If it was the server itself, it might provide more useful information.)

If it's an active component, it's only going to get worse as that component fries.

I did suggest he swap computers to see if the problem disappears, but that was four months ago. He needs to test the second computer with "streaming" because that's what's going to exacerbate the problem the most.

3 re-connections (all successful, by the way) in 6 seconds can hardly be mistaken for a DoS attack. Furthermore these were not the actual problem, which was being disconnected by the Server nearly 1 hour later. That's the one that required the restart.

Like I mentioned before, if the Server is rejecting the connection, there's probably something wrong with the login, token refresh, multiple connections to the same account, etc.

(By the way, it's a full 24 hours later now, and I still haven't had even one re-connect during that time.)

QUOTE:
if the Server is rejecting the connection, there's probably something wrong with the login,...

If the security box has his IP address on a DoS watch list, the behavior of his login is going to be different than yours. I think he's seeing some unique behavior.

After testing/forcing disconnects, I got the "ByServer" scenario. It's complaining about the Session ID.
I don't know why, but at least we've got something to go on again.

QUOTE:
Timestamp Source Message Exception
"10/16/2024 08:06:15:779" "Schwab" "Schwab Broker: Connected to Schwab."
"10/16/2024 08:06:30:897" "Tradier" "08:06:30.897 Initiating Tradier accounts socket...Start"
"10/16/2024 08:06:34:365" "Schwab" "Schwab Initial Streaming Connect"
"10/16/2024 08:06:34:564" "Schwab" "Connected"
"10/16/2024 08:20:04:586" "Tradier" "Streaming Disconnected - attempting Reconnect: Lost, CloseStatus: , SessionId: c7304e8d-f85e-4c3c-8296-f5e0a876aabd"
"10/16/2024 08:20:05:857" "Tradier" "Tradier Streaming Reconnected after Lost on sessionID: c7304e8d-f85e-4c3c-8296-f5e0a876aabd"
"10/16/2024 08:20:06:162" "Tradier" "Streaming Reconnect Successful"
"10/16/2024 08:20:06:398" "Tradier" "Tradier Streaming Reconnected after Lost on sessionID: c7304e8d-f85e-4c3c-8296-f5e0a876aabd"
"10/16/2024 09:17:44:495" "Tradier" "Streaming Disconnected - attempting Reconnect: Lost, CloseStatus: , SessionId: 5bcb41f8-3708-40ae-9bee-c6f6c9707fda"
"10/16/2024 09:17:44:495" "Tradier" "09:17:44.495 Tradier Broker socket disconnected: Lost; Reconnecting..."
"10/16/2024 09:17:45:874" "Tradier" "Tradier Streaming Reconnected after Lost on sessionID: 5bcb41f8-3708-40ae-9bee-c6f6c9707fda"
"10/16/2024 09:17:46:077" "Tradier" "Streaming Reconnect Successful"
"10/16/2024 09:17:46:316" "Tradier" "Tradier Streaming Reconnected after Lost on sessionID: 5bcb41f8-3708-40ae-9bee-c6f6c9707fda"
"10/16/2024 09:17:48:231" "Tradier" "09:17:48.231 Tradier Broker reconnected after Lost"
"10/16/2024 09:17:49:734" "Tradier" "09:17:49.734 Tradier Broker socket disconnected ByServer. Restart WealthLab to Reinitialize."
"10/16/2024 09:17:55:341" "Tradier" "Streaming Reconnect Successful"
"10/16/2024 09:17:55:578" "Tradier" "Tradier Streaming Reconnected after Lost on sessionID: 5bcb41f8-3708-40ae-9bee-c6f6c9707fda"

Now Tradier is disconnecting from broker during the day. Likely at 9:17:55 or some time after.

Here's what happened when I tried to reconnect ("Connect" button in OM).

QUOTE:
"10/16/2024 10:24:35:137" "Tradier" "10:24:35.137 Initiating Tradier accounts socket...Start"
"10/16/2024 10:24:36:610" "Tradier" "10:24:36.610 Tradier Broker socket disconnected ByServer. Restart WealthLab to Reinitialize."

I have to restart WL to reconnect, exactly as what happens on the disconnect on the first attempt each morning after the WL restart.

Note that I have zero disconnects from Schwab.

Post #16

QUOTE:
Or, a better idea is to simplify a Workspace that only has only:
1. the Order Manager with Tradier selected.
2. Streaming chart with Tradier

Try starting up to just that "mostly empty" workspace.
If that works, start building up your trading workspace by adding items one by one to try to find the "breaking point".

I started up WL in the morning with Tradier selected in the Order Manager, all Data Providers unchecked except Tradier, only one streaming chart active using Tradier, and all Strategies in the SM disabled. It still failed.

Thanks but updates aren't required. I've been working on the solution most of the week.
The main problem was the interpretation of Tradier's SessionID that "expires" after 5 minutes. We thought the session had to be refreshed constantly before 5 minutes expired otherwise we'd lose the session. That's not the case at all - the SessionID only needs to be updated if the payload changes. That's fixed, and I'm just cleaning up the Auto-reconnect for Streaming.

Yay!

Looking forward to the next release. BTW, Tradier has been having problems of its own. This morning, they actually shut down all trading for the remainder of the day because of a glitch they were having with a vendor. Just in case you are seeing some new weird stuff in your testing, it may be on their side.

QUOTE:
the SessionID only needs to be updated if the payload changes.

I'm just curious, but I'm assuming the http headers (and any http handshaking) aren't counted as part of the payload in a WebSocket connection. Is that correct?

If so, then if WL is simply "listening" to a stream, then the payload would never be changing. Right? Of course if WL makes any request to the server (like a polling request), that would be a payload change.

I'm wondering if the WL client drops out (e.g. bad network connection), if the server would continue to stream? I'm assuming if the http handshaking stops, then the server would drop the connection (and sessionID). That still raises the question whether the original sessionID is still active when the network connection is re-established? (I'm glad you're working on this, and not me.)

Tradier Build 36 is ready. This one should do the trick.

Woohoo!

Although I'm confident, I'm still crossing my fingers until you finish your smoke test.

fyi, after frequently disconnecting/reconnecting during some other dev today, I ran into "too many sessions". The scenario involves closing and restarting WL frequently in a short period of time. I don't see a way to avoid that outcome at the moment.