This week I upped my intraday trading velocity to 50+ trades per day. I started getting a session disconnect at some point during the day and was unable to reconnect, even with WL restart. It turns out that the Tradier backend server disconnected the session and would not reconnect. Tech support had to "flush" my open sessions so I could connect again. Here's the thread I had with the Tradier support team and their assessment of what is going on.
I hope this makes sense to you.
QUOTE:
to Support
This morning, my trading program connected to Tradier API successfully traded in/out of 2 positions, and then the API disconnected and won't reconnect. I've tried restarting the app without effect. I've shut down all trading so there are no requests going to Tradier, yet this error message still generated when the app tries to reconnect. Here's the error message I'm getting from Tradier in the log.
QUOTE:
Tech. replied
Jul 29, 11:29am
I just checked the streams connected to your account and it looks like you are well under the threshold for this error/limit. Are you still having trouble connecting? I can clear your current streams out if you are, just let us know.
All the best,
Tradier Support
to Tradier
I just started having the connect problems I had yesterday. Can't connect - "too many sessions"
Tech. replied
Jul 30, 5:06pm
I cleared out your streams, you are good to reconnect.
to Tradier
Yes, I can connect now. Thanks.
What is causing this? How do I prevent it from happening?
Tech. replied
Jul 30, 5:55pm
This is caused by a few different things, all of which are on the client side unfortunately.
1. A user having flawed connection logic in their code for an application/software that they are developing, that drops connections and creates new ones without closing out old connections.
2. A user running multiple programs connected to their tradier account leads to too many simultaneous instances of streaming account and market data
3. A user having internet connectivity or other computer issues that result in lots of dropped connections that when resumed, the trading software automatically re-establishes connections to streaming, but again all the orphaned connections from the drops still exist.
The main concept across all these potential problems is that the client side is dropping connections and then establishing new ones while the old connections still exist and are sending data to a client that is no longer listening to that particular stream but rather focused on the newest stream. The best thing for your to do is make sure your code has good closing connection logic, especially when you need to modify or drop a stream, that you are working with one platform at a time, and that your connectivity is good.
All the best,
Tradier Support
to Tradier
That's a helpful insight. What is the limit on the number of unclosed streams before the back end refuses additional connections?
Tech. replied
Jul 31, 2:45pm
Our published limits are 1 stream of each type: 1 account stream to trade orders, and 1 market data stream tracking whatever tickers and types you want. That being said we give generous allowances above that to prevent issues like this. The number of streams is linked to your overall Tradier brokerage account (so if you have 3 accounts such as; a taxable, an IRA, and a trust you do not get additional stream allowance), and is not tied into trades you may make or other actions, it is just the streaming connections you have running, either active or in the case of interruptions or bad code, orphaned.
All the best,
Tradier Support
I hope this makes sense to you.
Rename
I'm sure we only use one instance of a socket connection, but we'll look at more explicitly disposing the previous connection before reconnecting. The problem is if your internet isn't working...
The Internet is working because charts are still streaming because WL keeps attempting to reconnect and receives denial messages from Tradier server. Tradier tech support apparently can see these orphaned connections attached to my account and flushes them to get it working again.
Addendum
Happening again. Charts still streaming.
ADDENDUM 2
Further correspondence with Tradier Tech Team. Sounds like the market data streams are the ones being orphaned.
Addendum
Happening again. Charts still streaming.
QUOTE:
Timestamp Source Message Exception
"8/1/2024 08:59:24:051" "Dummy Broker" "Dummy Broker Broker: Welcome innertrader, you are now connected to our Dummy Broker!"
"8/1/2024 08:59:48:489" "Tradier" "Connected"
"8/1/2024 08:59:31:249" "Yahoo! Finance" "Connected"
"8/1/2024 08:59:46:696" "Tradier" "08:59:46.696 Initiating Tradier accounts socket..."
"8/1/2024 08:59:48:064" "Tradier" "08:59:48.064 Tradier Connected on sessionID: ff1e804d-70a8-4256-9b12-900bfe9c969c"
"8/1/2024 08:59:48:065" "Tradier" "08:59:48.065 Tradier accounts Socket connected: sessionId ff1e804d-70a8-4256-9b12-900bfe9c969c"
"8/1/2024 08:59:48:499" "Tradier" "08:59:48.499 Tradier Connection Lost. Reconnecting..."
"8/1/2024 08:59:48:503" "Tradier" "08:59:48.503 Tradier: too many sessions requested. Connect aborted."
"8/1/2024 08:59:51:913" "Tradier" "08:59:51.913 Initiating Tradier accounts socket..."
"8/1/2024 08:59:52:150" "Tradier" "08:59:52.150 Tradier Connected on sessionID: c4e567bf-862f-4b4c-b88f-f322ac142614"
"8/1/2024 08:59:52:150" "Tradier" "08:59:52.150 Tradier accounts Socket connected: sessionId c4e567bf-862f-4b4c-b88f-f322ac142614"
"8/1/2024 08:59:53:603" "Tradier" "08:59:53.603 Tradier Connection Lost. Reconnecting..."
"8/1/2024 08:59:53:603" "Tradier" "08:59:53.603 Tradier: too many sessions requested. Connect aborted."
"8/1/2024 09:00:04:854" "Tradier" "09:00:04.854 Initiating Tradier accounts socket..."
"8/1/2024 09:00:08:524" "Tradier" "09:00:08.524 Tradier Connected on sessionID: 2de1e0e3-48c9-4c12-978c-33a4c41b007e"
"8/1/2024 09:00:08:524" "Tradier" "09:00:08.524 Tradier accounts Socket connected: sessionId 2de1e0e3-48c9-4c12-978c-33a4c41b007e"
"8/1/2024 09:00:08:953" "Tradier" "09:00:08.953 Tradier Connection Lost. Reconnecting..."
"8/1/2024 09:00:08:953" "Tradier" "09:00:08.953 Tradier: too many sessions requested. Connect aborted."
"8/1/2024 09:00:25:947" "Tradier" "09:00:25.947 Initiating Tradier accounts socket..."
"8/1/2024 09:00:27:218" "Tradier" "09:00:27.218 Tradier Connected on sessionID: 552931c4-eef7-479c-945e-3e19f95ed634"
"8/1/2024 09:00:27:218" "Tradier" "09:00:27.218 Tradier accounts Socket connected: sessionId 552931c4-eef7-479c-945e-3e19f95ed634"
"8/1/2024 09:00:27:643" "Tradier" "09:00:27.643 Tradier Connection Lost. Reconnecting..."
"8/1/2024 09:00:27:643" "Tradier" "09:00:27.643 Tradier: too many sessions requested. Connect aborted."
"8/1/2024 09:00:30:272" "Tradier" "09:00:30.272 Initiating Tradier accounts socket..."
"8/1/2024 09:00:30:538" "Tradier" "09:00:30.538 Tradier accounts Socket connected: sessionId 224a8ee6-bb6b-4e73-8d11-c1bc0f9d799e"
"8/1/2024 09:00:30:537" "Tradier" "09:00:30.537 Tradier Connected on sessionID: 224a8ee6-bb6b-4e73-8d11-c1bc0f9d799e"
"8/1/2024 09:00:30:966" "Tradier" "09:00:30.966 Tradier Connection Lost. Reconnecting..."
"8/1/2024 09:00:30:967" "Tradier" "09:00:30.967 Tradier: too many sessions requested. Connect aborted."
"8/1/2024 09:01:15:637" "Tradier" "09:01:15.637 Initiating Tradier accounts socket..."
"8/1/2024 09:01:16:943" "Tradier" "09:01:16.943 Tradier accounts Socket connected: sessionId 3d64f68d-6e54-4fc1-977b-2c566c6609a4"
"8/1/2024 09:01:16:943" "Tradier" "09:01:16.943 Tradier Connected on sessionID: 3d64f68d-6e54-4fc1-977b-2c566c6609a4"
"8/1/2024 09:01:17:368" "Tradier" "09:01:17.368 Tradier Connection Lost. Reconnecting..."
"8/1/2024 09:01:17:369" "Tradier" "09:01:17.369 Tradier: too many sessions requested. Connect aborted."
"8/1/2024 09:02:39:993" "Tradier" "09:02:39.993 Initiating Tradier accounts socket..."
"8/1/2024 09:02:41:384" "Tradier" "09:02:41.384 Tradier accounts Socket connected: sessionId bf4edd85-9c8a-4358-bf19-4d0e39ab7172"
"8/1/2024 09:02:41:384" "Tradier" "09:02:41.384 Tradier Connected on sessionID: bf4edd85-9c8a-4358-bf19-4d0e39ab7172"
"8/1/2024 09:02:41:810" "Tradier" "09:02:41.810 Tradier Connection Lost. Reconnecting..."
"8/1/2024 09:02:41:818" "Tradier" "09:02:41.818 Tradier: too many sessions requested. Connect aborted."
"8/1/2024 09:02:52:026" "Tradier" "09:02:52.026 Initiating Tradier accounts socket..."
"8/1/2024 09:02:53:327" "Tradier" "09:02:53.327 Tradier accounts Socket connected: sessionId 771d92d0-8f9b-4627-9ef2-c5257908ed0f"
"8/1/2024 09:02:53:326" "Tradier" "09:02:53.326 Tradier Connected on sessionID: 771d92d0-8f9b-4627-9ef2-c5257908ed0f"
"8/1/2024 09:02:53:748" "Tradier" "09:02:53.748 Tradier Connection Lost. Reconnecting..."
"8/1/2024 09:02:53:748" "Tradier" "09:02:53.748 Tradier: too many sessions requested. Connect aborted."
"8/1/2024 09:03:12:993" "Tradier" "09:03:12.993 Initiating Tradier accounts socket..."
"8/1/2024 09:03:14:283" "Tradier" "09:03:14.283 Tradier accounts Socket connected: sessionId 3986ea0c-9c3d-4d25-b3ac-1b192db1dbf2"
"8/1/2024 09:03:14:283" "Tradier" "09:03:14.283 Tradier Connected on sessionID: 3986ea0c-9c3d-4d25-b3ac-1b192db1dbf2"
"8/1/2024 09:03:14:706" "Tradier" "09:03:14.706 Tradier Connection Lost. Reconnecting..."
"8/1/2024 09:03:14:706" "Tradier" "09:03:14.706 Tradier: too many sessions requested. Connect aborted."
ADDENDUM 2
Further correspondence with Tradier Tech Team. Sounds like the market data streams are the ones being orphaned.
QUOTE:
to Tradier
Unable to connect API again. Please flush connections from my account.
Please also provide any additional information you may have about the nature of the error, particularly about closing connections. Can you tell me how many connections you see?
Tech. replied
Aug 1, 1:17pm
I have closed out your connections. You had one account stream and 9 market data streams running.
Tradier Support
Further correspondence with Tradier tech with potential valuable technical information.
QUOTE:
10:52 AM (16 minutes ago)
Thank you for quick service. Is there an API call I can make from the browser or a program to close connections myself while Wealth-Lab engineering is investigating?
Tradier Tech. Support
11:05 AM (4 minutes ago)
to me
Tech. replied
Aug 2, 2:05pm
Unfortunately no, there is not an endpoint for closing out your own streams, you'll need to let us know if there is an issue. From our investigation it looks like the platform is not properly closing connections before opening new ones, you can communicate to them they need to use a form of WebSocket.close() in whatever WebSocket package they are using before they open a new stream with the same or an updated payload. https://developer.mozilla.org/en-US/docs/Web/API/WebSocket/close
Tradier Support
techsupport@tradier.com
The fundamental problem is when your failing network hardware drops the Internet connection, and WL cannot connect to that original port# that dropped again, then what is it suppose to do?
You're saying it should just do nothing because if it establishes a whole new connection on a different port#, the original port is left hanging. Perhaps you are right, and WL should just halt all internet traffic on the failing port for 2 minutes (or so) trying to reconnect to it before opening a new connection on another port and leaving the old port hanging.
What Tradier should do is automatically close the connection of the original port# after 2 minutes if WL is unable to reconnect to it because your network hardware hasn't recovered yet. And you should tell Tradier that.
---
We appreciate you network has serious problems. But if it goes silent for over a minute at a time, there's not much the software can do. There's no way the software knows if the old port# is still alive but your hardware hasn't recovered yet, or whether that old port# has been closed (timed out on the Tradier server side) and WL needs to reconnect with a new port# instead. The software is in a pickle as what the situation is.
What you need to do is get your network hardware fixed, and until you do, this problem won't go away. And I think you know that. And to physically locate the source of this network hardware problem requires a digital network analyzer. And if the tech troubleshooting it doesn't have one of those, then he's wasting his time--and yours too--because the hardware location is in an uncommon place; otherwise, you would have found it by now.
I would also try to reproduce the problem on a different computer just to be sure the computer hardware itself isn't the source of your network hardware problem. Or you could take your old computer to an entirely different location to see if the problem goes away.
You're saying it should just do nothing because if it establishes a whole new connection on a different port#, the original port is left hanging. Perhaps you are right, and WL should just halt all internet traffic on the failing port for 2 minutes (or so) trying to reconnect to it before opening a new connection on another port and leaving the old port hanging.
What Tradier should do is automatically close the connection of the original port# after 2 minutes if WL is unable to reconnect to it because your network hardware hasn't recovered yet. And you should tell Tradier that.
---
We appreciate you network has serious problems. But if it goes silent for over a minute at a time, there's not much the software can do. There's no way the software knows if the old port# is still alive but your hardware hasn't recovered yet, or whether that old port# has been closed (timed out on the Tradier server side) and WL needs to reconnect with a new port# instead. The software is in a pickle as what the situation is.
What you need to do is get your network hardware fixed, and until you do, this problem won't go away. And I think you know that. And to physically locate the source of this network hardware problem requires a digital network analyzer. And if the tech troubleshooting it doesn't have one of those, then he's wasting his time--and yours too--because the hardware location is in an uncommon place; otherwise, you would have found it by now.
I would also try to reproduce the problem on a different computer just to be sure the computer hardware itself isn't the source of your network hardware problem. Or you could take your old computer to an entirely different location to see if the problem goes away.
After "Connect aborted" the broker logic stops connecting, what must be happening is that WealthLab hits "ConnectBroker()" again - and that's how it gets into that loop.
We'll make a couple changes -
1. We won't try to connect more than once in 15 seconds.
2. If it hits "Connect aborted" due to "too many sessions requested" we'll set a flag to not attempt a reconnect for at least 5 minutes. You can try sooner by restarting.
3. The streaming socket wasn't disposing the connection before reconnecting. The accounts socket that we're talking about here was doing that, so it's hard to say how you got into this situation unless there really was a internet connection issue.
We'll make a couple changes -
1. We won't try to connect more than once in 15 seconds.
2. If it hits "Connect aborted" due to "too many sessions requested" we'll set a flag to not attempt a reconnect for at least 5 minutes. You can try sooner by restarting.
3. The streaming socket wasn't disposing the connection before reconnecting. The accounts socket that we're talking about here was doing that, so it's hard to say how you got into this situation unless there really was a internet connection issue.
QUOTE:
2. If it hits "Connect aborted" due to "too many sessions requested" we'll set a flag to not attempt a reconnect for at least 5 minutes.
Well, that's a good idea. The problem is the software can't tell when the client network hardware comes back to life again. But if you're getting "too many sessions requested", then you know the network hardware has been down for awhile and the software needs to back off on the retry frequency.
The physical layer of the Ethernet protocol, HDLC (High Level Data Link Control), employs a process called exponential back-off (base 2 logarithm) whenever there's a collision with another packet. You might try the same thing. Each time there's a connection failure on a maybe-still-active old port#, wait 5 seconds. If it fails again, then double that waiting time before retrying, and repeat.
IBM developed HDLC back in the late 70s, so I'm sure the exponential back-off paradigm has been well simulated and tested. I would just copy that approach.
---
Having such a long network hardware failure seems really unusual to me. I'm beginning to think it's the Ethernet hardware on his computer itself that's the source of the problem. He should try reproducing the problem on a different computer.
@Cone I shared your ideas to address this issue with Tradier Tech and this is the current thread:
QUOTE:
Tech. replied
Aug 5, 11:54am
That is helpful and hopefully on the last point "The streaming socket wasn't disposing the connection before reconnecting" this is something they are working to correct, ie in the way that they have it with the accounts socket.
All the best,
Tradier Support
I replied
Aug 5, 12:13pm
You wrote, " this is something they are working to correct, "
Please clarify: Who is "they"? Tradier engineering?
Tech. replied
Aug 5, 12:21pm
Sorry about that I thought it was Wealth-Lab saying they were working to do a better job of sending close signals.
All the best,
Tradier Support
Your Response
Post
Edit Post
Login is required