Obtaining-data-from-IB-TWS-for-non-US-exchanges

First, see Help (F1) > Interactive Brokers topic.

There are sections for each type of instrument, but skip to the bottom and see IBContracts.txt Record Format. Unless the symbol conforms to an option or futures format, you need to define specific contracts that aren't U.S. stocks in IBContracts.txt.

I don't recommend using the "Explicit Contract Specification" method except for a quick attempt to get data without editing IBContracts.txt.

If you have a large number of symbols in non-U.S. markets, you can auto-generate the records with a script. To help with conflicts, it would be best to use an symbol extension to uniquely identify symbols in non-US markets.

For example, ABC is a US and also an AU stock. You could uniquely identify ABC on ASX like this:

CODE:
ABC=ABC|USD|STK|SMART|NYSE  (this record exists by default)
ABC.AX=ABC|AUD|STK|SMART|ASX  (add records for each AU contract like this one)

In WealthLab, when you use ABC.AX, IB will pull the ASX contract instead of the US one.

Do you use Norgate Data?

Here's the code to Auto-generate all those ".AX" records. We're using the Yahoo! DataSet symbols and the Yahoo! extension for our non-US WealthLab symbol as a convenience.

1. Modify the constant strings at the top as required for the market
2. Run it on any 1 symbol in WealthLab
3. Copy the contract records from the Debug log to Clipboard
4. CLOSE WealthLab
5. Paste the clipboard to the bottom of IBContracts.txt and Save.

CODE:
using WealthLab.Backtest;
using System;
using WealthLab.Core;
using System.IO;
using System.Collections.Generic;

namespace WealthScript2 
{
   public class MyStrategy : UserStrategyBase
   {
      //Modify currency, primary market, and Yahoo! DataSet and run on ANY symbol
      const string _currency = "AUD";
      const string _market = "ASX";
      const string _yahooDataSet = "Australia & NZ - S&P ASX 200";      //"Australia & NZ - S&P ASX All Ordinaries"
      
      const string _formatString = "{0}={1}|{2}|STK|SMART|{3}";
      static bool _runOnce = false; 
      static List<string> _contractSymbols = new List<string>();
      static List<string> _conflicts = new List<string>();
      
      public override void Initialize(BarHistory bars)
      {
         if (_runOnce)
            return;
         _runOnce = true; 
         DataSet ds = DataSetFactory.Instance.Find(_yahooDataSet);
         
         foreach (string symbol in ds.Symbols)
         {
            if (_contractSymbols.Contains(symbol))
            {
               if (symbol.Length > 0)
               {
                  _conflicts.Add(symbol);
               }
            }
            else
            {
               string[] sym = symbol.Split('.');
               if (sym.Length > 1)
               {
                  WriteToDebugLog(String.Format(_formatString, symbol, sym[0], _currency, _market), false);
               }
            }
         }
      }

      public override void BacktestBegin()
      {
         WriteToDebugLog("1. Close WealthLab\n2.then add these records to IBContracts.txt:\n", false);

         string[] lines = File.ReadAllLines(Path.Combine(WLHost.Instance.DataFolder, "IBContracts.txt"));
         foreach (string line in lines)
         {
            string[] token = line.Split('=');
            _contractSymbols.Add(token[0]);
         }
      }

      public override void BacktestComplete()
      {
         if (_conflicts.Count < 1)
            WriteToDebugLog("\nNo conflicts");
         else
         {
            WriteToDebugLog("\n_conflicts.Count = " + _conflicts.Count);
            WriteToDebugLog("Conflicts for the following should be resolved before adding these records:");
            for (int n = 0; n < _conflicts.Count; n++)
            {
               WriteToDebugLog(String.Format(_formatString, _conflicts[n]));
            }
         }
      }

      public override void Execute(BarHistory bars, int idx)
      {
      }
   }
}

Many thanks and that makes sense. I do use Norgate so using .au extension would work well.

I will try the code and see how I go. Where is the location of the txt file?!

Not yet, but after posting that I thought of the same thing.
We'll have to build that into the next IB Provider.

That would be awesome...New to Wealth-Lab....can you please point me as to how to run the code in WL to obtain code list?

Found the file in File menu > Open WealthLab User Data Folder but no txt extension....has a bunch of individual US codes.

I figured it out....this is only a data formatter for the data set that you choose. Perfect..I will use it on Norgate.

Right. For the IB Provider update, I'll add both .AX and .AU extensions to point to "AUD" on the ASX, then you'll be able to trade it on IB without any symbol mapping.

You are a legend. Thanks

Would you know why I am still struggling to get data from the API?

16/06/2024 21:33:44:302 InteractiveBrokers
200: No security definition has been found for BHP.au

Even with IBContracts having the following record:
BHP.au=BHP|AUD|STK|SMART|ASX

and on IB Gateway, I am connected to "afarm" and "usfarm"...

with TWS API...same issue:

16/06/2024 21:59:24:805
Norgate Data
Current Datasets: Init (8.0.12)

--------
16/06/2024 21:59:25:176
Norgate Data
Current Datasets: Starting population

--------
16/06/2024 21:59:25:632
Norgate Data
Current Datasets: Population completed with 14 Current datasets

--------
16/06/2024 22:01:04:417
Interactive Brokers
Connected

--------
16/06/2024 22:01:04:624
InteractiveBrokers
Market data farm connections OK: cafarm, hfarm, eufarmnj, cashfarm, usfuture, afarm, jfarm, usfarm.nj, usopt, usfarm, hkhmds, fundfarm, ushmds, secdefnj

--------
16/06/2024 22:01:08:883
InteractiveBrokers
0.9 seconds difference: Local - TWS time

--------
16/06/2024 22:01:28:719
Norgate Data
Data Provider: Init (8.0.12)

--------
16/06/2024 22:01:28:719
Norgate Data
Data Provider: Connected to Norgate Data API

--------
16/06/2024 22:04:52:766
InteractiveBrokers
200: No security definition has been found for EVN.au

--------
16/06/2024 22:04:52:999
InteractiveBrokers
200: No security definition has been found for ORA.au

--------
16/06/2024 22:04:53:226
InteractiveBrokers
200: No security definition has been found for 360.au


... etc...etc.

It looks like it doesn't like the 'SMART' there.
I just finished the upgrade and these are the contract records that were generated by IB -

CODE:
BHP.au=BHP|AUD|STK|ASX|ASX||||0|
BHP.AX=BHP|AUD|STK|ASX|ASX|BHP|||0|

Let's see if we can sweet talk Glitch into pushing out IB Build 53 with this nice upgrade. With Build 53, you won't have to edit IBContracts except possibly for special situations.

That is an awesome news. May I suggest you also add .ASX as a suffix as I have seen some variants on that as well. Interesting that there are two formats for .AU and .AX so I have modified the code accordingly (Below for others to use). Do you think that Norgate's .au in lower case makes any difference in symbols matching?

Do you have ETA for next version release? Many thanks for all your help.

CODE:
using WealthLab.Backtest;
using System;
using WealthLab.Core;
using System.IO;
using System.Collections.Generic;

namespace WealthScript1 
{
   public class MyStrategy : UserStrategyBase
   {
      //Modify currency, primary market, and Yahoo! DataSet and run on ANY symbol
      const string _currency = "AUD";
      const string _market = "ASX";
      const string _yahooDataSet = "S&P ASX 300";      //"Australia & NZ - S&P ASX All Ordinaries"

      //const string _formatString = "{0}={1}|{2}|STK|SMART|{3}";
      
      const string _formatString_AX = "{0}={1}|{2}|STK|ASX|{3}|||0|";
      const string _formatString_AU = "{0}={1}|{2}|STK|ASX|||||0|";
      
      static bool _runOnce = false;
      static List<string> _contractSymbols = new List<string>();
      static List<string> _conflicts = new List<string>();
      
      public override void Initialize(BarHistory bars)
      {
         if (_runOnce)
            return;
         _runOnce = true;
         DataSet ds = DataSetFactory.Instance.Find(_yahooDataSet);

         foreach (string symbol in ds.Symbols)
         {
            if (_contractSymbols.Contains(symbol.ToUpper()))
            {
               if (symbol.Length > 0)
               {
                  _conflicts.Add(symbol.ToUpper());
               }
            }
            else
            {
               string[] sym = symbol.ToUpper().Split('.');
               if (sym.Length > 1)
               {
                  string ticker = sym[0].ToUpper();
                  string suffix = sym[1].ToUpper();

                  if (ticker.Length > 1 && suffix.Length > 1)  // Added validation for both ticker and suffix
                  {
                     if (suffix.Equals("AU", StringComparison.OrdinalIgnoreCase))
                     {
                        WriteToDebugLog(String.Format(_formatString_AU, symbol.ToUpper(), ticker, _currency), false);
                     }
                     else if (suffix.Equals("AX", StringComparison.OrdinalIgnoreCase))
                     {
                        WriteToDebugLog(String.Format(_formatString_AX, symbol.ToUpper(), ticker, _currency, _market), false);
                     }
                  }
                  else
                  {
                     // Handle cases where validation fails, e.g., log an error or take other action
                     WriteToDebugLog(String.Format("Invalid ticker or suffix length for {0} {1}", ticker, suffix), false);
                  }
               }
               else
               {
                  // Handle cases where symbol does not contain a period or has an insufficient length
                  WriteToDebugLog(String.Format("Invalid ticker or suffix length for {0}", sym), false);
               }
                        
                              
               //string[] sym = symbol.Split('.');
               //if (sym.Length > 1)
               //{
               //   WriteToDebugLog(String.Format(_formatString, symbol, sym[0], _currency, _market), false);
               //}
            }
         }
      }

      public override void BacktestBegin()
      {
         WriteToDebugLog("1. Close WealthLab\n2.then add these records to IBContracts.txt:\n", false);

         string[] lines = File.ReadAllLines(Path.Combine(WLHost.Instance.DataFolder, "IBContracts.txt"));
         foreach (string line in lines)
         {
            string[] token = line.Split('=');
            _contractSymbols.Add(token[0]);
         }
      }

      public override void BacktestComplete()
      {
         if (_conflicts.Count < 1)
            WriteToDebugLog("\nNo conflicts");
         else
         {
            WriteToDebugLog("\n_conflicts.Count = " + _conflicts.Count);
            WriteToDebugLog("Conflicts for the following should be resolved before adding these records:");
            foreach (string conflict in _conflicts)
            {
               string[] sym = conflict.ToUpper().Split('.');
               if (sym.Length > 1)
               {
                  string ticker = sym[0].ToUpper();
                  string suffix = sym[1].ToUpper();

                  if (suffix.Equals("AU", StringComparison.OrdinalIgnoreCase))
                  {
                     WriteToDebugLog(String.Format(_formatString_AU, conflict.ToUpper(), ticker, _currency), false);
                  }
                  else if (suffix.Equals("AX", StringComparison.OrdinalIgnoreCase))
                  {
                     WriteToDebugLog(String.Format(_formatString_AX, conflict.ToUpper(), ticker, _currency, _market), false);
                  }
               }
               else
               {
                  WriteToDebugLog(String.Format("Invalid symbol format for {0}", conflict.ToUpper()), false);
               }
            }
            //WriteToDebugLog("\n_conflicts.Count = " + _conflicts.Count);
            //WriteToDebugLog("Conflicts for the following should be resolved before adding these records:");
            //for (int n = 0; n < _conflicts.Count; n++)
            //{
            //   WriteToDebugLog(String.Format(_formatString_AX, _conflicts[n]));
            //}
         }
      }

      public override void Execute(BarHistory bars, int idx)
      {
      }
   }
}

Just tested both formats and still not working...from WL

17/06/2024 13:39:46:008
InteractiveBrokers
200: No security definition has been found for BHP.AU

From Contract File
BHP.AU=BHP|AUD|STK|ASX|ASX|||0|

I have also tried...with no luck
BHP.AU=BHP|AUD|STK|ASX||||0|

Am I missing spaces or any other formatting issues?

Use ".au" because that's the Norgate format.
Did you restart after editing IBContracts?

QUOTE:
May I suggest you also add .ASX as a suffix

I'll reject the suggestion because a 3 letter suffix could be confused with a base currency and I don't want to make special cases at this point. Our existing providers that support ASX use .AX and .au.

QUOTE:
Interesting that there are two formats for .AU and .AX

Like adding .ASX, you can use any name you want for the WealthLab symbol. The record in IBContracts is the contract that the WL Symbol "points to" at IB. You could use ANYTHING.USA for BHP on ASX if you wanted to - even the 3 letter suffix if you enter it yourself.

QUOTE:
Do you think that Norgate's .au in lower case makes any difference in symbols matching?

Yes, the symbol is case sensitive.

QUOTE:
Do you have ETA for next version release?

It's ready for Glitch to release it.

Many thanks...it is working!!

The only thing I would suggest is if we could add .AU (in capitals) as some places in WL do not allow you to put lower case (like adding quote in Quote Monitor manually) and by default types everything cap. This way, .au for Norgate and .AU for when you type stuff can work seamlessly.

Thanks for being so responsive and seeing this through.

1. It's not a good idea to put screenshots in PDFs.
2. "Side note" has another meaning = offtopic.

Please repost your deleted post in a new topic, attaching the image as PNG. Thanks.

Hi Eugene,

re-PDF.../thanks...new user so no idea.

re-side-note...it is a direct consequence of the changes posted in the update and the current problem..(I hope so..lol)

QUOTE:
it is a direct consequence of the changes

OK but this topic concerns IBKR, not the 3rd party's Norgate software and extension's errors. As a different issue it deserves a new report.

For other reasons, symbols need to be case sensitive, but maybe the suffix can be an exception.

QUOTE:
WL do not allow you to put lower case (like adding quote in Quote Monitor manually)

The Shift key should work in reverse there. We'll have to fix that. You can, however, drag symbols from DataSets (and entire DataSets) into the Quotes. Also, of course, you can Send to Quotes from Signals pages too.

Thanks Cone...yes indeed, I have found that workaround as well. Overall, it seems pretty reliable and holds connection well. I have tested it by having it talk to the API the whole day, stream and chart in 1 min intervals.

Yes...extension case-sensitivity fix would be a great enabler to just remove these annoying inconsistencies.

Thanks again for your prompt attention and all the help. As per Eugene, I will raise a new topic on Norgate data now reporting issue since we have introduced .au extensions