johnbens8
 ( 0.00% )
- ago
I am considering whether I need to build a local database myself to address the following issues:
Organizing historical fundamental data
Organizing Chinese stock data and fundamental data

I am curious if anyone here uses a self-built database, and what is the most recommended solution?

For example, using MySQL or Clickhouse, and then writing a data plugin? I am not very familiar with this, so any suggestions are welcome, thank you all!
0
50
4 Replies

Reply

Bookmark

Sort
Cone8
 ( 17.63% )
- ago
#1
Why is a db required? What's lacking with the built-in persistent storage mechanism for EventProviderBase?

In case it's not clear, in WealthLab, Event data is basically any data that isn't part of the BarHistory - like Fundamental data.
https://www.wealth-lab.com/Support/ExtensionApi/EventDataProvider
0
- ago
#2
QUOTE:
if anyone here uses a self-built database,... using MySQL or

Back in the old days, RAM memory was expensive. So the database was stored on disk rather than cached in RAM memory. In this context, MySQL was tasked with indexing the data (which is cached in RAM) and locating the record on disk to bring that specific data record into RAM to satisfy the request. So the job of MySQL was to associate which record on disk held the requested data via the cached index.

Today RAM memory is cheap, so programmers just cache the entire database in RAM so we avoid all the indexing to disk records business a traditional disk-oriented database would do.

My suggestion is to forget the traditional disk-oriented database (e.g. MySQL), buy more RAM, and cache your entire database there. And periodically during and at the end of your WL session, you can simply write your RAM cached database to disk to save it. Keep it simple.

The next question is whether or not the WL internal event database, discussed in Post #1, is fast enough for you? Well is it? If not, why not? Perhaps you need to reorganize it into a different data structure to make lookup faster. The .NET framework does have a number of different data structures, such as Dictionary<key,reference>, you can reorganize the data into for more lookup speed.

I would get your solution running first, then worry about reorganizing the cached database (with a different .NET data type) for higher lookup speed later. But if you're interested, you can look over what .NET has to offer. https://learn.microsoft.com/en-us/dotnet/standard/collections/commonly-used-collection-types
0
Glitch8
 ( 11.39% )
- ago
#3
Where are you sourcing the fundamental data?
0
- ago
#4
I use PostgreSQL in my trading tool. But it's not used for Event data or price data. For that I depends on WL itself and data providers such as Norgate.

I use local db mainly for the following reasons. I use WL for backtesting and daily signal generation but use my own software for execution.
https://www.wealth-lab.com/Discussion/Add-Trading-Preference-to-exit-position-based-on-executed-order-size-12192
https://www.wealth-lab.com/Discussion/Automatically-save-Order-Manager-history-12155
0

Reply

Bookmark

Sort