I want to do an Evolver Run with i.E. the Russel 1000.
After a few minutes my RAM is at 98 - 99% utilization (16GB). And two minutes later WL crashes. That happens every time to me when using big portfolios.
Our WL runs on a virtual server with 8 cores, so this should be sufficient.
Isn't there another way to solve this Evolver process in terms of programming? Apparently WL "runs" with all stocks of the dataset at one time and this causes ram overload=!
Perhaps running one stock after the other is better, so that the evaluate can check large data sets properly?
Thank you!
After a few minutes my RAM is at 98 - 99% utilization (16GB). And two minutes later WL crashes. That happens every time to me when using big portfolios.
Our WL runs on a virtual server with 8 cores, so this should be sufficient.
Isn't there another way to solve this Evolver process in terms of programming? Apparently WL "runs" with all stocks of the dataset at one time and this causes ram overload=!
Perhaps running one stock after the other is better, so that the evaluate can check large data sets properly?
Thank you!
Rename
8 cores has nothing to do with potential lack of memory which is only 16GB. It may be insufficient for running such a memory-consuming task as Evolver. Can you upgrade the virtual server, adding say 64GB RAM?
>> Perhaps running one stock after the other is better, so that the evaluate can check large data sets properly?<<
No it would be worse. It would make it impossible to do any kind of dynamic portfolio level simulations.
You simply need more RAM. 64g should be plenty.
No it would be worse. It would make it impossible to do any kind of dynamic portfolio level simulations.
You simply need more RAM. 64g should be plenty.
QUOTE:
No it would be worse. It would make it impossible to do any kind of dynamic portfolio level simulations.
That is not true. WL is simply not designed to handle it, that's all.
It is a question of programming and design. Every tool that is part of the software can operate with much less RAM. In return, there are "slight/acceptable" performance losses. Without any doubt.
Let's be clear and put some context on that. WealthLab is not designed to limit memory use and allows simulations to use all the memory a machine will give it.
You're not going to be able to run simulations that require 32 GB with 16 GB (the absolute minimum recommend for WealthLab).
Response to your edit:
Right, WealthLab is not designed to conserve memory. Any other digs you have? (I've even agreed with you up to here.)
You're not going to be able to run simulations that require 32 GB with 16 GB (the absolute minimum recommend for WealthLab).
Response to your edit:
Right, WealthLab is not designed to conserve memory. Any other digs you have? (I've even agreed with you up to here.)
I have nothing to criticise at all.
I can even understand that the focus in development is on constantly improving the content components.It is okay to require certain hardware resources. It's simply a descision, a design descision.
The limitations that sometimes occur "could" be solved programmatically. The issue is more common in the context of intraday data, where the data volumes are much larger. Hardware development will progress quickly so that intraday data in a one-minute context can be processed without problems, even for large markets/portfolios.
I can say that 5-minute scale for 15-20 years of data and 128 GB RAM for the Russell 1000/2000 is handled without problems most of the time. Good job, WL team.
I don't see a problem, but i see room for improvements.
Surely you don't want software that freezes, crashes or slows down extremely because the operating system is swapping.
I do not intend to spread bad vibes.
I can even understand that the focus in development is on constantly improving the content components.It is okay to require certain hardware resources. It's simply a descision, a design descision.
The limitations that sometimes occur "could" be solved programmatically. The issue is more common in the context of intraday data, where the data volumes are much larger. Hardware development will progress quickly so that intraday data in a one-minute context can be processed without problems, even for large markets/portfolios.
I can say that 5-minute scale for 15-20 years of data and 128 GB RAM for the Russell 1000/2000 is handled without problems most of the time. Good job, WL team.
I don't see a problem, but i see room for improvements.
Surely you don't want software that freezes, crashes or slows down extremely because the operating system is swapping.
I do not intend to spread bad vibes.
>> That is not true. <<
Yes it is true. The software would be worse if it processed each symbol one by one as the original poster requested. We’d completely lose the ability to do dynamic simulations at a portfolio level.
This is actually how WL6 did things and the new paradigm is a great improvement.
It’s easy to toss off criticisms on a forum but much harder to actually sit down and design a product yourself, but we do the best we can and stand by our work and claims.
Yes it is true. The software would be worse if it processed each symbol one by one as the original poster requested. We’d completely lose the ability to do dynamic simulations at a portfolio level.
This is actually how WL6 did things and the new paradigm is a great improvement.
It’s easy to toss off criticisms on a forum but much harder to actually sit down and design a product yourself, but we do the best we can and stand by our work and claims.
Guys, what is going on? Did you read my last post?
Glitch, no one said you had to process symbol by symbol. Come on.
As an idea, you can do a partial backtest and collect the results step by step.
Don't say that you have found the only solution that is possible, that would be ridiculous.
Glitch, no one said you had to process symbol by symbol. Come on.
As an idea, you can do a partial backtest and collect the results step by step.
Don't say that you have found the only solution that is possible, that would be ridiculous.
I was responding to this chain of comments:
OP: >>Perhaps running one stock after the other is better, so that the evaluate can check large data sets properly?<<
Me: No, that would be worse
You: That is not true
And breaking this up somehow into chunks is not viable, what with having to recalculate indicators, synchronization, etc, it just isn't something we'd consider.
OP: >>Perhaps running one stock after the other is better, so that the evaluate can check large data sets properly?<<
Me: No, that would be worse
You: That is not true
And breaking this up somehow into chunks is not viable, what with having to recalculate indicators, synchronization, etc, it just isn't something we'd consider.
QUOTE:
It’s easy to toss off criticisms on a forum but much harder to actually sit down and design a product yourself, but we do the best we can and stand by our work and claims.
I respect your work, as i wrote. This is a complete unessary comment on your side.
Even in this forum are developers and people who desigend, integrated software from different areas.
My comment was about memory consumption. And NOW I tell you, if after reading my post (#5) above you still tell me that better memory management is not possible, then it is all about your ego and has nothing to do with reality.
The vibe is changing completely unnecessarily!
It's not about my ego, I'm sure there is someone out there who could do things better than I could. I'm just saying I did the best that *I* can. Perhaps that's you, and if so I'd love to see your product!
We're not in a position now to completely change the entire backtesting architecture in the manner you suggested.
We're not in a position now to completely change the entire backtesting architecture in the manner you suggested.
I still don't get it. What is wrong with Post #5?
Translated into German, this means that there are good reasons for you (your team) to treat your software development strategy the way you do. I even argue that it is a waste of time to focus on memory management because time will solve the problem anyway. (better hardware). In conclusion, I am sure you are interested in improvements because it is also in your interest to avoid freezing, crashing or slow software! In the meantime, I also pay credit to you for a job well done.
So, what is the problem now? That I said there are possibilities that could be investigated? That is a problem for you?
And I'd be happy to talk about and demonstrate a product of mine in a video session if optimisation methods or classic AI are of interest.
Translated into German, this means that there are good reasons for you (your team) to treat your software development strategy the way you do. I even argue that it is a waste of time to focus on memory management because time will solve the problem anyway. (better hardware). In conclusion, I am sure you are interested in improvements because it is also in your interest to avoid freezing, crashing or slow software! In the meantime, I also pay credit to you for a job well done.
So, what is the problem now? That I said there are possibilities that could be investigated? That is a problem for you?
And I'd be happy to talk about and demonstrate a product of mine in a video session if optimisation methods or classic AI are of interest.
No problem, there are surely many ways to tackle any problem, each with its own set of tradeoffs. It's fun to theorize about how things might have been designed differently :)
Thank you for your feedback. If I had known my question would spark such a heated debate, I might not have asked it...
Yes, the discussion "evolved" in unusual directions! :D
Compared to Wealth-Lab 7, Wealth-Lab 8 introduced the migration to the .NET 6 framework. At the time, I experienced some noticeable performance improvements.
Eventually, Wealth-Lab will one day migrate to the .NET 7 (or even .NET 8) framework, and we can hopefully enjoy from some more improvements that the framework keeps introducing:
https://devblogs.microsoft.com/dotnet/performance_improvements_in_net_7/
Eventually, Wealth-Lab will one day migrate to the .NET 7 (or even .NET 8) framework, and we can hopefully enjoy from some more improvements that the framework keeps introducing:
https://devblogs.microsoft.com/dotnet/performance_improvements_in_net_7/
True, we hate to keep disrupting users with major new revisions like this but eventually we'll have to update to .NET7.
Your Response
Post
Edit Post
Login is required