For both the SMAC and particle swarm optimizers, there appear to be threads trying to access the same C:\Windows\temp\smac.log file at the same time.
In addition, the optimizer should be using the logical TEMP as the scratch device instead of C:\Windows\temp\smac.log in the first place.
In addition, the optimizer should be using the logical TEMP as the scratch device instead of C:\Windows\temp\smac.log in the first place.
CODE:
11/23/2024 09:38:13:975 Particle Swarm Exception in Optimize() AggregateException One or more errors occurred. (The process cannot access the file 'C:\Windows\temp\smac.log' because it is being used by another process.) (The process cannot access the file 'C:\Windows\temp\smac.log' because it is being used by another process.) (The process cannot access the file 'C:\Windows\temp\smac.log' because it is being used by another process.) (The process cannot access the file 'C:\Windows\temp\smac.log' because it is being used by another process.) (The process cannot access the file 'C:\Windows\temp\smac.log' because it is being used by another process.) Inner Exception: The process cannot access the file 'C:\Windows\temp\smac.log' because it is being used by another process. at System.Threading.Tasks.TaskReplicator.Run[TState](ReplicatableUserAction`1 action, ParallelOptions options, Boolean stopOnFirstFailure) at System.Threading.Tasks.Parallel.ForWorker[TLocal,TInt](TInt fromInclusive, TInt toExclusive, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Func`4 bodyWithLocal, Func`1 localInit, Action`1 localFinally) --- End of stack trace from previous location --- at System.Threading.Tasks.Parallel.ForWorker[TLocal,TInt](TInt fromInclusive, TInt toExclusive, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Func`4 bodyWithLocal, Func`1 localInit, Action`1 localFinally) at System.Threading.Tasks.Parallel.For(Int32 fromInclusive, Int32 toExclusive, ParallelOptions parallelOptions, Action`1 body) at SharpLearning.Optimization.ParticleSwarmOptimizer.Optimize(Func`2 functionToMinimize) at finantic.Optimizer.ParticleSwarm.Optimize2(ParameterList pl, Boolean resumePrevious) at finantic.Optimizer.ParticleSwarm.Optimize(ParameterList pl, Boolean resumePrevious) at WealthLab.Backtest.StrategyOptimizer.PerformOptimization(IOptimizerHost host, PositionSize posSize, BacktestSettings bts, Boolean resuming) at WealthLab8.ucOptimization.InvokeStruct(Object item, DoWorkEventArgs cfg)
Rename
Unfortunately, this is an older, known bug in the current build of finantic.Optimizers.
I work on a solution.
I work on a solution.
If the log file is write-only, you could see if there's a thread-safe StreamWriter implementation that would let several threads append logs to an output file.
If not, there's "probably" a thread-safe version of StringBuilder that might let you do the same thing in memory, although that wouldn't be the ideal solution.
There are some error-logging libraries that are setup to do append logging, but that would probably be an overkill when the AddLogItem method is available in WL that already does this.
If not, there's "probably" a thread-safe version of StringBuilder that might let you do the same thing in memory, although that wouldn't be the ideal solution.
There are some error-logging libraries that are setup to do append logging, but that would probably be an overkill when the AddLogItem method is available in WL that already does this.
DrKoch, you probably could get away with plain old locks. It looks like not a whole lot of data is output to smac.log. The little console test program below writes 40,000 lines to a file using a single thread and compares that to writing 5000 lines per thread in 8 threads. The difference is miniscule. On my machine its 10 ms versus 51 ms. You can't even make a cup of coffee in that time. Of course, if you're also reading the file during main SMAC processing that could/would alter the outcome.
CODE:
using System.Diagnostics; namespace TimeLocking; internal abstract class Program { private const int LinesToWrite = 5000; private const int ThreadsToUse = 8; private static readonly object MyLock = new(); private static readonly string? TempDir = Environment.GetEnvironmentVariable("TEMP"); private static readonly string? FileName = Path.Combine(TempDir, "timelockingtest.txt"); private static void Main(string[] args) { var file = File.CreateText(FileName); var sw = new Stopwatch(); sw.Start(); for (var i = 0; i < LinesToWrite * ThreadsToUse; i++) file.WriteLine($"Line {i}: This is some text\n"); file.Close(); sw.Stop(); Console.WriteLine($"Without locks time taken: {sw.ElapsedMilliseconds}ms"); TimeWithLocking(); } private static void TimeWithLocking() { var file = File.CreateText(FileName); // create 8 threads var threads = new Thread[ThreadsToUse]; for (var i = 0; i < ThreadsToUse; i++) { var i1 = i; // avoid closure issue threads[i] = new Thread(() => WriteToFile(i1, file)); } var sw = new Stopwatch(); sw.Start(); for (var i = 0; i < ThreadsToUse; i++) threads[i].Start(); // wait for each thread to finish for (var i = 0; i < ThreadsToUse; i++) threads[i].Join(); file.Close(); sw.Stop(); Console.WriteLine($"With locks time taken: {sw.ElapsedMilliseconds}ms"); } private static void WriteToFile(int i, StreamWriter file) { for (var j = 0; j < LinesToWrite; j++) lock (MyLock) { file.WriteLine($"Thread {i}: Line {j}: This is some text\n"); } } }
Many thanks for all your help and suggestions.
The thing is - in this case - the logging was used for debugging and performance checks during development only.
I just forgot to deactivate it before release. :(
The thing is - in this case - the logging was used for debugging and performance checks during development only.
I just forgot to deactivate it before release. :(
Your Response
Post
Edit Post
Login is required