- ago
For both the SMAC and particle swarm optimizers, there appear to be threads trying to access the same C:\Windows\temp\smac.log file at the same time.

In addition, the optimizer should be using the logical TEMP as the scratch device instead of C:\Windows\temp\smac.log in the first place.



CODE:
11/23/2024 09:38:13:975 Particle Swarm Exception in Optimize() AggregateException One or more errors occurred. (The process cannot access the file 'C:\Windows\temp\smac.log' because it is being used by another process.) (The process cannot access the file 'C:\Windows\temp\smac.log' because it is being used by another process.) (The process cannot access the file 'C:\Windows\temp\smac.log' because it is being used by another process.) (The process cannot access the file 'C:\Windows\temp\smac.log' because it is being used by another process.) (The process cannot access the file 'C:\Windows\temp\smac.log' because it is being used by another process.) Inner Exception: The process cannot access the file 'C:\Windows\temp\smac.log' because it is being used by another process. at System.Threading.Tasks.TaskReplicator.Run[TState](ReplicatableUserAction`1 action, ParallelOptions options, Boolean stopOnFirstFailure) at System.Threading.Tasks.Parallel.ForWorker[TLocal,TInt](TInt fromInclusive, TInt toExclusive, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Func`4 bodyWithLocal, Func`1 localInit, Action`1 localFinally) --- End of stack trace from previous location --- at System.Threading.Tasks.Parallel.ForWorker[TLocal,TInt](TInt fromInclusive, TInt toExclusive, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Func`4 bodyWithLocal, Func`1 localInit, Action`1 localFinally) at System.Threading.Tasks.Parallel.For(Int32 fromInclusive, Int32 toExclusive, ParallelOptions parallelOptions, Action`1 body) at SharpLearning.Optimization.ParticleSwarmOptimizer.Optimize(Func`2 functionToMinimize) at finantic.Optimizer.ParticleSwarm.Optimize2(ParameterList pl, Boolean resumePrevious) at finantic.Optimizer.ParticleSwarm.Optimize(ParameterList pl, Boolean resumePrevious) at WealthLab.Backtest.StrategyOptimizer.PerformOptimization(IOptimizerHost host, PositionSize posSize, BacktestSettings bts, Boolean resuming) at WealthLab8.ucOptimization.InvokeStruct(Object item, DoWorkEventArgs cfg)
0
128
4 Replies

Reply

Bookmark

Sort
- ago
#1
Unfortunately, this is an older, known bug in the current build of finantic.Optimizers.
I work on a solution.
1
- ago
#2
If the log file is write-only, you could see if there's a thread-safe StreamWriter implementation that would let several threads append logs to an output file.

If not, there's "probably" a thread-safe version of StringBuilder that might let you do the same thing in memory, although that wouldn't be the ideal solution.

There are some error-logging libraries that are setup to do append logging, but that would probably be an overkill when the AddLogItem method is available in WL that already does this.
0
- ago
#3
DrKoch, you probably could get away with plain old locks. It looks like not a whole lot of data is output to smac.log. The little console test program below writes 40,000 lines to a file using a single thread and compares that to writing 5000 lines per thread in 8 threads. The difference is miniscule. On my machine its 10 ms versus 51 ms. You can't even make a cup of coffee in that time. Of course, if you're also reading the file during main SMAC processing that could/would alter the outcome.

CODE:
using System.Diagnostics; namespace TimeLocking; internal abstract class Program { private const int LinesToWrite = 5000; private const int ThreadsToUse = 8; private static readonly object MyLock = new(); private static readonly string? TempDir = Environment.GetEnvironmentVariable("TEMP"); private static readonly string? FileName = Path.Combine(TempDir, "timelockingtest.txt"); private static void Main(string[] args) { var file = File.CreateText(FileName); var sw = new Stopwatch(); sw.Start(); for (var i = 0; i < LinesToWrite * ThreadsToUse; i++) file.WriteLine($"Line {i}: This is some text\n"); file.Close(); sw.Stop(); Console.WriteLine($"Without locks time taken: {sw.ElapsedMilliseconds}ms"); TimeWithLocking(); } private static void TimeWithLocking() { var file = File.CreateText(FileName); // create 8 threads var threads = new Thread[ThreadsToUse]; for (var i = 0; i < ThreadsToUse; i++) { var i1 = i; // avoid closure issue threads[i] = new Thread(() => WriteToFile(i1, file)); } var sw = new Stopwatch(); sw.Start(); for (var i = 0; i < ThreadsToUse; i++) threads[i].Start(); // wait for each thread to finish for (var i = 0; i < ThreadsToUse; i++) threads[i].Join(); file.Close(); sw.Stop(); Console.WriteLine($"With locks time taken: {sw.ElapsedMilliseconds}ms"); } private static void WriteToFile(int i, StreamWriter file) { for (var j = 0; j < LinesToWrite; j++) lock (MyLock) { file.WriteLine($"Thread {i}: Line {j}: This is some text\n"); } } }
1
- ago
#4
Many thanks for all your help and suggestions.
The thing is - in this case - the logging was used for debugging and performance checks during development only.

I just forgot to deactivate it before release. :(
1

Reply

Bookmark

Sort