finantic-Optimizer-The-process-cannot-access-the-file-error

2024-11-23T16:25:19Z - 2024-11-23T16:25:19Z ago

finantic.Optimizer: The process cannot access the file error

For both the SMAC and particle swarm optimizers, there appear to be threads trying to access the same C:\Windows\temp\smac.log file at the same time.

In addition, the optimizer should be using the logical TEMP as the scratch device instead of C:\Windows\temp\smac.log in the first place.

CODE:
11/23/2024 09:38:13:975
Particle Swarm
Exception in Optimize()
AggregateException
One or more errors occurred. (The process cannot access the file 'C:\Windows\temp\smac.log' because it is being used by another process.) (The process cannot access the file 'C:\Windows\temp\smac.log' because it is being used by another process.) (The process cannot access the file 'C:\Windows\temp\smac.log' because it is being used by another process.) (The process cannot access the file 'C:\Windows\temp\smac.log' because it is being used by another process.) (The process cannot access the file 'C:\Windows\temp\smac.log' because it is being used by another process.)  Inner Exception: The process cannot access the file 'C:\Windows\temp\smac.log' because it is being used by another process.
   at System.Threading.Tasks.TaskReplicator.Run[TState](ReplicatableUserAction`1 action, ParallelOptions options, Boolean stopOnFirstFailure)
   at System.Threading.Tasks.Parallel.ForWorker[TLocal,TInt](TInt fromInclusive, TInt toExclusive, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Func`4 bodyWithLocal, Func`1 localInit, Action`1 localFinally)
--- End of stack trace from previous location ---
   at System.Threading.Tasks.Parallel.ForWorker[TLocal,TInt](TInt fromInclusive, TInt toExclusive, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Func`4 bodyWithLocal, Func`1 localInit, Action`1 localFinally)
   at System.Threading.Tasks.Parallel.For(Int32 fromInclusive, Int32 toExclusive, ParallelOptions parallelOptions, Action`1 body)
   at SharpLearning.Optimization.ParticleSwarmOptimizer.Optimize(Func`2 functionToMinimize)
   at finantic.Optimizer.ParticleSwarm.Optimize2(ParameterList pl, Boolean resumePrevious)
   at finantic.Optimizer.ParticleSwarm.Optimize(ParameterList pl, Boolean resumePrevious)
   at WealthLab.Backtest.StrategyOptimizer.PerformOptimization(IOptimizerHost host, PositionSize posSize, BacktestSettings bts, Boolean resuming)
   at WealthLab8.ucOptimization.InvokeStruct(Object item, DoWorkEventArgs cfg)

141

4 Replies

Bookmark

Notify

Sort

DrKoch8

2024-11-24T13:48:34Z - 2024-11-24T13:48:34Z ago

Unfortunately, this is an older, known bug in the current build of finantic.Optimizers.
I work on a solution.

superticker8

2024-11-24T14:28:55Z - 2024-11-24T14:28:55Z ago

If the log file is write-only, you could see if there's a thread-safe StreamWriter implementation that would let several threads append logs to an output file.

If not, there's "probably" a thread-safe version of StringBuilder that might let you do the same thing in memory, although that wouldn't be the ideal solution.

There are some error-logging libraries that are setup to do append logging, but that would probably be an overkill when the AddLogItem method is available in WL that already does this.

paul19868

2024-11-24T17:52:59Z - 2024-11-24T17:52:59Z ago

DrKoch, you probably could get away with plain old locks. It looks like not a whole lot of data is output to smac.log. The little console test program below writes 40,000 lines to a file using a single thread and compares that to writing 5000 lines per thread in 8 threads. The difference is miniscule. On my machine its 10 ms versus 51 ms. You can't even make a cup of coffee in that time. Of course, if you're also reading the file during main SMAC processing that could/would alter the outcome.

CODE:
using System.Diagnostics;

namespace TimeLocking;

internal abstract class Program
{
    private const int LinesToWrite = 5000;
    private const int ThreadsToUse = 8;
    private static readonly object MyLock = new();
    private static readonly string? TempDir = Environment.GetEnvironmentVariable("TEMP");
    private static readonly string? FileName = Path.Combine(TempDir, "timelockingtest.txt");

    private static void Main(string[] args)
    {
        var file = File.CreateText(FileName);
        var sw = new Stopwatch();
        sw.Start();
        for (var i = 0; i < LinesToWrite * ThreadsToUse; i++)
            file.WriteLine($"Line {i}: This is some text\n");
        file.Close();
        sw.Stop();
        Console.WriteLine($"Without locks time taken: {sw.ElapsedMilliseconds}ms");

        TimeWithLocking();
    }

    private static void TimeWithLocking()
    {
        var file = File.CreateText(FileName);
        // create 8 threads
        var threads = new Thread[ThreadsToUse];
        for (var i = 0; i < ThreadsToUse; i++)
        {
            var i1 = i; // avoid closure issue
            threads[i] = new Thread(() => WriteToFile(i1, file));
        }

        var sw = new Stopwatch();
        sw.Start();
        for (var i = 0; i < ThreadsToUse; i++) threads[i].Start();

        // wait for each thread to finish
        for (var i = 0; i < ThreadsToUse; i++) threads[i].Join();

        file.Close();
        sw.Stop();
        Console.WriteLine($"With locks time taken: {sw.ElapsedMilliseconds}ms");
    }

    private static void WriteToFile(int i, StreamWriter file)
    {
        for (var j = 0; j < LinesToWrite; j++)
            lock (MyLock)
            {
                file.WriteLine($"Thread {i}: Line {j}: This is some text\n");
            }
    }
}

DrKoch8

2024-11-25T08:52:15Z - 2024-11-25T08:52:15Z ago

Many thanks for all your help and suggestions.
The thing is - in this case - the logging was used for debugging and performance checks during development only.

I just forgot to deactivate it before release. :(

Bookmark

Notify

Sort