Processing Thousands of Files Without Losing Any of Them

The problem is not the loop

I spent a good while on a service whose entire job was to ingest files. A partner system dropped record files onto a server continuously, and we had to read each one, extract structured data from it, and store it. Thousands a day, unattended, in the background.

The naive version of this is four lines and it’s the version every tutorial shows you:

foreach (var path in Directory.GetFiles(inbox, "*.xml"))
{
    var content = File.ReadAllText(path);
    Process(content);
}

That code is fine for a folder you’re processing by hand, once. Put it in a service that runs forever against a folder somebody else writes to, and essentially every assumption in it is wrong. It assumes the files are complete. It assumes none of them will throw. It assumes the folder is not being written to while you enumerate it. It assumes the process won’t die halfway through.

None of those held. This post is about what replaces each one. It’s the reliability half of the problem — the performance half, which nearly sank the same project, is its own article.

Enumerate lazily

Starting with the smallest fix, because it’s free.

Directory.GetFiles() builds the complete array before returning. Directory.EnumerateFiles() streams entries as it walks. On a folder with a few dozen files this is a rounding error. On a folder holding a backlog of a hundred thousand, GetFiles means you sit there allocating a large array and doing nothing useful until the whole walk finishes.

More usefully, EnumerateFiles lets you start work on file one while discovery is still running. That matters for the structure I’ll get to at the end.

Use the lazy one by default. There’s no case where the eager one is better, and one where it’s much worse.

A file existing is not a file being ready

This is the failure that cost me the most time, and it’s completely invisible in testing because when you create the test files, they’re already complete.

When a file appears in a directory listing, all you know is that a directory entry exists. You do not know that the process writing it has finished, or has flushed, or has closed its handle. Read it at the wrong moment and you get an empty file, a truncated one, or an IOException because the writer holds an exclusive lock.

Truncation is the dangerous one. A crash is fine — you’ll see it and retry. But a half-written XML file will frequently still parse, because the opening tags are all there and the closing ones are what’s missing, or the structure happens to be valid with only some records present. You get a clean parse of incomplete data and store it as though it were real.

There’s no reliable way to ask “is the writer done?” so you infer it: if you can open the file for exclusive read, nobody else has it open.

private static async Task<bool> WaitUntilReadableAsync(string path, CancellationToken ct)
{
    const int maxAttempts = 10;

    for (var attempt = 0; attempt < maxAttempts; attempt++)
    {
        try
        {
            // FileShare.None is the whole point — this throws while the writer holds it.
            using var stream = File.Open(path, FileMode.Open, FileAccess.Read, FileShare.None);
            if (stream.Length > 0)
                return true;
        }
        catch (IOException)
        {
            // Expected while the file is still being written. Fall through and back off.
        }

        await Task.Delay(TimeSpan.FromMilliseconds(200 * (attempt + 1)), ct);
    }

    return false;   // Still locked after ~11s — leave it, the next sweep will retry.
}

Note what happens on failure: nothing destructive. The file stays where it is and gets picked up by the next pass. A file that isn’t ready yet is not an error, it’s a not yet.

FileSystemWatcher is a hint, not a source of truth

If files arrive continuously, you want to react to them rather than poll every thirty seconds. FileSystemWatcher is the obvious tool, and it will quietly betray you in three ways.

It fires on creation, not completion — that’s the previous section, and it applies to every event the watcher raises.

Its buffer overflows. The watcher keeps an OS-level buffer of pending notifications. When files land faster than your handler drains them, that buffer fills and events are discarded. You don’t get an exception. You get an Error event, which you only see if you subscribed to it, and which cannot tell you which files you missed. The conditions that trigger this — a large batch arriving at once — are exactly the conditions where losing files matters most.

It can fire multiple times for one file. Depending on how the writing process behaves, one logical file can raise several Created and Changed events.

You can enlarge InternalBufferSize and keep the handler thin, and you should — but neither turns “usually” into “always.” So don’t build on the assumption that it does.

The framing that fixed this for me: the watcher is a latency optimisation. The periodic sweep is the source of truth.

// Fast path: react to arrivals within milliseconds.
_watcher.Created += (_, e) => _pending.Writer.TryWrite(e.FullPath);

// Safety net: whatever the watcher missed, this finds. Runs regardless.
_sweepTimer = new PeriodicTimer(TimeSpan.FromMinutes(1));

Once the sweep exists and is authoritative, the watcher no longer has to be perfect. It just has to be helpful most of the time, which it is. I stopped trying to make it reliable and the design got considerably simpler.

The same file will arrive twice

With a watcher and a sweep both feeding the same pipeline, duplicates are guaranteed rather than possible. A file lands during a sweep; the watcher queues it, then the sweep reaches it and queues it again.

Don’t try to prevent the overlap — coordinating “who owns this file” between two producers is fiddly and gets worse as you add a third. Make claiming a file atomic and let whoever gets there first win:

public sealed class FileClaimSet
{
    private readonly ConcurrentDictionary<string, byte> _inFlight
        = new(StringComparer.OrdinalIgnoreCase);

    public bool TryClaim(string path) => _inFlight.TryAdd(path, 0);
    public void Release(string path)   => _inFlight.TryRemove(path, out _);
}

That covers duplicates within one run of the process. It does nothing across restarts — the dictionary is gone. For that you need the content itself to be the identity:

private static string ContentHash(string content)
    => Convert.ToHexString(SHA256.HashData(Encoding.UTF8.GetBytes(content)));

Store that hash with a unique constraint on the destination table and let the database reject replays. Hash the content, not the path — the same record genuinely does arrive under different filenames, and a path-based check won’t catch it.

Cheap insurance, and it’s what lets you re-run a batch after a crash without thinking hard about it.

One bad file must not kill the batch

The naive loop has no error handling, so the first file that throws ends the run. In a batch of five thousand, file 1,847 will throw. It will be malformed, or in an encoding you didn’t expect, or zero bytes, or something you haven’t imagined.

Two rules.

The try/catch goes inside the loop, around one file. Obvious once stated, easy to get backwards when the natural place to put error handling is around the whole operation.

A file that fails goes somewhere specific. Not logged and skipped — moved, out of the inbox and into a quarantine folder, along with the exception. Two reasons: the inbox now only contains work that’s still pending, so its size is a meaningful metric; and quarantine is a queue a human can look at, which “an error in a log file from two weeks ago” is not.

foreach (var path in candidates)
{
    if (!claims.TryClaim(path)) continue;

    try
    {
        await ProcessAsync(path, ct);
        File.Move(path, Path.Combine(_processed, Path.GetFileName(path)));
    }
    catch (Exception ex) when (ex is not OperationCanceledException)
    {
        _log.LogError(ex, "Failed to process {File}", path);
        Quarantine(path, ex);          // out of the inbox, into a human's queue
    }
    finally
    {
        claims.Release(path);
    }
}

The when (ex is not OperationCanceledException) matters for a service that gets shut down. Cancellation is not a poison file, and quarantining files because someone stopped the service is a genuinely confusing thing to debug later.

Let the folder hold your state

That File.Move at the end of the success path is doing more than tidying up.

If you process files in place and mark them done somewhere else — a database column, an in-memory set, a log — you now have two sources of truth that can disagree. Crash between “wrote the record” and “marked it done” and you don’t know what happened.

Moving the file is the completion marker. A rename within the same volume is atomic: the file is either in inbox or in processed, never both and never neither. That gives you a state machine made of directories, with no extra bookkeeping:

inbox/ — not yet done. Whatever is in here is the work remaining.
processed/ — done successfully.
quarantine/ — failed, needs a human.

Crash recovery becomes “start up and read the inbox,” which is the same code path as normal operation, which means it’s the code path you’ve been testing all along. Recovery logic that only runs after a crash is recovery logic that has never been tested.

Two practical notes: keep the folders on the same volume or the move stops being atomic and becomes copy-then-delete. And put something rotating on processed/ — a folder with four hundred thousand files in it makes every subsequent enumeration slow, and eventually makes the folder painful to open at all.

Backpressure, or how to run out of memory

Last piece, and it’s the one that separates “works on a hundred files” from “works on a hundred thousand.”

If discovery pushes into an unbounded queue and processing drains it, and discovery is faster than processing — which it always is, since listing a directory is far cheaper than parsing a file — then the queue grows without limit. On a big enough backlog the service dies of memory exhaustion while looking perfectly busy.

A bounded channel fixes this by making the producer wait:

// Full queue blocks the writer instead of growing. That's the feature.
private readonly Channel<string> _pending = Channel.CreateBounded<string>(
    new BoundedChannelOptions(capacity: 1_000)
    {
        FullMode = BoundedChannelFullMode.Wait
    });

// Producer: discovery streams in and self-throttles.
await foreach (var path in EnumerateAsync(inbox, ct))
    await _pending.Writer.WriteAsync(path, ct);

// Consumer: drains at whatever rate it can manage.
await foreach (var path in _pending.Reader.ReadAllAsync(ct))
    await HandleAsync(path, ct);

Now memory is bounded by the channel capacity regardless of whether the backlog is a hundred files or a million. This is where lazy enumeration pays off — EnumerateFiles streams into the channel and throttles naturally, whereas GetFiles would have already materialised the entire list before the channel could apply any pressure.

What to actually monitor

A background service that is running is not a background service that is working. Mine could be alive, consuming CPU, and failing every single file, and nothing would have told me — because nobody looks at a service that isn’t complaining.

Process-level health checks don’t catch that. Queue-level ones do:

Inbox count over time. Flat is healthy. Climbing means you’re falling behind. Zero forever might mean the upstream feed died — which looks identical to “everything is fine” if you’re only watching for errors.
Age of the oldest file in the inbox. More useful than the count. Ten files that have been stuck for six hours is a much louder signal than four hundred that arrived a minute ago.
Quarantine count, and its rate of change. A sudden jump almost always means the upstream format changed. It’s the earliest warning you’ll get.
Files claimed but never released. Something is hanging rather than throwing, which is the failure mode logs are worst at surfacing.

Every one of those is a directory count or a simple query. None of it was in the original ticket, and all of it is what made the thing trustworthy enough to leave running unattended.

The shape of it

Strip away the specifics and the structure is:

Discover lazily → verify the file is complete → claim it exactly once → process it in isolation → move it to record the outcome → bound the queue between discovery and processing.

Six pieces, none of them clever. The naive four-line loop does one of them.

What’s worth noticing is that none of this is about speed. Every one of these is about not silently losing or corrupting data in a process nobody is watching — and that’s the failure that actually hurts, because you find out weeks later, from a report that doesn’t add up.

Speed was a separate fight on the same project, and a much stranger one: this pipeline was correct and took over half an hour to handle three hundred files. Threads didn’t fix it. That’s the next article.