12. Capstone: parallel file indexer

Goal

Build a concurrent search tool: it walks a directory tree, reads and tokenises thousands of files in parallel, builds an in-memory inverted index, and answers queries ("which files contain all of these words, best matches first?"). It must be data-race free under Swift 6 strict concurrency, tested, and measurably faster in parallel.

This exercises: structured concurrency (withTaskGroup), bounded parallelism, Sendable value types, designing around no shared mutable state, async testing with Swift Testing, deterministic output from nondeterministic execution, and timing with ContinuousClock.

What it does

$ indexer --stats ~/projects/myapp actor isolation
   14  Sources/App/Sync/CacheActor.swift
    6  docs/concurrency.md
    2  Sources/App/Model/User.swift

indexed 1873 files (412905 words, 21877 distinct terms)
skipped 2, jobs 8
build 84.3 ms
$ echo $?
0

Requirements

1. indexer [options] <directory> <query words...>; exit 0 if there are hits, 1 if none, 2 for usage errors.
2. Recursively list files, filtered by extension (--ext swift,md, default swift,md,txt), skipping hidden entries and build directories (.build, node_modules, ...).
3. Index files concurrently with at most --jobs N tasks running (default: CPU cores). Never create one task per file up front.
4. Files that are not valid UTF-8 or can't be read are skipped and reported on standard error, not fatal.
5. Search returns files containing every query word, ranked by total occurrences, ties broken by path — so the output is identical no matter how tasks were scheduled.
6. --top N limits results; --stats prints file/word/term counts, skip count, and build time.
7. Compiles in Swift 6 language mode with zero concurrency warnings and no @unchecked Sendable, no locks, no actors needed.

Design

The key design decision: share nothing.

 listFiles ──► [URL] ──►  ┌── task ──► FileIndex ──┐
                          ├── task ──► FileIndex ──┤   one loop (the parent)
   bounded pool of        ├── task ──► FileIndex ──┼──► SearchIndex.add(...)
   at most `jobs` tasks   └── task ──► skipped ────┘

• Each child task reads one file and returns an independent, Sendable value (FileIndex: path + word counts). Child tasks touch no shared state.
• Only the parent loop, running sequentially as results arrive, mutates the combined SearchIndex.
• Because of that, there is nothing to lock and nothing for the compiler to complain about. This is the "return results instead of sharing" rule from the actors lesson in action.
• Parallelism is bounded by priming the group with jobs tasks and adding one more each time one finishes.
• Completion order is nondeterministic, so anything user-visible is sorted (skipped, search hits) to keep the output stable.

An actor would also work (children await index.add(file)) but adds an actor hop per file and makes the merge a contended resource. Measure both — that's a stretch goal.

Project layout

indexer/
├── Package.swift
├── Sources/
│   ├── IndexKit/
│   │   ├── Model.swift        # tokenize, FileIndex, SearchIndex, SearchHit
│   │   └── Builder.swift      # listFiles, buildIndex (the concurrent part)
│   └── indexer/
│       └── main.swift         # arguments, timing, output
└── Tests/
    └── IndexKitTests/
        └── IndexTests.swift

The manifest

// swift-tools-version: 6.0
import PackageDescription

let package = Package(
    name: "indexer",
    platforms: [.macOS(.v13)],          // only affects Apple platforms; Linux and Windows ignore it
    products: [
        .executable(name: "indexer", targets: ["indexer"])
    ],
    targets: [
        .target(name: "IndexKit"),
        .executableTarget(name: "indexer", dependencies: ["IndexKit"]),
        .testTarget(name: "IndexKitTests", dependencies: ["IndexKit"]),
    ]
)

swift-tools-version: 6.0 selects the Swift 6 language mode, so data races are compile errors. The platforms line matters only on Apple systems: the Swift concurrency runtime and ContinuousClock need macOS 13 or newer, and without it the build fails with "only available in macOS 10.15 or newer".

Step 1: the model (pure and Sendable)

Sources/IndexKit/Model.swift:

/// Splits text into lowercase words made of letters and digits.
/// Single-character words are dropped: they are noise for search.
public func tokenize(_ text: String) -> [String] {
    var words: [String] = []
    var current = ""

    for ch in text {
        if ch.isLetter || ch.isNumber {
            current.append(ch)
        } else if !current.isEmpty {
            words.append(current)
            current = ""
        }
    }
    if !current.isEmpty { words.append(current) }

    return words.map { $0.lowercased() }.filter { $0.count >= 2 }
}

/// The word counts of a single file. A plain value: cheap to build on any
/// thread, and `Sendable` for free because every property is.
public struct FileIndex: Sendable, Equatable {
    public let path: String
    public let counts: [String: Int]

    public var totalWords: Int { counts.values.reduce(0, +) }

    public init(path: String, counts: [String: Int]) {
        self.path = path
        self.counts = counts
    }

    public init(path: String, text: String) {
        var counts: [String: Int] = [:]
        for word in tokenize(text) {
            counts[word, default: 0] += 1
        }
        self.init(path: path, counts: counts)
    }
}

public struct SearchHit: Sendable, Equatable {
    public let path: String
    public let score: Int

    public init(path: String, score: Int) {
        self.path = path
        self.score = score
    }
}

/// An inverted index: term -> (file path -> occurrences).
/// A value type, so it is only ever mutated in one place and needs no lock.
public struct SearchIndex: Sendable {
    private var postings: [String: [String: Int]] = [:]
    public private(set) var fileCount = 0
    public private(set) var wordCount = 0

    public init() {}

    public var termCount: Int { postings.count }

    public mutating func add(_ file: FileIndex) {
        fileCount += 1
        wordCount += file.totalWords
        for (term, count) in file.counts {
            postings[term, default: [:]][file.path] = count
        }
    }

    /// Files containing *every* query term, best first (total occurrences, then path).
    public func search(_ query: String, limit: Int = 10) -> [SearchHit] {
        let terms = Set(tokenize(query))
        guard !terms.isEmpty else { return [] }

        let lists = terms.map { postings[$0] ?? [:] }
        guard let rarest = lists.min(by: { $0.count < $1.count }) else { return [] }

        var hits: [SearchHit] = []
        for path in rarest.keys {
            var score = 0
            var matchesAll = true
            for list in lists {
                guard let count = list[path] else {
                    matchesAll = false
                    break
                }
                score += count
            }
            if matchesAll { hits.append(SearchHit(path: path, score: score)) }
        }

        hits.sort { $0.score != $1.score ? $0.score > $1.score : $0.path < $1.path }
        return Array(hits.prefix(limit))
    }
}

• Everything is a struct holding Strings, Ints, and dictionaries of them, so Sendable conformance is straightforward; we write it explicitly because the types are public.
• SearchIndex.search starts from the rarest term's posting list and intersects: for multi-word queries that touches far fewer entries than scanning all files.
• private(set) stops outside code from corrupting the counters.

Step 2: the concurrent builder

Sources/IndexKit/Builder.swift:

import Foundation

public struct Skipped: Sendable, Equatable {
    public let path: String
    public let reason: String
}

public struct BuildResult: Sendable {
    public var index: SearchIndex
    public var skipped: [Skipped]
}

/// Recursively lists files, skipping hidden entries and build directories.
/// An empty `extensions` set means "every file".
public func listFiles(under root: URL, extensions: Set<String>) -> [URL] {
    let fm = FileManager.default
    guard let walker = fm.enumerator(
        at: root,
        includingPropertiesForKeys: [.isDirectoryKey],
        options: [.skipsHiddenFiles]
    ) else { return [] }

    let ignoredDirectories: Set<String> = [".build", "node_modules", "build", "DerivedData"]
    var files: [URL] = []

    for case let url as URL in walker {
        let isDirectory = (try? url.resourceValues(forKeys: [.isDirectoryKey]))?.isDirectory ?? false
        if isDirectory {
            if ignoredDirectories.contains(url.lastPathComponent) { walker.skipDescendants() }
            continue
        }
        if extensions.isEmpty || extensions.contains(url.pathExtension.lowercased()) {
            files.append(url)
        }
    }
    return files.sorted { $0.path < $1.path }
}

private enum Outcome: Sendable {
    case indexed(FileIndex)
    case skipped(Skipped)
}

/// Pure with respect to everything except the file it reads.
private func process(_ url: URL) -> Outcome {
    do {
        let data = try Data(contentsOf: url)
        guard let text = String(data: data, encoding: .utf8) else {
            return .skipped(Skipped(path: url.path, reason: "not valid UTF-8"))
        }
        return .indexed(FileIndex(path: url.path, text: text))
    } catch {
        return .skipped(Skipped(path: url.path, reason: error.localizedDescription))
    }
}

/// Indexes `files` using at most `concurrency` simultaneous tasks.
///
/// Design: child tasks produce independent `FileIndex` values; only this
/// function's loop ever touches the combined `SearchIndex`. No shared mutable
/// state means no locks, no actor, and no data races.
public func buildIndex(
    of files: [URL],
    concurrency: Int = 4,
    onProgress: ((_ done: Int, _ total: Int) -> Void)? = nil
) async -> BuildResult {
    var index = SearchIndex()
    var skipped: [Skipped] = []
    var done = 0
    let limit = max(1, concurrency)

    await withTaskGroup(of: Outcome.self) { group in
        var remaining = files.makeIterator()

        // Prime the pool, then add one new task for each task that finishes.
        for _ in 0..<limit {
            guard let url = remaining.next() else { break }
            group.addTask { process(url) }
        }

        while let outcome = await group.next() {
            switch outcome {
            case .indexed(let file): index.add(file)
            case .skipped(let item): skipped.append(item)
            }
            done += 1
            onProgress?(done, files.count)

            if let url = remaining.next() {
                group.addTask { process(url) }
            }
        }
    }

    skipped.sort { $0.path < $1.path }   // completion order is arbitrary; make output deterministic
    return BuildResult(index: index, skipped: skipped)
}

Read buildIndex slowly:

1. withTaskGroup(of: Outcome.self) creates a scope; no child escapes it.
2. The priming for _ in 0..<limit starts at most limit children. Each group.addTask { process(url) } captures only a URL — a Sendable value.
3. while let outcome = await group.next() receives one finished result at a time, on the parent. This loop is the only place index, skipped, and done are mutated.
4. After each completion we add exactly one more task, keeping the pool full without exceeding limit.
5. onProgress is called from the parent, so the caller's closure needs no @Sendable.
6. process does blocking file I/O inside a cooperative-pool thread. With concurrency capped at the core count that's acceptable for a CLI; a server would use async I/O or a dedicated queue (see the async/await lesson).

Step 3: the command-line front end

Sources/indexer/main.swift:

import Foundation
import IndexKit

func usage() -> String {
    """
    usage: indexer [options] <directory> <query words...>

    Builds an in-memory index of the text files under <directory> using
    concurrent tasks, then lists the files that contain every query word.

    options:
      --jobs N     maximum concurrent tasks (default: number of CPU cores)
      --ext a,b    only index these extensions (default: swift,md,txt)
      --top N      show at most N results (default: 10)
      --stats      print timing and index statistics
    """
}

func fail(_ message: String) -> Never {
    FileHandle.standardError.write(Data((message + "\n").utf8))
    exit(2)
}

func milliseconds(_ duration: Duration) -> Double {
    let parts = duration.components
    return Double(parts.seconds) * 1_000 + Double(parts.attoseconds) / 1e15
}

var jobs = ProcessInfo.processInfo.activeProcessorCount
var extensions: Set<String> = ["swift", "md", "txt"]
var top = 10
var showStats = false
var positional: [String] = []

var arguments = ArraySlice(CommandLine.arguments.dropFirst())
while let argument = arguments.popFirst() {
    switch argument {
    case "--jobs", "--top":
        guard let text = arguments.popFirst(), let value = Int(text), value > 0 else {
            fail("\(argument) needs a positive number\n\(usage())")
        }
        if argument == "--jobs" { jobs = value } else { top = value }
    case "--ext":
        guard let text = arguments.popFirst() else { fail("--ext needs a value\n\(usage())") }
        extensions = Set(text.split(separator: ",").map { $0.lowercased() })
    case "--stats":
        showStats = true
    case "-h", "--help":
        print(usage())
        exit(0)
    case _ where argument.hasPrefix("--"):
        fail("unknown option '\(argument)'\n\(usage())")
    default:
        positional.append(argument)
    }
}

guard positional.count >= 2 else { fail(usage()) }
let rootPath = positional[0]
let query = positional.dropFirst().joined(separator: " ")

var isDirectory: ObjCBool = false
guard FileManager.default.fileExists(atPath: rootPath, isDirectory: &isDirectory), isDirectory.boolValue else {
    fail("indexer: '\(rootPath)' is not a directory")
}

let root = URL(fileURLWithPath: rootPath)
let clock = ContinuousClock()
let start = clock.now

let files = listFiles(under: root, extensions: extensions)
let result = await buildIndex(of: files, concurrency: jobs)
let built = clock.now

let hits = result.index.search(query, limit: top)
let prefix = root.standardizedFileURL.path

if hits.isEmpty {
    print("No files contain: \(query)")
} else {
    for hit in hits {
        let shown = hit.path.hasPrefix(prefix) ? String(hit.path.dropFirst(prefix.count + 1)) : hit.path
        let score = String(hit.score)
        print(String(repeating: " ", count: max(0, 5 - score.count)) + score + "  " + shown)
    }
}

for item in result.skipped {
    FileHandle.standardError.write(Data("skipped \(item.path): \(item.reason)\n".utf8))
}

if showStats {
    let index = result.index
    print("""

        indexed \(index.fileCount) files (\(index.wordCount) words, \(index.termCount) distinct terms)
        skipped \(result.skipped.count), jobs \(jobs)
        build \(String(format: "%.1f", milliseconds(built - start))) ms
        """)
}
exit(hits.isEmpty ? 1 : 0)

Top-level code in main.swift can use await directly. All user-facing formatting lives here; the library never prints.

Step 4: tests

Tests/IndexKitTests/IndexTests.swift:

import Foundation
import Testing
@testable import IndexKit

@Suite("Model") struct ModelTests {
    @Test func tokenizerLowercasesSplitsAndDropsNoise() {
        #expect(tokenize("Hello, World! a x2 C++ is_fun") == ["hello", "world", "x2", "is", "fun"])
    }

    @Test func fileIndexCountsWords() {
        let file = FileIndex(path: "a.txt", text: "swift swift Swift rocks")
        #expect(file.counts == ["swift": 3, "rocks": 1])
        #expect(file.totalWords == 4)
    }

    @Test func searchRequiresEveryTermAndRanksByOccurrences() {
        var index = SearchIndex()
        index.add(FileIndex(path: "a.txt", text: "swift actors are great"))
        index.add(FileIndex(path: "b.txt", text: "swift swift swift actors"))
        index.add(FileIndex(path: "c.txt", text: "only swift here"))

        #expect(index.search("swift actors") == [
            SearchHit(path: "b.txt", score: 4),
            SearchHit(path: "a.txt", score: 2),
        ])
        #expect(index.search("swift").map(\.path) == ["b.txt", "a.txt", "c.txt"])   // 3, 1, 1 — ties by path
        #expect(index.search("missing").isEmpty)
        #expect(index.search("   ").isEmpty)
        #expect(index.search("swift", limit: 1).count == 1)
    }

    @Test func statisticsAccumulate() {
        var index = SearchIndex()
        index.add(FileIndex(path: "a", text: "one two two"))
        index.add(FileIndex(path: "b", text: "two three"))
        #expect(index.fileCount == 2)
        #expect(index.wordCount == 5)
        #expect(index.termCount == 3)
    }
}

@Suite("Concurrent build") struct BuilderTests {
    /// Every test gets its own directory: Swift Testing runs tests in parallel.
    func makeCorpus(fileCount: Int) throws -> URL {
        let dir = FileManager.default.temporaryDirectory
            .appendingPathComponent("indexer-\(UUID().uuidString)", isDirectory: true)
        try FileManager.default.createDirectory(at: dir, withIntermediateDirectories: true)
        for n in 0..<fileCount {
            let text = "common file\(n) " + String(repeating: "word\(n % 5) ", count: n + 1)
            try text.write(to: dir.appendingPathComponent("f\(n).txt"), atomically: true, encoding: .utf8)
        }
        return dir
    }

    @Test func resultDoesNotDependOnConcurrency() async throws {
        let dir = try makeCorpus(fileCount: 40)
        defer { try? FileManager.default.removeItem(at: dir) }

        let files = listFiles(under: dir, extensions: ["txt"])
        #expect(files.count == 40)

        let serial = await buildIndex(of: files, concurrency: 1)
        let parallel = await buildIndex(of: files, concurrency: 8)

        #expect(serial.index.fileCount == 40)
        #expect(parallel.index.fileCount == 40)
        #expect(serial.index.wordCount == parallel.index.wordCount)
        #expect(serial.index.termCount == parallel.index.termCount)
        #expect(serial.index.search("common word3", limit: 100) == parallel.index.search("common word3", limit: 100))
    }

    @Test func skipsNonUTF8FilesAndReportsThem() async throws {
        let dir = try makeCorpus(fileCount: 2)
        defer { try? FileManager.default.removeItem(at: dir) }
        try Data([0xFF, 0xFE, 0xFD]).write(to: dir.appendingPathComponent("binary.txt"))

        let result = await buildIndex(of: listFiles(under: dir, extensions: ["txt"]), concurrency: 2)
        #expect(result.index.fileCount == 2)
        #expect(result.skipped.count == 1)
        #expect(result.skipped.first?.path.hasSuffix("binary.txt") == true)
    }

    @Test func progressIsReportedForEveryFile() async throws {
        let dir = try makeCorpus(fileCount: 10)
        defer { try? FileManager.default.removeItem(at: dir) }

        var seen: [Int] = []
        _ = await buildIndex(of: listFiles(under: dir, extensions: []), concurrency: 3) { done, total in
            #expect(total == 10)
            seen.append(done)
        }
        #expect(seen == Array(1...10))
    }

    @Test func listFilesFiltersExtensionsAndIgnoresBuildDirectories() throws {
        let dir = try makeCorpus(fileCount: 1)
        defer { try? FileManager.default.removeItem(at: dir) }
        let ignored = dir.appendingPathComponent(".build", isDirectory: true)
        let node = dir.appendingPathComponent("node_modules", isDirectory: true)
        for folder in [ignored, node] {
            try FileManager.default.createDirectory(at: folder, withIntermediateDirectories: true)
            try "x".write(to: folder.appendingPathComponent("skipme.txt"), atomically: true, encoding: .utf8)
        }
        try "x".write(to: dir.appendingPathComponent("code.swift"), atomically: true, encoding: .utf8)

        let txt = listFiles(under: dir, extensions: ["txt"]).map(\.lastPathComponent)
        #expect(txt == ["f0.txt"])
        let all = listFiles(under: dir, extensions: []).map(\.lastPathComponent)
        #expect(all == ["code.swift", "f0.txt"])
    }
}

• resultDoesNotDependOnConcurrency is the most important test: it builds the same corpus with concurrency: 1 and concurrency: 8 and demands identical results. A concurrency bug that corrupts or loses data would show up here.
• Each filesystem test uses its own temp directory because Swift Testing runs tests in parallel.
• progressIsReportedForEveryFile shows that progress callbacks arrive on the parent in order: 1, 2, 3 … 10.

Try it

swift test
swift build -c release

# Index this course's Markdown (or any project):
.build/release/indexer --stats docs concurrency actors
.build/release/indexer --jobs 1 --stats docs concurrency actors     # compare build times
.build/release/indexer --ext swift --top 5 Sources Task

Self-check

• [ ] swift build shows no warnings in Swift 6 mode, and you did not use @unchecked Sendable.
• [ ] You can explain why index.add(file) is race-free without a lock (only the parent task touches it).
• [ ] --jobs 1 and --jobs 8 produce identical search output on a large directory.
• [ ] Running with Thread Sanitizer is clean: swift test --sanitize=thread.
• [ ] You measured a speedup for a directory with a few thousand files and can describe what limits it (disk cache, the sequential merge, the number of cores).
• [ ] You can say what happens if you cancel the surrounding Task while buildIndex runs.

Stretch goals

1. Cancellation: check Task.isCancelled in process, call group.cancelAll() when cancelled, and return a partial result flagged as incomplete. Add a test.
2. Actor comparison: write ActorIndex where children await index.add(file). Benchmark against the merge-in-parent design with 20k small files and explain the difference.
3. Chunking: group files into batches of 32 per task and measure whether it helps with many tiny files.
4. Streaming results: return an AsyncStream<FileIndex> and update a progress bar as files complete.
5. Phrase search ("actor isolation" as consecutive words) by storing token positions.
6. Persistence: make SearchIndex Codable, save it to disk, and reload it, with a version number and a modification-time check to re-index only changed files.
7. Ranking: replace raw counts with TF-IDF, and test ranking quality on a small hand-labelled corpus.
8. Benchmark with package-benchmark, recording wall time and allocations for 1, 2, 4, 8 jobs.
9. Async file reading with FileHandle.bytes, compared with blocking reads.
10. Memory-map large files (Data(contentsOf:options: .alwaysMapped)) and measure.

When you're done

You can design a concurrent Swift program where the compiler proves the absence of data races, test it deterministically, measure it, and explain the trade-offs. That is the summit of this course: you now know the language from print("Hello") to ownership, macros, and structured concurrency.

Stop studying and go build something real. Contribute to an open-source Swift project, port a tool you rely on, or ship the app you've been putting off.