12. Capstone: Parallel File Indexer

Goal

Build a production-quality concurrent application end to end: a full-text search tool that walks a directory tree, tokenizes thousands of files in parallel, builds an inverted index, and answers multi-word queries ranked by TF-IDF.

This exercises: RAII and value semantics, move semantics, std::jthread and atomics, the memory model, data-race-free design, std::filesystem, algorithms and ranges, std::format, CMake packaging, Catch2, sanitizers (ASan, UBSan, TSan), and performance measurement.

What it does

$ indexer -j 4 -n 3 /usr/include mutex condition_variable
indexed 4151 files (38.3 MiB, 0 skipped), 261931 terms
scan 10.9 ms | build 384.1 ms | search 0.08 ms

 0.16623  /usr/include/c++/13/bits/std_mutex.h
 0.03863  /usr/include/c++/13/bits/atomic_futex.h
 0.01951  /usr/include/x86_64-linux-gnu/c++/13/bits/stdc++.h

Measured on a 4-core machine (Release build):

Threads	Build time	Speedup
1	1136 ms	1.0×
2	487 ms	2.3×
4	373 ms	3.0×
8	351 ms	3.2×

(The 1-thread run also pays for a cold file cache; always repeat measurements.) Why doesn't it scale perfectly? The single-threaded merge phase and disk I/O are serial — Amdahl's law in action. Improving that is one of the stretch goals.

Requirements

1. indexer [-j THREADS] [-n TOP] DIR QUERY...
2. Recursively collect files with configured extensions; skip unreadable files and files over a size limit without failing.
3. Tokenize: lowercase ASCII words of [a-z0-9_], at least 2 characters.
4. Tokenize files in parallel across -j threads (default: hardware concurrency).
5. Build an inverted index term → [(document, count)].
6. Queries use AND semantics and are ranked by TF-IDF, ties broken by path.
7. Determinism: the index and results must be identical for any thread count.
8. Report timings for each phase to stderr; results to stdout.
9. Exit code: 0 matches found, 1 no matches, 2 usage or fatal error.
10. Clean under ASan + UBSan and TSan, warning-free with -Wall -Wextra -Wpedantic -Wshadow -Wconversion.

Design

            collect_files (serial, sorted)
                        │
          ┌─────────────┼─────────────┐
          ▼             ▼             ▼
      worker 1      worker 2  ...  worker N      ← map: tokenize files,
     (claims files through one atomic counter)      write results[i]
          └─────────────┼─────────────┘
                        ▼
            merge in file order (serial)         ← reduce: add_document
                        │
                        ▼
                  Index::search

Key decisions:

• Map/reduce with no shared mutable state. Each worker writes only to results[i] for the files it claimed. Distinct elements of a vector are distinct memory locations, so there is no data race and no lock.
• Work distribution by an atomic counter (fetch_add). Workers that get small files simply claim more — natural load balancing without a task queue.
• std::jthread joins at the end of a scope. Thread completion happens-before join() returns, so the merge phase safely reads every result. memory_order_relaxed on the counter is enough: it only hands out indices; the join provides the synchronization for the data.
• Merge in file order so document ids — and therefore tie-breaking and output — never depend on thread scheduling.
• The index is immutable after building, so concurrent search calls would be safe without locks (a stretch goal uses this).

Project layout

indexer/
├── CMakeLists.txt
├── src/
│   ├── CMakeLists.txt
│   ├── main.cpp                  # CLI: args, timing, output
│   └── indexer/
│       ├── tokenize.hpp / .cpp   # text → term counts (pure)
│       ├── index.hpp / .cpp      # inverted index + ranking (pure, single-threaded)
│       └── build.hpp / .cpp      # filesystem + parallelism
└── tests/
    ├── CMakeLists.txt
    ├── test_tokenize.cpp
    ├── test_index.cpp
    └── test_build.cpp            # includes a serial-vs-parallel equivalence test

Concurrency lives in one function (build_index). Everything else is ordinary single-threaded code that's easy to test.

Build files

CMakeLists.txt

cmake_minimum_required(VERSION 3.24)
project(indexer VERSION 1.0.0 LANGUAGES CXX)

set(CMAKE_CXX_STANDARD 20)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
set(CMAKE_CXX_EXTENSIONS OFF)

find_package(Threads REQUIRED)

add_subdirectory(src)

if(PROJECT_IS_TOP_LEVEL)
  include(CTest)
  if(BUILD_TESTING)
    add_subdirectory(tests)
  endif()
endif()

src/CMakeLists.txt

add_library(indexer_lib
  indexer/tokenize.cpp
  indexer/index.cpp
  indexer/build.cpp
)
add_library(indexer::lib ALIAS indexer_lib)
target_include_directories(indexer_lib PUBLIC ${CMAKE_CURRENT_SOURCE_DIR})
target_link_libraries(indexer_lib PUBLIC Threads::Threads)
target_compile_options(indexer_lib PUBLIC
  $<$<CXX_COMPILER_ID:GNU,Clang,AppleClang>:-Wall -Wextra -Wpedantic -Wshadow -Wconversion>
  $<$<CXX_COMPILER_ID:MSVC>:/W4>
)

add_executable(indexer main.cpp)
target_link_libraries(indexer PRIVATE indexer_lib)

tests/CMakeLists.txt

include(FetchContent)
FetchContent_Declare(
  Catch2
  GIT_REPOSITORY https://github.com/catchorg/Catch2.git
  GIT_TAG        v3.7.1
)
FetchContent_MakeAvailable(Catch2)

add_executable(tests test_tokenize.cpp test_index.cpp test_build.cpp)
target_link_libraries(tests PRIVATE indexer_lib Catch2::Catch2WithMain)

include(Catch)
catch_discover_tests(tests)

Tokenizing

src/indexer/tokenize.hpp

#pragma once

#include <string>
#include <string_view>
#include <unordered_map>

namespace indexer {

// Word -> number of occurrences in one document.
using TermCounts = std::unordered_map<std::string, int>;

// Split text into lowercase ASCII words ([a-z0-9_]+), dropping words shorter
// than min_length, and count them.
TermCounts count_terms(std::string_view text, std::size_t min_length = 2);

// Lowercase a single query term the same way the tokenizer does.
std::string normalize(std::string_view word);

}  // namespace indexer

src/indexer/tokenize.cpp

#include "indexer/tokenize.hpp"

#include <cctype>

namespace indexer {

namespace {

bool is_word_char(unsigned char c) { return std::isalnum(c) || c == '_'; }

char lower(unsigned char c) { return static_cast<char>(std::tolower(c)); }

}  // namespace

TermCounts count_terms(std::string_view text, std::size_t min_length) {
    TermCounts counts;
    std::string word;
    auto flush = [&] {
        if (word.size() >= min_length) {
            ++counts[word];
        }
        word.clear();   // keeps capacity: no reallocation per word
    };
    for (char ch : text) {
        auto c = static_cast<unsigned char>(ch);
        if (is_word_char(c)) {
            word += lower(c);
        } else if (!word.empty()) {
            flush();
        }
    }
    flush();
    return counts;
}

std::string normalize(std::string_view word) {
    std::string out;
    out.reserve(word.size());
    for (char ch : word) {
        out += lower(static_cast<unsigned char>(ch));
    }
    return out;
}

}  // namespace indexer

Note word.clear() keeps the string's capacity, so the hot loop doesn't allocate for every word.

The index

src/indexer/index.hpp

#pragma once

#include <cstdint>
#include <filesystem>
#include <string>
#include <string_view>
#include <unordered_map>
#include <vector>

#include "indexer/tokenize.hpp"

namespace indexer {

using DocId = std::uint32_t;

struct Posting {
    DocId doc;
    int count;
};

struct Hit {
    std::filesystem::path path;
    double score;
};

// An immutable-after-build inverted index: term -> documents containing it.
// Not thread-safe for concurrent add_document; build.cpp adds from one thread.
class Index {
public:
    DocId add_document(std::filesystem::path path, const TermCounts& counts);

    // Documents containing ALL terms, ranked by TF-IDF, best first.
    std::vector<Hit> search(const std::vector<std::string>& terms, std::size_t limit = 10) const;

    std::size_t document_count() const { return paths_.size(); }
    std::size_t term_count() const { return postings_.size(); }

private:
    std::vector<std::filesystem::path> paths_;
    std::vector<int> doc_lengths_;   // total terms per document
    std::unordered_map<std::string, std::vector<Posting>> postings_;
};

}  // namespace indexer

src/indexer/index.cpp

#include "indexer/index.hpp"

#include <algorithm>
#include <cmath>
#include <unordered_map>

namespace indexer {

DocId Index::add_document(std::filesystem::path path, const TermCounts& counts) {
    auto id = static_cast<DocId>(paths_.size());
    paths_.push_back(std::move(path));
    int length = 0;
    for (const auto& [term, count] : counts) {
        postings_[term].push_back({id, count});
        length += count;
    }
    doc_lengths_.push_back(length);
    return id;
}

std::vector<Hit> Index::search(const std::vector<std::string>& terms, std::size_t limit) const {
    if (terms.empty()) {
        return {};
    }

    // Gather the posting list of every term; any missing term means no results.
    std::vector<const std::vector<Posting>*> lists;
    for (const auto& raw : terms) {
        auto it = postings_.find(normalize(raw));
        if (it == postings_.end()) {
            return {};
        }
        lists.push_back(&it->second);
    }

    // Score documents: sum over terms of tf * idf.
    const auto n_docs = static_cast<double>(paths_.size());
    std::unordered_map<DocId, std::pair<std::size_t, double>> scores;   // doc -> (terms matched, score)
    for (const auto* list : lists) {
        const double idf = std::log(1.0 + n_docs / static_cast<double>(list->size()));
        for (const auto& [doc, count] : *list) {
            const double tf = count / static_cast<double>(doc_lengths_[doc]);
            auto& [matched, score] = scores[doc];
            ++matched;
            score += tf * idf;
        }
    }

    std::vector<Hit> hits;
    for (const auto& [doc, entry] : scores) {
        if (entry.first == lists.size()) {   // AND semantics: every term present
            hits.push_back({paths_[doc], entry.second});
        }
    }

    auto by_score = [](const Hit& a, const Hit& b) {
        return a.score != b.score ? a.score > b.score : a.path < b.path;   // deterministic ties
    };
    if (hits.size() > limit) {
        std::ranges::partial_sort(hits, hits.begin() + static_cast<std::ptrdiff_t>(limit), by_score);
        hits.resize(limit);
    } else {
        std::ranges::sort(hits, by_score);
    }
    return hits;
}

}  // namespace indexer

TF-IDF in one line: a term matters more in a document when it's a large fraction of that document (term frequency) and when few documents contain it (inverse document frequency).

Parallel build

src/indexer/build.hpp

#pragma once

#include <filesystem>
#include <string>
#include <vector>

#include "indexer/index.hpp"

namespace indexer {

struct BuildOptions {
    unsigned threads = 0;                              // 0 = hardware_concurrency()
    std::vector<std::string> extensions{".txt", ".md", ".cpp", ".hpp", ".h", ".c", ".py"};
    std::uintmax_t max_file_size = 8 * 1024 * 1024;    // skip files larger than this
};

struct BuildStats {
    std::size_t files_indexed = 0;
    std::size_t files_skipped = 0;                     // unreadable or too large
    std::uintmax_t bytes_read = 0;
};

// Every regular file under root whose extension is in options.extensions, sorted.
std::vector<std::filesystem::path> collect_files(const std::filesystem::path& root,
                                                 const BuildOptions& options);

// Tokenize files in parallel and build an index. Document ids follow the order of `files`,
// so the result is identical for any thread count.
Index build_index(const std::vector<std::filesystem::path>& files, const BuildOptions& options,
                  BuildStats* stats = nullptr);

}  // namespace indexer

src/indexer/build.cpp

#include "indexer/build.hpp"

#include <algorithm>
#include <atomic>
#include <fstream>
#include <iterator>
#include <optional>
#include <system_error>
#include <thread>

namespace fs = std::filesystem;

namespace indexer {

namespace {

// Result of tokenizing one file; empty optional = skipped.
struct FileResult {
    std::optional<TermCounts> counts;
    std::uintmax_t bytes = 0;
};

FileResult process_file(const fs::path& path, std::uintmax_t max_size) {
    std::error_code ec;
    auto size = fs::file_size(path, ec);
    if (ec || size > max_size) {
        return {};
    }
    std::ifstream in{path, std::ios::binary};
    if (!in) {
        return {};
    }
    std::string text(static_cast<std::size_t>(size), '\0');
    in.read(text.data(), static_cast<std::streamsize>(size));
    text.resize(static_cast<std::size_t>(in.gcount()));
    return {count_terms(text), text.size()};
}

}  // namespace

std::vector<fs::path> collect_files(const fs::path& root, const BuildOptions& options) {
    std::vector<fs::path> files;
    auto wanted = [&](const fs::path& p) {
        return std::ranges::find(options.extensions, p.extension().string()) !=
               options.extensions.end();
    };
    if (fs::is_regular_file(root)) {
        files.push_back(root);
        return files;
    }
    const auto opts = fs::directory_options::skip_permission_denied;
    for (const auto& entry : fs::recursive_directory_iterator{root, opts}) {
        std::error_code ec;
        if (entry.is_regular_file(ec) && wanted(entry.path())) {
            files.push_back(entry.path());
        }
    }
    std::ranges::sort(files);
    return files;
}

Index build_index(const std::vector<fs::path>& files, const BuildOptions& options,
                  BuildStats* stats) {
    unsigned n_threads = options.threads != 0 ? options.threads
                                              : std::max(1u, std::thread::hardware_concurrency());
    n_threads = static_cast<unsigned>(std::min<std::size_t>(n_threads, std::max<std::size_t>(files.size(), 1)));

    // Map phase: workers claim files through an atomic counter and write to
    // their own slot in `results`. Distinct slots => no locks, no data races.
    std::vector<FileResult> results(files.size());
    std::atomic<std::size_t> next{0};
    {
        std::vector<std::jthread> workers;
        workers.reserve(n_threads);
        for (unsigned t = 0; t < n_threads; ++t) {
            workers.emplace_back([&] {
                for (auto i = next.fetch_add(1, std::memory_order_relaxed); i < files.size();
                     i = next.fetch_add(1, std::memory_order_relaxed)) {
                    results[i] = process_file(files[i], options.max_file_size);
                }
            });
        }
    }   // jthreads join here: every write to `results` happens-before the reads below

    // Reduce phase: single-threaded merge in file order => deterministic ids.
    Index index;
    BuildStats local;
    for (std::size_t i = 0; i < files.size(); ++i) {
        if (results[i].counts) {
            index.add_document(files[i], *results[i].counts);
            ++local.files_indexed;
            local.bytes_read += results[i].bytes;
        } else {
            ++local.files_skipped;
        }
        results[i] = {};   // free memory as we go
    }
    if (stats != nullptr) {
        *stats = local;
    }
    return index;
}

}  // namespace indexer

The executable

src/main.cpp

// indexer: build a full-text index of a directory in parallel, then search it.
#include <charconv>
#include <chrono>
#include <exception>
#include <format>
#include <iostream>
#include <string>
#include <string_view>
#include <vector>

#include "indexer/build.hpp"

namespace fs = std::filesystem;

namespace {

struct Options {
    fs::path root;
    std::vector<std::string> query;
    indexer::BuildOptions build;
    std::size_t top = 10;
};

unsigned parse_unsigned(std::string_view s, std::string_view flag) {
    unsigned value{};
    auto [ptr, ec] = std::from_chars(s.data(), s.data() + s.size(), value);
    if (ec != std::errc{} || ptr != s.data() + s.size()) {
        throw std::invalid_argument(std::format("{} expects a number, got '{}'", flag, s));
    }
    return value;
}

Options parse_args(int argc, char* argv[]) {
    Options opts;
    std::vector<std::string_view> args(argv + 1, argv + argc);
    for (std::size_t i = 0; i < args.size(); ++i) {
        auto arg = args[i];
        auto value = [&]() -> std::string_view {
            if (i + 1 >= args.size()) {
                throw std::invalid_argument(std::format("{} needs a value", arg));
            }
            return args[++i];
        };
        if (arg == "-j" || arg == "--threads") {
            opts.build.threads = parse_unsigned(value(), arg);
        } else if (arg == "-n" || arg == "--top") {
            opts.top = parse_unsigned(value(), arg);
        } else if (arg.starts_with('-')) {
            throw std::invalid_argument(std::format("unknown option: {}", arg));
        } else if (opts.root.empty()) {
            opts.root = arg;
        } else {
            opts.query.emplace_back(arg);
        }
    }
    if (opts.root.empty() || opts.query.empty()) {
        throw std::invalid_argument("missing DIR or QUERY");
    }
    return opts;
}

int run(const Options& opts) {
    using clock = std::chrono::steady_clock;
    auto ms = [](clock::duration d) { return std::chrono::duration<double, std::milli>(d).count(); };

    auto t0 = clock::now();
    auto files = indexer::collect_files(opts.root, opts.build);
    auto t1 = clock::now();
    indexer::BuildStats stats;
    auto index = indexer::build_index(files, opts.build, &stats);
    auto t2 = clock::now();
    auto hits = index.search(opts.query, opts.top);
    auto t3 = clock::now();

    std::cerr << std::format(
        "indexed {} files ({:.1f} MiB, {} skipped), {} terms\n"
        "scan {:.1f} ms | build {:.1f} ms | search {:.2f} ms\n\n",
        stats.files_indexed, static_cast<double>(stats.bytes_read) / (1024.0 * 1024.0),
        stats.files_skipped, index.term_count(), ms(t1 - t0), ms(t2 - t1), ms(t3 - t2));

    if (hits.empty()) {
        std::cout << "no matches\n";
        return 1;
    }
    for (const auto& hit : hits) {
        std::cout << std::format("{:8.5f}  {}\n", hit.score, hit.path.generic_string());
    }
    return 0;
}

}  // namespace

int main(int argc, char* argv[]) {
    try {
        return run(parse_args(argc, argv));
    } catch (const std::invalid_argument& e) {
        std::cerr << "indexer: " << e.what()
                  << "\nusage: indexer [-j THREADS] [-n TOP] DIR QUERY...\n";
        return 2;
    } catch (const std::exception& e) {
        std::cerr << "indexer: " << e.what() << '\n';
        return 2;
    }
}

Tests

tests/test_tokenize.cpp

#include <catch2/catch_test_macros.hpp>

#include "indexer/tokenize.hpp"

using namespace indexer;

TEST_CASE("count_terms lowercases and splits on punctuation") {
    auto counts = count_terms("Hello, hello WORLD! std::vector<int> x");
    CHECK(counts.at("hello") == 2);
    CHECK(counts.at("world") == 1);
    CHECK(counts.at("std") == 1);
    CHECK(counts.at("vector") == 1);
    CHECK(counts.at("int") == 1);
    CHECK_FALSE(counts.contains("x"));   // shorter than min_length
}

TEST_CASE("count_terms handles empty input and trailing words") {
    CHECK(count_terms("").empty());
    CHECK(count_terms("end").at("end") == 1);
}

TEST_CASE("normalize lowercases") {
    CHECK(normalize("MiXeD_Case") == "mixed_case");
}

tests/test_index.cpp

#include <catch2/catch_test_macros.hpp>

#include "indexer/index.hpp"

using namespace indexer;

TEST_CASE("search requires every term and ranks by tf-idf") {
    Index index;
    index.add_document("a.txt", count_terms("apple banana apple"));
    index.add_document("b.txt", count_terms("apple cherry"));
    index.add_document("c.txt", count_terms("banana cherry cherry"));

    SECTION("single term") {
        auto hits = index.search({"apple"});
        REQUIRE(hits.size() == 2);
        CHECK(hits[0].path == "a.txt");   // apple is 2/3 of a.txt, 1/2 of b.txt
        CHECK(hits[1].path == "b.txt");
    }

    SECTION("AND semantics") {
        auto hits = index.search({"banana", "CHERRY"});
        REQUIRE(hits.size() == 1);
        CHECK(hits[0].path == "c.txt");
    }

    SECTION("unknown terms match nothing") {
        CHECK(index.search({"apple", "durian"}).empty());
        CHECK(index.search({}).empty());
    }

    SECTION("limit") {
        CHECK(index.search({"cherry"}, 1).size() == 1);
    }
}

tests/test_build.cpp

#include <catch2/catch_test_macros.hpp>

#include <filesystem>
#include <fstream>
#include <random>

#include "indexer/build.hpp"

namespace fs = std::filesystem;
using namespace indexer;

namespace {

struct TempDir {
    fs::path path;
    TempDir() {
        std::random_device rd;
        path = fs::temp_directory_path() / ("indexer-test-" + std::to_string(rd()));
        fs::create_directories(path);
    }
    ~TempDir() { fs::remove_all(path); }
    TempDir(const TempDir&) = delete;
    TempDir& operator=(const TempDir&) = delete;

    void write(const fs::path& name, const std::string& content) const {
        fs::create_directories((path / name).parent_path());
        std::ofstream{path / name} << content;
    }
};

}  // namespace

TEST_CASE("collect_files filters by extension and sorts") {
    TempDir dir;
    dir.write("b.txt", "x");
    dir.write("sub/a.md", "x");
    dir.write("image.png", "x");

    auto files = collect_files(dir.path, {});
    REQUIRE(files.size() == 2);
    CHECK(files[0].filename() == "b.txt");
    CHECK(files[1].filename() == "a.md");
}

TEST_CASE("parallel build gives the same results as a single-threaded build") {
    TempDir dir;
    std::mt19937 gen{7};
    const std::vector<std::string> vocab{"alpha", "beta", "gamma", "delta", "epsilon", "zeta"};
    std::uniform_int_distribution<std::size_t> pick{0, vocab.size() - 1};
    for (int f = 0; f < 200; ++f) {
        std::string text;
        for (int w = 0; w < 50; ++w) {
            text += vocab[pick(gen)] + ' ';
        }
        dir.write("doc" + std::to_string(f) + ".txt", text);
    }

    auto files = collect_files(dir.path, {});
    BuildOptions serial;
    serial.threads = 1;
    BuildOptions parallel;
    parallel.threads = 8;

    BuildStats stats;
    auto a = build_index(files, serial);
    auto b = build_index(files, parallel, &stats);

    CHECK(stats.files_indexed == 200);
    CHECK(a.term_count() == b.term_count());
    for (const auto& q : std::vector<std::vector<std::string>>{{"alpha"}, {"beta", "gamma"}}) {
        auto ha = a.search(q, 20);
        auto hb = b.search(q, 20);
        REQUIRE(ha.size() == hb.size());
        for (std::size_t i = 0; i < ha.size(); ++i) {
            CHECK(ha[i].path == hb[i].path);
            CHECK(ha[i].score == hb[i].score);
        }
    }
}

TEST_CASE("oversized files are skipped, not fatal") {
    TempDir dir;
    dir.write("big.txt", std::string(1000, 'a'));
    dir.write("small.txt", "hello world");

    BuildOptions opts;
    opts.max_file_size = 100;
    BuildStats stats;
    auto index = build_index(collect_files(dir.path, opts), opts, &stats);
    CHECK(stats.files_indexed == 1);
    CHECK(stats.files_skipped == 1);
    CHECK(index.search({"hello"}).size() == 1);
}

The serial-vs-parallel test is the most important one: it pins down the determinism requirement and, under TSan, exercises the concurrent path with many files.

Build, test, measure

# Release build for measuring
cmake -S . -B build-rel -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=OFF
cmake --build build-rel -j
for j in 1 2 4 8; do ./build-rel/src/indexer -j $j /usr/include mutex > /dev/null; done

# Sanitizer builds: all must pass
cmake -S . -B build-asan -DCMAKE_BUILD_TYPE=Debug \
      -DCMAKE_CXX_FLAGS="-fsanitize=address,undefined -fno-sanitize-recover=all"
cmake --build build-asan -j && ctest --test-dir build-asan --output-on-failure

cmake -S . -B build-tsan -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_FLAGS="-fsanitize=thread"
cmake --build build-tsan -j && ctest --test-dir build-tsan --output-on-failure

Then profile the release build (perf record -g ./build-rel/src/indexer ...) and find out where the time actually goes.

Build it in stages

1. Serial first. count_terms, Index, and a single-threaded build_index with tests. Get correct results before making anything parallel.
2. Parallelize the map phase with std::jthreads and an atomic counter. Add the serial-vs-parallel equivalence test. Run it under TSan.
3. Break it on purpose: have workers push_back into a shared vector without a lock. Watch TSan report it. Then fix it with a mutex, measure, and compare with the slot-per-file design.
4. Measure speedup for 1, 2, 4, 8 threads. Profile. Is it CPU, memory allocation, or I/O?
5. Harden: unreadable files, huge files, binary files, symlink loops, an empty directory, a query with no matches, bad flags.

Stretch goals

• Parallel merge: shard the index by hash(term) % N and let each worker build one shard; search queries all shards. Measure the new speedup — did Amdahl's serial fraction shrink?
• Faster tokenizing: memory-map files (mmap / CreateFileMapping), avoid per-word std::string allocations with std::string_view into the file buffer plus a string pool, or use a faster hash map (boost::unordered_flat_map, absl::flat_hash_map).
• Persistence: serialize the index to a binary file and load it on the next run, with a format version header. Fuzz the loader.
• Interactive mode: build once, then answer queries from stdin — concurrently from a thread pool, relying on the immutable index.
• Phrase queries ("condition variable"): store positions in postings.
• Incremental updates: re-index only files whose modification time changed.
• Package it (lesson 11): install rules, find_package(indexer), a vcpkg port.
• Benchmarks with Google Benchmark for count_terms and search, tracked in CI.

Self-check

• [ ] Explain why build_index has no mutex and no data race, and which operation makes the merge phase's reads safe.
• [ ] Explain why memory_order_relaxed is sufficient for the work counter.
• [ ] All tests pass under ASan+UBSan and under TSan.
• [ ] Output is identical for -j 1 and -j 8.
• [ ] You measured the speedup, profiled the hot spots, and can say what limits scaling.
• [ ] You can add a feature from the stretch goals without introducing a race.

Congratulations — you've finished the course. Stop studying and go build something real.