🔗 Porting Internal Linking to Haskell
🎯 The Mission
🏗️ The internal linking module was the last major stub in the Haskell port.
📝 Its job: traverse the content graph starting from the most recent daily reflection, identify genuine book references using Gemini AI, and insert wikilinks automatically.
🧩 The TypeScript implementation was about 1,000 lines of battle-tested logic that needed a faithful translation into idiomatic Haskell.
🧠 Planning the Approach
📐 Three candidate plans emerged before writing any code.
🔹 Plan one was a direct line-by-line translation from TypeScript. This risked creating unidiomatic Haskell that fights the type system instead of leveraging it.
🔹 Plan two was a full redesign using monadic abstractions and category-theoretic patterns. While elegant, it risked behavioral divergence from the well-tested TypeScript implementation.
🔹 Plan three, the chosen path, balanced fidelity with idiom. Translate the core algorithms faithfully, but express them using standard Haskell patterns: pure functions for text processing, IORef-based BFS for traversal, and algebraic data types for results.
🏗️ Architecture at a Glance
📚 The module defines four key data types: ContentEntry for book metadata, LinkCandidate for discovered title matches, FileResult for per-file outcomes, and LinkingResult for the aggregate run summary.
🔍 The pipeline flows through five stages. First, buildContentIndex scans the books directory for markdown files and extracts titles from frontmatter, stripping emojis to create plain search targets. Second, bfsTraversal starts from the most recent reflection file and follows wikilinks and markdown links through traversable directories. Third, for each visited file, maskProtectedRegions replaces frontmatter, code blocks, inline code, existing links, headings, URLs, and bold markers with space characters, preserving exact character positions. Fourth, Gemini AI identifies which books are genuinely referenced versus mere word coincidences. Fifth, applyReplacements inserts wikilinks from end to start to maintain correct character offsets.
🎭 The Masking Challenge
⚠️ The biggest surprise was regex compatibility. The TypeScript implementation relies heavily on non-greedy quantifiers and character class shortcuts that POSIX Extended Regular Expressions do not support.
🔧 The solution was a hybrid approach. Simple patterns like inline code and URLs still use regex-tdfa. Complex patterns like frontmatter blocks, fenced code, markdown links, and wikilinks use hand-written Text-based parsers that scan for delimiters directly. This actually improved reliability since delimiter-matching parsers cannot accidentally match across boundaries.
✅ A property-based test confirms the critical invariant: maskProtectedRegions always produces output of exactly the same length as its input. Every character position in the original maps to the same position in the masked version.
🤖 Gemini Integration
🧪 The Gemini prompt asks the AI to identify books genuinely referenced as literary works, not just words that happen to match titles. For example, the word “diplomacy” in a sentence about international relations should not trigger a link to a book titled Diplomacy.
🔄 Rate limit handling uses exponential backoff starting at five seconds, doubling up to sixty seconds, with three retry attempts. Daily quota exhaustion is detected and halts the pipeline gracefully.
📦 The response parser handles Gemini’s sometimes messy output: stripping markdown code fences, finding the outermost JSON array brackets, and falling back gracefully on parse failures.
📊 Test Coverage
✅ The test suite adds 49 new tests covering constants, emoji stripping, regex escaping, wikilink formatting, context extraction, all eight categories of protected region masking, link detection, candidate discovery, body extraction, analysis tracking, replacement application, prompt construction, and four property-based tests.
🔬 Property tests verify that masking preserves length, emoji stripping never increases length, wikilinks always contain the entry title, and replacement with all-false validations returns the original content unchanged.
🔑 Key Design Decisions
🚫 No mutable variables outside IO. All text processing functions are pure.
📏 Character position preservation is the fundamental invariant. Masking replaces with spaces rather than removing text. Replacements proceed from end to start.
🧮 Longest-match-first ordering prevents shorter titles from stealing matches that belong to longer ones. If both “Fast and Slow” and “Thinking, Fast and Slow” appear in the index, the longer title gets priority.
🔒 Conservative linking philosophy: better to miss a genuine reference than insert a wrong link. Gemini acts as a gatekeeper, and the pipeline respects a maximum inference request limit of one file per run.
📖 Book Recommendations
🔹 Similar
- 📕 Real World Haskell by Bryan O’Sullivan, Don Stewart, and John Goerzen
- 📗 Haskell Programming from First Principles by Christopher Allen and Julie Moronuki
- 📘 Algorithm Design Manual by Steven Skiena
🔸 Contrasting
- 📕 The Pragmatic Programmer by David Thomas and Andrew Hunt
- 📗 Working Effectively with Legacy Code by Michael Feathers
🔮 Creatively Related
- 📕 Thinking, Fast and Slow by Daniel Kahneman
- 📗 The Design of Everyday Things by Don Norman
- 📘 Godel, Escher, Bach by Douglas Hofstadter