🛡️ Never Again: Multi-Layered Safeguards Against Vault Data Loss
🌪️ The Aftermath
🔥 Hours after the catastrophic data loss incident documented in the previous post, the vault owner recovered their files from another device and restored their vault. 😮💨 A close call with years of notes nearly wiped out by a bidirectional sync on a partial cache.
🧱 This post documents the three layers of defense now built into both the Haskell and TypeScript implementations to ensure this class of failure can never happen again.
🔬 Recap of the Root Cause
🧨 The root cause was deceptively simple. 🔄 When the warm cache sync failed, the code fell back to cold cache mode and ran sync-setup on the existing partial cache directory. 📂 That directory only contained the small subset of files managed by our automation, not the full vault. 🗑️ Bidirectional sync then treated every remote file not present locally as a local deletion and propagated those deletions to the remote vault.
🧹 Layer One: Clean Slate on Cold Cache Fallback
🎯 The most impactful single fix directly addresses the root cause.
📋 Before running sync-setup, the cold cache path now completely removes and recreates the vault directory.
🏗️ In Haskell, the coldCacheSync function now calls removeDirectoryRecursive on the vault directory and then createDirectoryIfMissing to start from a clean empty directory.
🏗️ In TypeScript, the equivalent code uses fs.rmSync with recursive and force flags, followed by fs.mkdirSync to create a fresh directory.
🧠 The reasoning is straightforward. 🆕 If we are running sync-setup, we are starting fresh. 🚫 There should be zero local files influencing what bidirectional sync considers as the local state. ⬇️ With an empty directory, sync will only download remote files, never delete them.
📊 Layer Two: File Count Baseline Tracking
📈 After every successful vault pull, whether warm cache or cold cache, both implementations now record the total number of non-hidden files in the vault to a marker file called .vault-sync-file-count.
📏 This baseline establishes what a healthy vault looks like immediately after pulling from the remote. 🔢 Any subsequent push operation can compare the current file count against this baseline to detect anomalies.
🙈 The count intentionally excludes hidden files and directories (those starting with a dot) since the .obsidian configuration directory and our own marker files should not factor into the health check.
🛑 Layer Three: Pre-Push Circuit Breaker
🔍 Before every push operation, a validation function checks two conditions.
1️⃣ If no baseline exists (first sync), the circuit breaker uses an absolute minimum threshold of 50 files. 📉 A vault with fewer than 50 files is almost certainly not a healthy full vault and pushing from it would be dangerous.
2️⃣ If a baseline exists, the circuit breaker calculates the percentage of files lost since the pull. 🚨 If more than 30 percent of files have disappeared, the push is refused with a descriptive error message.
⚡ When the circuit breaker triggers, it throws an error that halts the push operation entirely. 📢 The error message includes the baseline count, current count, and the threshold that was exceeded, making diagnosis straightforward.
🧮 Choosing the Thresholds
🔢 The minimum safe file count of 50 is deliberately conservative. 📚 A vault with years of notes should have thousands of files. 🎯 Fifty is low enough to never trigger on a legitimately small vault but high enough to catch the scenario where only automation-managed blog posts exist locally.
📐 The 30 percent maximum drop threshold accounts for normal variation. 🆕 The automation adds files (blog posts, images, updated frontmatter) but should never delete files in bulk. 📉 A drop of more than 30 percent strongly indicates something has gone wrong with the local vault state.
🔄 Defense in Depth
🛡️ These three layers operate independently and catch different failure modes.
🧹 Layer one prevents the specific root cause by ensuring cold cache sync always starts from a clean directory.
📊 Layer two detects drift by establishing what normal looks like.
🛑 Layer three acts as a last line of defense by refusing to propagate destructive changes regardless of how the local state got corrupted.
🤝 Even if layer one somehow fails, layers two and three would catch the resulting file loss before it reaches the remote vault.
🏗️ Implementation Parity
✅ Both the Haskell and TypeScript implementations have identical safeguards.
🔧 The Haskell version uses removeDirectoryRecursive, doesDirectoryExist, listDirectory, and doesFileExist from System.Directory.
🔧 The TypeScript version uses fs.rmSync, fs.existsSync, fs.readdirSync, and fs.statSync from the Node.js fs module.
📊 Both implementations share the same thresholds: 50 files minimum, 30 percent maximum drop.
🧪 All 245 Haskell tests and all 1298 TypeScript tests continue to pass.
📝 Lessons Learned
🧠 Bidirectional sync is inherently dangerous when the local state may be incomplete. 🛡️ The defense must operate at multiple levels because any single safeguard can fail.
📏 File count tracking is a cheap but effective canary for vault health. ⏱️ It adds negligible overhead to every sync operation but provides a critical safety net.
🚫 Never run sync-setup on a directory that contains stale state from a previous sync. 🧹 Either clear it completely or verify it thoroughly.
🔒 The most important principle: never push unless you can verify that what you are pushing is safe. 🛑 Circuit breakers that refuse to act are far better than circuits that propagate destruction.
📚 Book Recommendations
Similar
- 📘 Release It! Design and Deploy Production-Ready Software by Michael Nygard
- 📘 Site Reliability Engineering by Betsy Beyer, Chris Jones, Jennifer Petoff, and Niall Richard Murphy
- 📘 The Art of Monitoring by James Turnbull
Contrasting
- 📘 Antifragile: Things That Gain from Disorder by Nassim Nicholas Taleb
- 📘 Normal Accidents: Living with High-Risk Technologies by Charles Perrow
Creatively Related
- 📘 The Design of Everyday Things by Don Norman
- 📘 Thinking in Systems: A Primer by Donella Meadows