2026-04-05 | 🖼️ Expanding the Image Backfill Horizon 🌅
🎯 The Mission
🔍 Every piece of published content on this site deserves a cover image, but until now the automated image backfill system only covered five content directories: reflections, ai-blog, and the three blog series.
📚 Meanwhile, nearly a thousand books, dozens of articles, software projects, topics, and more sat quietly in the library without any visual identity.
🚀 Today we quadrupled the throughput and expanded the reach of the image generation pipeline to cover nearly all published content types.
🔧 What Changed
⬆️ Quadrupled the Rate
🎛️ The image backfill task runs every hour, around the clock, and previously generated one image per run.
📈 We bumped that to four images per run, taking us from twenty-four images per day to ninety-six.
📂 Eight New Content Types
🆕 Eight library content directories are now eligible for cover image generation: articles, books, bot-chats, games, products, software, tools, and topics.
🚫 Three content types remain excluded for good reasons. Videos already use the video embed itself as a visual cover, people are excluded because we do not want AI hallucinating likenesses of real humans, and presentations are collections of slides that may benefit from manually chosen images.
🧩 A Domain-Specific File Predicate
🔀 The old scanning logic required every eligible file to have a date prefix in the format year-month-day. That works great for blog posts and reflections, but library content like books and articles use descriptive filenames instead.
✨ We introduced a new predicate called shouldHaveImage that accepts any markdown file not on the exclusion list, regardless of whether it has a date prefix. Dated blog content still gets sorted to the front of the processing queue, so newer posts continue to receive priority treatment.
🛡️ An important safety guard remains in place: today’s daily reflection is never eligible for a cover image until it receives a creative title, which happens around 10 PM Pacific. The checkCandidate function skips any reflection whose title is still the bare date.
📊 Enhanced Logging
🔍 We improved the logging throughout the image generation pipeline to give clear, data-centric visibility into what is happening at each step.
📈 Progress messages now show the current image count relative to the session maximum, like one of four or three of four, so you can see exactly how far along each run is.
⚠️ Quota exhaustion and provider switching events include the provider index and total count, making it easy to understand how the fallback chain is being utilized.
🛑 Terminal conditions like all providers exhausted or no more candidates now log a summary of generated, skipped, and error counts.
❌ Error messages include the specific file that failed, not just the error text, making it straightforward to diagnose issues.
📊 By the Numbers
🔢 Here is a breakdown of the current backlog across content types.
📝 In the blog content category, reflections account for about 264 files needing images, ai-blog has about 4, and the three blog series are fully covered with zero remaining.
📚 In the library content category, books dominate with roughly 955 files, followed by topics at 87, articles at 82, bot-chats at 49, software at 33, products at 6, games at 1, and tools at 1.
🧮 The grand total is approximately 1,478 files needing cover images.
⏱️ At ninety-six images per day, the entire backlog should be cleared in about fifteen days, or roughly two weeks.
💰 Free Tier Quota Analysis
🆓 The image generation pipeline uses a provider chain with automatic fallback, and several of these providers offer generous free tiers.
🤗 Hugging Face provides around twenty to fifty images per day on their free tier, which amounts to roughly ten cents per month of equivalent compute.
🤝 Together AI offers a free FLUX.1-schnell endpoint that is rate-limited but generally supports fifty to two hundred images per day.
🌸 Pollinations.ai requires no API key at all and is community-supported with effectively unlimited but variable reliability.
🧠 Google Gemini has a free tier that supports roughly fifty to one hundred image generation requests per day.
📊 Combined across all providers, the free capacity is estimated at 150 to 350 or more images per day. At our current rate of ninety-six images per day, we are comfortably within free tier limits and have headroom to increase further if desired.
📝 Logging Guidelines
✍️ While working on this change, we also updated the project’s AGENTS.md to codify logging best practices: logs should be thorough but not spammy, easy for a human to read and understand including the decisions being made in code, data-centric, and maintain a high signal-to-noise ratio.
🏁 Summary
🎉 This change expands the scope of the image backfill system from five content directories to thirteen, quadruples the generation rate, adds detailed logging for quota and rate-limit observability, and sets us on track to have every piece of published content wearing a unique cover image within about two weeks.
📚 Book Recommendations
📖 Similar
- Designing Data-Intensive Applications by Martin Kleppmann is relevant because it explores the architecture of systems that process data at scale, much like our provider chain processes images across multiple APIs with fallback logic and rate limiting.
- Release It! by Michael T. Nygaard is relevant because it covers patterns for building resilient production systems including circuit breakers and rate limiters, which are central to how the image backfill pipeline handles quota exhaustion across providers.
↔️ Contrasting
- The Art of Doing Nothing by Véronique Vienne offers a perspective that sometimes restraint and simplicity are more valuable than automation, a useful counterpoint to a system that relentlessly generates images for every piece of content.
🔗 Related
- Automating Inequality by Virginia Eubanks explores the societal implications of automated decision-making systems, relevant to our deliberate choice to exclude AI-generated images of people to avoid hallucinated likenesses.
- 🧑💻📈 The Pragmatic Programmer: Your Journey to Mastery by David Thomas and Andrew Hunt covers engineering best practices like self-documenting code and data-driven design that align with the logging guidelines we codified in this change.