Engineering notes from maintaining large content archives — published as a dataset and mirrored here as a static page.
One source of truth between the sitemap index and the chunk generator. Includes the
incident write-up from a ~462K-URL archive where the index promised 50 chunks and the
generator produced 43 — and the integrity-over-popularity filter that fixed it
(num_pages ≥ 1, not favorites counts).
Remote incremental crawl, staging table, single-statement
ON CONFLICT ... IS DISTINCT FROM merge. The two footguns that bit us:
unqualified columns in the WHERE clause, and row-by-row updates at volume.
Store base model, sampler, steps, CFG, prompt and LoRA stack as structured data at ingest time — never as a display string. The design rule behind recipe-first AI galleries.