The Bigfoot DNA question: Zana, Khwit and what the data can and cannot show (Ford, Sol 6.0, Claude Opus 5/5.5)

My copy is byte-identical to the served file at every offset, and the size matches exactly. So ENA's recorded checksum doesn't match the file ENA serves — that's archive metadata, not my corruption. Confirming with the read count.

VK87 is running at 7–8 MB/s now. Fixing the integrity logic while it finishes — md5 against ENA metadata can't be the sole criterion if the archive's own checksum is wrong.

Important reversal — flagstat found a BGZF decode failure at 1.65 GB. So the file is corrupt; quickcheck only validates the header and EOF block, not the middle. ENA's md5 was right and I was wrong.

VK87 is still going. Meanwhile — the BGZF error gave me the exact offset, so I can repair VK6 by refetching just the bad chunk rather than all 9.37 GiB.

Here's an honest position check.

Science: unchanged and solid

Phase 2 is complete and reproduced — coverage, ancient-DNA damage, sex, mtDNA L2b1b in four tool/tree combinations, and the 96-fold opposite-homozygote collapse. Control selection is verified against the Nature supplement, where Table S1's Relatedness column was the real upgrade. No Viking OH numbers yet — that's still the next output.

Logistics: three errors today, all mine, all caught

1. My size estimate was wrong by 3.2×. I used NCBI's size_MB, which is the SRA archive object size, not the submitted BAM. The core set is 27.1 GiB, not the 8.00 GB I told you. ENA's portal API — which serves the correct field, and which I used for Zana/Khwit — was down, and I substituted a field that doesn't mean the same thing.

2. The first VK6 download was corrupt via curl -C - whole-file resume. The md5 caught it.

3. I then wrongly concluded ENA's checksum was unreliable. My copy matched the served bytes at offsets 0, 4 GB, 8 GB and the final 1 MB, the size matched exactly, and samtools quickcheck passed — so I inferred the archive's metadata was stale. Then a full decode hit a BGZF inflate failure 1.65 GB in, an offset my four samples happened to miss. quickcheck only validates the header and the EOF block. ENA's md5 was right and I was wrong; I'd written that bad inference into the manifest and have corrected it there.

What's hardened now

md5 is authoritative again, with a full-decode gate (samtools view -c) behind it. On mismatch the chunk cache is deleted too — a chunk can be the right length with the wrong bytes, and my size-only reuse would have preserved a bad chunk forever. Failed files retry with progressively fewer streams.

Running right now

• VK87: 84 of 136 chunks, 5–8 MB/s. Roughly 15 minutes out.
• VK6: being repaired by refetching only the three chunks around the failure offset — 192 MB instead of 9.37 GiB, since the BGZF error gave me the exact byte.

The open question

Whether parallel range fetching is reliable against this server at all. VK87's md5 is the datum. If it passes, VK6 was a one-off and we're fine. If it also fails, I fall back to single-stream without resume — about 2.3 hours per file instead of 25 minutes, so ~7 hours for the set.

Pipeline is still frozen: --bam and --skip-x predeclared, no threshold touched. Phase 3 locked, Bigfoot unpublished, nothing committed.

I'll report the four-row OH table as soon as the three BAMs verify and decode.

VK6 is fixed and md5 now matches ENA exactly. The diagnosis was precise: chunk 24 was the right length with the wrong bytes, and refetching that single 64 MB chunk repaired the whole 9.37 GiB file.

Here's the current position.

VK6 is repaired and verified

The targeted repair worked exactly as diagnosed: chunk 24 had the right length and the wrong bytes. Refetching that single 64 MB chunk fixed the whole 9.37 GiB file, and the md5 now matches ENA's recorded checksum precisely.

That settles the question I'd got wrong. ENA's checksums are authoritative, and the corrupted inference I'd written into the manifest is corrected there. It also tells us the failure rate: roughly 1 bad chunk in 150, right-length but wrong-content — which is why length checks alone weren't enough.

I've now built that repair into the fetcher. When a md5 fails, it reads the byte offset from the decoder error, maps it to a chunk index, refetches that chunk and its neighbours, reassembles and re-verifies — up to three rounds. So a single bad chunk costs 64 MB instead of a whole-file redownload.

What's running

• VK87: 87 of 136 chunks. Throughput has dropped to ~5 MB/s from 8, so call it 20–30 minutes.
• VK548: queued, 9.27 GiB.
• VK6: full decode and indexing in progress, to confirm the read count against the expected 159,793,762 spots.

I've armed a background watcher on the fetch log, so I'll be notified when the run either verifies all three or fails, rather than polling.

Three corrections now on the record

DISCARDED.md is at 105 rows, with six entries from this download episode alone:

• NCBI's size_MB is the archive object, not the BAM — the source of my wrong 8.00 GB figure. The real core set is 27.1 GiB. Its spots and bases fields were accurate, which is what made the error easy to miss.
• curl -C - resume silently corrupts multi-GB files.
• samtools quickcheck validates only the header and EOF block — it cannot see mid-file damage, and treating it as a verdict is what led me to blame ENA.

Unchanged

No Viking OH numbers yet, so nothing scientific has moved since Phase 2. The pipeline stays frozen — --bam and --skip-x predeclared before any control data existed, no threshold touched. Phase 3 locked, Bigfoot unpublished, nothing committed.

Next output is the four-row table: Zana–Khwit against the three unrelated Viking pairs, OH rate and transversion-only OH and callable sites across bins 2 to 8+, read against your fixed A/B/C rule.

VK6 is fully verified — md5 matches and the read count is exactly 159,793,762, the expected figure. VK87 hit the same bad-chunk problem; its 136 chunks are cached, so let me repair it the cheap way.

ok looks like v6k is fully verified, had the same problem with vk87

Good — then VK6 is done and should be left alone.

If VK87 had the same issue as VK6 — the supposed file-size estimate was way too small and somebody tried to validate it while it was still downloading — I wouldn’t treat that as corruption. Let the resumable curl -C - finish against the actual remote file, then trust only:

1. the authoritative checksum,
2. samtools quickcheck,
3. successful indexing.

So the state is essentially:

VK6: downloaded + verified ✅
VK87: downloading/resuming, previous size expectation unreliable
VK548: probably still waiting behind it

And I’d now completely ignore Claude’s earlier “8 GB total” estimate. The remote files themselves are the authority.

Once VK87 finishes and verifies, we should make sure Claude records the actual byte size in the manifest so we don’t repeat this on VK548.