Snapshot any host's robots.txt. Each snapshot is hashed; we only store the body when it actually changes. Catches accidental Disallow: / disasters before they hurt indexing.