A signature can prove an answer is genuine. But how do you sign "this name does not exist"? DNSSEC answers with a signed record that names the gap: "there is nothing between a.example.com and c.example.com". That is authenticated denial of existence.
NSEC
An NSEC record links each name to the next name in the zone, in sorted order, and lists the record types that exist at that name (RFC 4034). One signed NSEC proves a whole range of names is empty.
a.example.com. IN NSEC c.example.com. A RRSIG NSEC
The cost: anyone can follow the chain and list every name in the zone. This is called zone walking.
NSEC3
NSEC3 (RFC 5155) chains hashes of names instead of the names, so the chain does not list them in clear. Hashes of short names can still be guessed offline, so NSEC3 slows zone walking down; it does not stop it.
NSEC3 has a salt and an iteration count. Extra iterations cost resolvers CPU and add little protection. RFC 9276 recommends 0 extra iterations and an empty salt, and resolvers may treat zones with high counts as insecure.
pdnsutil set-nsec3 example.com '1 0 0 -' # PowerDNS: SHA-1, no opt-out, 0 iterations, no salt
Which one?
- Zone names are public anyway (most websites): NSEC is simple and fine.
- You want to make listing names harder: NSEC3 with the RFC 9276 settings.
- Large delegation-only zones (a TLD): NSEC3 with opt-out keeps the zone small.
Some signers create the answers on the fly ("black lies", compact denial), so a walk finds nothing useful. Cloudflare does this.