How Both TCP and Ethernet Checksums Fail
Evan Jones (via Hacker News):
At Twitter, a team had a unusual failure where corrupt data ended up in memcache. The root cause appears to have been a switch that was corrupting packets. Most packets were being dropped and the throughput was much lower than normal, but some were still making it through. The hypothesis is that occasionally the corrupt packets had valid TCP and Ethernet checksums. One "lucky" packet stored corrupt data in memcache. Even after the switch was replaced, the errors continued until the cache was cleared.
I was very excited to hear about this error, because it is a real-world example of something I wrote about seven years ago: The TCP checksum is weak. However, the Ethernet CRC is strong, so how could a corrupt packet pass both checks? The answer is that the Ethernet CRC is recalculated by switches. If the switch corrupts the packet and it has the same TCP checksum, the hardware blindly recalculates a new, valid Ethernet CRC when it goes out.
There is an RFC that compares different checksumming algorithms which was written when iSCSI was being finalized (~2002):
https://tools.ietf.org/html/rfc3385
Another interesting RFC is how the checksum was changed after SCTP was supposedly “finalized”, but before it became wide spread (relatively speaking, as it’s not really used a lot):
https://tools.ietf.org/html/rfc3309
They went from Adler-32 to CRC-32c. iSCSI can negotiate the algorithm, but CRC-32c is a MUST in the base RFC.
Back in 2004 Koopman and Chakravarty did a survey for use in embedded networks:
http://users.ece.cmu.edu/~koopman/roses/dsn04/koopman04_crc_poly_embedded.pdf