Chat Wars
David Auerbach (via Nitesh Dhanjani and Nicolas Seriot):
Our client just ignored it, but the AOL client responded to this gobbledygook with a shorter version of the same gobbledygook. I didn’t know what it was. It was maddening. After staring at it for half a day, I went over to Jonathan, a brilliant server engineer on our team, and asked what he thought. He looked at it for a few minutes and said, “This is code.” As in, actual x86 assembly code. The repeated 90s were what tipped him off: they signify an empty instruction in x86 Assembler.
The pieces then came together. Normally, these protocol messages sent from the server to the client are read and understood as data, not as code. But AOL’s client had a security bug in it, called a buffer overflow. The buffer is a place where a program temporarily stores data while running some operation. However, it’s all too easy in lower-level languages to allow in more input than the buffer can actually accommodate. In this case, very large protocol messages could flood it, overwriting the client code and arbitrarily controlling the functioning of the client program—this is why it’s called a buffer overflow, and it’s a huge security hole, since it gives the server control of the client PC. In the wrong hands, the server can choose to shut down or corrupt or do other terrible things to your computer. AOL knew about this bug in their program and now they were exploiting it! That was what all those double zeros were for—they were just filling up space in the program’s buffer until they hit the end of the AOL client’s buffer and started overwriting executable code with the remainder of the protocol message. AOL was causing the client to look up a particular address in memory and send it back to the server. This was tricky, vastly trickier than anything they’d done so far. It was also a bit outside the realm of fair play: exploiting a security hole in their own client that our client didn’t have!
[…]
Someone had the bright idea of telling the press about the buffer overflow, figuring that if people knew that AOL’s client could and in fact was executing whatever a server sent to it, AOL would be forced to patch their client and could no longer use it to determine that Messenger was an impostor.
