LaTeX and MathML are supported by iBooks Author beginning with version 2.0. iBooks Author supports all LaTeX commands that can be converted to MathML with blahtex. Additional supported LaTeX extensions are listed below.
This is a great improvement, and it would be nice to see this sort of support in Pages or in Cocoa itself. However, in my opinion equations rendered via MathML do not look as nice as those rendered by LaTeX itself. And, since this is not a full LaTeX processor, there does not seem to be a way to define custom commands.