% Is it possible to transform an XML document to a binary format using
% XSL?
Not really. A lot could be done using numeric entity references and
extension functions, but the goal of xslt is to produce result trees
consisting of text, elements, and other things that can be found in
typical XML files.
% Can one use XSL-FO to do this?
It really depends on the binary format you're talking about. If it's,
say, the binary format expected as input to a laserjet, you could use
xsl:fo, with appropriate formatting software, to do the job. If you
want to represent the data from an XML file in a particular binary
format (say, a word-processing format), then xsl:fo is not going to
help in any way.