Currently my sanitizing function just escapes <, >, ', and " in the

If you are taking in a string and want to ensure that it is encoded in
XML as itself (in character data) rather than markup then you need
to escape < and & (and > if it follows ]]) you don't need to escape " or
' unless you are putting the string in attribute values.

Are these characters recognized by the XSLT engine
if they are hex or unicode encoded?

All XML text is unicode encodes in one way or another, so it's not quite
clear what you mean there. Encoding issues are resolved by the XML
parser before XSLT really sees the input. If you are taking unknown text
you should be escaping & as &amp; so then a character ref such as &#a0;
would be escaped tp &amp;#a0;.

but I was wondering if anyone knows of other vectors by which
attackers can enter

attacks are as likely to come from what is inserted into XML character
data as from any XML markup that is inserted. Specifically if the
stylesheets are generating html then if there is a danger of script
being inserted you need to quote (or disable) possible script syntax.