The normal extension language for the java XSLT engines is java rather
than javascript (but you need to check the docs for each processor)

Xalan can be extended using JavaScript. It does so, of
course, in a slightly different way that MSXML. You'll
need bsf.jar (the Bean Scripting Framework) an a JavaScript
implementation (i used rhino http://www.mozilla.org/rhino/)
I think bsf.jar is already distributed with Xalan.

Here is an example on how to define and use JavaScript
functios in Xalan:

<?xml version='1.0' encoding='ISO-8859-1' ?>
<xsl:stylesheet version="1.0"
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  xmlns:lxslt="http://xml.apache.org/xslt"
  xmlns:counter="ext1"
  extension-element-prefixes="counter">
  <lxslt:component prefix="counter" functions="count">
    <lxslt:script lang="javascript">
   var i = 1;
   function count() {
      i++;
      return i;
   }
    </lxslt:script>
  </lxslt:component>

  <xsl:template match="/">
    <xsl:value-of select="counter:count()" />
    <xsl:text>&#x0A;</xsl:text>
    <xsl:value-of select="counter:count()" />
  </xsl:template>
</xsl:stylesheet>

Note that you should really know what you are doing,
your stylesheets will be non-protable and have a good
chance to grow into a maintenance nightmare.