Not sure if by "fail" you mean the <xsl:when test tests as false or you
get a type error reported. And also not sure if by "an integer" you
mean
a value of that type in the xpath type system or something that looks
like an intger, eg "123".

  <xsl:template match="mods:extent[@unit='page']">
    <xsl:param name="prefix"/>
    <xsl:param name="suffix"/>
    <xsl:value-of select="$prefix"/>
    <span class="pages">
      <xsl:text> pp. </xsl:text>
      <xsl:value-of select="mods:start"/>
      <xsl:if test="mods:end">
        <xsl:text></xsl:text>
        <xsl:value-of
select="bib:number-condense(mods:start,mods:end)"/>
      </xsl:if>
    </span>
    <xsl:value-of select="$suffix"/>
  </xsl:template>

  <xsl:function name="bib:number-condense">
    <xsl:param name="begin"/>
    <xsl:param name="end"/>
    <xsl:choose>
      <xsl:when test="$begin castable as xs:integer">
        <xsl:variable name="begin" select="$begin" as="xs:integer"/>
        <xsl:choose>
          <xsl:when test="$begin gt 100 and $begin mod 100 and $begin
idiv 100 eq $end idiv 100">
            <xsl:value-of select="$end mod 100"/>
          </xsl:when>
          <xsl:otherwise>
            <xsl:value-of select="$end"/>
          </xsl:otherwise>
        </xsl:choose>
      </xsl:when>
      <xsl:otherwise>
        <xsl:value-of select="$end"/>
      </xsl:otherwise>
    </xsl:choose>
  </xsl:function>

It has not failed yet (as in, Saxon never stops in its tracks with a
type error, and it formats the page ranges as I expect them to be
formatted), but it's not been widely tested.