I tried to iterate over the cells and store the number of labels in a
"variable" (i.e. a param), keeping track of the maximum number seen for
any cell, but my attempts to keep track of the maximum value seen so far
aren't working.

You have a wron understanding of XSLT variables and parameters.
Variables aren't actually variable, they can be assigned and
that's it. Parameters are passed to templates or (in case of
global parameters) from the outside into the style sheet.

    <xsl:if test="$vgroup_size > $max_vgroup_size">
      <xsl:param name="max_vgroup_size" select="$vgroup_size"/>

This is actually illegal. Your processor should have raised
an error. A xsl:param element must be an immediate child
of a xsl:template or xsl:stylesheet, in case of the former
it may only be preceeded by other xsl:param elements. You
can't use it elsewhere.

How should I be approaching this? Thanks....

This has been rehashed thousands of times here, check the
archives.
You'll have to use a recursive template, or search for
FXML by Dmitre Novatchev (plenty of references in the
archive).