I am converting a very large XML file to CSV (e.g. GB in size which
can't fit into memory), I am not sure if it is the correct way to use
XSL to do my job.

Currently I have a Java program which parse the XML file (SAX) and
write to a file.

For a very large file I think SAX is better as it is not constrained by
memory. Otherwise XSL might be convenient. Or be considered overkill.

In any case, I'd reuse any CSV writing functionality that is available
instead of trying to handle that part in XSL. So I have a Perl script
call an XSL transformation which then calls back into the host language
to use the CSV writing functionality bundled with Perl. So the role of
XSL in this scenario is ancillary: gather values and pass them to Perl.