I use the following Java options to change the default memory use for both stack area and main area to, respectively, 20Mb and 500Mb:

-Xms20m -Xmx500m

Be aware that such generous settings usually require a
corresponding amount of physical RAM in the machine, in
this case 640MB or more. For most JVMs performance
suffers dramatically once they run out of physical RAM
and start swapping.

If FOP needs all of the physical memory of the machine
and more, tweaking the FO and images are a more promising
strategy to get the job done.