Hello,
   border-width for table cells in xsl fo seems to
work fine as long as it is greater than 0.8pt. Once I
start decreasing say for example 0.5pt, then the
border width becomes inconsistent in all the rows. Has
anyone seen that problem ?

Some context would be useful in order to judge this:
what tools are you using to process and display the FO?

Most likely you hit known effects in Acrobat Reader, which
seems to be prone to round off errors on monitors. If this
is the case, print it on a good laser.

I tried using px instead of pt, but still the problem
exsists.

Unless you are rendering your FO to bitmaps or computer
monitors exclusively, using px is unwise anyway.