as long as you can afford to care for them, there is never too many, ragdolls or otherwise
It is recognized that larger colonies of cats have a greater risk for communicable diseases. Somewhat arbitrarily, a multicat household, with such higher risks, is considered five or more cats.