This is just a guess...
The term physical culturist would include strength training, conditioning, bodybuilding, gymnatics, healthy eating, stretching, and positive thinking. The guys back then wanted to be well-rounded in every aspect of the physical and mental well-being.
I'm betting when bodybuilding and muscle mass became more important (Thank you Weider - lol) that the term lost favor.
But that's just a guess.