It has to be an inner pursuit, not an external one. If your self-esteem is based on your muscles and your strength, how are you going to feel when you run into the many, many, many guys that are twice as strong as you or twice as big as you. Will these individuals squash your self-esteem?
The pursuit of strength and size should be a life long journey that brings fulfillment in the steps along the way. Do not look for admiration from others to validate your own self worth. There just aren't that many people in your life that will care that you get bigger and stronger, and those that do care just aren't really worth it.
Strength and size are worthy goals, just don't base your self-esteem on such things. Get an education, marry a fine woman, raise respectable kids; these are goals worthy of EVERYBODY's admiration.