I have a friend/training partner who used to only do arms and chest. A year later he looks back at how stupid it was. He's still got small legs, but he's seen more gains on his upper body since he started training a bit more all around. He's got a better back since he's started deadlifting more often.
Really, I think it's stupid to avoid any part of your body(well, I avoid calves). If you're stronger and bigger all over, that can only be a plus. Plus, a girl may be attracted by your arms, but big arms and small this and that aren't too impressive or manly. Sex has more to do with hip drive too... So yes, it is stupid not to train every body part for multiple reasons.