No its awesome for gaining size and strength.
The difference is the absorption rate and the effect it has on blood sugar level. Complex carbs are absorbed slower and makes your blood sugar more stable over time. Sugars peak your blood sugar fast and can also cause a rapid drop, leeding to the desire to eat more sweets.
Also by peaking your blood sugar you release insulin to take care of amino acids and blood glucose, insulin is the most anabolic hormone in the body and its also what makes you fat if you combine fast carbs with fats in your diet.
This was just my broscience 0.2cent =)