I wouldnt really say its going to hinder your gains much, probably sending your bloodsugar levels for a rollercoaster ride though.
Why don't you just cut all those unnecessary calories out? It will definitely help with keeping bodyfat levels down.
We lift weights and manipulate our diets so that we'll look good naked. Sure, it's healthy too, and we'll probably live a longer and more productive life than the average guy, but mostly it's about the naked thing.