It also depends on your goals, too. If your goal is fat loss for example, then working out first thing in the morning is probably preferable than later in the day when you will be burning food instead of fat (depending on your diet and workout routine of course). If your goal is maximum strength, then finding your peak energy time of day is probably best as has been mentioned. If your goal is overall fitness, then it would probably be beneficial to always change up your times to force your body into continual adaptations. Etc. Your goals will be your main driver.
But, scheduling is a huge concern as well unless you are a professional athlete. If you have life issues that make it difficult to find the perfect time to workout any given day, so be it. My personal opinion as a pretty busy person is that whenever you can get to the gym is the best time, and while it might not be the absolute best time for me for that, or any, day, at least I got it done.
Better to have many workouts done at the worst times, than few workouts done at the very best times.
Wasn't that Confucious?