My rule of thumb is to rest as long as you need to, but never as long as you want to.
I likely don't rest enough, especially between some bigger lifts, but I like to think that I am increasing my endurance as well by not having long periods of rest between sets.
"Look son, there comes a time in every man's life when he has to make a decision. Do you want to be big, powerful, jacked, yoked-up, have women everywhere want you and men fear you . . . or do you want to do crossfit?"