Yeah, If I workout the day before diving, particularly if its a harder workout, then my bottom time will suck. Lately I just take that day off, or do yoga.
By the way, yoga has made me much more flexible and improved balance, I feel its improved everything atheletic. Like doing stuff on top of a kayak is quick n easy. I recommend it