The conditions here are very different...
But I've noticed that fish tend to be a lot more curious when I'm just checking out a stingray on the bottom, or playing with a tiny crab on the bottom, or filming some colorful reef fish with my head mounted gopro. I guess what I'm trying to say is that they get more curious when you're simply "having a look around", as opposed to hiding stealthily.
I've always maintained that fish (like many other animals) can sense your intention and body language very quickly. I suppose I'm lucky that here in the Arabian gulf, fish don't tend to be too scared of divers. We just have to worry about being able to see the fish in the super green and murky waters.