I recently found that www.mcmaster.com sells metric drill bits in 0.1mm increments. That's 0.004", which makes it really easy to find the perfect size bit when you need to drill a hole that is just a tiny bit larger or smaller than a "standard" size.
For example, a 9/32" shaft is 0.281" in diameter. So if you needed something that would slip easily over it, you could use a 7.2mm drill which is 0.284".
For the $3 to $4 that a typical bit costs, it could save a lot of frustration.