For want of a press

Drilling a hole through a piece of wood seems simple. But I keep doing it wrong. I've been trying to make 1cm holes to fit my 1cm dowels. Since I don't have a drill press, it is impossible (at least with my level of skill) to be perfectly vertical and stable while drilling. This means that the side where I started is slightly larger than 1cm. The dowels still fit snugly but you can see the shadow around them. On the back side, the wood kept splintering as the large bit poked through. I tried putting a piece of tape, but that didn't help. I managed one or two times to drill from both sides, but that meant that both sides had that slightly larger radius from the drilling inaccuracies. 
However, the shadow line disappears after sanding because it gets filled with sawdust.