How do you like your coffee? I like mine to have a mule's kick in it. That's fine...

According to The Complete Guide to Coffee Grind Size, the grind size’s impact has to do with physical processes taking place on a fairly small scale during brewing. When water is trying to get those flavor particles out of an individual coffee ground and into your cup, it can’t just magically teleport to the center of the ground. It has to start from the outside of that coffee particle and wash away those soluble coffee solids first.

So, say you have one coffee particle and you cut it in half. You’re left with the same amount of total coffee, but there’s now a whole lot more surface area from the inside of that particle that the water immediately has access to. The more pieces you divide that coffee particle into, the more surface area will be exposed (while the total mass of coffee stays the same). So in any brew method, finer coffee grounds will extract faster, while coarser coffee grounds extract slower.