Gravel used on railway tracks is known as ballast. As rail tracks heats and cools, there is a lot of expansion and contraction, the ballast is there to hold the track from moving. Crushed rock works best as it has countless sharp edges form the broken rocks that actually dig into the sides and bottom of the ties and hold it in place.
It is also used to dissipate the vibration produced by trains traveling at high speeds. In effect, the gravel layer acts as a cushion and damps the vibrations so that they do not travel long distances. If the rails are laid on a solid base, these vibrations can travel long distances and lead to cracks on the base as well as on nearby buildings.