If diffusion applies than how come when you are at the top of a mountain there isn't as much oxygen?

Because air has weight and the air way up there pushes down on the air down here, compressing it a considerable amount. That means that the higher you go, the less air there is above you pushing down on the air around you, and so there is less pressure. Its why air planes have to be pressurized.