When you look towards the horizon you are looking through a much greater thickness of air. The air does absorb some light. Dense air near surface absorbs more, and if you look towards the horizon you are looking thought a great distance of dense air.
It is not "pollution" per se, though atmospheric aerosols and smoke can exacerbate the effect. Water vapour also absorbs light, and of course any haze or mist will absorb more.
If there are wildfires, dust or air pollution, the "extinction" of stars will be greater. Smoke can travel very long distances (I've known smoke from wildfires in Spain to have a very noticeable effect in the UK). In deserts, there can be windblown dust that has a similar effect. However, even with pristine air, the extinction of stars near the horizon will still occur.