One other option for destroying atoms is called photodisintegration. When massive stars near the end of their life, just before they blow themselves up in a supernovae, the energy of photons in the core is high enough to smash apart a heavy atom (like iron) into lighter atoms.
There are several processes at work:
Fission (of unstable elements) increases the number of atoms.
Fusion (occurs in stars and supernovae) decreases the number of atoms.
you could claim the formation of neutron stars decreases the number of atoms: it reduces atoms to just neutrons
you could claim the formation of black holes decreases the number of atoms: ...