Our best current understanding of the formation of the Moon has it forming quite close to Earth. Because even then its orbit around the Earth took more than one day (Earth was spinning faster, also), the effect of tides has always been for the Earth to transfer some of its rotational energy to the Moon. This slows the Earth's rotation and makes the Moon move outwards into a higher (and slower) orbit.
It's been around four billion years since the Moon last suffered a major impact, so collisions have not played a significant role since then.
Expanding (a lot) on this:
How do we measure Lunar ages? Two ways: Cratering and isotope dating. The cratering technique was developed by William K. Hartmann of the University of Arizona and makes use of the amazingly simple fact that when two craters overlap, you can tell which is older. Combine that with the observed fact that while the cratering rate varies with time, it changes smoothly and the distribution of craters is essentially random. Using that and a lot of painstaking analysis of photography, the order of events on the Moon can be determined quite closely (E.g., this crater, then this mare, then those craters, etc.) as well as getting rough estimates of the actual time.
Isotope dating of Lunar meteors (bits of Lunar rock blasted into space by impacts) and Apollo samples lets us put precise times to various bits of the sequence resulting in a reliable chronology going back nearly to the Moon's formation.
Arguably there were four stages in the Moon's history:
Stage 1: Formation of the Moon by Giant Impact
The Moon forms out of the debris in orbit after a Mars-sized impactor strikes the Earth about 4.5 billion years ago during the formation of the Solar System. This was the final giant impact involving either the Earth or the Moon.
Stage 2: The Early Years
When formed, the early Moon was around 25,000 miles from Earth and the huge tides raised by the Moon in the Earth and by the Earth in the moon rapidly (by astronomical standards, anyway) further out. The Moon (and Earth) continued to be hit by big rocks (dinosaur-killer-sized and bigger) but not by things in the many hundreds of miles in diameter.
Stage 3: The Late Heavy Bombardment
We can tell from cratering patterns and from dating Lunar samples, that 4.1 to 3.8 billion years ago there was an upsurge in big rocks (dinosaur-killer-sized) hitting the Moon. We also see evidence of this on other planets, though not on Earth since so little remains of the Earth's crust from that era.) We don't know why the LHB happened, though there are lots of theories. (There are also scientists who think the LHB was not as heavy and more spread out, but they are currently in a minority.)
Stage 4: Retirement
Since the end of the LHB, the Moon has had few new craters created and has simply slowly receded from Earth (enjoying a serene and fairly uneventful old age.) There are still craters being formed now and then -- the freshest big one, Tycho, is only 108 million years old! Another recent one, Aristarchus, is around 400 million years old.
These are typical of the last few billion years: Very occasional craters caused by rocks in the 10-50 mile rage. They make a big hole, but happen rarely (ca. even hundred million years) and have had no significant impact (pun intended) on the Moon's orbit.