How do we know the Moon was much closer than it is now?
Theory and evidence.
On the theoretical side, it is hypothesized that the Moon formed from a collision between a Mars-sized object and the proto-Earth. Per this hypothesis, the Moon formed after this collision a handful of Earth radii from the center of the Earth. The Moon is now about 60 Earth radii from the center of the Earth, so if this hypothesis is correct, the Moon has receded from the Earth by quite a bit.
Also on the theoretical side, tidal forces make objects that orbit a planet closer than planet synchronous altitude spiral in while objects that orbit a planet further out from synchronous altitude spiral out. The Moon is spiraling out due to these tidal interactions.
On the evidential side, the Moon is very similar to the Earth in terms of geology and chemistry. We know this as a fact due to observations of the Moon and due to Moon rocks brought back to Earth. This similarity gives incredible credence to the hypothesis that the Moon formed from a giant impact, and formed at a much closer distance that the current separation between the Earth and the Moon.
Also on the evidential side, there are tidal rhythmites frozen in ancient stone. These rhythmites show that the Earth's rotation rate used to be much faster than the current rate of one rotation per 24 hours. Some of these tidal rhythmites date back to over half of the Earth's age. The cause of the slowdown in the Earth's rotation rate is the transfer of rotational angular momentum to the Moon's orbit.