Because the year is actually slightly less than exactly 365.25 days, the standard "one leap-day every four years" pattern makes us overshoot a little; if we didn't fix it, the days would slip slightly in the opposite direction. To even out the excess, we have to skip a leap year every century -- so to make it easy to find, we just say we skip the leap day if the year ends in 00. So 1896 was a leap year, 1900 was not, 1904 was. We skipped the 1900 leap day because that put us back on schedule. We don't need to then add another "make-up" leap-day later; that would defeat the purpose of skipping the day in 1900.
But skipping one leap day per century still isn't quite right, so to get even more accurate, we added another rule that if the year is divisible by 400, it is a leap year after all. Since 2000 is divisible by 400, we did NOT skip the leap day that time, so 1996, 2000, and 2004 were all leap years. The next time we skip a leap day will be 2100.
That system gets us to an accuracy of just about one day every 3,000 years. We could keep adding more rules to get even more accurate, but there's no point to that -- the axial procession of the earth, which is slightly variable, means that over a few thousand years, we're going to wind up with calendar shifts building up no matter what we do. We'll have to adjust for them as we go along and see how well it's keeping up.