One obvious reason is that the bulk of a white dwarf is much hotter than indicated by the surface temperature. About 99% of a white dwarf's mass is at temperatures about 100 times higher than the surface (photospheric) temperature. There is thus a very large thermal reservoir available to keep the surface warm.
The 1% on top acts like a insulating blanket - like a jacket on a hot water tank - so that the white dwarf cannot lose energy at a rate faster than the blackbody radiation from its (relatively) cool surface.
Once that is established, the cooling time, given by the ratio of the thermal energy content to the rate at which heat is radiated, is obviously going to be very long for a given surface temperature. The thermal energy is proportional to the mass multiplied by the internal temperature (very large), whereas the cooling luminosity is proportional to the surface area (small for a white dwarf) multiplied by the fourth power of the surface temperature of the white dwarf.
To add more complexity, the time for white dwarfs to become "black dwarfs" (objects too cool to emit visible light, say cooler than a surface temperature of 2000 K) may not be as long as you think - "only" about the current age of the universe.
The reason for is that white dwarfs crystallise at interior temperatures of about $3\times 10^6$ K, when their surfaces are of order 30,000 K. However, the Debye temperatures of their crystalline interiors has almost the same value. Below this temperature a quantum mechanical effect (quantisation of the oscillation modes of the crystal - a.k.a. phonons) drastically reduces the heat capacity of their interiors leading to a greatly reduced cooling timescale and an accelerated fall in temperature (both the interior and the surface) with time.
The exact temperatures for crystallisation and "Debye cooling" depend on density and hence on the white dwarf mass, but some example cooling curves are shown below. They show the catastrophic cooling that takes place once the white dwarf enters the Debye cooling regime. This plot shows that the oldest white dwarfs in our Galaxy should already be "black dwarfs" if they have more than a solar mass.
Typical Galactic white dwarfs are actually a bit below the lowest mass in this plot (more like 0.6 solar masses), so the sudden drop in temperature will occur after about 15-20 Gyr for most white dwarfs.
