TL;DR: Anything between 1,5 hours and infinity.
Let's assume the moon would be hit in its perigee by an object of the same mass and speed but opposite direction of movement relative to Earth.
Let's also assume a sizeable chunk of debris left by this colossal impact would remain at the last known position of the moon but with zero orbital velocity. (Maybe the impacting asteroid was made of cheese?) This chunk of debris will be "the moon" for the purpose of this answer.
Next after the incident "the moon" will plummet towards Earth, accelerated by a force of about 1G. This is because gravity doesn't decrease by much for a given distance and 1G is the force exerted by Earth. Actually "the moon" also exerts a force but for simplicity, let's assume it only cancels the effect of distance.
The acceleration of the moon is thus about 9.81 m/s² with a starting distance of the moons perigee (~270.000km if I remember correctly, being too lazy to look it up on wikipedia). If I'm not mistaken "the moon" will take (sqrt(distance/acceleration)=5246,23 seconds) about 1,5 hours to reach Earth. Maybe it'll be a bit less for Earth's radius. It will also arrive with a speed beyond Mach 50 and thus actually "impact" Earth's atmosphere, i.e. experience a resistance equivalent to the sound barrier plus extreme compressive heating, likely to rip it apart.
This is the fastest way for moon to be impacted and then crash into Earth. However the question asked for the slowest way: Well, by decreasing the mass and/or velocity of the asteroid impacting the moon, we can "fine-tune" the effect to take any time between 1,5 hours (complete stop w.r.t. Earth/orbit, see above) and infinity (still having a stable orbit). For moon-crashes later than 1,5 hours after the initial impact, the moon would need to be put in an unstable orbit, e.g. orbiting through low density areas of Earth's atmosphere once in a while.
Also other answers have mentioned ways for the moon to get destroyed or ripped apart in the process of deorbiting it, which definitely apply. I just wanted to focus on the aspect of timeline.