My question isn't why it doesn't have a circular orbit, and why an elliptical, my question is why do they move at all? What caused the debris in space to apply such a force on the planets that it has been circling around the Sun billions of years.
This is a simple explanation to give a basic understanding as to why the solar system rotates.
It's caused by gravity and velocity. If you put two objects with mass in space a distance apart and at rest, they will accelerate directly toward each other, until kerpow! they hit.
But, if they have some small, random relative velocity to begin with that's not directly toward or away from each other, then no amount of acceleration toward each other will zero that out. It means they will pass each other at some finite distance and keep going. If the random velocity is low enough, this dance will repeat and they will move in elliptical orbits around their center of mass.
This kind of motion can be expresses in terms of its angular momentum and in this example (as in most) it is going to be a conserved quantity, so it's always going to be there.
So when things in the universe like solar systems or galaxies initially form, they have some random, residual, collective motion, and this angular momentum in the center of mass frame of whatever blob is coalescing doesn't go away. Objects in the solar system keep rotating because there's no other option, at least in the short term.
If there was a large amount of gas in the solar system, that might slow things down a bit, and there is a teeny tiny loss of energy due to gravitational waves, although for planets this is so extremely small it can be ignored. But basically the "wheels" of the celestial sphere are well lubricated and solar system bodies will continue in their orbits for millions and perhaps billions of years because there's nothing we know about right now to stop them.
It's a simple matter of statistics. All the particles that did not move in a near-elliptical orbit in the first place either drifted away or fell into the sun. What you see is what's left.
As to why orbits self-maintain, see uhoh's discussion of, basically, Keplerian dynamics.