It's simply because the sun and planets were formed out of a big pile of dust. Originally the dust was spinning. So once the dust became planets, it kept spinning around. That's all there is to it - that's why the solar system is spinning around.
It's was once a pile of dust that was spinning a bit; it's still spinning.
You then ask,
"For example, if you turned off gravity, why would the planets carry on moving?"
THat's just the same as asking "if I spin something around on a string, and break the string, why does it keep moving?"
For that matter, it's the same as asking "What is momentum?" So, if you push something ...... why does it keep moving?
At this stage in history, we have utterly no clue, at all, what the heck time, space, matter, and momentum are. "Why does momentum do what it does?" is for now just one of those super-deep questions like "What is time" or "So what caused the big bang" or "What is gravity" or "What's the explanation for this quantum stuff?" or indeed ... "What is momentum"?
So, why do the planets keep moving?: answer "momentum". If you want to know "what causes momentum?", that is one of the basic total mysteries.
For now, nobody has a clue. You may as well ask ... what is space, what is time, etc.
Regarding momentum, you might like to read up about the so-called "Mach's conjecture". The famous Einstein was, like you, fascinated by the question "WTF is momentum anyway?" One sort of general thinking-point originating with this smart guy called Mach is that momentum could have something to do with "all the other mass in the universe". Nobody has a clue about this, and it's just a vague general idea.
Isaac Newton was a pretty smart guy (if a bit whacky), and in the end he found gravity so mysterious, he just put it down to God. He probably found momentum as mysterious, and he was one of the first guys to think about it clearly.
An interesting point: actually everything astronomers look at (all galaxies, all structures of galaxies) in fact does not (!) behave the way small things (our solar system, as in your question) behaves in terms of gravity and momentum. This is usually explained by invisible unknown matter ("dark matter") or for a few scientists, that gravity works differently than we think presently. So the fact is with issues like gravity/momentum you ask about ... not only do we have no clue "why momentum works" but when you look through a telescope, issues like "orbits" work totally differently anyway!