The picture was so much cleaner 20 to 25 years ago. I'll present that nice clean picture first. Stars form from the gravitational collapse of huge clouds of interstellar gas. Those gas clouds inevitable have some net non-zero angular momentum. This forces the gas cloud to change shape from being more or less spherical to being disk-shaped. (Why? That's a different question. Ask it.)
While this protoplanetary disk continued to feed mass to the growing protostar, it also set the stage for the formation of planets. The gas cloud was mostly primordial hydrogen and helium, but it also contained heavier elements thanks to stellar fusion and supernovas in the billions of years that preceded the formation of our solar system.
Those heavier elements behave quite differently than do hydrogen and helium. They have chemistry. The planets started as microscopic clumps of mass of these heavier elements, bound together chemically. These microscopic clumps occasionally collided, eventually forming larger clumps of mass. These larger clumps in turn collided with one another, forming even larger clumps of mass. Eventually the clumps became large enough that they interacted gravitationally, making them grow even larger. This process continued, eventually forming protoplanets, and then planetary embryos, and finally planets.
Temperature in the protoplanetary disk were high near the forming protostar but dropped precipitously with increasing distance from the protostar. At some point, volatiles such as water, ammonia, methane, and carbon dioxide become as solid as rock. This is the ice line, aka the snow line or frost line. Asteroids inside of Ceres' orbit tend to be rocky. Asteroids outside of Ceres' orbit tend to be icy.
Planets that form outside the ice line can grow very quickly and then they can grow very, very large. The stuff that comprises the protoplanetary disk orbits the growing protostar at something other than the rate suggested by Kepler's laws thanks to the pressure of all that stuff in the disk. Thanks to the square-cube law, larger objects aren't as subject to that pressure. Those larger objects orbit at a Keplerian rate. Planets that form outside the ice line grow very quickly and then sweep up gas and ice because they are orbiting at a different speed than the immediate surroundings. The result is gas giants such as Jupiter and Saturn and further out, ice giants such as Uranus and Neptune. Planetary growth is a much more difficult process and much slower process inside the ice line. That's why Mercury, Venus, the Earth, and Mars are rocky and so much smaller than Jupiter, Saturn, Uranus, and Neptune.
That's the pretty picture. The not so pretty picture:
Why are Mercury and Mars so much smaller than Venus and the Earth?
Simulations suggest that the rocky planets should all be more or less the same size. That is not the case in our own solar system, let alone elsewhere.
How could Uranus and Neptune have formed?
Simulations cannot recreate Uranus and Neptune at their current distances from the Sun. The material in the protoplanetary disk should have been too sparse at those distances to form large planets.
Much, much worse, what's the deal with all the weird exoplanets scientists have found?
Scientists have found Jupiter-sized objects orbiting very close to their sun, Neptune-sized objects orbiting where the simple model would have only rocky planets forming, and planets in highly inclined (and sometimes retrograde) orbits that don't make sense.
These simulations (which have become to be very good) and the plethora of exoplanets have pushed the theory of how planets form back into the “that’s funny” stage. ("The most exciting phrase to hear in science, the one that heralds new discoveries, is not “Eureka!” but “That’s funny...”", a quote widely attributed to Isaac Asimov.)
