You're touching on a lot of related topics which isn't recommended for Stack Exchange. It's better to keep it to one question and specific. That said, I'll give this an answer as best I can, as I've been interested in this subject for a while and have read up on it a bit. The powers that be might close this question for being too broad and/or lacking specifics on the specific details of the rogue planet that affects Earth's orbit.
The subject of the orbital systems around different stars is of interest to astronomers and it has been modeled, many times in fact and observed, though it's very difficult to get a good look at other solar systems. Most of what has been observed is probably incomplete and a lot of what has been modeled involves some guess-work so it's an ongoing field of study.
I have to think that astronomers have simulated solar systems and come
to some conclusions about that. Is anyone familiar with that and can
say something about it?
Yes, astronomers have studied and modeled this. It's worth pointing out that models will only give the right answers if you ask the right questions. The fact that astronomers didn't expect, but found many hot Jupiters is one example of the problem with running models vs learning how real systems are likely to be set up.
This is one of my favorite videos on solar-system formation which touches on your question, though this one focuses on inner-planet formation, it's entirely mathematical modeling and it suggests that 3 gas giants tends towards instability while two can be stable. It's a little old now, but I think it's still accurate information. The modeling begins about 20 minutes into the video and the 3 gas giant chaos is shown around 24 minutes.
Scholarpedia also has a nice article on the current solar system's stability.
When you write this: "I have the idea that, in a complex system, orbits would tend toward roughly circular and in the same plane."
I think it's safe to say that there are both stable and unstable systems and an unstable system, after it kicks enough planets or other material out, will become a stable system.
I think it's also safe to say that the proto-planetary disk is what causes planets to form on the same plane, but planetary migration may change that. The sum of inclination should remain fixed, but ejected can change that.
Our solar system, the 8 known planets orbit about 6 degrees off the Sun's equator and planet 9 is one possible explanation for that, but an ejected planet could explain it too.
So I have the idea that in a complex system, where the different
bodies can affect each other's orbits,
This is probably true and it's still true in our solar system. Jupiter and Saturn affect Venus, Earth and Mars Milankovich cycles. Mars, being the closest to Jupiter is especially prone to large axial tilt shifts and eccentricity shifts. Jupiter might, in time, even pull Mercury entirely out of it's orbit where it could move significantly. The odds may be low, but if that happens, once it starts, Mercury could pass close enough to Venus or Earth and undergo a gravity assist and basically end up anywhere, possibly crashing into another planet or crashing into the sun.
There is a circularization effect caused by tides, but it only applies in 2 body systems where the rotation is faster than the Orbit and the tidal effect pushes the orbiting object outwards. There seems to be a circularizing effect for mutually tidally locked systems like Pluto-Charon but planetary perturbations prevent planets from having neat circular orbits in multi-planet systems. They can still be long term stable, but there is eccentricity variation.
The only planet in our solar system with a nearly perfect circular orbit is Neptune and I suspect it's because it's the outer-most planet. Outer planets tend to perturb the orbits of inner planets, but less so the other way around. At least, that seems true. I can't give a good mathematical reason for that, it's more loosely deducted on my part.
The inner 4 planets have eccentricity that changes over time (one of the Milankovich cycles, driven primarily by outer planets). Mars' line looks more straight, but it's on a different axis, it actually changes the most.
Source of image
they would tend to become circular and in a plane
I wouldn't make this assumption. Though the proto-planetary disk likely begins in a plane over the star's equator, some systems have been found that don't orbit in a plane. A plane may be most common, but I suspect that's because all systems begin that way more than they tend to become that way.
See here and here
To your main question,
Suppose a rogue planet came zipping through the solar system and
changed the earth's orbit a little, including the plane of the orbit.
I would think that if a rogue planet changed Earth's eccentricity, that Jupiter over time would return Earth to about where it was. Maybe with the inclination too. I couldn't find any good information on inclination changes over time by orbital perturbation.
If Earth's semi-major axis was changed, that seems to be more constant and that would change the length of Earth's year and might might be a much more permanent change. Likewise, if the Semi-major axis was moved into resonance, say with Jupiter or Venus or Mars, . . . that might be bad, as it might open the door to repeating perturbations and instability.
That's my understanding anyway. Corrections welcome.
