I'm thinking, this is the gist of your question
Earth feels a zero net force. Will it help earth to fly away?
First, zero net gravitational force between two large objects is certainly possible, well, not exactly zero, not for more than an instant anyway, but close to zero, absolutely possible, but whether it leads to an object flying away is more complicated. It depends on the relative motion of the 3 objects.
The Moon, for example, orbits the Earth, but from the Moon's point of view, the Sun is about 333,000 times more massive than the Earth and about 388 times further away, (on average) when the Moon is between the Earth and the Sun (390 times when the Moon is on the opposite side of the Earth, again on average. There's some variation in there).
Because gravitation drops with the square of the distance, 388 times more distant means about 151,000 times less G force at that relative distance, but with 333,000 times more mass, the Moon actually experiences over twice the gravitational tug from the Sun than it gets from the Earth, so, even though, from the Moon's surface, the Earth is much larger than the Sun, the sun's mass is sufficient to exert the greater gravitational pull.
So, if, by some magical power, you were to grab a hold of the Earth and stop it from moving and grab a hold of the Moon and stop it too, then let the Moon go, the Moon would fall more towards the Sun than the Earth cause the gravitational pull in that direction is over twice as much. (Ask this great magical being not to let go of the Earth, because if he does, the Earth would fall into the Sun too).
That's not quite the same as your scenario but it points out that zero net gravitation doesn't govern where an object ends up. The Moon orbits both the Earth and the Sun, and it's in a stable orbit around the earth even though it's feeling more gravitation from the Sun. That's because the Moon is inside the stable part of the Earth's Hill Sphere.
In your scenario, however, a passing object the mass of another star could certainly pull the Earth away from the Sun. It wouldn't even need to achieve a net zero gravitation to accomplish that, nor would it need to be nearly so massive.
The picture below covers the Earth orbiting around the Sun. If you bring the net Gravity to zero, in theory the "F" in the diagram shrinks to zero and the Earth continues straight in direction V for that time period, increasing it's distance from the Sun. Source
The Earth's tangential velocity relative to the sun is 30 km/s and it's escape velocity is just the square root of 2 times that, about 42.5 km/s, so an acceleration of the Earth of 12.5 km/s or moving the Earth to an orbit a bit outside Mars' orbit and keeping the velocity the same would both work (or some combination of the two).
The model is a bit more complicated because a gravitational assist, which would also happen in your scenario and a gravity assist can work both ways, increasing or decreasing the orbital velocity. It's possible, depending on direction of the pass, that a passing star could push the Earth closer to the Sun, even passing outside, if it slows the earth's velocity by gravity assist. Drawing it away isn't the only possible outcome.
More on gravity assists here, Short and Longer.
As James Kilfinger points out, stars passing that close is extremely extremely rare so this kind of thing, for all practical purposes, virtually never happens. It's much more rare than a dinosaur killing meteor for example. It's hugely unlikely.