This is partly space exploration and partly physics.
Rockets do not normally go straight up. They start off going up, to get into thinner air, then they steer to turn themselves eventually to a horizontal path, until they are going fast enough to stay in orbit.
In general relativity, an object that is free to move in the Earth's gravitational field will appear to accelerate towards the ground as it is following a path (called a geodesic) in curved spacetime. This is not specific to rockets; a cricket ball will follow a curved path when thrown due to the curvature of spacetime around the Earth. And if one is far from any gravitational field, then a cricket ball will travel in a straight line.
To directly answer your question: The path of a rocket going straight up will not look curved, it will look like it is going straight up.