Our nearest star Proxima Centauri is 4.243 light years away from Earth.
Does that mean we are seeing light that is 4.243 years old everyday?
Yes, the speed of light in vacuum (or c) is 299,792,458 m/s and one light-year is the distance the light travels in one Julian year (365.25 days), which comes out as 9.4605284 × 1015 meters. Since c is the maximum speed at which all energy, matter, and information in the Universe can travel, it is the universal physical constant on which the light-year (ly) as one of the astronomical units of length is based.
If we calculate this distance for a 4.243 ly distant object, that comes out as
That is the distance the light traveled since it was last reflected of (or in our case emitted from, since Proxima Centauri is a red dwarf star) the surface of a celestial object to be 4.243 Julian years later visible at our observation point, in this case our planet Earth from where the distance to Proxima Centauri you quoted was measured.
The more powerful the telescope, the further into the past we can see because the light is much older! This goes the same regardless of the distance of the object you're observing, but astronomy is particularly neat in this regard and we can observe objects that are so distant that we see them from the time when they were still forming.
For further reading on other units used to measure far away objects you might be interested in reading this question on the parsec.
All light we see is from the past. The light from a light bulb at 3 meter distance arrives 10 ns after it left the bulb in your eye. For short distances this delay is negligible (10 ns is 10 billionths of a second), but at astronomical scale it becomes significant. Light from the Sun takes 8 minutes and 20 seconds to reach the Earth, so when we see the Sun it's the Sun like it was 8 minutes ago. If the Sun would suddenly die we wouldn't notice for 8 minutes.
The same goes for other stars in our Galaxy. The light from a star at 4 light years takes 4 years to reach us; it's the definition of a light year.
One could make the following comparison: suppose there's a town at 100 car years from where you live. That means it takes a car 100 years to reach you. When a car from that town reaches you today it left in 1914. It won't be a 2010 sedan, but a Ford T. As the car arrives you're looking 100 years in the past.
This looking into history is very convenient for cosmologists. You want to know what galaxies looked like 13.5 billion years ago, when the Universe was still young? Well, look for light that's been underway for that time. It left the galaxy under study 13.5 billion years ago and shows you what that galaxy looked like at the time. It doesn't tell you anything about the current state of it. It may have collided with another galaxy or absorbed by a black hole. There's no way of knowing other than waiting another 13.5 billion years, until the light emitted now will reach us.
Another interesting thing to observe from that far past is the Cosmic Microwave Background Radiation (CMB). It's the radiation from the Big Bang, which has been underway for 13.8 billion years. Of course today the Big Bang is history, but thanks to the "limited" light speed this history is continuously underway to us.
Actually, the light hitting us from Proxima Centauri is not necessarily 4.243 years old. Perhaps some of the photons arriving here were created in the photosphere of Proxima. But some of them will have been created in the center of the star, and these photons may take many years to arrive at the photosphere, where they are then "emitted".
For our sun, it is written (in Wikipedia's article about our Sun):
Similarly, many of the photons arriving from Proxima may be many tens of thousands of years old. Their travel time from Proxima's photosphere is only a small part of their journey to Earth.
A deeper answer is "yes and no". In the frame of reference of the light itself the journey from Proxima to here is instantaneous. In our frame of reference it takes four years - this is all bound up in relativity and the nature of spacetime.
But in the everyday sense we are indeed looking back in time at light from the stars.