Let's walk through the stages of post-main sequence evolution. For reference, the images, and much of the content is being pulled from An Introduction to Modern Astrophysics by Carroll and Ostlie. I'll break this into low mass stars ($\sim1\:\mathrm{M_\odot}$) and intermediate mass stars (($\sim5\:\mathrm{M_\odot}$)
Low Mass Evolution

Above is an H-R Diagram representing the path of a low mass star through the luminosity-temperature phase space. The star leaves the Zero-Age Main Sequence (ZAMS) once the core of the star has run out of hydrogen and fusion of hydrogen into helium ceases. As you stated, you begin to have a slight core contraction and then hydrogen in a shell around the core begins to burn, producing a helium shell. This hydrogen shell burning causes the core to continue to grow in size and at some point, you reach the Schonberg-Chandrasekhar limit, essentially the mass limit for the core which is capable of supporting the gravitational pressure of itself and the envelope. Once the core becomes too massive, it rapidly contracts, releasing a lot of gravitational potential energy which is dumped into the envelope, causing it to heat up. At this point, the star is at the end of the Sub-Giant Branch (SGB) portion of the path and just about to enter the Red Giant Branch (RGB).
With a collapsing core (and continued burning of the hydrogen shell) producing huge amounts of energy and an envelope which is heating up, the star needs to reach an equillibrium. This is achieved by the envelope expanding drastically. This, I would argue, is when the star becomes a red giant. After the core contraction energy has been dumped into the envelope and the envelope has expanded due to being heated up. A lot more stuff happens after this point, including a dredge-up from the core and the helium flash, but that is outside the scope of your question.
Intermediate Mass Evolution

This is the H-R Diagram representing the path of an intermediate mass star as it leaves the ZAMS. Most of the description I provided above is applicable here, in that the intermediate mass star follows the same process. There are only a few differences in the details. The major difference you may pick out is that as the star moves along the SGB, it actually decreases in luminosity towards the end before venturing up the RGB. This is simply due to the fact that after the core contracts and releases energy into the envelope, the envelope is so much larger than the low mass case, that it can absorb a lot more energy before needing to re-equillibrate. For a time, the core contracts, dumping energy into the envelope, and the overall star luminosity decreases because the core's energy has not found its way out of the star yet, but the star is now smaller. Once the envelope reaches its peak energy consumption it then expands, just as in the low mass state $-$ it simply takes more time and energy to reach this point.