How does the dense atmosphere of Venus affect the planet's rotation?
I don't know all the details, but I can give a partial answer, with links to further reading if interested.
Tidal locking, and orbits slowing down due to a tidal bulge and tidal forces and/or a denser or greater gravitational gradient on one side of the tidally locked object are all pretty straight forward, and I don't think that part needs to be explained too much. The tidal bulge has mass and that mass creates drag on the rotation (or speeds it up if the bulge is behind the object causing the tides).
And it's worth adding that rocky worlds are gravitationally lumpy, where the heavier side (gravitational gradient), tends to, over time and in tidal locking, face the object it orbits. The near side of the Moon, for example, is slightly denser on average than the far side. See here for a better explanation and here for gravitational map of the moon (the Earth side is on the left in the picture below).
Venus' atmosphere, particularly it's upper atmosphere, behaves very differently. There's still an atmospheric tidal bulge, but because of the heat from the sun, the hot gas on the sun side of Venus upper atmosphere has lower density, so the tidal bulge may have less mass, not more mass, even with greater size. Atmosphere doesn't behave like a typical tidal bulge orbiting close to a star because the hot atmosphere is less dense and the night-side of Venus' upper atmosphere is more dense. The (by mass) true tidal bulge is on the far side of Venus, away from the sun, not the near side.
Another effect is that the heat from the sun causes something like a jet-stream on Venus, a very powerful jet stream. The wind speed on Venus' upper atmosphere is faster than any hurricane on Earth and the upper atmosphere tends to rotate in the same direction around Venus. This solar driven rotation (may be) a factor in Venus' surface having a slow retrograde rotation where, without an atmosphere it might be tidally locked.
It's difficult to imagine an upper atmosphere wind could "spin" a planet, and it would only do so very very slowly, (Venus' atmosphere is about 1/10,000th it's total mass, not too far off from the ratio of Earth's oceans to Earth's total mass) and the tidal slowing of a planet is very gradual, at Venus distance taking perhaps 100 million years to slow a rocky body of that size to tidal locking. So the rotation may be a balance between Venus' solar driven jet stream and it's gradual tidal locking.
At least, that's what I came up with doing some reading on this. From Wikpedia:
The rotation period of Venus may represent an equilibrium state
between tidal locking to the Sun's gravitation, which tends to slow
rotation, and an atmospheric tide created by solar heating of the
thick Venusian atmosphere
and link to the source, which is in French but English translation is available. (see footnote 100 and 101)
On Venus "jet-stream" like upper atmosphere, note that while the jet-stream on Earth is driven by surface temperature variation perhaps more than solar and by Earth's Coriolis effect, and it varies by seasons. Venus doesn't have surface temperature variation to speak of and essentially no Coriolis effect due to it's slow rotation, so it's jet-stream tends to be much more along latitude lines and much less curved than Earth's. That's not to say that there's no variation. Venus' wind seems to operate on a 5 day cycle and it appears to have grown stronger in the years since we've been observing Venus. Neither of those is well understood.
Article here
At the very top of the cloud layers on Venus, wind speeds reach 355
km/hour (or 100 meters/second). This is the same the jet stream here
on Earth. As you descend through the cloud layers, though, the wind
speeds pick up. In the middle layer, the winds can reach speeds of
more than 700 km/hour. That’s faster than the fastest tornado speed
ever recorded on Earth.
But then as you descend further down through the clouds, the
thickening atmosphere slows the winds down, so that they act more like
currents in the ocean than winds in the atmosphere. Down at the
surface, the winds only move at a few km/hour. That’s not much, but
the thick atmosphere can still kick up dust and push around small
rocks.
and
The winds on Venus travel in a westerly direction, the same backwards
direction that Venus rotates. Seen from above, Venus rotates in a
clockwise direction. This is backwards from the other 7 planets, which
rotate counter-clockwise.
another article here.
and, related, quote from Wikipedia and source article.
Venus's rotation has slowed down by 6.5 min per Venusian sidereal day
in the 16 years between the Magellan spacecraft and Venus Express
visits
So, short summary, Venus atmosphere may play a key role in it's reverse rotation, or, maybe it's rotation was reversed by an impact a long time ago and it's still slowing down from that impact. The impact theory might make the most sense, but impacts large enough to spin a planet are quite rare so, it shouldn't be assumed. I don't believe it's known 100% whether Venus' atmosphere caused it's backwards rotation, but it seems reasonable and possible. My personal favorite idea, though it may not be likely, is the gyroscope flipping hypothesis. (Posting video cause it's really cool, but it has nothing to do with your question).
https://www.youtube.com/watch?v=1n-HMSCDYtM
