I read in Neil deGrasse Tyson's book Astrophysics for People in a Hurry that scientists can tell if a star has a planet orbiting it because the light appears to shake.
So if in the case of a binary star which is quite obviously shaking because its orbiting another star how would I know a planet is orbiting the 2 stars?
The light from the two stars would be Doppler shifted in a sinusoidal pattern (for a circular orbit). The signals from the two stars would be in anti-phase and would oscillate at the orbital period of the binary.
The planet would cause a small pull on the binary system which would superimpose a further small sinusoidal signal (again, assuming a circular orbit), but at the longer orbital period of the planet around the binary system.
The process is illustrated below by adding two (red and blue) sine waves together, where the blue sine wave has a much smaller amplitude and lower frequency than the red sine wave. These sine waves encode the Doppler shift that would be measured from the absorption lines of one of the binary components. Thus the red wave would indicate the Doppler shift due to the motion of the binary system and the blue sine wave the additional modulation due to the smaller motion induced by the tug of the planet as it orbits. The black line then gives the superposition of these (they can basically be added if the velocities are non-relativistic) and you can see that it is subtly different from a monochromatic sine wave of fixed amplitude.
For non-circular orbits, the pattern of Doppler shifts would be non-sinusoidal but still periodic in a similar way.
You check the periodic frequency shift with the Doppler effect. The light from the two stars is not very suitable because each light source moves. Much better: The binary system (total mass $m_A\approx 2m_{\odot}$) generates a gravitational wave with high power that appears to come from the stationary center of gravity of both stars. There is no source of error when evaluating the GW.
Let's assume that this GW can be measured and that the two suns take 2.315 days to orbit. Then the pair emits a GW with constant frequency $f_{GW}=10~\mu$Hz. If a planet $B$ orbits the pair at a distance of $r_B$, the binary system orbits the common center of gravity of the entire system with radius $r_A$. Because of center of gravity theorem $m_A\cdot r_A=m_B\cdot r_B$ applies. Because the planet forces the center of mass of the binary system to move, the instantaneous value of $f_{GW}$ is sometimes larger and sometimes smaller than the average value. The maximum frequency deviation is
$\Delta f = f_{GW} {\left(\sqrt{\frac{v_{GW}+v_{orbit}}{v_{GW}-v_{orbit}}}-1 \right)}$
With $\Delta f$ you can calculate: a) at what speed does the binary system rotate around the common center of gravity of the triple. b) at what speed does the planet rotate around the common center of gravity of the triple.
With these intermediate results you can calculate the orbital period and the mass of the planet.
