The condition $$\sum_{i=1}^{3}m_iq_i'=0$$
of this paper generalizes to
$$\sum_{i=1}^{n}m_iq_i'=0,$$
meaning the barycenter should be at the origin of the frame.
Equation (1) generalizes to
$$H=\frac{1}{2}\sum\frac{|p_i'|^2}{m_i}-\sum_{i=1}^n\sum_{j=1, j\neq i}^n\frac{m_im_j}{|q_i'-q_j'|}.$$
The first part is the total kinetic energy of the system relative to the barycenter, written in terms of momenta: $$E=\frac{1}{2}mv^2=\frac{1}{2}m\left(\frac{p}{m}\right)^2=\frac{1}{2}\frac{p^2}{m}.$$
The second part is the sum of all self-potential energy between each pair of (spherical) objects, with Newton's gravitational constant $G$ set to 1.
See also n-body problem on Wikipedia.
The 3-dimensional 3-body (or n-body) problem is like the planar version. Just take 3-dimensional vectors for $p_i'$ and $q_i'$.