For mass transfer in a binary system, one star must fill its Roche lobe. What determines whether a star in a binary system will fill its Roche Lobe? How will I calculate it? I can't find any mathematical formula that shows any equation of mass transfer.
One should distinguish the "Roche limit" from the "Roche lobe," as the latter is what you need when you have two objects of similar mass, like a binary star. The Roche lobe is an effective equipotential in a frame that cororates with the orbit of the binary (assumed circular). The equipotential is "effective" because it includes both stellar gravities, and also the centrifugal force from the rotating frame. The Roche lobe is not only an effective equipotential, it is the last one that is closed-- any equipotential further out connects to infinity and cannot correspond to material in equilibrium that is corotating with the orbit. From the side, this last equipotential looks like a kind of infinity sign. Material outside that "infinity sign" has to start moving in the rotating frame, bringing in coriolis forces and in generally leading to complicated motion. However, if close to the Roche lobe, the gas tends to flow through the nodal point at the center of the "infinity", so tends to be transferred from the domain of gravitational attraction of one star to the other. That's "mass transfer."
Setting up the calculation of the Roche lobe is fairly easy because you only need the two gravities and the centrifugal force at the orbital period. But it is a difficult set of equations to solve, and even at the research frontier, is usually approximated rather than calculated exactly. No doubt this is the source of your difficulty.
Why one star fills its Roche lobe is a consequence of stellar evolution, often as one star attempts to evolve into a red giant or asymptotic giant. Before it can reach its giant radius, its outer layers encounter the Roche lobe, and pass along it through the central node (the "Roche point") and may ultimately end up attracted to the other star. This is what is thought to have happened in Algol, for example, which is a binary where the star with the greater mass now started out as the star with the lesser mass (because it gained from its companion). The rate of mass transfer is also difficult to compute, because it depends on how fast the mass-loser is evolving. It is only its evolutionary rate that causes the "Roche overflow," so the mass transfer rate is pegged to the evolutionary rate, which is largely due to rate of fusion that adds mass to its core.
If you're up to deriving the Roche limit, here is something that might be useful to your study. This webpage offers its derivation, based on the Newton's equations.