I had this question for a long time. But otherwise the question basically sums the whole thing up.
The problem with associating virtual particles with either dark matter or dark energy is that, quite literally, virtual particles aren't real. By that I mean, they are intended to change the effects of energy, like how the energy is transported and where it goes, but they shouldn't alter the amount of energy. As such, virtual particles are just a kind of picture, a way to view the terms in a particular way of writing the equations. Many physicists reject that picture, they think it gets taken too literally, while others embrace it wholeheartedly, but it shouldn't appear in the stress energy that is used in general relativity.
But of course, the problem is that general relativity may be wrong, especially since it does not seem very consistent with quantum mechanics. So the idea is, maybe there is some way to correct general relativity where virtual particles do have a more direct role. Then the people who tend to dismiss their relevance would have to reconsider. But so far it's not at all clear that this will be the right way to fix GR, because you run into significant problems.
The problems you run into with dark matter being virtual particles is that most of the effects of dark matter are seen on completely Newtonian scales, like the gravity in a galaxy. On that scale, one should not need GR, and no one is suggesting that we need to fix Newtonian gravity by including virtual particles, that would not be consistent with everything Newton did. For example, we see no effects of dark matter within our own solar system, so why would we need to fix Newtonian gravity on the galactic scale but not the solar system scale? We would need to do more than include virtual particles, we'd need a whole new kind of gravity (sometimes called "MOND", modifications to Newtonian dynamics"). If you are going to do MOND, it opens a full range of possibilities, there's really no reason to be constrained by the virtual particle concept.
The problems you run into if you try to associate virtual particles with dark energy is that the answer you get, if you proceed in the most straightforward way possible, is that the dark energy should either be zero (if you don't think virtual particles count in the energy of the vacuum), or something more than 100 orders of magnitude too large (if you think they do count). So how do you take that virtual particle concept and whittle down its effects from something more than 100 magnitudes too large to something tiny but not zero? It's not at all clear this is the best starting point!