Why is dark energy preferred to the cosmological constant?

Question

Dark energy and the cosmological constant are often identified, but Peebles and Ratra explain that

"Einstein did not consider the cosmological constant to be part of the stress-energy term... a new constant of nature, $\Lambda$, appears in the addition to Einstein’s original field equation. One can equally well put Einstein’s new term on the right hand side of the equation, and count [...] as part of the source term in the stress-energy tensor. The distinction becomes interesting when $\rho_\Lambda$ takes part in the dynamics, and the field equation is properly written with $\rho_\Lambda$, or its generalization, as part of the stress-energy tensor."

In other words, Einstein treated $\Lambda$ term as a modification of GR with a "new constant of nature", while dark energy may potentially manifest itself in some other ways. Later they write about dark energy:

"The idea that the universe contains close to homogeneous dark energy that approximates a time-variable cosmological “constant” arose in particle physics, ...in cosmology..., and on both sides by the thought that $\Lambda$ might be very small now because it has been rolling toward zero for a very long time. The idea that the dark energy is decaying by emission of matter or radiation is now strongly constrained by the condition that the decay energy must not significantly disturb the spectrum of the 3K cosmic microwave background radiation."

So if I understand this correctly the idea of a dynamical effect came from suspecting that $\Lambda$ may be time dependent, but so far that was not detected. And since "one can equally well put Einstein’s new term" on either side of the equation mathematically both interpretations are equivalent. Still, the dynamical interpretation seems to be overwhelmingly preferred.

Are there empirical reasons for conjecturing dark energy, some spatial non-uniformity in expansion perhaps? If not, what are the theoretical reasons for preferring it to the cosmological constant of modified GR?

Answer 1

A cosmological constant should be considered a special case of dark energy. The effective stress-energy tensor for a cosmological constant is proportional to the metric $g_{\mu\nu}$, so in a local inertial frame will be proportional $\mathrm{diag}(-1,+1,+1,+1)$. This is equivalent to perfect fluid with energy density and pressure directly opposite one another, but more importantly, it is the only possible form for the stress-energy that would give the exact same energy density and pressure in all local inertial frames.

If by 'dark energy', we understand it to mean all the contributions to stress-energy in the above form, then there is no reason for this to be constant, and plenty of reasons why it might not be, as this situation is not exceptional in fundamental physics. For example, there could be false vacua with various different energy densities, and they must be invariant across inertial frames.

In particular, the basic idea of inflation considers a flat FRW universe with expansion driven by a scalar field $\phi$ at a local extremum of its potential, $V'(\phi_0) = 0$, which yields an exponential expansion with constant energy density $T^0{}_0 = V(\phi_0)$. More refined models, such as slow-roll inflation, could therefore be directly interpreted as a time-varying dark energy density, while eternal inflation would also include spatial variability. There's plenty of other inflationary models besides.

One the interpretational flip-side, one could always have a $\Lambda$ that corresponds to the energy density of the true vacuum, and the rest as separate contributions on top of that. It's just not as useful in a cosmological context compared to grouping all 'dark energies' together, as all stress-energy gravitates equally.