But you are asking about the speed of light, since the product $\mu_0 \epsilon_0$ determines the speed of light in vacuum, which is fixed.
Before 2019, both the speed of light and $\mu_0$ were defined as fixed numbers in SI units and therefore by definition, $\epsilon_0$ was fixed.
With the revision of SI units, the speed of light is still a defined number but both $\mu_0$ and $\epsilon_0$ are determined by experiment (though their product must still be a defined constant). They are experimentally determined to about 1 part in $10^{10}$ via the fine structure constant $\alpha$. There is no very strong evidence that they or any other physical constants vary in different parts of the universe.
See https://en.wikipedia.org/wiki/Vacuum_permittivity#Redefinition_of_the_SI_units
More fundamentally, it isn't that meaningful to ask whether physical constants with units vary, either in space or time, since they are measured in terms of other things that could potentially vary instead. The only really meaningful question is to ask whether dimensionless combinations of physical constants, such as the fine structure constant $\alpha$, vary.
There have been suggestions that $\alpha$ might vary with time or direction. Since the fine structure constant is
$$\alpha = \frac{e^2}{4\pi \epsilon_0 \hbar c} \ ,$$
then any variation in $\alpha$ could now be interpreted as a variation $\epsilon_0$, since the other constants are fixed in SI units. Or you could rewrite $\epsilon_0$ as $1/\mu_0 c^2$ and interpret it as a variation in the permeability of the vacuum, or some combination of the two.
The results of investigations into high redshift gas clouds along the lines of sight of quasars have proved some of the best probes of this effect. Th emost recent work I have seen (Wilczynska et al. 2020) suggests no variation with time, but there might be a change in direction at the level of 1 part in $10^5$. The present rate of change (if any) must be less than part in $10^{16}$ per year.