Consider an initially stationary particle of matter and suppose a 1 Earth mass black hole flies past it at speed $v$ on a trajectory that passes the initial position of the particle at distance $r$. The particle will be mainly affected by the gravity during a time period of roughly $r/v$ (up to some "geometric" constants), during which time it will accelerate at roughly $GM/r^2$. This means that it will move a distance about $GM/r^2 \times (r/v)^2 = GM/v^2$. So if this is small compared to $r$ it will be left "more or less" where it was, rather than being ripped away. So we can expect a hole of diameter a few times $GM/v^2$. $GM$ is about $4\times 10^{14}$ so to keep the hole diameter to say 1m you could need $v$ about $2\times 10^7 m/s$ which is about $0.07c$.
We can also estimate the typical velocity change achieved by our test particle from this interaction, which is $GM/r^2 \times r/v = GM/rv$, giving a KE per unit mass of about $$G^2M^2/r^2v^2$$.
So, suppose the Earth has density $\rho$ and radius $R$ the cylindrical shell of thickness $dr$ will mass $R\rho rdr$ (still ignoring "small" constants like $2\pi$). That shell will acquire kinetic energy $G^2M^2R\rho dr/rv^2$ from the interaction. Integrating from $GM/v^2$ to $R$ we get a total energy deposited in the parts of the Earth which are not "ripped away". $$\frac{G^2M^2R\rho}{v^2}\log\frac{Rv^2}{GM}$$
Using $R = 6\times 10^6$, $GM = 4\times 10^{14}$, $\rho = 5000$, and $v = 2\times 10^7$ (all SI units) we get about $10^{25}J$, not enough to actual destroy the Earth (Earth's gravitational binding energy is about $10^{32}J$) but about $10^{10}$ megatons or an earthquake of about 13 on the Richter scale.
A black hole moving 1000 times slower (typical solar system velocity) would essentially destroy the Earth.
