Could an asteroid or a rogue planet have enough magnetism and/or gravity to spin the Earth near the speed in which they are passing by Earth?
Yes, a little, but not enough to matter.
Magnets can spin other magnets. Think of magnets as a bar magnet with a north and a south, and you can think of them as linear not spherical if that helps with the imagery, because in a sense, they are linear with a distinct North and South magnetic pole.
Earth's magnetic field, for example has a distinct longitude and latitude that moves over time, but it's fixed at any given time.
For your question to work, it's easier to imagine a very powerful magnetic object flying past, and to think of the Earth's internal magnet as a compass arrow that wants to line up. Magnets want to line up with magnetic field lines and a force would be created by a passing strong magnetic field that could orient the Earth's magnetic field to align with it.
You might think that this would turn the Earth, but it wouldn't. Earth's magnetic field isn't "the Earth" it happens in the Earth's core, and it may happen mostly in the region where Earth's inner and outer core meet and the outer core is liquid (dense and viscous but still not solid). The force would be applied to the magnetism inside the Earth, perhaps giving the Earth's inner core a slight tug and twirl and perhaps re-orienting the flowing charged liquid iron just outside of the core, but this force wouldn't turn the Earth as a whole, it would generate a force on the core and the core doesn't spin precisely with the Earth's surface anyway so any movement would likely not show up on the surface, for some time.
Given that a percentage of earth's magnetism is carried by the liquid outer core, then the effect on rotation would be even less, more of a re-orientation than a direct angular momentum tug.
It's also difficult to generate a magnetic field strong enough to move something as massive as the Earth's core very much anyway. Something to keep in mind as well is that the magnetic field drops by the cube of the distance, not the square, so you'd need a very powerful magnet, very close, to have any effect at all. Anything large enough to have that strong a natural magnetic field, the gravity would have a much bigger effect than any interaction between the magnetism.
But to answer your question, a tiny bit of angular momentum would be transferred when two magnetic fields fly past each other in space, so there would be some variation in rotation, but it would be tiny.
Magnetic fields as strong as Earth's are rare, you'd probably need a gas or ice giant planet or large rogue planet, and the interaction between passing magnetic fields is very weak compared to the gravitational interaction. Compared to it's mass, Earth is a very very weak magnet, so using magnetism to spin it would be ineffective and slow, but but in theory, if we wanted to set up a current and generate a magnetic field around the Earth, such a project could in theory spin the Earth's inner core like a induced magnetic field spins a magnet in a generator, but that would also depend on how permanent the magnetism is inside the Earth and it might not be fixed, but instead, a product of flowing magnetic metals. Permanent magnets are easier to rotate than charged flowing liquid . . . but I digress.
It would be an interesting calculation if somebody wanted to do the math behind this, but even without doing the math I can tell you that the effect would be teeny-tiny.