Disable all observers in your test suite by default. They should not be complicating your model tests because they should have separate concerns anyway. You don't need to unit test that observers actually fire, because ActiveRecord's test suite does that, and your integration tests will cover it.

Rails' observers were partly inspired by aspect-oriented programming -- they're about cross-cutting concerns. If you're putting business logic in observers that is tightly coupled to the models they're observing, you're doing it wrong IMO.