That's probably because your face produced more sebum after you used a cleanser. Soap and cleansers tend to strip the face of sebum, and to compensate for that, it produces more sebum which leads to clogged pores. So when you stopped washing your face regularly, your glands naturally adjusted and didn't need to produce more sebum because it had a steady and consistent supply.

huh So the answer is actually to stop washing.

No, just don't wash as much. You could wash every day and be fine, but once you stop for a while and let your body revert back to its natural flow of sebum, it'll come back to screw you once you start washing your face again.