I think it was because he found [u]one[/u] man that actually had faith in him. And when Noah built the ark, everyone thought he was crazy. Then since nobody came in despite Noah's warnings, they never came in.