We're scared of what we don't know. He had his territory, but was scared to venture into the unknown. Kinda like when Pippin and Merry left the shire and said, "this is the farthest I've ever been away from home".