PS3 had blu ray going for them but $600 was serious money to drop on a console. The thing 360 allowed was customization over time and a price point that was more reasonable. People were going to pay $600 for a computer first before buying a gaming machine.
Money talks and the consumer wants value and quality. XBOX 360 catered to both of them which allowed them to take the lead and never look back. In terms of which was the better machine, for ME it was PS3. I had a 360 but after a red ring and only finding FPS on the system I stayed with PS3. Not bashing the 360 but it didn't cater to my interests as a gamer.
Don't forget, XBOX actually was pushing for HD-DVD (remember that medium, ha!). That flopped in a hurry.