There is one thing that always gets me in the problematic situation - estimation.
When is it done exactly?
Let's say I have 5 user stories in product backlog. When do I estimate them - during current sprint while the team is working on current user stories? If not, then how do I know how many Story Points each has? Is it done during Sprint Planning? Then it's not enough time.