That problem usually happens to my 12 year old brother who has no patience. The game picks up all motion you make so you have to be careful. When you start the swing, be sure the controller is level point down. If now, then it will start going back itself. The same goes with putting.
I have had no problems with this game. It is one of the best I have. When I first started, I got frustrated when I goofed on big shots but HEY thats golf for ya . It happens.