I just beat that temple yesterday, but I can't tell exactly where you are from what you described. I do know that you can't get across the bridges until you get the boomerang. Go back and look at the places that you can get to. There is a rock wall or two that you can blow up with those bomb-spider things.