I did that quest with 15 agility and didn't need to leave to bank.
The only spot that quest ever had an issue was where you get the blue orb, and it's been fixed for some time.
You need to be sure to search the rocks, not just walk over them, as it sets off the traps.

I'm on the part to meet the dwarf. I keep falling below on last gap. Always the same. Ill try again tonight.

When I did the quest I was 40 agility, I failed 2-4 times but its all rng really cause there isn't a certain level of agility for the quest that will allow you make it 100% of the time. So it is unfortunate that you keep failing but all I can really suggest is to keep working on it.

The lower the agility level you got the lower the chance to fail an obstacle , ik it sounds wierd but its how it works on this server . Barbarian Outpost Agility Course is the only place where the higher agility level you got the less you fail , while Wilderness Agility Course the higher level you got the higher chance to fail . Its the same for woodcutting the higher your wc level the higher the chance to fail and cut down a tree. I did underground pass few days ago with 52 agility and only failed once at the blue orb .I did  underground pass 3 times on this server , two times on an acc with 52 agility , both times failed only once , and once with 88 agility , had to bank and grab more food to finish the quest.

I'm on the part to meet the dwarf. I keep falling below on last gap. Always the same. Ill try again tonight.

So.. you're going to meet the dwarf. You fall below. Walk to him.. that's where he is. Below. People always try to make it to him from top and walk down. Just walk to him from bottom.

Had I known that it would have saved me lots of time.thank you.