4 minutes is still quite far away. "Below" simply means you need to go down a level (using a ladder or staircase). If you're on ground level it could mean that it's in a dungeon, cave or basement.
Every in-game tile is counted as 0.01 minute. Moving will change the number of minutes and/or direction when clicking on the clue again. (teleporting around can save you a lot of time. Try to see if there's cities or teleport locations in that direction and try again to excludes areas and pinpoint the location).