Dean, let me reiterate a few points already made and add a couple of my own.
First, almost everyone will have to look at the fretboard when they first start playing. Eventually, you gain the muscle memory of where the frets are and you don't have to look as much.
Second, many of us play different instruments with different VSLs, so we can't rely on that muscle memore as much as those who play a single instrument all the time. I have five dulcimers I play regularly, only two of which are standard-sized dulcimers. The others are made to fill different tonal ranges, and their sizes vary. All five of those dulcimers have different VSLs.
Third, some songs are more complicated than others. For simple tunes played around the nut, I can play without looking at the fretboard. But other songsinvolve chords higher up the freboard and/ora lot of movement from lower frets to very high frets. It is very difficult to play those kinds of tunes without looking at the fretboard. When I play guitar, I can generally play chordal accompaniment even with bass runs and little licks without looking at what I'm doing. But if I have to take a solo, then my eyes will be glued to the fingerboard since my fingers will be moving around a lot.
Fourth, some of us don't use tab or sheet music. This might sound odd, but I learn the songs I learn. What I mean is that I use tab or music when I first learn a song, and perhpas I have to refer to that to refresh my memory of a song I haven't played ina while, but while I play I don't look at music. I find I can play faster and more precisely by concentrating solely on the fretboard and my finger placement. I don't want the distraction of looking at a piece of paper and then having to translate that paper to action on the fretboard. My fingers fretting and picking strings is all I want to think about.