When plucked, the string will vibrate to produce sound. I'm certainly no expert, but I suspect the buzzing is related to two factors: 1.) the direction in which you are plucking the string and 2.) the space between the string and nearby frets.
If the plucking finger is pulled off "parallel" to the fingerboard you will have less buzzing because the string vibrates from side to side making it less likely to contact nearby frets. If the plucking finger is pulled up in a "perpendicular" direction from the fretboard you may have more buzzing because the string vibrates up and down making it more likely to make contact with nearby frets.
Obviously the space between the string and nearby frets will also play a part. The string is closest to the frets at the peghead end of the fretboard. This may explain why you are experiencing the buzzing only at the 1st and 2nd fret.
Since the string only buzzes when you do pull-offs I suspect it is related to number 1 above, but it may be related to how forcefully you pluck the string.