This is probably why:
There are several things going on when you see someone looking at you, all of which happen very quickly. (This applies to actually seeing someone looking at you, not “sensing it” from behind or in the periphery.)
Primates (including humans) are unique in the degree to which the eyeball can move around in the eye socket. This allows visual attention to be shifted quickly without physically moving the head. Primates and certain other mammals can tell when another animal is looking at them, but humans are particularly good at doing this from a distance. In fact, humans have the added ability to be able to tell where someone is looking, even when it is not at them.
It is easy to see why this skill confers an evolutionary advantage: By being able to do this, you can essentially “read out” the location of another animal's attention. If you are a social animal, and the one looking at you is a superior, you'd better behave. Or if it is an inferior, you are being challenged and need to respond so you don't lose your place in the status hierarchy.
From: How Does a Person Know When Someone Is Looking at Her?