The present study investigated brain activity during eye contact using functional near-infrared spectroscopy (fNIRS). Although eye contact is important for smooth and effective social communication, little is known about its neural correlates. In the present experiment, subjects observed an experimenter sitting in front of them, occasionally making eye contact. The subjects' eye movement was recorded using an eye tracker to determine the timing of eye contact. Brain activity in the temporal area, believed to be involved in gaze perception, was measured with fNIRS. The results revealed that temporal areas were more activated during non-eye-contact (averted) gaze shifts compared to periods of eye contact. This finding suggests that temporal cortex processing of eye contact differs from the processing of non-eye-contact gaze shift. We discuss how the simultaneous application of fNIRS and eye tracking can extend social cognitive neuroscience research.