He doesn't have to gesture to show the emptiness of the space they're in- "No one else is here." His tone carries with it an almost accusatory, questioning tone; if no one else is here, what reason did you have to stand around, and what reason do you have not to stare like anyone who did?

It's bugging him dammit!