I feel like his walls are weakening, and he feels it too so he decided to leave before hiding some of his other feelings became more challenging. I may be failing, but from going through this series again (I wasn't really thinking about it until now, so I could be wrong) but in lot of the cases where she was being obvious and most people would feel embarrassed and joyful, a sweat drop appears on him and his face or he looks troubled.

Either that is worry, or that is him struggling to keep a straight face. I may be very wrong, but still, he seems to be holding back a lot.