I think you are reading too much in to this.

He was making a joke and not trying to hurt you.

He wasn't trying to make you feel bad or ruin your day.

I am sure you ruined his though. And you just reduced the odds of him engaging in playful banter with you in the future.

And women wonder why men clam up.