What should I say when a guy's harmless question makes me uneasy?

Well, you should just be honest. If he's joking then you'll get a good laugh afterwards but you never know, you don't want to hurt his feelings (: