When guys make comments about how tiny I am...?

I'm very petite... 5'1 and 98 pounds. Women never really mention my size but I hear a lot of guys make comments, seemingly neutral, about my size. Someone said "watch out, she's a tough one" and the other one said "the tiny ones usually are." Last night my crush said, "my coat is bigger than you!" When they say that, are they subtly insulting me or do guys not think it's such a bad thing?