It's not "when did you stop..." It's all about the guy and whether he's a creep or not. Certain men who I am close to can make outright horndog rude comments to me and it's funny and flattering. But when he-who-shall-not-be-named was giving me attention, it was not flattering and he was just a creep.
And like Zara said, looks have almost nothing to do with it. There was a guy a few months ago that I made the mistake of agreeing to have drinks with. He was physically cute as a button, but he quickly devolved into creep because of his obsessive personality and how it manifested itself. That makes him no longer cute and just a creepy idiot.
So, gents, if you are continually getting shot down, it's not your looks that are the problem.