It is not uncommon for the person you met the first time to say to you that 'you have accent.' It refers to your way of pronounciation, which may be Southern, Northern, or Foreign accent. In this case, it is a mere refererce to your state or country of origin; nothing more.
But when a person you've known for years, or a co-worker you worked with for months; or in some cases, a person you met first time, all of a sudden, says to you: "You have accent!" "I don't understand your accent!" In this case, it is a direct reference to your identity. And it may mean one or all of the following:
1. 'You are not one of us!'
2. 'Remember you're a foreigner.'
3. 'How on earth you dare to tell me this.' (Although he/she clearly understands that what you said was right/true).
4. "I would rather prefer to hear it from someone else."
You are a Pharmacy Tech and Albert came to the window:
You: "What can I help you?"
Albert: "I want Sudafed."
You: "May I see your ID please?"
Albert: "Why do you need ID. It's just a Sudafed!"
You: "It is the State Law."
Albert: "I don't understand what you are saying."
You: "I said the State requires ID in order to buy a Sudafed."
Albert: "You have accent!" "I don't know what you're talking about." And he walked away.