Saying this refers to the fact that someone is ugly, strange, and/ or fat.
You can also refer to someone as a potato if they are dull, uninteresting, or boring. Such as:
"Mary is such a potato"
"David asked me to prom, but he's a potato so I'm not going"
"Our professor is such a potato."
"Dude I'm a potato"