This phrase comes from cricket terminology. In cricket, a shortball is a fast, rising delivery that causes discomfort to the batsman, and can sometimes result in an injury from being struck by the ball.
In social situations, a short ball can be most loosely defined as anything that causes annoyance, discomfort, or rejection.
Sometimes, when saying "Shortball!', people use an accompanying hand action as if to fend the ball away from their chin, similar to a batsman in cricket. Experienced shortballers sometimes only perform the hand action without saying the word, or just abbreviate the phrase to "Short!"
1: "Hey mate, are you coming to the pub tonight?"
2: "No mate, I have to watch some paint dry at my house"
1: "Shortball there"
1: "Dude, I got locked outside and had to sleep in the snow!"
2: "Shortball there"
1: "So I got back to her place, and she just wanted to watch movies and talk about feelings!"