Generally stands for 'What the fuck'. Most people use a question mark afterwards to get the point through. Rather than using the same term for the other 'w's, who, when, where, and why, it makes more sense to actually state the word and follow it with 'tf'
Capitalization doesn't really matter.
This term can also be likened to 'What the shit?' which is more comical and has a tantamount meaning.
(Intended as an addendum to andyaml's already great definition)
Another popular workaround of the STFY misinterpretation is the new catchphrase "Shut the fuck you," combining perhaps two of the most potent online threats for a salient, comical and effective deterrent.
<TrendyHipster> Hey you, guy.
<TrendyHipster> SHUT THE FUCK YOU
<Wallflower> Hey, that guy's pretty trendy
<Wallflower(1)> Yeah, and hip, too!