It can be anywhere from decreased ability to function due to lack of sleep, to the process of going batshit insane.
Most commonly it's seen used for people under the effects of Hallucinogens (i.e. psychedelics, dissociatives, and deliriants)
Another use is if they are panicking due to some sort of shock, paranoia, or fear.
Generally anything that effects a persons conscience to a degree where it can't function at it's normal performance can cause them to start 'tripping out'.
It is not reserved for use on or by someone under the influence of drugs.
P1: Hey man, Dave took my shrooms, have you seen him around?
P2: Sorry man, I haven't, he's probably tripping out by now.
Dave: Duuuuuude, I'm tripping out!
Used a lot in mainstream college-student-made horror movies.
Sometimes by the first/second person to die or later by one of the victims who's loosing his grip on reality.