It is important to remember that there are different levels of being a poge depending on who you are talking to. Starting at the front line, each echelon considers the folks behind him a poge. Boiled down like this, it becomes glaringly apparent that the only folks who are NOT poges are grunts but non-grunts feel the need to decrease their poge level by pointing fingers at someone who is more of a poge than they are. They have an inner desire to be a door-kicker but that desire didn't run deep enough to actually join the Infantry and they usually make fun of grunts while at the same time trying to be like them.
It is also important to understand that any member of the service will, under diress, admit that the service as a whole could not function at all if it were not for the mighty poge. They are administrators, facilitators and have their fingers in every aspect of the grunt's life with the exception of actually pulling the trigger for him.
A typical poge works in some sort of office and performs administrative duties while enjoying cold AC and hot coffee. He does personal favors for the Sergeant Major, calls staff NCO's by their first name when discussing them with his peers, always has creases in his uniform and highly polished boots. His workday is 0700-1630 or so, but because of the "fluid" nature of his job he may disappear somewhere around 1300 on Friday and not be seen until formation Monday morning. He gets first crack at all the new gear that comes through Supply, never mind the fact that it'll most likely never see neither light of day nor speck of dirt. He enjoys the power his MOS gives him over people who require his services from time to time and likes to let them know how vital he is to the process by sandbagging requests he doesn't deem to be important. Generally an unsavory, whiny pencil-necked suckass travelling in the CO's vacuum.
Few grunts would literally define the term as "any non-grunt", however. For example, pilots are not grunts, nor are EOD, artillery, medics, engineers, and a myriad of other personnel who are exposed to hostile fire during the normal course of their duties. Grunts respect anyone who pulls their load, regardless of their MOS.
The classic poge is all about the "hide and slide", skating out of duty when the opportunity arises but more than willing to pick up a ribbon or two for "what the unit did". The first Gulf War was a fuckin' breeding ground for these people - it was a great "Put Your Boot In The Sand And Get A Medal" war for those not directly involved in the fighting, and when they rotated back to CONUS they had all sorts of "desert warrior" stories to tell about how rough it was.
It is usually derogatory in nature but can be spoken as either an epithet or in general conversation as good-natured ribbing. Call a poge a "grunt" and they love it, but call a grunt a "poge" and see what happens :-)
Grunt 2 - "I always knew you were a fuckin' poge at heart"
"Man, those fuckin' poges have it easy - hot chow, cots, a fuckin' GP tent, must be nice."
A great example can be found in Stanley Kubrick's movie "Full Metal Jacket", where "grunt" and "poge" are used in the same line:
Chili: You weren't on Operation Hastings, Payback. You weren't even in country.
Private Payback: Oh, eat shit and die, you fuckin' Spanish American, you fuckin' poge. I was THERE man, I was in the shit with the grunts.