This is taken from the slang term for Pyjamas - JimJams - but is re-applied using the word Gym instead.
Gym-Jams are items of clothing that you initially purchase as expensive shit for looking good at the gym, but end up using as pyjamas. Usually the items start out being used at the gym, then as they start to fade and become a bit crappy looking, slowly become relegated into schlepping-around-the-house gear, then into fully-fledged jammies.
Girl - I forgot my bag in the car - can you go out and get it for me? I'm not dressed properly.
Guy - why don't you go? No one will notice cuz you're wearing your Gym-Jams.