A difficult situation. Specifically, circumstances that require a great deal of finesse to come out of unscathed. Physically, this term references the inherent difficulty of solving Oriental puzzle boxes, but this instance the phrase is used in reference to uncomfortable social situations.
My brother's been crashing at my house for the last four months. He's really been wrecking the place, no matter how many times I talk to him about it. If it was anybody else, I wouldn't hesitate on telling him to get out, but he's family and I don't want to be a dick. I'm really stuck in a trick box.