it's your shit.. as in, all of it..
it implies leaving/moving.
it's true that there are small bags that are used to hold things.. but generally when people say 'I'm gonna pack up my ditty bag" they're not being literal - it's slang for leaving.
'he gives me anymore lip and i'm gonna pack up my ditty bag!'
A small bag that contains various odds and ends such as a mirror, flashlight, glue, paper clips, rubber bands, twine, shoe string, bandaids, and tape. The bag is usually taken on a trip, particularly camping, where it can be used for a variety of purposes.
Joe: Oh no, the car won't start.
MacGyver: It's the ignition. I'll use the batteries from the flashlight in my Ditty Bag to start it.
Joe: Oh no, my luggage won't stay together.
MacGyver: Use one of the zip ties in my Ditty Bag to hold it together.
Joe: Oh no, I have a head ache.
MacGyver: I'll give you some of the asprin from my Ditty Bag.
Joe: Oh no, my space suit is leaking.
MacGyver: I'll plug the whole with some of the silly puddy from my Ditty Bag.