I have heard this expression before but wonder where it comes from? I've heard it in rap lyrics before. I believe it refers to your parents as in, "my old man, I wouldn't mess with him". Like, don't mess with ol' papa dukes.

Literally everyone, ever. Including the writer of this definition.