Boobie hatch is an ancient term for small hatch covers intended to keep a species of seabird called a Boobie from entering a ship through portholes and other smaller ventilation opening.
The usage to mean insane asylum is probably related to events happening when things enter through a boobie hatch. Either its drunken sailor activity (urination, etc) or a very messy situation involving an excited seabird thrashing around madly in a confined space with a bunch of sailors.
My mother taught me how to be a real subtle lady. She used to flash all the guys when she purposely bent over with an open booby hatch under a loose shirt. The other women never knew.
She always used to say never fall asleep with an open booby hatch or you might wake with a common sea bird perched on a nipple...that why they called them boobies.