Boobie hatch is an ancient term for small hatch covers intended to keep a species of seabird called a Boobie from entering a ship through portholes and other smaller ventilation opening.
The usage to mean insane asylum is probably related to events happening when things enter through a boobie hatch. Either its drunken sailor activity (urination, etc) or a very messy situation involving an excited seabird thrashing around madly in a confined space with a bunch of sailors.
Someone left a (boobie) booby hatch open when we docked. A boobie crawled in and then all hell broke loose after if the bird landed on the Captain's bowl of soup and shat in it. It was all arms and legs thrashing about for 10-15 minutes. And someone lost an eye to its beak before cook crushed the boobie against a bulkhead with a skillet. Yes a regular mad house all because of an unsecured (boobie) booby hatch.