Like "before I/we forget," "by the bye" means the speaker (or writer) is going to change the subject. It comes from "by" (meaning "near") "the bye" (the closing, where we say "goodbye.") So it means, "before we say bye, let me also tell you...."
"... and that's allI heard about Brenda and Eddie. By the bye, have you heard from Brenda's friend Billy?"
An expression of surprise and joy, by presumably referring to the nipples of a Babylonian god that were, for the sake of this expression, inordinately minty.