Having or making a finger baby is simply the act of fingering as performed by a lesbian to another. Lesbians are women and they do not have penises so they cannot actually make a baby. Rather, they finger each other, thus colloquially sex can be called making a finger baby, or often, making finger babies. See the "finger babies" entry.
"You're hot. I want to have your finger baby."