It is a term people use when someone is being silly. It is normally said with affection to the person being silly. I have absolutely no idea why the saying mentions a goose. maybe because people see geese as being silly. Who knows?
Octavious: "I had seen an octopus fly a kite one day."
Kevin: "Octopus don't fly kites, you silly goose!"