British origin.
Evolving from the term of abuse bugger, meaning a sodomite or
irritating person. To play 'silly buggers' is to
generally act the fool, lark about, waste time or generally mess things up. As
illustrated by English band The Bus Station Loonies song "Playing Silly Buggers" (1995).