British origin.
Evolving from the term of abuse bugger, meaning a sodomite or irritating person. To play '
silly buggers' is to generally act the fool, lark about, waste
time or generally mess things up. As illustrated by English band The Bus Station Loonies song "Playing Silly Buggers" (1995).