Evolving from the term of abuse bugger, meaning a sodomite or irritating person. To play 'silly buggers' is to generally act the fool, lark about, waste time or generally mess things up. As illustrated by English band The Bus Station Loonies song "Playing Silly Buggers" (1995).
"They were too busy playing silly buggers to get the job completed in time".
To act the fool. This is not a sexual reference as one might assume.
"Roono and Rosier are playing silly buggers again"