The Buffs (note the capitalization) refers to the 3rd Regiment of Foot in the British Army, later renamed the Royal East Kent regiment until it was amalgated with other regiments after the Second World War (the current successor regiment being the Princess of Wales's Royal Regiment). Before the late 19th century introduction of khaki uniforms British infantry famously wore red jackets. To tell different regiments apart they had different coloured facings - that is to say the inner lining visible when parts of the jacket like the sleeves where folded over. The 3rd Regiment of Foot nickname came from their buff coloured (a pale yellow) facings. This name was officially recognised and popularised during the period when regiments were named after their colonel. There were two Colonel Howard's Regiments so one was called the Buff Howards and the other the Green Howards.
The phrase itself originated in the 1858 when the adjutant of the Buffs was administrating a parade of his regiment under the gaze of a rival regiment, the 21st Fusiliers. Not wanting to be embarrassed by an indisciplined parade he shouted out 'steady the Buffs!' to get his men into order. It became a common phrase in the British Army and was popularised by Kipling
"Steady the Buffs! I lost my leg to a lion in Tanganyika but you don't see me blubbing like a nancy boy.
"Sorry, I was being a trifle self-indulgent for a bit there"