Stay Brown to stay party tough