Terry towelling is a soft material that is often used in baby nappies (or diapers).
By labelling someone a "terry cloth" you are indeed accusing them of being "very soft", or as eloquently put by Jim Jones at the end of Hate Me Now (a Nas diss by Cam'Ron ft Jim Jones), a "pussy ass nigga wit a dick".
Terry: Please pass me that terry cloth.
Barb: You named your cloth after yourself?
Terry: Yes and no Barb it's complicated.
Barb: How do you mean?
Terry: It's always been called a terry cloth but also I name all my cloth's Terry. Will you pass the cloth this burgundy is going to stain the carpet!