In Turkey, Persia, and other areas of the Middle East, the harem was a section of a palace where the women of the palace lived. Contrary to popular belief, the harem was not a brothel; that's simply a Western viewpoint. The harem did house the wives (and sometimes concubines) of the Shah, but his female relatives also lived there.
Still, there are many cases where a group of females linked sexually to one male (or males linked to one female) is referred to as a harem.
female version of this is reverse harem
The term serraglio (Italian from Persian sarāy "palace, enclosed courts") carries a similar meaning.
In modern colloquial (and humorous) English, "harem" may denote a number of female followers of a charismatic man.
"From Arabic harIm (literally "something forbidden") and haram (literally "a sanctuary")"
Although its original meaning was the part of a household reserved for females - including perhaps a man's mother, sisters, and daughters, as well as wife or wives - it is commonly used to mean a group of more than one woman who are all available sexually to a single man. This situation is Polygyny ("multiple women.") Sometimes it also refers to a group of men who are available to a single woman. This situation is Polyandry ("multiple men.") Either may be called Polygamy ("multiple marriage.")
People who think it derives from Latin don't know what they're talking about, which is no great crime - no one knows everything. People who think it derives from Latin and say so on a public forum like this not only don't know that they don't know what they're talking about, but are displaying and perpetuating their ignorance.