The phrase emerged in England in the 19th century. Hatmakers in those days used a lot of felt that was treated with chemicals including lead, arsenic, and mercury.
Unfortunately, those chemicals are highly toxic. The symptoms of such poisoning include palsy, confused speech, and distorted thinking. Today, making hats is a much safer profession, but the phrase survives.
An interesting alternate explanation of the phrase derives hatter from Anglo-Saxon atter (poison), which is related to adder (a poisonous snake whose bite was thought to cause insanity).