How do I let him down gently?

I need to shake off a clingy man that I don't really even fancy but have recently had a short term relationship with. Being 'friends' doesn't seem to help, he thinks he's in love with me. He's a wonderful man but not right for me. We have an intense connection but he wouldn't fit into my lifestyle. To be more specific he's into dodgy dealings, drugs and doesn't take too much care of his appearance. I'm high maintenance, do everything honestly and don't do drugs. I need to let him down gently but how without hurting his feelings? I'd like to still be friends with him if possible. He just doesn't take no for an answer and pressures people to get what he wants. I can't just ignore him because we have mutual friends too.