My ex Lover is has turned me life into hell?

I had an affair with a married man (I am married too) and now it's over he is running around town bad mouthing me, making up stories, telling lies and his version of events. Every time I've tried to retaliate it always seems to blow up in my face. I know I have done the wrong thing but why should I be made out to be the baddie. I have tried to keep this whole matter private and feel like everyone is believing him because they are only hearing one version of events. He is blaming me for instigating the whole thing, when he made the initial phone call and is saying that he tried to stop it but I kept ringing him and making me out that I'm am a psycho. I have lost half my family and most of my friends because of this and feel like he has lost nothing. I am really depressed and don't feel like this is fair at all. Any comments!