Attracted to a married man now his wife hates me.?

I was attracted to this married man. I know that it is a sin. coveting and adultery. I was NEVER planning on acting on it. In fact I completely avoided him. I limited my contact with him and tried not to make eye contact. I'm not sure if I avoided him to much, because I noticed that he started watching me all the time. I have caught him staring at me a couple of time (not sure if he was trying to figure out why i was avoiding him so much). Anyway, I noticed that his wife started watching me to. I felt like I had committed a sin against both her and God, so I tried being nice to her. When I would catch her staring at me I would smile. She in turn would give me a nasty look and on one occasion called me a *****. Any hints on how I could have avoided the situation (I didn't want to be attracted to him. but I was...aside from that any other suggestions on how to avoid such a situation).