What's the meaning of his actions?

His a friend of mine we have the same circle of friends we work at the same company . I know a lot of things about him his a nice person. I know his wife they live just across our house and I live together with our other friends, we go out often together w/ our common friends watch movies, shopping etc. One time while we're inside the car his wife is not there he tried to hold my hand but i pull my hand away I didn't gave any malice to it , i just ignore it as if nothing happen and I didn't change how I treat him. I just notice this to him that when he show affection(like hugging, whispering) to her wife in front of us , He's looking straight at me so I just try to look away , I don't know how to react it's so awkward...