Can somebody seriously help me?

My boyfriend's father is serving a very long prison term for vehicular manslaughter. My boyfriend is planning on getting a new car, and when he does, he wants to go visit his father...and he wants me to come with him. He doesn't want me to go to the prison with him, but he wants me to travel with him. I don't want to go...period. Not only is his father incarcerated in a different state, but I've only been dating the guy for a couple of months. How do I tell him that I don't want to go without breaking his heart?