Am I wrong in not talking to my father? Has been 5 years.?

No your right to not have this man in your life. I grew up in a similar situation, and I don't talk to my dad very often..but there may come a time when you can forgive him, even if he has never apologized. Hopefully this happens for you, but if not, you do have the right as an adult to not speak to him.