Am I wrong to deny my sons father from seeing him?

I have a 3 year old son from a previous relationship, we live 4 hours away from his dad. I had to move because of the military. I still have my family living in the same city as my ex so normally whenever I would drive down and visit I would bring my son to see his dad. We had an arrangement of every two weeks we'd switch but I was always the one making the drive down. I got tired of doing that so I started asking him to meet me halfway, he made up excuses several times so I talked to my lawyer about it and told me since our custody agreement states that I'm the primary physical and legal guardian than I didn't even have to do that and I was being way too nice and to let my sons dad make the drive to pick up our son. For the past 4 weeks now my ex has been telling my son that he's coming to pick him up. He hasn't, and when it comes down to it he either asks me to meet him halfway or he comes up with an excuse to why he cannot make the drive. First it was his truck being stolen then it was he didn't have anyone to ride with him (it's a 4 hour drive one way so he didn't want to drive 8 hours alone). I'm just getting tired of him getting my sons hopes up and not coming through so I've decided (along with my husband) to not let him see him period. Until he can come up and start making the effort to pick him up. We're going to visit my parents this weekend which is in the same city as my sons dad but I didn't tell him. I feel like if I tell him I'm coming down he'll come and try to get our son from my mothers but I don't want him to think I'm making this easy on him. Am I being unreasonable here?