What is the deal with this?

You have to be honest with him and tell him that if he doesn't start giving you some time then you are going to leave. There are 2 ways to go about saying that, the nice approach or the straight to the point approach. I would go with the latter but I'm very honest and blunt about things.