Can you use someone as a refernce without asking permission?

I was babysitting for one of my neighbors for some time. There was an issue which I won't discuss to save time and keep privacy. Basically, in the end, I had to stop working for him even though he wanted me to continue. A situation arose which I handled quite well (he told me so and asked me to continue babysitting), but I didn't want the situation to arise again and felt it showed a lack of respect for me on his part. Now, I am applying to other jobs. I would like to use him as a reference, but I am not sure if I can do so without asking permission. I would have to give his phone number out on application forms. I though it was implied that as an employer you could be used as a reference, but my friend cautioned against it. I am worried about asking him because we have not talked in months and I feel it could be awkward.