Neighbor's fence is across my line?

I purchased my house 4 years ago and I'm friendly with my next door neighbor. I did a survey on my house but I was never able to make the survey. It appears that the neighbor's fence is clearly "over the line" into my property. It is a small 25x100 lot. I believe I have at least a half a foot or very least 6 inches on my side of the house that runs alongside his driveway. I also noticed an old fence post in the ground which indicated the old fence was 6 inches towards his side. Right now I am completely boxed in. I mentioned this to him a year ago and he admitted it and said he would remove it if it bothered him. I didn't press the issue because I wanted to be a good neighbor and on good terms. I recently had a falling out with him over another minor issue. I am upset at him and in light of our property taxes being raised yet again, I want to have my property line restored. I want to know how can I go about confronting him on this. I have the survey (even though I don't know how to read it) and his admission as evidence. Does anyone have any advice. Thank you.