Gosh isn't it hard being a cat owner.
I think you should just say to your neighbour, please do not attract my cat in to your house, do not feed him, I miss him and I love him.

I feed some other cats, the nieghbours knows, I feel bad now, but I worry if i didn't feed them sometimes, maybe they'd starve, I never let them in though. Yours is obviously well looked afterand wouldn't, so just go and take him.