You could ask him nicely? It's got to be difficult, I'd try to keep my cat in for a couple of days and see if the neighbour is still having problems (without letting the neighbour know I was keeping my cat in) incase it's not your cat but the other one. I don't have a cat flap but have to 'manually' let any of my cats in and out but I'd assume if I did have one then I'd have visitors inviting themselves in unwanted and any problems that entailed. Good luck.