maybe you could cat proof your garden, so your cat cant get out and others cant get in.

that way he has his freedom and you dont need to worry about him.