You are quite correct in keeping him in until the dreaded 'SNIP' Well done!

Seriously, you should risk assess whether for you it's ok to let him out. Risk assessment is looking at the seriousness of something happening as against the likelihood of it happening in your situation.

You also have to take into consideration your feelings and how you would cope if something awful happened.

For instance, is your cat a pedigree? In your neighbourhood what is the likelihood of him being stolen?

You may want to read this:

The Indoor Outdoor Debate

Just for your information, my cats are pedigree, they are neutered, insured and microchipped. I make sure that they are only let out in daylight, and are in at night. That last will never be 100%, though to date it is. What I am saying is that I cannot cater for the unexpected, but I will try.